Named comparisons

Can Whisper transcribe audio with loud background music?

Usually yes, for speech a person could still make out, and the thing to watch for is not the failure you are expecting. Whisper is an open-source speech recognition family from OpenAI, trained on a lot of ordinary messy audio instead of clean studio speech, so a music bed under a voice is normally handled. The risk is what happens at the other end of the scale: when there is no intelligible speech in a stretch of audio, a model whose job is to produce text will often still produce text, and the sentence it invents reads exactly like the sentences it got right.

The only answer that is true for your audio is the one you measure. Take the worst 90 seconds you have, transcribe it, and read the output with the audio playing. That takes ten minutes and settles the question better than any general claim, including this one.

Loud means two different things

There is a difference between music that is loud in the mix and music that is loud in the same place as the voice. A dense wall of sound under a clearly recorded voiceover is often fine, because the voice still occupies its own part of the spectrum. A moderate music cue that sits right on top of a distant speaker can be much worse, because the parts of the signal the model needs are covered.

This is why the room matters more than the level. Stadium audio with a PA cue running under an interview, a live event where the presenter is off-mic, a field recording where the music is coming from the same speakers as everything else: these are all harder than a mixed promo with a music bed, even when the promo sounds louder.

The failure mode worth planning for

Generative speech recognition works by predicting a likely sequence of text. That is what makes it good at guessing through a noisy passage, and it is the same property that makes it produce fluent, confident, entirely fictional lines over music with no speech in it at all. You may also see a phrase repeat several times in a row, or timestamps that drift after a long non-speech stretch and stay wrong for a while afterwards.

None of that looks like an error in the output file. It looks like a transcript. If the transcript is going anywhere a human will trust it, someone has to check the sections that correspond to music, applause and silence, because those are exactly the sections most likely to be wrong.

What actually helps

Cut the audio down to the parts with speech before transcribing, using voice activity detection or just by hand, so the model is never asked what a music-only passage says. Try separating the speech from the music first, but test it against the raw mix rather than assuming it wins. Run the same clip through more than one model size and compare the outputs against each other, since the places where they disagree are a good map of where the audio is hard.

If you are running this yourself rather than through a service, the tradeoffs of self-hosting are the same ones covered in choosing between open models and a hosted API, and the arithmetic of doing it at archive scale comes down to the per-hour numbers more than to the model choice.

How good does it have to be

The honest question is what the transcript is for. A caption file has to be right, because errors are visible to the audience and someone will screenshot them. A search index does not: a transcript that misses two words in ten still surfaces the right clip, and nobody ever reads it.

That distinction changes what you should worry about. Transcripts also do not cover everything you might need to find, since a stretch where the music drowns the speaker is often a stretch where something clearly visible is happening on screen. Vivu indexes footage by what is in it and not only by what was said, so material with unusable audio is still reachable, and it returns the timecode rather than the text.

When this question does not apply to you

If the audio is one file and one interview, transcribe it and read it. If the music is something you added in the edit, go back and transcribe the version from before you added it. And if the recording is genuinely unintelligible to you when you listen closely, no model will do better; that is not a tooling problem.

So: run the test on your worst minute. If the errors land in places you can live with, the answer to the question in the title is yes for your audio, whatever anyone else's experience has been. If the errors land in the middle of the lines you actually need, the fix is upstream, in how the audio was captured, and no amount of model tuning will substitute for it.

FAQ

Should I remove the background music before running Whisper?

Not automatically. Separation can help when the voice is genuinely buried, but it also introduces artifacts of its own, and a recognition model sometimes copes better with the untouched mix. Run both versions of the same difficult minute and compare them before committing to a pipeline, because the answer depends on how your material was mixed and does not generalize between projects.

Why does my transcript contain sentences nobody said?

Because the model is producing likely text, and a stretch of music, applause or silence has no speech for it to recognize but still gets a prediction. This is a known behavior of speech recognition models that generate text sequences, and it shows up most on audio that alternates between talking and long non-speech passages. Trimming out the non-speech sections before transcription prevents most of it.

Do the timestamps stay reliable on music-heavy audio?

Treat them as approximate. Timestamps tend to drift around long stretches without clear speech and can stay offset afterwards, which matters if you are using them to drive an edit or to jump to a moment. If you are relying on timing, spot-check a few marks against the actual audio at the start, middle and end of the file rather than trusting the whole run.

Is transcription the right tool for a music video or a montage?

Usually not. Material where the audio is mostly music has very little speech to recognize, so you get a thin transcript and a high proportion of invented lines. For that kind of footage, what identifies a moment is what is on screen, and text-based indexing is the wrong handle to be reaching for.