# How does multimodal video search work? > It works by not searching the video. Before any question is asked, the system pulls each file apart into parallel streams of evidence: the speech… Canonical URL: https://vivu.ai/guide/how-does-multimodal-video-search-work It works by not searching the video. Before any question is asked, the system pulls each file apart into parallel streams of evidence: the speech, the images, text that appears on screen, sometimes non-speech audio like a crowd or a door. Each stream is converted into numbers that can be compared with numbers made from a sentence, and the whole thing is stored against timecodes. When you type "the moment the founder holds up the prototype", nothing rewatches anything. Your sentence becomes numbers, those numbers get compared against the index, and what comes back is a ranked list of positions inside files. ## What multimodal actually refers to The word describes how many kinds of evidence the index holds. Speech is the easiest and the one every system has, because speech-to-text is mature and the output is text you can already search. Visual content is the one that makes the category interesting: models trained on paired images and captions can place a picture and a description in the same space, so "person in a hard hat pointing at a screen" can match a frame nobody ever labeled. On-screen text is a third stream, and it matters more than people expect, since slide titles, lower thirds, and product packaging carry names that were never said out loud. Audio events are the least common. Each of these is a separate pass with separate failure modes. A system can be strong on speech and weak on visuals, and the word "multimodal" on a page tells you nothing about the balance. ## The indexing pass is where the work happens The expensive part runs once, when material enters the system. Frames get sampled, the audio gets transcribed, embeddings get computed, and everything is written to a store with timecodes attached. This is why the first pass over a back catalog takes real time and the queries afterward feel instant: at query time there is no video processing at all, only a comparison between your sentence and an index that already exists. That ordering has a consequence worth planning for. Whatever was not indexed cannot be found, so the question "what happens when new footage arrives" is an operational question rather than a technical detail. The answer is either that someone runs the pass, or that arrival triggers it. ## Why results are timecodes and not files Similarity matching is continuous, so the system does not decide a clip either matches or does not. It scores segments and returns the highest, which is why results come as ranked moments with confidence attached and why the second and third results are often worth checking. A file-level result would throw away the only thing that saved you time, which is knowing where in a 90-minute recording to look. [Vivu](https://vivu.ai/platform) returns the matching time range along with the surrounding context rather than a list of files that scored well, which is the difference between an answer you can use and a shortlist you still have to sit through. It stops there, at the timecode. The cutting is still yours. ## Where it goes wrong Three failure modes account for most disappointment. Abstract queries fail, because "the emotional part of the interview" is a judgment and the index holds observations. Near-identical takes fail, because six versions of the same setup are, to the model, six versions of the same thing, and picking the one where the line landed is not a retrieval problem. Proper nouns fail unless they were spoken or written on screen, since no amount of visual understanding tells a model which of two people is your CEO. Knowing this changes how you phrase queries. Concrete and visual beats interpretive every time, which is the same discipline that makes [phrase search across transcripts](https://vivu.ai/guide/search-video-by-spoken-words) work well. ## How it differs from the two things it gets confused with Keyword search over transcripts matches strings, so it finds the exact phrase and misses the paraphrase, and it is blind to anything silent. Tag search matches a vocabulary someone chose in advance, so it is precise within that vocabulary and empty outside it. Multimodal retrieval trades exactness for coverage: you can ask for things nobody planned for, and you accept a ranked shortlist instead of a definitive yes. ## When you don't need it An archive one person can hold in their head does not need an index, and neither does material with a rigid structure you can navigate by name. If every question you ask is about a line somebody said, transcripts alone are cheaper and more precise. The inflection point is the archive nobody can enumerate anymore, combined with questions that are not about dialogue. ## Working out whether the mechanism fits your problem Write down the last five searches you wanted to run, then sort them: said out loud, visible on screen, or a matter of opinion. The first group needs transcripts. The second is what this machinery is for. The third is not a search problem, and no product will solve it, which is worth accepting early. If most of your list is about finding the good few seconds inside something long, that is a related but distinct job, and [what automated highlight finding can and cannot decide](https://vivu.ai/guide/can-ai-find-highlights-in-long-videos) is the better starting point. If you are considering building this yourself, [the components you would be assembling](https://vivu.ai/guide/what-apis-exist-for-video-indexing-and-search) are worth pricing before you commit. ## FAQ ### Do I need transcripts before multimodal search will work? No. The visual side of the index is built from frames and works on footage with no audio at all. Transcripts add a strong second signal, so material with dialogue tends to return better results than silent material, and systems that combine both usually rank higher when a query matches on two streams at once. If you already have transcripts, they are worth feeding in rather than regenerating. ### Why did my search return an obviously wrong clip? Because the match is by similarity, not by truth. The system found the segment closest to your sentence in its index, and if nothing close exists it still returns whatever is nearest. Low-confidence results that look random usually mean the thing you asked for is not indexed: it was never said, never visible, or never happened in the material. Rephrasing in concrete visual terms fixes a surprising share of these. ### Is multimodal search the same thing as AI tagging? No, though the same models often power both. Tagging produces labels from a fixed set that you then filter on, so it is a metadata layer sitting on top of your files. Multimodal search compares free-form sentences to an index of the content itself, with no vocabulary decided in advance. Tagging gives you consistent facets for browsing; retrieval gives you an answer to a question you did not anticipate. ### Does the index have to be rebuilt when I add footage? The existing index does not, but the new material has to be processed before it can be found. How that gets triggered is the practical difference between systems: some expect a manual import step, others watch a storage location and pick up arrivals. Ask about it directly, because an index that silently falls behind is worse than no index, since people stop trusting the results and go back to scrubbing.