# How to do multimodal video search without Twelve Labs > There are three ways, and the right one depends on how much of the stack you want to own. Canonical URL: https://vivu.ai/guide/how-can-i-do-multimodal-video-search-without There are three ways, and the right one depends on how much of the stack you want to own. You can assemble it from open-weight models and a vector database. You can compose it from cloud services that each do one part. Or you can use a product where the index is already running and the query is the only thing you write. All three produce natural-language search over footage. They differ in who does the ongoing work. ## First, decide what multimodal has to mean here The word covers more ground than most teams need. Full multimodal retrieval means the query can match on the picture, the speech, on-screen text, and sometimes the audio track that has no words in it. Plenty of archives only need two of those, and the plan gets much cheaper once you know which two. A useful test: take twenty real requests from the last few months and mark what each one would have to match on. If eighteen of them are things somebody said, you are building transcript search with a visual fallback, and the approach in [matching queries against spoken audio](https://vivu.ai/guide/search-video-by-spoken-words) carries most of the load. If half of them are visual and nobody was talking, you need embeddings and there is no way around it. ## Route one: build it from open weights Frames get embedded with a CLIP-family image-text model such as SigLIP, speech gets transcribed with Whisper, both land in a vector store, and a query gets embedded with the same model and matched by similarity. That is [the same shape a hosted product runs](https://vivu.ai/guide/how-does-ai-search-inside-video-footage), with you owning each piece of it. This is well-trodden and every piece has a maintained open-source implementation. The cost is not the models. It is sampling policy, deciding how many frames per second to embed and how to group them into segments that are worth returning. It is GPU capacity at ingest, which is spiky and awkward to size. It is evaluation, because without a labeled query set you have no way to tell whether a change made retrieval better or worse, and you will change things. Teams that run this well treat it as a service with an owner. Teams that treat it as a project ship it once and watch it rot. ## Route two: compose cloud services Instead of hosting models, wire together a hosted transcription service, a hosted frame-labeling service, and your own retrieval layer. The operational burden drops a lot. The ceiling drops too, because label-based visual search only answers questions that fit the vendor's fixed vocabulary, and your interesting queries generally do not. This route works best as a first version. It gets a searchable archive in front of people quickly, and the queries they type against it become the evaluation set you need if you later move to route one. ## Route three: skip the pipeline The last option is to use something where indexing is not a thing you operate. The trade is control: you get the vendor's segmentation, their model choices, and their notion of relevance, and you cannot tune the retriever. In exchange you skip the entire ingest and reprocessing problem, which is where self-built systems consume their budget. Vivu is one implementation of this route: footage is uploaded to a project and indexed once in the cloud, and it stays searchable from one session to the next with no reprocessing on your side. In practice the query side becomes a conversation: after [Vivu](https://vivu.ai/mcp) is added to Claude under Settings → Connectors, anyone who can put the shot into words can ask for it there and get back time ranges they can watch on the results page. Whether that is a fair trade depends on how specific your relevance requirements are. A team with unusual footage and a precise definition of a good hit will out-tune a general product. A team that just wants to find things will not. ## When none of this is the answer If your archive is under a few hundred files and one person knows it, search infrastructure is a solution looking for a problem. If everything you shoot is scripted and you already keep scripts, a text search over scripts joined to filenames answers most requests for free. And if your real problem is that nobody can agree what the files are named, multimodal retrieval will find the footage and you will still have the naming argument afterward. The honest split: build it yourself when retrieval quality is a competitive matter for your product and you can staff someone to own it. Compose cloud services when you need something usable this month and want the option to change your mind. Buy a product when finding footage is a cost center rather than a feature, which is the situation most teams are actually in, even the ones who arrived asking which model to fine-tune. Work out which sentence describes you before you compare vendors, because the comparison looks completely different depending on the answer. ## FAQ ### Is CLIP good enough for video search on its own? For visual similarity, usually yes at small and medium scale. CLIP-family models handle queries like "person in a hard hat outdoors" well because that is close to what they were trained on. They are weak in two places that matter for archives. They have no sense of time, so a query about an action rather than an appearance, someone handing over a document, tends to return frames that merely look related. And they do not read speech at all, so anything defined by what was said needs a transcript index running alongside. Most working systems fuse the two rather than picking one. ### How much footage does it take before self-hosting stops making sense? There is no single threshold, but the shape is predictable. Cost scales with hours ingested, and effort scales with the number of distinct sources feeding the archive. A hundred hours from one camera setup is easy to self-host. Ten hours a week arriving from six freelancers in four formats is where teams lose their weekends. Watch the second number rather than the first. Volume is a hardware problem you can solve with money. Source variety is an operations problem, and it is the one that turns a working pipeline into a permanent chore. ### Can I run multimodal video search entirely on-premise? Yes. Every component has a self-hostable version: open-weight embedding models, open-weight speech recognition, and vector databases that run on your own hardware. Nothing has to leave your network, which is why regulated and security-sensitive organisations tend to go this way. Plan for GPU capacity at ingest and for a model upgrade path. On-premise systems drift because upgrading a model means reprocessing the archive, and without a reason forcing the issue that reprocessing keeps getting deferred until the index is years behind the state of the art. ### What should I test before committing to any approach? Build a query set from real requests before you evaluate anything. Twenty to fifty queries taken from actual past asks, each with the correct answer already known and written down as a file and a timestamp. Then run every candidate against that same set and count how often the right moment appears in the top handful of results. Vendor demos are built on footage that flatters the system, and self-built prototypes are tested on the queries the builder had in mind. A fixed query set from your own archive is the only comparison that survives contact with your actual work.