Self-driving teams rarely start by searching raw video. First they narrow the footage using what the vehicle already logged: events like hard braking or a driver takeover, plus tags for location, time of day and weather. Then they search what's left by content, either by querying the output of object detectors or with embedding-based search that matches a text or image query against frames. What comes out is a list of candidates, and people review each one before it goes into a training, evaluation or QA set. The hard cases are scenarios that leave no trace in the logs, and those are where content search is worth having.
Route 1: triggers and logged events
The cheapest filter is the one the fleet already produced. If a scenario reliably causes a takeover, an emergency stop or a planner flag, you can query for those events and pull the few seconds around each one. This works for scenarios you already know to look for. It misses everything the vehicle handled without incident, and many rare scenarios are exactly that: a mattress on the shoulder, someone pushing a shopping cart across the road, a traffic light partly hidden behind a truck. If nothing fired, nothing gets pulled.
Route 2: querying detector output
Teams that run perception models offline over recorded video can query the results, for example frames with a cyclist next to a bus, or frames inside a construction zone. Queries over labels are fast and precise within the model's vocabulary. That vocabulary is also the limit. A detector finds the classes it was built for, and rare scenarios are often rare because no class covers them, or because the model misreads them.
Route 3: embedding search you build yourself
The more open-ended approach is to encode frames or short clips with a model that puts images and text in the same vector space, store the vectors, and search with a sentence or an example frame. This finds things nobody wrote a class for. It also takes real engineering: choosing clip length, re-encoding everything when the model changes, and tuning how many near misses reach the review queue. The cost of generating embeddings is worth working out before you commit, and how-does-multimodal-video-search-work explains the mechanics.
Route 4: hosted natural-language search over uploaded video
The last option hands the embedding work to a service. You upload camera video, the service indexes it once, and you query in plain language. Vivu is one example: a query like "a pedestrian stepping out from between parked cars" returns time ranges from the uploaded video, each with a one-line reason, and a person still reviews them before anything goes into a set. It works on video files, so camera streams stored in ROS bag or MCAP logs have to be exported to video first. Teams that work in an AI assistant or agent can call the same search over MCP and have the agent collect the candidate ranges into a review list. For more on that setup, see how MCP connectors for video search work.
What a result looks like, and why review stays manual
Content search returns candidates ranked by how well they match. Two patterns show up. A quick pass often returns whole clips. That tells you whether the scene is in the footage at all, but someone still has to scrub for the moment. A slower, more precise pass narrows the result to the seconds where the thing appears and explains each match. One practical pattern is to run the quick pass to find which clips are relevant, then the precise pass to locate the moment. The precise pass takes longer and uses more of the search budget.
Either way, the reasons only describe what's visible. They don't say whether a scene was dangerous or why the vehicle reacted the way it did, and they don't replace a label. A reviewer still decides whether each candidate belongs in the set. Borderline cases, like a pedestrian near the curb who never steps out, need a written rule before review starts. Otherwise two reviewers will sort them differently.
When content search doesn't help
Some scenarios aren't visible in the frame. Anything defined by vehicle state, like a late brake or a small lane offset, belongs to the logs, not the video. Questions about how often a scenario happens across the fleet need systematic labels, because a search returns the best matches and not a complete count. And if your triggers already catch a scenario reliably, a second search layer adds cost without adding new cases.
Deciding where to spend the effort
Start by writing the rare scenario down as one sentence. If the sentence names a logged event, query the logs. If it names a class your detector knows, query the detections. If it describes something you can see but never built a class for, that's the case for content search. Then the remaining choice is whether to build the embedding pipeline yourself or use a hosted one.
FAQ
Can natural-language search find scenarios the perception model was never trained on?
It can often surface them, because it matches a description against what's visible and doesn't depend on a fixed class list. It won't find every instance, and it also returns look-alikes, so treat the results as candidates for review.
Do ROS bag or MCAP recordings need to be converted before video search?
For tools that work on video files, yes. Export the camera streams to standard video files first. The rest of the log isn't part of a video search.
Can a video search tell us why the vehicle reacted the way it did?
No. It can find the moment and describe what's in the frame. Explaining the vehicle's behavior takes the logs and an engineer's analysis.
Is it better to search with text or with an example frame?
Text works when you can put the scenario into words. An example frame helps when the scene is hard to describe, such as unusual glare or lighting, and you want more scenes like it. DIY embedding setups often support both.