Footage for training and evaluation

How to search dashcam video for specific scenarios

With dashcam recordings the video is usually most of what you have, so searching for a scenario happens in three layers. First filter on whatever the files carry beyond the picture: timestamps, event flags, a GPS track if one was recorded. Then find the scenario in the picture itself, either by watching or by describing it to a video search tool. Finally, have a person review every candidate before it goes into a training, evaluation or QA set. The first two layers produce candidates. Only the third produces data you can use.

What a dashcam file gives you

Published answers to this question mostly describe the data-mining stack of a full self-driving program, built on logged sensor streams, automated triggers and large internal tooling. Dashcam footage doesn't look like that. A typical file is a video with a date, time and sometimes speed burned into the frame, maybe a separate GPS track, and maybe a flag from the camera's own motion sensor when it registered a jolt. The scenario you're after, whether it's a cyclist merging from the right, a pedestrian stepping out between parked cars, or glare on a wet road at dusk, is visible in the video and recorded nowhere else.

If your camera streams sit inside ROS bag or MCAP logs, export them to ordinary video files before searching, and keep a mapping back to the log timestamps so every candidate can be traced to its source.

Write the scenario the way it looks

Search goes better when each scenario is written as something a camera would see, not as a category name. "Near miss" is a judgment. "Pedestrian steps off the curb from between two parked cars while the vehicle is moving" is a picture. Write your scenario list in that second style before you search. The list does double duty: it's the query set, and it's the definition your reviewers check candidates against.

Route 1: metadata and event flags

Filter on time of day, date ranges, location if you have GPS, and any motion-sensor flags. This costs almost nothing and works for scenarios that leave a signature, like a hard stop. Its limit is that most visual scenarios leave none. A pedestrian who steps out and then waits doesn't trip anything, and flagged clips lean toward events that happened to produce a jolt.

Route 2: watching

People watch the footage at speed and log timestamps in a spreadsheet. Each individual call is good, but it doesn't scale, and over a long review the definition of the scenario drifts between reviewers unless the written list is kept in front of them.

Route 3: building your own search

A team with engineering time can sample frames, embed them with an open vision-language model, and search the frames by text. You own the whole thing and can tune it. The costs are the build, the upkeep, and the fact that frame hits have to be stitched back into time ranges a reviewer can actually watch.

Route 4: a search layer over uploaded video

The last route is to upload the recordings to a tool that indexes the video once and accepts plain-language descriptions, which returns candidate time ranges to open. This works on the picture only, so it adds nothing for signals the video doesn't show. In Vivu's case, recordings uploaded to a project are indexed once and a description from your scenario list returns candidate time ranges to review. Because Vivu can be called over MCP from Claude, ChatGPT or another agent, the agent can run your whole scenario list in turn and collect the candidates into one review list. Building the list is the agent's work, and deciding what enters the set stays with your reviewers.

For how programs with full sensor logs approach the same problem, see the fuller pipelines self-driving programs use.

Review before anything enters a set

A reviewer opens each candidate, checks it against the written definition, trims it to the relevant stretch, and records a short reason for accepting or rejecting it. Keep the rejects. They show where a description is too loose, and tightening the wording usually does more for the next round than anything else. How search inside video works explains why the phrasing of a query changes what comes back.

When you don't need a search tool

If the scenario reliably leaves a flag or a GPS signature, filter on that and watch what's left. If the volume of footage is small enough to review by hand in the time you have, watching is simpler and needs no upload. If the scenario is defined by something the camera can't see, video search won't find it however you phrase the query.

So look at your scenario list. If most entries are things a person would recognize by looking, searching the picture is worth setting up. If most are defined by what the vehicle did, start with the signals and keep video for review.

FAQ

Can I search dashcam footage by the speed or time burned into the frame?

Burned-in overlays are part of the picture, so a tool that reads on-screen text may pick them up. Treat what it reads as a hint, and confirm against the file's own metadata or GPS track before relying on a value, because overlays can be misread or can drift from the true time.

How do I search dashcam video that's stored in ROS bag or MCAP files?

Export the camera streams to standard video files first, since video search works on video files, not robotics logs. Keep a table that maps each exported file and its timestamps back to the original log, so every candidate you accept can be traced to its source.

Is natural-language search reliable enough to build an evaluation set?

Use it to generate candidates, not to decide what goes into the set. The set's quality comes from people reviewing every candidate against a written definition and logging why each one was accepted or rejected.

Can video search find scenarios I haven't thought to describe?

No. Describing a scenario finds what you describe. To surface scenarios nobody has named yet, you need other sources: event flags, cases where your own system's output turned out wrong, or random samples that people watch. Once one of those turns up something new, write it down as a description and search for more like it.