---
name: legged-robot-terrain-shortlists
description: "Sort a batch of legged robot test run videos into one candidate CSV per terrain (stairs, gravel, grass or leaves, wet shiny floor) with Vivu. Use when checking eval set terrain coverage."
---Legged robot terrain shortlists with Vivu
This skill turns a batch of body camera videos from quadruped or humanoid test runs (the forward camera stream your team has already exported from its logs to mp4) into one CSV per terrain class on the team's list: stairs going up or down, gravel or loose rock, grass or a layer of fallen leaves, and wet shiny ground that reflects like a mirror. Claude measures the batch, prices it against the user's Vivu plan, uploads it to a private Vivu project, runs one precise search per terrain, cuts a contact sheet at half second steps for every window Vivu returns, labels each window looks right, looks wrong or can't tell, and writes every candidate into its terrain's CSV with an empty reviewer_verdict column. The data engineer decides which runs count toward the eval set's terrain coverage.
The value is in what only the picture holds. A test run log has IMU readings, joint angles, foot contacts and a site name; none of them says whether the robot walked up a staircase, crossed a gravel strip or stepped through a puddle on that run. Two runs at the same site can cover different ground, and exported camera streams have no audio or captions to search. The camera on a legged robot sits low and bobs with the gait, so the ground fills the lower half of the picture and is often blurred, which is exactly where picture search is weakest. In our test run the first wordings for stairs and gravel returned more look alikes than real ones (ribbed concrete ramps, pebbles under river water), and the tighter stairs wording then missed a staircase a person sees at once. So every candidate is labeled from its frames, rejected candidates stay in the CSV, and a short list never means the batch has no such ground.
When to use
Use when someone asks which of this week's test runs went over stairs, gravel, grass or wet floors, to check terrain coverage of a locomotion eval set, to pull candidate stretches of a given ground type for review, or to sort a batch of robot camera videos by terrain before labeling.
Not for these:
- A one off question about one run ("did this run hit the stairs?"): search Vivu directly and look at the frames.
- Falls, slips, stumbles, gait quality or why a run failed. The skill lists ground types that are visible in the picture; it judges nothing about the robot or its controller.
- Safety judgments, operator scoring, or anything real time or on the robot. The CSVs are review lists made after the runs.
- Frame accurate terrain labels or segmentation masks: use the team's annotation tool. This skill makes review lists, not labels.
- Hundreds of hours at once. Pick a subset from the run logs first (sites, dates, controller versions); one run of this skill is sized for about an hour of footage.
- Snow, sand, night, thermal or depth streams as the main material: untested. The skill can run, but treat its output as unmeasured and have the reviewer look at every row.
Working principles
- Report measured numbers, not estimates. When a number is an estimate, say so.
- Nothing is verified until it has been checked against the source video. Vivu's results are candidates, Claude's labels are a first pass, and the reviewer's verdict is the decision.
- Stop and tell the user when a required capability or tool is missing. Do not guess around it.
- Ask the user before anything that is expensive to redo (the batch and its index minutes, before upload) and before writing the full CSVs (one sample row first). The skill posts and sends nothing; if the user asks Claude to put the CSVs somewhere, ask again before each write.
- Labels come from frames, never from Vivu's reason text, which describes the terrain it was asked for even when the frame shows something else.
- The CSVs keep every candidate and carry no recall or precision figures. A search misses some stretches and caps how many come back, so an empty or short list does not prove the batch has none.
What you need before starting
Check each item at the start of the run and tell the user plainly what is missing before doing anything else.
| Requirement | Why | How to check |
|---|---|---|
| Vivu connector with write access | create a private project, open its upload page, search | vivu_get_account shows can_create_projects: true (tool names may carry a server prefix). A write call failing with "has not granted vivu.write" means the user reconnects Vivu and allows write access |
| The run videos as mp4 files on the user's computer | Vivu indexes video files, and every sheet is cut from the local file | ffprobe -v error -show_entries format=duration -of csv=p=0 FILE prints a length for each file; ROS bag or MCAP camera topics must be exported to video by the team's own tools first |
| A shell on that computer with ffmpeg and ffprobe | measure the batch and cut contact sheets | ffmpeg -version, ffprobe -version |
| An upload path | move the files into Vivu | vivu_open_upload_page plus the user's own browser, or a browser tool that can attach local files |
| Someone who reviews the CSVs | Claude's labels are a first pass; the reviewer decides | ask who fills reviewer_verdict; the default is the user |
This skill needs Claude Code on the user's computer (the terminal or the Code tab of Claude Desktop), because it reads local files and runs ffmpeg. Claude on the web and cloud sessions cannot reach the files. It downloads nothing, so no residential IP is needed. Nothing recurs, so no scheduler is involved; run it once per batch.
Inputs to collect
Ask for anything missing, most important first.
- BATCH_DIR: the folder that holds the exported mp4 files for this batch. Required.
- The terrain classes and what counts for each. Default: the four classes in Step 4 with the label rules in Step 5. If the team defines a class differently (for example, a single curb counts as stairs, or wet grass counts as wet ground), the team's definition wins; write it into config.json before labeling.
- BATCH_NAME: a short name such as a site and a date. It names the working folder and the Vivu project. Default: the name of BATCH_DIR.
- Batch size. Default: up to 60 minutes of video, or what remains of the plan this month, whichever is smaller.
- Who reviews the CSVs. Default: the user.
Files and state
Keep everything in one working folder next to BATCH_DIR:
terrain-BATCH_NAME/
config.json BATCH_DIR, classes with query, maximum_results and label rules, Vivu project id
files.csv source_file, duration_s, listed_name, video_id
results/ search results without the result page link, one JSON per class
sheets/ one folder per class with contact sheets
CLASS.csv one per class: every candidate, Claude's label, empty reviewer_verdict
state.json steps done, files uploaded, classes searched with job ids, classes cut off, candidate keys labeled
The commands in Steps 2 to 5 run from the folder that holds both BATCH_DIR and terrain-BATCH_NAME/, so the relative paths in them resolve.
A rerun reads state.json first and skips finished steps. A file already marked uploaded is never uploaded again, a class with a saved result is not searched again, and a candidate key (video_id, start_ms and class) that already has a label is not labeled again. The next batch gets its own folder and its own Vivu project, so its searches do not return last week's candidates.
Step 1: Check the Vivu connector and the setup
Goal: confirm every requirement before spending anything.
- Call vivu_get_account. If the tool does not exist, tell the user to add the Vivu connector in Claude (
https://mcp.vivu.ai/mcp) and stop. If the account does not showcan_create_projects: true, or a later write call fails with "has not granted vivu.write", ask the user to reconnect Vivu and allow write access, then stop until they have. - Run
ffmpeg -versionandffprobe -version. - List BATCH_DIR and run the ffprobe length command on one file.
Done when vivu_get_account shows can_create_projects: true, ffmpeg and ffprobe print their versions, and ffprobe prints a length for a file in BATCH_DIR.
Step 2: Measure the batch and price it
Goal: an indexed set the user's plan can pay for, approved before upload.
- Run the ffprobe length command on every mp4 in BATCH_DIR and write files.csv with source_file and duration_s. Add listed_name: the file name with each space, bracket and plus sign replaced by an underscore, because that is how Vivu will list it.
- Call vivu_get_usage and show one table:
| This batch | Plan allowance | Remaining this month | |
|---|---|---|---|
| Index minutes | sum of duration_s / 60 | from vivu_get_usage | from vivu_get_usage |
| Search credits | 5 per class, 20 for the default four (estimate), plus 5 for each rewording | from vivu_get_usage | from vivu_get_usage |
Plan facts: Free is $0 a month with 20 index minutes a month and 50 search credits a month; Premium is $30 a month with 180 index minutes and 500 search credits. A precise search uses 5 credits in total. A typical batch of ten runs of four minutes each is about 40 index minutes (estimate), which is more than Free covers in a month and fits in Premium.
- If the batch does not fit, offer levers in this order: pick a smaller subset with the run logs (the sites, dates or controller versions the eval set needs most), split the batch across two runs or two months, and only then a larger plan. Never drop files the user asked for without saying which ones.
- Confirm the Compliance items with the user.
The account used in our test run returns no allowance figures, so the comparison with a real plan and the user's approval were not exercised in our test run.
Done when the user approves the file list and its index minutes, and config.json records both.
Step 3: Upload and index
Goal: every file in the batch indexed in a private Vivu project for this batch.
- Call vivu_list_projects and reuse a project named "Terrain BATCH_NAME" if one exists. Otherwise call vivu_create_project with that name and visibility "private". The default visibility is organization, which shows the project to everyone in the user's Vivu organization, and test footage from labs and private sites is often sensitive.
- Call vivu_open_upload_page with the project ID immediately before uploading. The link expires in 180 seconds and works once, so never post or store it. Give it to the user to open in their own browser and select the files, or open it in a browser tool that can attach local files. Claude in Chrome accepts at most 10 MB per upload call and robot camera videos are usually larger, so they normally go through the user's own browser or the Vivu web app. Never split or recompress a file to fit.
- Poll vivu_list_videos until every file shows ready. Match each Vivu file name to files.csv by listed_name and record its video_id there and in state.json.
In our test run the 10 videos (32.29 minutes) were all ready 265 seconds after the upload started. They went through the same upload page with an automated browser, so opening the link in the user's own browser was not exercised in our test run.
Done when every file in files.csv shows ready and has a video_id.
Step 4: Search each terrain class
Goal: a saved list of candidate windows for every class.
| Field | Query | Mode | maximum_results |
|---|---|---|---|
| stairs | a flight of steps on the path ahead: a repeating pattern of flat treads and vertical risers, each step a horizontal band, stacked one above the next from the bottom of the picture toward the middle. A smooth or ribbed concrete slope, a pile of rocks, a cave or a wooden footbridge does not count | precise | 40 |
| gravel | the walking surface in the bottom half of the picture is dry loose stone: pebbles, gravel or broken rock pieces lying on top of each other with gaps between them. Stones under water, a muddy grass track, sand and smooth pavement do not count | precise | 40 |
| grass_leaves | the ground right in front of the low camera is covered with grass or a layer of fallen leaves | precise | 40 |
| wet_reflective | wet, shiny ground: puddles or rain soaked pavement in front of the camera reflect the sky, lights or buildings like a mirror | precise | 40 |
The stairs and gravel wordings describe surface texture and where it sits in the picture, and name the look alikes they exclude, because the plainer wordings pulled in ramps, rock heaps and pebbles under water (see the worked example). The searches run side by side, one per class, because each class is its own list for the reviewer. Within a class the chain is: the precise search finds candidate windows, the contact sheet shows the ground across each window, Claude labels it, and the reviewer decides. Every search is precise because only precise returns a time window; fast returns whole files with an empty reason, and the batch is already a chosen subset. Exported robot camera streams have no audio and no on screen text about the ground, so there is no second modality to switch to.
maximum_results is also the ceiling on how many candidates come back. 40 leaves room for an hour of footage with several stretches per run; a search that returns exactly 40 was cut off, and the batch holds more candidates than one search can return. Then tell the user, mark the class cut off in state.json, and offer to split the batch into two smaller batches with their own projects (quote the index minutes again and wait for approval as in Step 2). In our test run no search came close to 40, so a cut off search was not exercised in our test run.
Run each query with vivu_search_videos (project_id, query, mode, maximum_results). It returns a job ID. Call vivu_get_search_results until complete is true; each status call can wait up to 45 seconds, so a pending search is not a stalled one. Save each completed result as results/FIELD.json without its result_page_url field, and record the job_id in state.json. Show the Vivu result page link in the live reply only; it expires after four hours, so it never goes into a saved file or a CSV.
Run each class once. If the team changes a wording, the new wording is untested until its first candidates have been through Step 5.
Done when every class has a completed result saved in results/, and every class whose search returned exactly its maximum_results is marked cut off in state.json.
Step 5: Make contact sheets and label every candidate
Goal: every candidate labeled looks right, looks wrong or can't tell from what its frames show.
For each result in results/FIELD.json, cut contact sheets of its window from the local file at two frames a second:
mkdir -p terrain-BATCH_NAME/sheets/FIELD
ffmpeg -v error -y -ss START -to END -i FILE -vf "fps=2,scale=384:-2,tile=5x4" terrain-BATCH_NAME/sheets/FIELD/rRANK_STEM_STARTMS_%02d.png
FIELD is the class, FILE the local source file (BATCH_DIR/ followed by its source_file), STEM its name without .mp4, RANK the result_number written with two digits (r01, r09) so the files sort by rank, STARTMS the result's start_ms, and START and END the result's start_ms and end_ms divided by 1000. Each sheet holds 20 tiles covering 10 seconds of the window; the last sheet is padded with black tiles. Tile k on sheet n (both counted from 0, left to right and top to bottom) is at START + 10 * n + k / 2 seconds. Look at every sheet of a window: windows are often longer than the stretch they point at, and the ground that matches can be only two or three seconds of it. When a tile is too blurred by the gait to tell, take one full size frame at that tile's second with ffmpeg -v error -y -ss SECONDS -i FILE -frames:v 1 -q:v 3 terrain-BATCH_NAME/sheets/FIELD/rRANK_STEM_SECONDS_full.png.
Label rules (defaults; the team's definitions from Inputs win):
- stairs looks right when a run of at least two steps (treads and risers) is on the path ahead or under the camera. It looks wrong for ribbed or grooved ramps, stepped walls or abutments beside the path, rock heaps, wooden footbridge slats, and a single curb.
- gravel looks right when dry loose stones, pebbles, gravel or broken rock cover the ground under or just ahead of the camera. It looks wrong for stones seen through water, muddy tracks, sand, cobbles set in a street, and boulders beside the path.
- grass_leaves looks right when grass, low ground plants or a layer of fallen leaves covers most of the ground in front of the camera. It looks wrong for a dirt trail with leaves only at the edges and for leaves floating in a puddle.
- wet_reflective looks right when water on the ground or a wet surface mirrors the sky, lights or buildings. It looks wrong for open water (a river or the sea), a dry polished floor and glare from the sun on dry ground.
- can't tell when the full size frame still does not settle it (motion blur, the ground out of view, the camera pointing at the sky). The reviewer looks at the source.
Write a note of what the frames show for every candidate, such as "stone steps with a railing from 01:11 to 01:14, seen from a distance" or "riprap under the camera, last seconds a concrete ramp". Never copy the reason into the note. Record each candidate key and its label in state.json.
Worked example from our test run
No licensed public footage from a legged robot's own camera turned up that was long enough for a test, so the test corpus was public video under a Creative Commons license used as a stand in: 10 videos (32.29 minutes), made of a clip from a small quadruped robot walking indoors with a camera on board, clips from cameras on dog harnesses (knee high and bobbing with the gait, the closest public match to a quadruped body camera), and clips added for stairs and wet ground that are not at robot height (an eye level walk ending on stone stairs, a third person hiking video, a camera resting on wet pavement, rainy street shots). Audio was removed and the files renamed so the names said nothing about the ground. Every stretch of stairs, gravel, grass or leaves and wet shiny ground was marked by hand on contact sheets before any search, together with look alikes (a wooden footbridge, a stepped concrete bridge abutment, cobbles, beach sand, surf, a river, a parquet floor). The corpus had few real stairs and a single clear gravel stretch, so the counts below say little about recall.
- stairs, first wording ("the camera walks low over a staircase ... step edges as parallel lines"): 5 candidates, 1 real and 4 false, and 1 of the 2 staircases marked by hand was missed. The false ones were a ribbed concrete ramp, a stepped concrete bridge abutment beside the path, a heap of broken stone slabs and a dark bunker corridor, each with a reason describing steps.
- stairs, with the wording in the table, after 1 rewording on the same corpus: 1 candidate, 1 real and 0 false, and 1 staircase missed. The look alikes were gone, but the missed one was the long town staircase that the first wording had found; the one returned was seen from a distance behind a hiker.
- gravel, first wording: 5 candidates, 2 real and 3 false (pebbles seen through shallow river water twice, and a muddy grass track whose reason said loose gravel).
- gravel, with the wording in the table, after 1 rewording on the same corpus: 2 candidates, 2 real and 0 false, and none of the 1 gravel stretch marked by hand missed.
- grass_leaves: 8 candidates, 6 real and 2 false (leaves floating in a puddle, a dirt trail with leaves only at the edges), and 1 stretch marked by hand was missed (a grass track that came back under gravel instead). One real window was a 62 second stretch of grass, so windows can be long.
- wet_reflective: 5 candidates, 5 real and 0 false, and none of the stretches marked by hand was missed. The parquet floor, the river and the surf were not returned. A glossy dry tile floor was not in the corpus, so telling wet floors from shiny tiles is untested.
- A search for snow, which the corpus did not have, returned 0 candidates.
- The 10 videos were all ready 265 seconds after the upload started. The CSVs from the final wordings held 16 rows in total.
Done when every candidate in results/ has at least one contact sheet in sheets/ and a label in state.json.
Step 6: Write one CSV per class and hand it over
Goal: a review list per class that a data engineer can work through without opening Vivu.
Show the user one sample row and the field mapping, and wait for an OK before writing the rest:
source_file,rank,start_mmss,end_mmss,search_reason,contact_sheets,claude_label,claude_note,reviewer_verdict
RUN_07.mp4,2,02:45,03:11,"The dog navigates across a dry walking surface composed of loose broken rock pieces and stones piled along the riverbank before stepping onto the concrete ramp.",sheets/gravel/r02_RUN_07_165000_01.png;sheets/gravel/r02_RUN_07_165000_02.png,looks right,"riprap under the camera for most of the window; the last seconds are a ribbed concrete ramp",
The row above is adapted from our test run (file names shortened); the user's rows carry their own file names.
| Column | Source | If unavailable |
|---|---|---|
| source_file | files.csv, matched by listed_name | keep the row with Vivu's file name, label it can't tell with the note "no local file matched", and tell the user |
| rank | result_number in results/FIELD.json | none |
| start_mmss, end_mmss | start_ms and end_ms, written MM:SS | none |
| search_reason | the result's reason, copied as is; it is Vivu's description, not evidence | blank |
| contact_sheets | the sheet files from Step 5, separated by semicolons | none; the row cannot be labeled without them |
| claude_label | Step 5 (inferred from frames) | can't tell |
| claude_note | what the frames show, in Claude's words (inferred) | blank |
| reviewer_verdict | left empty for the reviewer | empty |
claude_label and claude_note are Claude's reading of the frames, not a measurement. Borderline cases (dry heath counted as grass, a rough stone path that might be steps) depend on the team's definition, so the reviewer goes through every row, the looks right ones included, before anything counts toward coverage.
Write FIELD.csv for every class with one row per result, sorted by rank, including the rows labeled looks wrong: a reviewer may disagree with a label, and the rejected rows show what the search confuses. The CSV uses the user's own file names and times, never a Vivu result page link, so it stays usable after the link expires.
Then tell the user, per class, how many candidates came back, how many Claude labeled looks right, looks wrong and can't tell, and whether the list was cut off at maximum_results. Say plainly that an empty or short list does not mean the batch has no such ground, and that stairs were the class most often missed. Do not add recall or precision figures: the hand marks needed to measure them do not exist for the user's batch.
The CSVs stay in the working folder. Passing them on is the user's step. If the user asks Claude to upload or post them (a shared drive, a tracker, a chat), name the destination, show the rendered first rows and wait for a yes before each write.
In our test run the CSVs were written from the dry run's results and sheets; showing the sample row to a user and handing the CSVs over were not exercised in our test run.
Done when every class has a CSV whose row count equals the number of results in results/FIELD.json, the user approved the sample row and the mapping, and state.json marks the batch done.
Compliance
- Use test footage the team owns or has the rights to analyze. For public videos used as extra material, check the license and the platform's terms first and keep them for internal analysis only. Have the user confirm this before Step 3.
- People who walk through the frame (operators, bystanders, people in the building) did not agree to be in a search index. The skill lists ground types only; it does no face or identity recognition and never searches for people. Follow the team's data policy for people in view (some teams blur or crop before any upload), and confirm before Step 3. Runs filmed where children are present (schools, homes, public parks) need that policy to cover minors, which usually means guardian consent or blurring first; if it does not, leave those runs out.
- The skill makes no safety judgment, does not say why a run failed or whether a controller is good, and does not score operators. The CSVs are review lists for people, not annotation files, and they carry no recall or precision figures to quote outside the team.
- The videos stay in the user's Vivu project until the user deletes them. Create the project as private; the default is visible to the whole Vivu organization. Delete videos or the project only when the user asks, and confirm first.
- The skill writes only local files. Any upload or post of the CSVs goes through the approval in Step 6.
Known failure modes
| Symptom | Cause | Fix |
|---|---|---|
| (observed) a ribbed concrete ramp came back as stairs, with a reason about "concrete steps/ribbed staircase" | the reason describes the class asked for, not the frame | label from the sheet; ramps and stepped walls beside the path are looks wrong |
| (observed) the plain stairs wording returned 5 candidates, 1 real and 4 false | step like lines (ramps, abutments, rock heaps) match a loose description | use the tread and riser wording in the table |
| (observed) the tighter stairs wording returned 1 candidate, 1 real, and missed the staircase the first wording found | exclusions and a stricter description push out some real stairs | an empty or short stairs list does not prove the batch has none; say so in the summary |
| (observed) pebbles seen through shallow river water and a muddy track came back as gravel: 5 returned, 2 real, 3 false | the plain gravel wording ignores water and mud | use the dry surface wording in the table, and label stones under water looks wrong |
| (observed) leaves floating in a puddle came back as leaf cover | the search matched leaves, not ground | label it looks wrong under the default rule |
| (observed) one wet ground window covered 122 seconds of a 151 second clip, and a grass window ran 62 seconds | windows do not match the length of the stretch | look at every sheet and set start_mmss and end_mmss from the tiles if the team needs tight edges |
| a tile is too blurred to tell the ground | the camera bobs with the gait | take a full size frame at that second; label can't tell if it still does not settle it |
| upload page asks to sign in or shows an error | the one time upload link expires after 180 seconds | request a new link right before opening it |
| a file is rejected by the browser upload tool | the tool accepts at most 10 MB per call | the user adds the file in their own browser or the Vivu web app |
| "has not granted vivu.write" | Vivu connected read only | the user reconnects Vivu with write access |
| a result's file name does not match BATCH_DIR | Vivu replaced a space, bracket or plus sign in the name | match on listed_name in files.csv |
| a search returns exactly its maximum_results | the batch holds more candidates than one search returns | mark the class cut off, say so in the summary, and offer to split the batch |
| labels on indoor glossy floors, snow, sand or night footage look unreliable | this footage is untested | treat every row as unmeasured and have the reviewer check each one |