Home setting shortlists

---
name: home-setting-shortlists
description: "Sort a batch of head camera household demo videos into candidate CSVs per home setting (kitchen, bathroom, laundry, dim room, daylight) with Vivu. Use when building a robot training subset."
---

Home setting shortlists with Vivu

This skill turns one collection round of head camera household demonstrations (mp4 files your collectors recorded in their own homes and your team has downloaded) into one CSV per home setting on your list: kitchen, bathroom, laundry, a dim room, and a room lit by daylight, plus an optional cluttered counter class. Claude measures the batch, prices it against the user's Vivu plan, uploads it to a private Vivu project, runs one precise search per setting, cuts a contact sheet across every window Vivu returns, labels each window looks right, looks wrong or can't tell from what the frames show, and writes every candidate into its setting's CSV with an empty subset_decision column. The data engineer who builds the training or eval subset makes the call.

The value is in what only the picture holds. The collection sheet says the task and the region; the file name is a collector ID and a timestamp. Neither says the demo was recorded at a crowded kitchen counter, in a small bathroom, at a laundromat, by a single lamp or next to a sunny window, and demos usually have no narration about the room. A head camera mostly sees hands and the counter, so the room has to be read from whatever background is in frame. Vivu's search is used only to shortlist: in our test run a cluttered counter search returned an empty new kitchen, and a daylight window spent most of its time on a shower door before the camera reached the RV's windows. So every candidate is labeled from its frames, rejected candidates stay in the CSV, and a short list never means the batch had no such setting.

When to use

Use when someone asks to sort a batch of egocentric or head camera demos by room, to find the demos recorded in bathrooms or laundry rooms, to check the setting mix of a collection round, or to pull dim and daylight examples for a lighting balanced subset.

Not for these:

1. A one off question about one recording ("which room is this?"): open the file or search Vivu directly and look at the frames.
2. Finding a particular object in the scene (a red bowl, a mug): object and color searches behave differently and are not covered here.
3. People: the skill never searches for, counts or identifies the collector or household members, and does not score collectors.
4. Action or task labels, success or failure of a demo, and frame accurate segments: use the team's annotation tool.
5. Hundreds of hours at once. Pick a subset from the collection sheet first (3 to 5 demos per region and task is a good start); one run of this skill is sized for about an hour of footage.
6. Night footage, infrared and fisheye distortion beyond a normal action camera: untested. The skill can run, but have the reviewer look at every row.

Working principles

1. Report measured numbers, not estimates. When a number is an estimate, say so.
2. Nothing is verified until it has been checked against the source video. Vivu's results are candidates, Claude's labels are a first pass, and the reviewer's subset_decision is the decision.
3. Stop and tell the user when a required capability or tool is missing. Do not guess around it.
4. Ask the user before anything that is expensive to redo (the batch and its index minutes, before upload) and before writing the full CSVs (one sample row first). The skill posts and sends nothing; if the user asks Claude to put the CSVs somewhere, ask again before each write.
5. Labels come from frames, never from Vivu's reason text, which describes the setting it was asked for even when the frame shows something else.
6. The CSVs keep every candidate and carry no recall or precision figures. A search misses some stretches and caps how many come back, so an empty or short list does not prove the batch has none.

What you need before starting

Check each item at the start of the run and tell the user plainly what is missing before doing anything else.

This skill needs Claude Code on the user's computer (the terminal or the Code tab of Claude Desktop), because it reads local files and runs ffmpeg. Claude on the web and cloud sessions cannot reach the files. It downloads nothing, so no residential IP is needed. Nothing recurs, so no scheduler is involved; run it once per collection round.

Inputs to collect

Ask for anything missing, most important first.

1. BATCH_DIR: the folder that holds the mp4 files for this batch. Required.
2. The settings and what counts for each. Default: the five classes in Step 4 with the label rules in Step 5, cluttered counter off. If the team defines a setting differently (for example, a laundromat does not count as laundry, or dim means under a given light level), the team's definition wins; write it into config.json before labeling.
3. BATCH_NAME: a short name such as the collection round and region. It names the working folder and the Vivu project. Default: the name of BATCH_DIR.
4. Batch size. Default: up to 60 minutes of video, or what remains of the plan this month, whichever is smaller.
5. Who reviews the CSVs. Default: the user.

Files and state

Keep everything in one working folder next to BATCH_DIR:

home-BATCH_NAME/
  config.json      BATCH_DIR, settings with query, maximum_results and label rules, Vivu project id
  files.csv        source_file, duration_s, listed_name, video_id
  results/         search results without the result page link, one JSON per setting
  sheets/          one folder per setting with contact sheets and zoomed frames
  SETTING.csv      one per setting: every candidate, Claude's label, empty subset_decision
  state.json       steps done, files uploaded, settings searched with job ids, settings cut off, candidate keys labeled

The commands in Steps 2 to 5 run from the folder that holds both BATCH_DIR and home-BATCH_NAME/, so the relative paths in them resolve.

A rerun reads state.json first and skips finished steps. A file already marked uploaded is never uploaded again, a setting with a saved result is not searched again, and a candidate key (video_id, start_ms and setting) that already has a label is not labeled again. The next collection round gets its own folder and its own Vivu project, so its searches do not return last round's candidates.

Step 1: Check the Vivu connector and the setup

Goal: confirm every requirement before spending anything.

1. Call vivu_get_account. If the tool does not exist, tell the user to add the Vivu connector in Claude (https://mcp.vivu.ai/mcp) and stop. If the account does not show can_create_projects: true, or a later write call fails with "has not granted vivu.write", ask the user to reconnect Vivu and allow write access, then stop until they have.
2. Run ffmpeg -version and ffprobe -version.
3. List BATCH_DIR and run the ffprobe length command on one file.

Done when vivu_get_account shows can_create_projects: true, ffmpeg and ffprobe print their versions, and ffprobe prints a length for a file in BATCH_DIR.

Step 2: Measure the batch and price it

Goal: an indexed set the user's plan can pay for, approved before upload.

1. Run the ffprobe length command on every mp4 in BATCH_DIR and write files.csv with source_file and duration_s. Add listed_name: the file name with each space, bracket, plus sign and percent sign replaced by an underscore, because that is how Vivu will list it.
2. Call vivu_get_usage and show one table:

Plan facts: Free is $0 a month with 20 index minutes a month and 50 search credits a month; Premium is $30 a month with 180 index minutes and 500 search credits. A precise search uses 5 credits in total. A typical round of 35 demos of two minutes each is about 70 index minutes (estimate), which is more than Free covers in a month and fits in Premium.

3. If the batch does not fit, offer levers in this order: pick a smaller subset with the collection sheet (3 to 5 demos per region and task), split the round across two runs or two months, and only then a larger plan. Never drop files the user asked for without saying which ones.
4. Confirm the Compliance items with the user.

The account used in our test run returns no allowance figures, so the comparison with a real plan and the user's approval were not exercised in our test run.

Done when the user approves the file list and its index minutes, and config.json records both.

Step 3: Upload and index

Goal: every file in the batch indexed in a private Vivu project for this batch.

1. Call vivu_list_projects and reuse a project named "Home settings BATCH_NAME" if one exists. Otherwise call vivu_create_project with that name and visibility "private". The default visibility is organization, which shows the project to everyone in the user's Vivu organization, and footage from inside people's homes is sensitive.
2. Call vivu_open_upload_page with the project ID immediately before uploading. The link expires in 180 seconds and works once, so never post or store it. Give it to the user to open in their own browser and select the files, or open it in a browser tool that can attach local files. Claude in Chrome accepts at most 10 MB per upload call and demo videos are usually larger, so they normally go through the user's own browser or the Vivu web app. Never split or recompress a file to fit.
3. Poll vivu_list_videos until every file shows ready. Match each Vivu file name to files.csv by listed_name and record its video_id there and in state.json.

In our test run the 12 videos (23.8 minutes) were all ready 226 seconds after the upload started. They went through the same upload page with an automated browser, so opening the link in the user's own browser was not exercised in our test run.

Done when every file in files.csv shows ready and has a video_id.

Step 4: Search each setting

Goal: a saved list of candidate windows for every setting.

The query for each field, word for word (one per line, field name first):

kitchen: the camera is in a home kitchen: a counter where food is being prepared, a kitchen sink, a stove or kitchen cabinets are in view
bathroom: the camera is in a bathroom: a shower, bathtub, toilet or bathroom sink vanity is in view
laundry: washing machines or clothes dryers fill the view, in a laundry room or laundromat
dim: the room is dark, lit by a single lamp or a stove flame, so most of the picture is near black
daylight: an indoor room lit by daylight coming through windows
cluttered_counter: a kitchen counter or table crowded with many loose items around the work area

The searches run side by side, one per setting, because each setting is its own list for the reviewer. Within a setting the chain is: the precise search finds candidate windows, the contact sheet shows what the camera saw across each window, Claude labels it, and the reviewer decides. Every search is precise because only precise returns a time window; fast returns whole files with an empty reason, and the batch is already a chosen subset. Household demos usually have no narration about the room and no text on screen, so there is no second modality to switch to. Finding a room or a lighting condition rests on Vivu's visual search, which our capability notes rate weak: it misses some stretches, and its reason can describe what was asked for rather than what is on screen. The numbers in Step 5 come from one small test corpus, which is why every window is checked on contact sheets before it gets a label.

Each wording is the first one we tried; none was reworded. Cluttered counter is off by default: how crowded is crowded is a matter of degree, and in our test run it was the class with a false row and a row too dark to judge, so turn it on only when the reviewer will judge it from the sheets. A setting the team adds (a bedroom, a garage) is untested until its first candidates have been through Step 5.

maximum_results is also the ceiling on how many candidates come back. 40 leaves room for an hour of footage with several stretches per demo; a search that returns exactly 40 was cut off, and the batch holds more candidates than one search can return. Then tell the user, mark the setting cut off in state.json, and offer to split the batch into two smaller batches with their own projects (quote the index minutes again and wait for approval as in Step 2).

In our test run no search came close to 40, so a cut off search was not exercised in our test run.

Run each query with vivu_search_videos (project_id, query, mode, maximum_results). It returns a job ID. Call vivu_get_search_results until complete is true; each status call can wait up to 45 seconds, so a pending search is not a stalled one. Save each completed result as results/FIELD.json without its result_page_url field, and record the job_id in state.json. Show the Vivu result page link in the live reply only; it expires after four hours, so it never goes into a saved file or a CSV.

Done when every setting has a completed result saved in results/, and every setting whose search returned exactly its maximum_results is marked cut off in state.json.

Step 5: Make contact sheets for every candidate and label it

Goal: every candidate labeled looks right, looks wrong or can't tell from what its frames show.

For each result in results/FIELD.json, cut contact sheets of its window from the local file at two frames a second:

mkdir -p home-BATCH_NAME/sheets/FIELD
ffmpeg -nostdin -v error -y -ss START -to END -i FILE -vf "fps=2,scale=384:-2,tile=5x4" home-BATCH_NAME/sheets/FIELD/rRANK_STEM_STARTMS_%02d.png

FIELD is the setting, FILE the local source file (BATCH_DIR/ followed by its source_file), STEM its name without .mp4, RANK the result_number written with two digits (r01, r09) so the files sort by rank, STARTMS the result's start_ms, and START and END the result's start_ms and end_ms divided by 1000. Each sheet holds 20 tiles covering 10 seconds of the window; the last sheet is padded with black tiles. Tile k on sheet n (both counted from 0, left to right and top to bottom) is at START + 10 * n + k / 2 seconds. Look at every sheet of a window. Keep -nostdin when the command runs inside a loop that reads the candidate list, because otherwise ffmpeg reads that list as input.

Setting windows are often long, because a demo stays in one room for minutes. For a window longer than 40 seconds, use fps=0.5 in the same command (one tile every 2 seconds); then tile k on sheet n is at START + 40 * n + 2 * k seconds.

A head camera points down at the hands most of the time, so the clue to the room is often a strip of background at the edge of the frame. When a tile might show a sink, a tub edge or a washer but the sheet does not settle it, extract a zoomed full size frame at that tile's second:

ffmpeg -nostdin -v error -y -ss SECONDS -i FILE -frames:v 1 -vf "crop=iw/2:ih/2:X:Y,scale=iw*2:-2" home-BATCH_NAME/sheets/FIELD/rRANK_STEM_SECMS_zoom_PART.png

SECONDS is the time of the tile, SECMS the same time in milliseconds, X is 0 for the left half or iw/2 for the right half, Y is 0 for the top half or ih/2 for the bottom half, and PART names the crop (for example left_bottom) so two crops of the same second get different files.

In our test run the sheets settled every candidate except one too dark to judge, so the zoomed frame was not exercised in our test run.

Label rules (defaults; the team's definitions from Inputs win):

1. kitchen looks right when a sink, stove, fridge, kitchen cabinets or a counter with cooking work is in most of the window. A head camera looking down at a counter with food, boards and utensils counts; say in the note when no sink or stove is in frame. A dining table alone looks wrong.
2. bathroom looks right when a shower, tub, toilet or bathroom vanity is in most of the window. A long stretch inside a cabinet with the room only implied is can't tell unless the first frames show the bathroom; say so in the note.
3. laundry looks right when washers or dryers fill most of the window. Say in the note whether it is a home laundry room, a building laundry room or a laundromat, because many teams want homes only.
4. dim looks right when most of each frame is near black and only a small lit area shows. A normally lit room at evening looks wrong.
5. daylight looks right when an indoor room is lit through windows that are in frame or clearly light the room. Outdoor shots, and rooms where no window or daylight shows, look wrong. A window where the window light appears only for a few seconds is can't tell.
6. cluttered_counter looks right when the counter or table around the work area is crowded with many loose items. A few items on an empty counter look wrong; a borderline counter is can't tell, and the reviewer decides.
7. can't tell when the zoomed frame still does not settle it (too dark, hands fill the frame, motion blur). The reviewer looks at the source.

Write a note of what the frames show for every candidate, such as "sink, air fryer and stove from head height" or "laundromat, not a home: row of top load washers". Never copy the reason into the note. Record each candidate key and its label in state.json.

Worked example from our test run

The test corpus was public household footage published under a Creative Commons license, used in place of a robot team's collection because none was available: 12 videos (23.8 minutes) cut from home cooking and cleaning videos, with the audio removed and the files renamed so the names said nothing about content. Fewer than half were head or chest camera point of view (dumpling sessions at a crowded counter, an air fryer kitchen, an RV shower, a laundromat); the others were handheld house walkthroughs, a tripod kitchen shot, a short film in a building laundry room and cinematic hand close-ups, most at 360p. No reusable real one lamp home footage was found, so one dim clip is a copy of a kitchen clip with the exposure lowered; the other dim stretches are a gas stove in a dark kitchen. Settings were marked by hand on contact sheets before any search, and a garage search was run as a control, with no garage anywhere in the batch.

1. kitchen: 12 candidates, 12 real and 0 false, and 3 kitchen stretches marked by hand missed (short glimpses of empty kitchens in the walkthroughs). Every head camera clip looking down at a counter came back, including the darkened one. The widest window ran 117 seconds.
2. bathroom: 7 candidates, 7 real and 0 false, 0 missed. One window spent most of its 52 seconds inside an under-sink cabinet.
3. laundry: 4 candidates, 4 real and 0 false, 0 missed; the laundromat and the building laundry room are not homes.
4. dim: 3 candidates, 3 real and 0 false, 0 missed, but one of the dim stretches is the darkened copy, and its window stopped at 01:02 of a clip that stays dark to the end.
5. daylight: 11 candidates, 10 real, 0 false and 1 can't tell, and 1 bedroom stretch missed. The can't tell window was mostly a shower door and a spray bottle, with the RV's windows only in the last seconds, and the laundromat's glass front also counted as real, though a laundromat is not a home.
6. cluttered_counter: 6 candidates, 4 real, 1 false (an empty new kitchen), 1 too dark to judge, and 1 busy counter stretch missed.
7. The garage control returned 0 windows. The 12 videos were all ready 226 seconds after the upload started, and the six CSVs held 43 rows in total.

Done when every candidate in results/ has at least one contact sheet in sheets/ and a label in state.json.

Step 6: Write one CSV per setting and hand it over

Goal: a review list per setting that a data engineer can work through without opening Vivu.

Show the user one sample row and the field mapping, and wait for an OK before writing the rest:

source_file,rank,start_mmss,end_mmss,search_reason,contact_sheets,claude_label,claude_note,subset_decision
HC_04.mp4,9,01:02,01:21,"The camera pans to reveal indoor rooms in an RV (bedroom and living/kitchen area) illuminated by daylight entering through the windows.",sheets/daylight/r09_HC_04_62000_01.png;sheets/daylight/r09_HC_04_62000_02.png,can't tell,"frosted shower door and spray bottle for most of the window; RV windows and a shaded bedroom window only after 01:12",

The row above is from our test run's daylight CSV; the user's rows carry their own file names.

claude_label and claude_note are Claude's reading of the frames, not a measurement. Whether a room counts for the subset, or a counter is crowded enough, depends on the team's definition, so the reviewer goes through every row, the looks right ones included, before anything enters a dataset.

Write FIELD.csv for every setting with one row per result, sorted by rank, including the rows labeled looks wrong: a reviewer may disagree with a label, and the rejected rows show what the search confuses. A demo can appear in several CSVs (a dim kitchen is in kitchen.csv and dim.csv), and one long window can cover more than one room; the note says so. The CSV uses the user's own file names and times, never a Vivu result page link, so it stays usable after the link expires.

Then tell the user, per setting, how many candidates came back, how many Claude labeled looks right, looks wrong and can't tell, and whether the list was cut off at maximum_results. Say plainly that an empty or short list does not mean the batch has none, that brief glimpses of a room in a walkthrough are the easiest to miss, and that windows can stop before a long stretch ends. Do not add recall or precision figures: the hand marks needed to measure them do not exist for the user's batch.

The CSVs stay in the working folder. Passing them on is the user's step. If the user asks Claude to upload or post them (a shared drive, a tracker, a chat), name the destination, show the rendered first rows and wait for a yes before each write.

In our test run the CSVs were written from the dry run's results and sheets; showing the sample row to a user and handing the CSVs over were not exercised in our test run.

Done when every setting has a CSV whose row count equals the number of results in results/FIELD.json, the user approved the sample row and the mapping, and state.json marks the batch done.

Compliance

1. Use demo footage the team collected under its own consent and collection agreement, or has the rights to analyze. For public videos used as extra material, check the license and the platform's terms first and keep them for internal analysis only. Have the user confirm this before Step 3.
2. Collectors and household members appear on camera, and homes show addresses, mail, photos and screens. Handle them within the consent the collection agreement covers. The skill sorts recordings by setting only; it does no face or identity recognition, never searches for people and does not score collectors. Follow the team's data policy for people in frame (some teams blur faces or redact screens before any upload). Children are common in homes: if a recording shows children, follow the guardian consent terms of the agreement, and if the batch is meant to gather footage of children, stop and do not upload it. Confirm before Step 3.
3. The videos stay in the user's Vivu project until the user deletes them. Create the project as private; the default is visible to the whole Vivu organization. Delete videos or the project only when the user asks, and confirm first.
4. The CSVs are review lists for people, not annotations or labels for training. The skill writes only local files; any upload or post of the CSVs goes through the approval in Step 6.

Known failure modes