Teleop capture problem list

---
name: teleop-capture-problem-list
description: "Find teleop episodes whose cameras recorded badly (view blocked, camera crooked, black, frozen, blown out) with Vivu and ffmpeg, and get a problem sheet for collection ops."
---

Teleop capture problem list with Vivu

This skill takes one teleop collection batch (one video file per episode and camera, exported from your logs) and returns a problem sheet for collection ops: one row per recording problem with the episode file, the camera, the problem type, the window as MM:SS, a contact sheet (one frame every 0.5 seconds, or start, middle and end frames when the window is the whole file), where the finding came from (a Vivu search checked on frames, or a local ffmpeg check), and an empty decision column for re-collect, drop or keep. Black, frozen and overexposed video is found locally with ffmpeg and costs no Vivu credits. A blocked or crooked camera is found with one Vivu search per problem type, and every window is checked on its own frames before it reaches the sheet.

The value is in what only the footage shows. A collection log records the task name, the success flag and the joint states; it does not record that the operator's arm or the robot arm covered the outside camera for half the episode, that a camera was mounted sideways, or that the wrist lens was pressed against the object. A run flagged as a success can still be useless for training because the camera saw nothing. The sheet lists the problems and leaves every decision to the people who run collection.

When to use

Use when someone asks to find bad recordings in a teleop batch, check which episodes had a blocked or knocked camera, build a re-collect list for collection ops, or screen a LeRobot, RLDS or ROS bag batch for camera problems before training. For one episode, open the video. This skill does not check task labels (use a label spot-check for that), produce annotations or label exports, score operators, or say why an episode failed.

Working principles

1. Report real counts. When a number is an estimate (credits, minutes for a future batch), say so.
2. A Vivu result is a candidate until Claude has looked at the frames inside its window. Rejected candidates go to a separate list with the reason; they never reach the problem sheet.
3. Stop and tell the user when a required tool or capability is missing. Do not guess around it.
4. Ask the user before anything that costs index minutes or search credits (Step 4), before uploading (Step 5), and before the sheet leaves the working folder (Step 7).
5. The sheet is for the data team and collection ops. It names files, cameras and windows; it carries no operator names, scores or failure causes.

What you need before starting

Check each item at the start of the run and tell the user plainly what is missing before doing anything else.

The skill runs in Claude Code on the user's computer (terminal or the Code tab of Claude Desktop), because it needs a shell and the local episode files. It needs no residential IP, since nothing is downloaded from YouTube, and it sets up no scheduled task: run it once per batch.

Inputs to collect

Ask for anything missing, most important first.

1. The batch folder (required).
2. Which cameras are external (fixed beside the robot) and which are wrist cameras, and how the file names say so (required). Default naming: EPISODE_CAMERA.mp4, for example ep080_ext1.mp4 and ep014_wrist.mp4.
3. Problem types. Default: all six (blocked view, camera off, wrist lens covered, black, frozen, overexposed).
4. Scope. Default: every file in the batch. Step 4 offers smaller scopes when the plan does not cover it.
5. The working folder. Default: capture-check/ next to the batch folder.
6. The Vivu project name. Default: "Capture check BATCH_ID", created private.

Files and state

Keep everything in one working folder:

capture-check/
  config.json             batch id, batch folder, camera naming rule, problem types, Vivu project id
  files.csv               episode_file, camera, path, duration_s, stored_name, video_id
  state.json              local checks done, uploaded video ids, finished searches with job ids, checked windows
  h264/                   H.264 copies of AV1 files (Step 2), same base names; nothing is written into the batch folder
  local/                  detect_NAME.log and bright_NAME.txt from Step 3
  results/PROBLEM.json    raw result of each problem search, saved as returned
  sheets/                 contact sheets and frame strips for every window
  capture_problems.csv    the problem sheet
  rejected.csv            candidates rejected on their frames, with the reason

state.json lets a rerun resume: skip files whose local check log exists, skip uploads whose video id is recorded, skip searches that already have a saved result, and skip windows already checked. Each batch gets its own working folder, so a problem is never listed twice.

Step 1: Check the Vivu connector and the tools

Goal: confirm that everything in the table above is in place.

1. Call vivu_get_account. If the tool does not exist, tell the user to add the Vivu connector in Claude (https://mcp.vivu.ai/mcp) and stop. If can_create_projects is not true, or a later write fails with "has not granted vivu.write", ask the user to reconnect Vivu with write access and stop.
2. Run ffmpeg -version and ffprobe -version. If either is missing, ask the user to install ffmpeg.
3. Confirm the batch folder can be read and the camera naming rule matches the files.

Done when vivu_get_account shows can_create_projects: true, both ffmpeg commands print a version, and every file in the batch maps to an external or a wrist camera.

Step 2: Line up the batch

Goal: files.csv with one row per file, its camera and its length.

1. If the batch is still in ROS bag, MCAP or RLDS form, export each camera stream to one mp4 per episode with the team's own export script. Log export was not exercised in our test run: the public dataset we used already ships one mp4 per camera per episode.
2. LeRobot datasets encode video as AV1. In our test run we re-encoded each file to H.264 before uploading; uploading AV1 directly was not tested. This is a format conversion, not a way to fit an upload limit:

ffmpeg -v error -i EPISODE_FILE -c:v libx264 -pix_fmt yuv420p -crf 18 -an OUTPUT_FILE

EPISODE_FILE is the exported file. OUTPUT_FILE is capture-check/h264/ plus the same base name. When a copy exists, every later step uses the copy.

3. Measure each file:

ffprobe -v error -show_entries format=duration -of csv=p=0 EPISODE_FILE

4. Show the user the count per camera and any file whose camera could not be read from its name.

Done when files.csv has one row per file with camera and duration_s, and the user has seen the counts per camera.

Step 3: Run the local checks

Goal: black, frozen and overexposed windows for every file, without spending Vivu credits.

One ffmpeg pass per file reports all three:

ffmpeg -hide_banner -nostats -i EPISODE_FILE -vf "blackdetect=d=0.5:pix_th=0.10,freezedetect=n=0.003:d=1,signalstats,metadata=select:key=lavfi.signalstats.YAVG:value=180:function=greater,metadata=print:key=lavfi.signalstats.YAVG:file=capture-check/local/bright_NAME.txt" -an -f null - 2> capture-check/local/detect_NAME.log

NAME is the file's base name. Then read the results:

1. Black: lines with black_start and black_end in detect_NAME.log.
2. Frozen: lines with freeze_start and freeze_end. Black video does not change, so a freeze that sits inside a black window is the same problem; keep the black row only.
3. Overexposed: bright_NAME.txt lists every frame whose average brightness (YAVG, 0 to 255) is above 180, with pts_time. Merge runs of consecutive frames into windows; a window shorter than half a second is noise. We tried negate plus blackdetect for overexposure first and it found nothing: an overexposed frame still has dark areas, so the average is the better signal.
4. Make a contact sheet for each window (Step 6 has the command) and look at it before adding the row; a dark room can trip blackdetect and a white wall can trip the brightness check.

In our test run the 12 real camera streams raised no local alarm, and on a separate test clip where we injected black, a freeze and overexposure with ffmpeg, each of them was found at the injected time.

The thresholds come from that single injected clip. Tune pix_th, n and the brightness threshold if your cameras run dark or bright.

Done when every file has a detect log and a bright file in local/, and every window found has a contact sheet and a row ready for the sheet.

Step 4: Price the batch and ask

Goal: the user approves the cost before anything is uploaded.

1. Index minutes = the sum of duration_s divided by 60, for the files that will be searched (skip files that are black or frozen end to end; Step 3 already settled them). Search credits = one precise search per problem type; a precise search uses 5 credits. Label it an estimate, since a rewording costs another search.
2. Call vivu_get_usage for the plan and what remains this month. For reference, Free is 20 indexing minutes a month and 50 search credits a month; Premium is $30 a month for 180 indexing minutes a month and 500 search credits a month.
3. Show one table:

4. If it does not fit, offer these levers in order, each with its saving: search only the external cameras (the wrist search is the least tested); search only today's new episodes; sample a fixed number of episodes per rig; move to a larger plan last. For example, 50 episodes of about 40 seconds on one camera is about 33 index minutes, more than a Free month and inside Premium; three problem types are an estimated 15 credits.

Approval was not exercised in our test run (it ran unattended on an admin account that reports no remaining allowance).

Done when the user has seen the table and said yes to a scope.

Step 5: Create a private project and upload

Goal: every file in the approved scope shows ready in Vivu.

1. Ask the user to confirm the Compliance points before uploading.
2. Call vivu_list_projects and reuse the project for this batch if one exists. Otherwise call vivu_create_project with the project name and visibility "private". The default visibility is the whole organization, which is wrong for lab and home recordings.
3. Call vivu_open_upload_page with the project ID right before the upload. The upload_url is a one time sign in link that expires in 180 seconds, so request it only when the user or the browser tool is ready, never post it in a message, and request a new one if it lapses.
4. Hand the link to the user to open in their own browser and select the files, or open it with a browser tool that can attach local files. Claude in Chrome takes at most 10 MB per upload call, so send files in batches under that. If a file cannot go through, ask the user to add it in the Vivu web app. Never split or recompress a file to fit.
5. Poll vivu_list_videos until every video shows ready. Vivu replaces spaces and brackets in file names with underscores, so match rows back by the episode index and camera in the name and check that duration_ms agrees with duration_s. Record stored_name and video_id in files.csv and the video ids in state.json.

In our test run the upload page link was opened by an automated browser on our machine, not by a user, and the compliance confirmation was not exercised.

Done when vivu_list_videos shows every approved file as ready and each one is matched to a row in files.csv.

Step 6: Search each problem type and check every hit

Goal: for each problem type, the windows whose frames show the problem.

The standing queries, one per problem type:

The chain: the local checks in Step 3 settle the problems that need no understanding of the picture, for free; one precise search per remaining problem type then finds the windows across the whole project; the 0.5 second contact sheet confirms or rejects each window. There is no fast pass: fast returns whole files with no reason, and every file here is short and gets checked anyway. The footage has no speech or on-screen text, so the flip in this skill is from what the log says (success) to what the camera shows (nothing usable).

1. For each problem type, call vivu_search_videos with project_id, query, mode "precise" and maximum_results equal to the number of files in the project (at most 100). The value is the recall ceiling: a smaller number silently drops problems. If the batch has more than 100 files, split it into projects of 100 or fewer.
2. Call vivu_get_search_results with the job ID until complete is true. Each call can wait up to 45 seconds, so a pending search is not a stalled one. Save the returned JSON unchanged as results/PROBLEM.json and record the job ID in state.json. The Vivu result page link from a completed search can go in the live reply; it expires after four hours, so it never goes in the sheet.
3. A hit on a file of the wrong camera (a wrist file for blocked_view or camera_off, an external file for wrist_covered) is false: write it to rejected.csv with the reason "wrong camera" and count it.
4. For every other window, make a contact sheet at one frame every 0.5 seconds:

ffmpeg -v error -y -ss START -t LENGTH -i EPISODE_FILE -vf "fps=2,scale=320:-1,tile=6x4" -frames:v 1 capture-check/sheets/NAME_PROBLEM.png

START and LENGTH are the window start and length in seconds (from start_ms and end_ms). For windows longer than 12 seconds, raise the tile rows (6x6 holds 18 seconds) or use fps=24/LENGTH so the sheet still holds 24 tiles. When the window is the whole file (camera_off usually is), a strip of the start, middle and end frames is enough:

ffmpeg -v error -y -ss 0.2 -i EPISODE_FILE -ss MID -i EPISODE_FILE -ss END -i EPISODE_FILE -filter_complex "[0:v][1:v][2:v]hstack=inputs=3" -frames:v 1 capture-check/sheets/NAME_PROBLEM.png

MID is half the duration, END the duration minus 0.3 seconds. Look at every sheet. Blocked view is real only when something near the lens covers more than half the picture and hides the spot where the gripper works; the arm being large in the frame while the table stays visible is not a problem. Camera off is real when the picture is rotated or tilted far off level, or the work area is out of frame; a camera that faces a wall because the task is on the wall is fine. Judge from the frames, not from the reason text, which describes the query back.

5. Tell the user the counts per problem type: returned, real, false.

How far to trust the search: telling a blocked or crooked camera from a normal one is visual search, which is weak for Vivu, and a person in front of the lens, a camera knocked mid episode and a covered wrist lens were not in our test batch. That is why every window goes through the frames, why the sheet says where each row came from, and why an empty result is not proof the batch is clean: an empty result does not prove the footage has no such moment. Partial blockings, where the arm covers about half the picture, were mostly not returned in our test run; if they matter to you, page through the contact sheets of the files with the most arm motion.

Worked example from our test run

In our test run on 12 camera streams from a public robot manipulation dataset (5.18 minutes in total; external cameras and wrist cameras, low resolution, no audio; home and lab scenes), all files were ready within 125 seconds of the upload starting. We wrote the truth from contact sheets before any search. The batch had two clear blocked views (the robot arm swinging close to the outside camera), a few partial ones, one camera mounted sideways, one tilted far off level, hard negatives (the robot base large in the frame, a camera facing a wall for a light switch task) and wrist close-ups of cloth and towels during normal grasps. Nobody stepped in front of a lens in this dataset, no camera was knocked mid episode, and no wrist lens was covered.

The blocked_view search returned 3 windows: 3 real, 0 false, 0 missed against the clear truth. One of the 3 was a partial blocking that our pre-written rule counts as real; the other partial blockings were not returned. No wrist close-up and no hard negative came back. The median window was 8.7 seconds, wider than a blocking, so the contact sheet is what shows when it happens.

The first camera_off wording ("the camera has been knocked or mounted crooked ...") returned 6 windows that each span the whole file: 2 real, 4 false. The false ones were every wrist file in the batch (a wrist view always looks tilted) and the light switch camera. After 1 rewording on the same corpus, naming the fixed camera and describing what a tilted picture looks like, it returned 3: 2 real, 1 false, the light switch camera again. The table above uses the reworded query, and step 3 drops hits on files from the wrong camera on top of it.

The wrist_covered search returned 0, and the wrist close-ups were not returned. With no covered lens in the batch, this says nothing about recall; treat that search as untested.

The run used 4 precise searches, an estimated 20 credits; a batch with all three problem types is an estimated 15 credits.

Done when every problem type has a saved result, every returned window is judged real or false in state.json, and the user has seen the counts.

Step 7: Build the problem sheet

Goal: capture_problems.csv with one row per confirmed problem, and rejected.csv.

1. Show the user one sample row and the field mapping below, and wait for a yes before writing the full sheet:

Sample row from our test run:

episode_file,camera,problem,window,contact_sheet,source,check_note,decision
ep080_ext1.mp4,external,blocked view,00:13-00:25,sheets/ep080_ext1_blocked_view.png,Vivu search + frame check,arm fills most of the picture 16-20 s; can and gripper hidden,

2. Write capture_problems.csv: confirmed Vivu windows plus the local check windows, sorted by episode file. File names are the only link to the video; no Vivu links. Write rejected.csv with every rejected window and its reason.
3. Tell the user the rows per problem type and the number rejected, and say which problem types were searched and which were skipped.
4. The sheet stays in the working folder. If the user wants it in a shared drive or a tracker for collection ops, name the destination, show the rendered file and wait for approval; that was not exercised in our test run.

Done when capture_problems.csv has one row per confirmed problem with an existing contact sheet path, rejected.csv lists every rejected window, and the user has the files.

Compliance

1. Rights to the footage: use it on data your team owns or may process. For public datasets, follow the dataset license and keep the videos for internal checks. Confirm before Step 5.
2. People on camera: teleop recordings often catch operators or people walking past. Handle them under the team's data policy, and leave out episodes that show people who did not agree to be recorded. Confirm before Step 5.
3. Minors: if a home collection might show children, get explicit guardian consent or leave those episodes out. Confirm before Step 5.
4. Where the videos go: uploaded files stay in the user's Vivu project until the user deletes them. The skill creates the project as private and never deletes projects or videos unless the user asks, and then confirms first.
5. What the skill does not do: no face or identity recognition, no operator scoring, no failure diagnosis, no annotation or label export.

Known failure modes