Warehouse aisle situation shortlists

---
name: warehouse-aisle-situation-shortlists
description: "Sort warehouse robot camera clips into candidate CSVs per aisle situation (person in view, pallets on the floor, loose debris) with Vivu. Use when curating AMR eval or training data."
---

Warehouse aisle situation shortlists with Vivu

This skill turns a batch of front camera clips from warehouse robots (AMRs, AGVs or autonomous forklifts), exported from the robot logs to mp4 and usually cut around logged stops or slowdowns, into one CSV per aisle situation: a person on the floor in view, a pallet or a stack of goods standing on the floor, and loose debris on the floor (packaging scraps, stretch wrap, a dropped bag or box). Claude measures the batch, prices it against the user's Vivu plan, uploads it to a private Vivu project, runs one precise search per situation, cuts contact sheets across every window Vivu returns (a frame every half second, every 2 seconds for windows over 40 seconds), labels each window looks right, looks wrong or can't tell, and writes every candidate into its situation's CSV with an empty reviewer_verdict column. The perception data engineer decides what goes into the eval or training set.

The value is in what only the picture holds. The robot log says when the robot stopped or slowed down, its speed and maybe an obstacle distance; it does not say whether the obstacle was a picker, a pallet left at the end of an aisle or a torn piece of stretch wrap. Exported camera streams have no audio or captions to search, and file names carry only a robot number and a time. Vivu's search is used only to shortlist: in our test run the pallet search returned goods sitting on flow rack levels and robots carrying totes as pallets on the floor, and a search for parked forklifts returned the camera's own stopped forklift. So every candidate is labeled from its frames, rejected candidates stay in the CSV, and a short list never means the batch had no such moment.

When to use

Use when someone asks to find the clips where a person was in front of the robot, to pull stops caused by pallets or goods on the floor for an eval set, to find debris on the floor before labeling, or to sort a batch of stop events by what the camera saw.

Not for these:

1. A one off question about one clip ("was someone in front of the robot here?"): search Vivu directly and look at the frames.
2. Whether a pallet or a person is inside the robot's lane or beside it. The pallet search finds pallets in view; the note says where they are, and the reviewer decides.
3. Parked forklifts, pallet jacks and carts. A search for them in our test run returned 12 windows, 3 real and 7 false (the camera's own stopped forklift, a worker pulling a jack), so the skill does not offer it.
4. Identifying people, scoring pickers or judging whether a stop was right or safe. The skill only lists that someone was in view.
5. Frame accurate labels, masks or bounding boxes, or anything on the robot in real time: use the team's annotation tool and its own detectors.
6. Hundreds of hours at once. Pick a subset from the logs first (sites, shifts, stop types); one run of this skill is sized for about an hour of clips.
7. Night, low light, thermal and depth streams, and cameras mounted at robot height: mostly untested. The skill can run, but have the reviewer look at every row.

Working principles

1. Report measured numbers, not estimates. When a number is an estimate, say so.
2. Nothing is verified until it has been checked against the source video. Vivu's results are candidates, Claude's labels are a first pass, and the reviewer's verdict is the decision.
3. Stop and tell the user when a required capability or tool is missing. Do not guess around it.
4. Ask the user before anything that is expensive to redo (the batch and its index minutes, before upload) and before writing the full CSVs (one sample row first). The skill posts and sends nothing; if the user asks Claude to put the CSVs somewhere, ask again before each write.
5. Labels come from frames, never from Vivu's reason text, which describes the situation it was asked for even when the frame shows something else.
6. The CSVs keep every candidate and carry no recall or precision figures. A search misses some moments and caps how many come back, so an empty or short list does not prove the batch has none.

What you need before starting

Check each item at the start of the run and tell the user plainly what is missing before doing anything else.

This skill needs Claude Code on the user's computer (the terminal or the Code tab of Claude Desktop), because it reads local files and runs ffmpeg. Claude on the web and cloud sessions cannot reach the files. It downloads nothing, so no residential IP is needed. Nothing recurs, so no scheduler is involved; run it once per batch.

Inputs to collect

Ask for anything missing, most important first.

1. BATCH_DIR: the folder that holds the exported mp4 clips for this batch. Required.
2. The situations and what counts for each. Default: the three classes in Step 4 with the label rules in Step 5. If the team defines a class differently (for example, only people inside the robot's lane count), the team's definition wins; write it into config.json before labeling.
3. Event times. If the team's cutting tool puts the stop time in the file name or a sidecar CSV, say where, and Claude copies it into the event_time column. Default: leave event_time empty.
4. BATCH_NAME: a short name such as a site and a week. It names the working folder and the Vivu project. Default: the name of BATCH_DIR.
5. Batch size. Default: up to 60 minutes of video, or what remains of the plan this month, whichever is smaller.
6. Who reviews the CSVs. Default: the user.

Files and state

Keep everything in one working folder next to BATCH_DIR:

warehouse-BATCH_NAME/
  config.json      BATCH_DIR, classes with query, maximum_results and label rules, event time source, Vivu project id
  files.csv        source_file, duration_s, listed_name, video_id, event_time
  results/         search results without the result page link, one JSON per class
  sheets/          one folder per class with contact sheets and zoomed frames
  CLASS.csv        one per class: every candidate, Claude's label, empty reviewer_verdict
  state.json       steps done, files uploaded, classes searched with job ids, classes cut off, candidate keys labeled

The commands in Steps 2 to 5 run from the folder that holds both BATCH_DIR and warehouse-BATCH_NAME/, so the relative paths in them resolve.

A rerun reads state.json first and skips finished steps. A file already marked uploaded is never uploaded again, a class with a saved result is not searched again, and a candidate key (video_id, start_ms and class) that already has a label is not labeled again. The next batch gets its own folder and its own Vivu project, so its searches do not return last week's candidates.

Step 1: Check the Vivu connector and the setup

Goal: confirm every requirement before spending anything.

1. Call vivu_get_account. If the tool does not exist, tell the user to add the Vivu connector in Claude (https://mcp.vivu.ai/mcp) and stop. If the account does not show can_create_projects: true, or a later write call fails with "has not granted vivu.write", ask the user to reconnect Vivu and allow write access, then stop until they have.
2. Run ffmpeg -version and ffprobe -version.
3. List BATCH_DIR and run the ffprobe length command on one file.

Done when vivu_get_account shows can_create_projects: true, ffmpeg and ffprobe print their versions, and ffprobe prints a length for a file in BATCH_DIR.

Step 2: Measure the batch and price it

Goal: an indexed set the user's plan can pay for, approved before upload.

1. Run the ffprobe length command on every mp4 in BATCH_DIR and write files.csv with source_file and duration_s, plus event_time when the team gave a source for it. Add listed_name: the file name with each space, bracket, plus sign and percent sign replaced by an underscore, because that is how Vivu will list it.
2. Call vivu_get_usage and show one table:

Plan facts: Free is $0 a month with 20 index minutes a month and 50 search credits a month; Premium is $30 a month with 180 index minutes and 500 search credits. A precise search uses 5 credits in total. For example, a batch of 35 stop clips of 1.5 minutes each is about 53 index minutes (estimate), which is more than Free covers in a month and fits in Premium.

3. If the batch does not fit, offer levers in this order: pick a smaller subset with the logs (the sites, shifts or stop types the dataset needs most), cut shorter windows around each stop, split the batch across two runs or two months, and only then a larger plan. Never drop clips the user asked for without saying which ones.
4. Confirm the Compliance items with the user.

The account used in our test run returns no allowance figures, so the comparison with a real plan and the user's approval were not exercised in our test run.

Done when the user approves the file list and its index minutes, and config.json records both.

Step 3: Upload and index

Goal: every file in the batch indexed in a private Vivu project for this batch.

1. Call vivu_list_projects and reuse a project named "Warehouse BATCH_NAME" if one exists. Otherwise call vivu_create_project with that name and visibility "private". The default visibility is organization, which shows the project to everyone in the user's Vivu organization, and footage from a customer's warehouse with workers in it is sensitive.
2. Call vivu_open_upload_page with the project ID immediately before uploading. The link expires in 180 seconds and works once, so never post or store it. Give it to the user to open in their own browser and select the files, or open it in a browser tool that can attach local files. Claude in Chrome accepts at most 10 MB per upload call and camera clips are often larger, so they normally go through the user's own browser or the Vivu web app. Never split or recompress a file to fit.
3. Poll vivu_list_videos until every file shows ready. Match each Vivu file name to files.csv by listed_name and record its video_id there and in state.json.

In our test run the 12 videos (23.6 minutes) were all ready about 365 seconds after the upload started. They went through the same upload page with an automated browser, so opening the link in the user's own browser was not exercised in our test run.

Done when every file in files.csv shows ready and has a video_id.

Step 4: Search each situation class

Goal: a saved list of candidate windows for every class.

The query for each field, word for word (one per line, field name first):

person_in_aisle: a person standing or walking on the warehouse floor in front of the camera, with most of their body visible; the camera operator's own hands or arms, a face filmed close up selfie style, and people seen only in a mirror do not count
pallet_on_floor: a pallet, a pallet load, or a stack of cartons or crates standing on the floor outside the racking, in an aisle, a staging area or an open floor area; goods on rack shelves, a load carried on forklift forks, and totes riding on robots or conveyors do not count
floor_debris: loose packaging, torn plastic stretch wrap, cardboard scraps, spilled goods or a fallen box lying loose on the floor where forklifts and people move; goods stacked on pallets, totes on robots, cartons on shelves, painted floor markings, stains and shiny floor reflections do not count

The searches run side by side, one per class, because each class is its own list for the reviewer. Within a class the chain is: the precise search finds candidate windows, the contact sheet shows what the camera saw across each window, Claude labels it, and the reviewer decides. Every search is precise because only precise returns a time window; fast returns whole files with an empty reason, and the batch is already a subset picked from the logs. Exported robot camera streams have no audio and no on screen text about the scene, so there is no second modality to switch to.

All three wordings ran once on our test footage without rewording (see the worked example). The pallet wording asks for pallets in view, not for pallets sticking into the aisle: the lane is a spatial judgment that searches in our earlier outdoor tests got wrong, so the reviewer judges it from the sheets. Keep the exclusions unless the team's definition differs.

maximum_results is also the ceiling on how many candidates come back. 40 leaves room for an hour of clips with several moments per clip; a search that returns exactly 40 was cut off, and the batch holds more candidates than one search can return. Then tell the user, mark the class cut off in state.json, and offer to split the batch into two smaller batches with their own projects (quote the index minutes again and wait for approval as in Step 2). In our test run no search came close to 40, so a cut off search was not exercised in our test run.

Run each query with vivu_search_videos (project_id, query, mode, maximum_results). It returns a job ID. Call vivu_get_search_results until complete is true; each status call can wait up to 45 seconds, so a pending search is not a stalled one. Save each completed result as results/FIELD.json without its result_page_url field, and record the job_id in state.json. Show the Vivu result page link in the live reply only; it expires after four hours, so it never goes into a saved file or a CSV.

Run each class once. If the team changes a wording, the new wording is untested until its first candidates have been through Step 5.

Done when every class has a completed result saved in results/, and every class whose search returned exactly its maximum_results is marked cut off in state.json.

Step 5: Make contact sheets for every candidate and label it

Goal: every candidate labeled looks right, looks wrong or can't tell from what its frames show.

For each result in results/FIELD.json, cut contact sheets of its window from the local file at two frames a second:

mkdir -p warehouse-BATCH_NAME/sheets/FIELD
ffmpeg -v error -y -ss START -to END -i FILE -vf "fps=2,scale=384:-2,tile=5x4" warehouse-BATCH_NAME/sheets/FIELD/rRANK_STEM_STARTMS_%02d.png

FIELD is the class, FILE the local source file (BATCH_DIR/ followed by its source_file), STEM its name without .mp4, RANK the result_number written with two digits (r01, r09) so the files sort by rank, STARTMS the result's start_ms, and START and END the result's start_ms and end_ms divided by 1000. Each sheet holds 20 tiles covering 10 seconds of the window; the last sheet is padded with black tiles. Tile k on sheet n (both counted from 0, left to right and top to bottom) is at START + 10 * n + k / 2 seconds. The tile position gives each frame's time, so the command needs no text overlay. Look at every sheet of a window.

Windows can be long: a person who stands in view for a minute, or a staging area the camera keeps passing, comes back as one window. For a window longer than 40 seconds, use fps=0.5 in the same command (one tile every 2 seconds); then tile k on sheet n is at START + 40 * n + 2 * k seconds.

Scraps of wrap and paper are a few pixels on a tile, and a fisheye cab camera makes far pallets small. When a tile shows something that might be debris, a pallet or a person but the sheet does not settle it, extract a zoomed full size frame at that tile's second:

ffmpeg -v error -y -ss SECONDS -i FILE -frames:v 1 -vf "crop=iw/2:ih/2:X:Y,scale=iw*2:-2" warehouse-BATCH_NAME/sheets/FIELD/rRANK_STEM_SECMS_zoom_PART.png

SECONDS is the time of the tile, SECMS the same time in milliseconds, X is 0 for the left half, iw/4 for the middle or iw/2 for the right half, Y is 0 for the top half, ih/4 for the middle or ih/2 for the bottom half, and PART names the crop (for example center) so two crops of the same second get different files.

Label rules (defaults; the team's definitions from Inputs win):

1. person_in_aisle looks right when a person stands or walks on the floor in view. Say in the note where they are (in the aisle ahead, beside it, far away) and do not describe who they are. It looks wrong for the camera wearer's hands, a driver seen only inside a cab, a face in a selfie or a person only in a mirror.
2. pallet_on_floor looks right when a pallet, a pallet load or a stack of goods stands on the floor in the picture. Say in the note whether it is in the travel lane, at the aisle edge or in a staging area, because "in the aisle" is the reviewer's call. It looks wrong for goods on rack or flow rack levels, a load on forks and totes on robots.
3. floor_debris looks right when a loose item (scrap, wrap, a dropped bag or box) lies on the floor. It looks wrong for floor markings, stains, reflections and goods on a pallet.
4. can't tell when the zoomed frame still does not settle it (darkness, motion blur, the item hidden by the mast). The reviewer looks at the source.

Write a note of what the frames show for every candidate, such as "white wrap scraps on the yard floor at 00:52-00:55, left of the travel path" or "goods on flow rack levels, nothing on the floor". Never copy the reason into the note. Record each candidate key and its label in state.json.

Worked example from our test run

The test corpus was public warehouse footage published under a Creative Commons license, used in place of robot camera exports because none was available: 12 videos (23.6 minutes) cut from forklift cab recordings at a night produce market, a handheld walk through a distributor warehouse, an equipment walkaround, two robot picking warehouses with presenters standing in the aisles, and a low ride through a tote robot warehouse, with the audio removed and the files renamed so the names said nothing about content. These are cab, handheld and presenter shots, mostly in good light; a robot's fixed camera sees aisles from lower down. People and pallets were marked by hand on contact sheets before any search, with look alikes (gloved hands, a selfie, people in mirrors, goods on rack shelves, totes on robots).

1. person_in_aisle: 21 candidates, 21 real and 0 false, and 4 of the person moments marked by hand missed (a presenter standing in an aisle, a picker at a robot station, a far worker in a dark yard, and the first seconds after a cut). No window came back for hands, the selfie or mirror-only people. Most people in this footage face the camera in bright light, which is the easy case.
2. pallet_on_floor: 18 candidates, 16 real and 2 false, and 0 of the pallet moments marked by hand missed. The false ones were goods sitting on flow rack levels and robots carrying totes. One window ran about 115 seconds over a clip where pallet stacks are always in view, and some real windows showed stacks far away in a goods area.
3. floor_debris: 3 candidates, 3 real by their half second sheets (torn white scraps on a night yard floor, twice, and a bag the driver dropped). Our hand marks before the search had missed all three, so the recall of this class is unknown; treat its list as a sample, not a sweep.
4. A parked forklift search was tried and dropped: 12 candidates, 3 real and 7 false.
5. The 12 videos were all ready about 365 seconds after the upload started. The three CSVs held 42 rows in total.

Done when every candidate in results/ has at least one contact sheet in sheets/ and a label in state.json.

Step 6: Write one CSV per class and hand it over

Goal: a review list per class that a data engineer can work through without opening Vivu.

Show the user one sample row and the field mapping, and wait for an OK before writing the rest:

source_file,event_time,rank,start_mmss,end_mmss,search_reason,contact_sheets,claude_label,claude_note,reviewer_verdict
WH_02.mp4,,3,00:47,00:56,"Pieces of loose white plastic wrap and packaging debris are visible on the floor around moving forklifts and personnel.",sheets/floor_debris/r03_WH_02_47000_01.png,looks right,"white scraps on the floor around a worker pulling a pallet jack at 00:52-00:55",

The row above is from our test run; the user's rows carry their own file names and, when given, event times.

claude_label and claude_note are Claude's reading of the frames, not a measurement. Whether a pallet blocks the lane, or a scrap is big enough to stop a robot, depends on the robot and the team's definition, so the reviewer goes through every row, the looks right ones included, before anything enters a dataset.

Write FIELD.csv for every class with one row per result, sorted by rank, including the rows labeled looks wrong: a reviewer may disagree with a label, and the rejected rows show what the search confuses. The CSV uses the user's own file names and times, never a Vivu result page link, so it stays usable after the link expires.

Then tell the user, per class, how many candidates came back, how many Claude labeled looks right, looks wrong and can't tell, and whether the list was cut off at maximum_results. Say plainly that an empty or short list does not mean the batch has none, that small debris is the easiest to miss, and that parked vehicles were not searched. Do not add recall or precision figures: the hand marks needed to measure them do not exist for the user's batch.

The CSVs stay in the working folder. Passing them on is the user's step. If the user asks Claude to upload or post them (a shared drive, a tracker, a chat), name the destination, show the rendered first rows and wait for a yes before each write.

In our test run the CSVs were written from the dry run's results and sheets; showing the sample row to a user and handing the CSVs over were not exercised in our test run.

Done when every class has a CSV whose row count equals the number of results in results/FIELD.json, the user approved the sample row and the mapping, and state.json marks the batch done.

Compliance

1. Use camera footage the team owns or has the rights to analyze. Footage recorded in a customer's warehouse may be covered by the customer contract or data agreement; check it allows uploading to a cloud service for analysis. For public videos used as extra material, check the license and the platform's terms first and keep them for internal analysis only. Have the user confirm this before Step 3.
2. Pickers, drivers and visitors on camera did not agree to be filmed for a dataset. The skill only records that a person is in view; it does no face, badge or identity recognition, does not score or track workers, and never says who someone is. Follow the team's data policy for people in frame (some teams blur faces before any upload). Do not use this skill to collect footage of particular people; if the batch is meant to gather footage of children, stop and do not upload it. Confirm before Step 3.
3. The skill makes no safety judgment, does not say whether a stop or slowdown was right, and does nothing on the robot or in real time. The CSVs are review lists for people, not annotation files.
4. The videos stay in the user's Vivu project until the user deletes them. Create the project as private; the default is visible to the whole Vivu organization. Delete videos or the project only when the user asks, and confirm first.
5. The skill writes only local files. Any upload or post of the CSVs goes through the approval in Step 6.

Known failure modes