.png)
Robot datasets often contain synchronized videos, actions, states, and a coarse task name. What they usually do not contain is the dense timeline we want when inspecting, filtering, training, or debugging. Creating this timeline in a fast, automated, and accurate way is difficult without relying entirely on human annotators.
The difficulty lies in finding the few frames where subtask trajectories change: the first stable grasp that is not a mistake, the first frame of a drop, or the transition from recovery back into the intended task. When the right boundaries are not found, poor grounding and hallucinations can propagate through the rest of the labeling procedure.
We previously relied on human annotation, and an obvious but key realization was that human annotators do not watch a video uniformly from beginning to end. They scrub and jump around. They inspect one camera, switch to another, slow down near transitions, and stop when the boundary is good enough.
That observation shaped the method we used to create our labeler. Instead of forcing a vision-language model to annotate sampled video frames in one shot, we made annotation a bounded tool-use loop. The model sees an initial sample, decides whether evidence is missing, requests more timestamps, receives those frames, and then revises its segmentation.
In short: the model labels videos the way a good annotator uses a scrubber.
A high-level task name is often too coarse, and raw actions are often too low-level. A demonstration labeled only as "fold cloth" or "move object to target" contains several qualitatively different phases. Treating the whole video as one unit hides the structure that makes it useful.
Dense subtask intervals help in several places:
This lines up with a broader direction in robot learning: using intermediate structure between high-level goals and low-level actions. RT-H uses language-level motions between task commands and low-level actions. Embodied Chain-of-Thought trains policies to reason through plans, subtasks, motions, and visual evidence. Subtask-Aware Visual Reward Learning from Segmented Demonstrations uses segmented demonstrations for dense reward learning. Vision Language Models are In-Context Value Learners shows how visual examples can help models reason about progress through robot trajectories.
The obvious automated baseline is to sample frames at a fixed rate, send them to a VLM, and ask for intervals. Sometimes interval boundaries are snapped to the nearest sampled frame. This by itself is not too bad. But because the VLM can miss certain key transition events, it may trick itself into believing a subtask is much longer than it actually is, or that it ends prematurely.
The opposite approach is to show many more frames. But this has its own issues. Long videos across multiple cameras can exceed practical request-size limits. Packing frames into mosaics or strips can address coverage, but it compresses local detail, which can cause grasps or slips to be overlooked.
We came up with this idea by asking: what if a coding or computer-use agent had access to our annotation tool? It could scrub the timeline of a video and keep collecting frames until it had enough information to figure out what subtask intervals should be.
Almost all agent harnesses now work this way. They continue looping, inspecting their own state and using tools until some stopping condition is met. As background, LangChain's The Art of Loop Engineering describes agents as models calling tools in loops, with verification loops wrapped around the work. Jeremy Theocharis makes the same point from the human-process side in All You Need is Loops (and Humans): stop giving the model only a destination; give it a process and a definition of done.
Our eureka moment was realizing we did not need to force one-shot correctness. Just give the VLM access to the fundamental "tool" a human annotator has: jump to specific timestamps.
request_more_timepoints(timestamp_s, reason)That action allows a one-shot classifier to search locally around suspected boundaries, effectively turning it into an automated annotator.
To make this work, first we made sure the model could not invent labels independently for every video. If it did, one episode might use "pick up object," another might use "lift object," and another might use "extract object" for the same behavior.
Our pipeline first builds a local taxonomy for the task family. It describes:
The taxonomy can be built automatically, manually, or in a hybrid mode. The VLM can look at a collection of episodes to infer the subtask menu itself, or we can provide the menu. Hybrid mode is usually strongest. A human provides starting hints, and the model uses sampled visual evidence to refine those hints into descriptions, cues, and references.
Example guidance can look like this:
{
"notes": [
"Use these as starting hints, not fixed labels.",
"Keep every hinted mistake as a possible mistake label, even if sampled evidence does not show it.",
"A retry is not a mistake by itself; retrying remains part of the original subtask being attempted.",
"Create a separate mistake interval only for the visible mistake event itself."
],
"preliminary_subtasks": [
{
"subtask": "reach toward the object",
"examples": [
{
"episode": 212,
"start_timestamp_s": 9.9,
"end_timestamp_s": 12.95
}
]
}
],
"preliminary_mistakes": [
{
"mistake": "mistake: slip or improper grasp causing failure to pick up",
"examples": [
{
"episode": 212,
"start_timestamp_s": 16.6,
"end_timestamp_s": 17.8
}
]
}
]
}We found that if the subtask menu also comes with visual references, selected automatically or manually, fine boundary detection greatly improves. It can be the difference between ending a subtask at near completion and ending it exactly when the defining state is reached.
During taxonomy generation, the model selects representative start and end frames for each subtask when possible. Those frames are passed into later episode-labeling calls as visual anchors. This is a practical annotation use of the multimodal in-context-learning pattern studied in Vision Language Models are In-Context Value Learners: instead of only telling the model what a label means, we show it examples of what the boundary looked like in this dataset.
Reference frames can include multiple camera views. A main camera may show the global object state, while a wrist or side camera shows the decisive contact. The taxonomy records the timepoint, camera slots, rationale, and confidence for each reference.
We choose not to stitch images together. Earlier experiments used strips and grids, but stitching made images very wide and risked losing detail by compressing key frames into a composite. The current approach keeps camera frames separate.
For each selected timepoint, the prompt can include multiple views, such as main, left, and right. Each image is resized only to fit within a configured maximum dimension while preserving aspect ratio. Each image also carries concise metadata:
For taxonomy generation, sampling has two sources:
For seed labeling, sampling is controlled by a configurable timepoint budget or cadence. The model can then request additional timestamps around ambiguous boundaries.
Recent video-understanding work also arrived at the conclusion that the representation sent to the VLM matters. Apollo treats sampling and representation as central design choices for video-LMMs. Streaming Dense Video Captioning frames long-video understanding as temporally localized event description. Scale Labs' large-scale dense video captioning writeup makes the same point in a manipulation-video annotation setting. Adaptive Keyframe Sampling and Q-Frame both focus on selecting the frames most likely to preserve task-relevant evidence.
Once a taxonomy is accepted, videos can be labeled independently and in parallel.
The seed-labeling prompt receives:
Episode frames are sent in order. There is no group randomization and no stitching. The model first describes what is visible at each sampled timepoint to ground itself, then either returns final intervals or asks for more frames.
The model has two response modes:
request_more_timepointsfinal_labelsOn the first pass, the model sees a sparse set of frames. If the evidence is enough, it returns final labels. If not, it requests specific timestamps and explains why each timestamp matters.
Examples:
We usually allow the initial call plus two follow-ups to stay within budget. This is where we differ from most agentic loops. It is more of a for loop than a while loop. In practice, most labeling runs return by the first follow-up call.
Conceptually:
accepted taxonomy
-> initial sampled frames
-> model labels or requests more timepoints
-> sampled requested frames
-> model revises labels or requests more timepoints
-> sampled requested frames
-> final labels
-> validation
-> review/exportThis is the annotation-specific version of iterative frame search. VideoAgent uses an agent loop to retrieve video evidence for question answering. Universal Video Temporal Grounding with Generative Multi-modal Large Language Models targets the related problem of localizing language-described moments in video. Here, the model retrieves more evidence for boundary finding, but the output is not a natural-language answer. It is a validated interval segmentation over a fixed taxonomy.
We recently used this on an in-house robot-video dataset collected for a partner workflow. The task family involved configuring a metal chain so it fit a predefined alignment.
The taxonomy pass sampled representative episodes, incorporated human-provided subtask and mistake hints, proposed a local subtask menu, and then ran a critique pass against additional visual evidence.
Figure 3. Example playback of automatically generated subtask intervals over robot video.
The request loop mattered in practice. Many episodes did not need extra evidence, but many did. Many videos needed at least one follow-up to pinpoint a transition.
This method substantially reduces the need for manual annotation, but it does not make human review irrelevant. Underlying datasets can contain messy or unclean episodes with one-off events or issues. Those can be routed to a focused review process.
Each episode can be marked with review-oriented metadata such as:
This metadata proved useful for dataset cleanup and a small amount of targeted post-labeling review.
We originally experimented with initial sampling biased toward robot state events: gripper transitions, large end-effector accelerations, and large joint-state changes. However, uniform sampling had a similar spread, and the VLM could often infer when state events would occur and request those frames. There is still room here for more efficient sampling.
We decided to treat mistakes as "subtasks" themselves. There is an argument that it may be better to treat them as their own labels, and it is an argument I like because that is what our RL pipeline has been doing: valid/good states for useful task progress, recovery, or efficient action, and invalid/bad states for mistakes, inefficient action, or behavior that should not be reinforced. For ease of implementation and to save on calls, we merged the two labeling tasks for now. But there is definitely room to have another loop purely for mistake or "am-I-taking-efficient-actions" detection.
Subtask annotation should be a loop, not a one-shot prediction.
Robot videos contain small, high-stakes temporal events. A sparse first pass will miss some of them. A human annotator handles that by scrubbing to the right moment. The labeler gives a VLM the same basic ability through a structured request_more_timepoints action.
The result is a practical pipeline for turning raw videos into timestamped subtask intervals: define the taxonomy once, label episodes against it, let the model ask for missing evidence, validate every output, and route uncertain cases to humans.
The breakthrough was not a longer prompt. It was changing the shape of the job from "answer from these frames" to "search for enough evidence to label the episode."
