1 · Put both toy shirts together at the front-left
Tasks
Every task attempted in an episode, in plain language, with where it starts and ends and whether it succeeded.
Argus
Argus is our open-source annotation and data quality pipeline. It densely labels every episode end-to-end, capturing action phases, key events and more.
Dense annotation
Argus labeled every episode of our open-source dataset, DiverseUMI-100, from the tasks attempted to every action and state change.
Labels every episode
Argus produces a full timeline of actions, state changes, failures and recoveries for each recording.
Right wrist camera
Withdrawboth arms · idle
Actions
Each arm's actions throughout the episode and its contribution to task progression.
State changes
What changed in the scene and when: an object moved, opened, stacked or knocked over, and where it ended up.
Failures and recoveries
Every missed grasp, drop and failed placement, when it happened, and how the operator recovered from it.
A harness, not a prompt
Most of Argus' accuracy gains come from our harness, not the underlying model: what it sees, what it is asked, and how outputs are checked.
- LeRobot v3
- MCAP
- MP4
- MKV
- TAR
Reads any recording
LeRobot, MCAP, plain video and archives, split into episodes by exact timestamps and labeled as recorded.
Looks closer where it matters
A quick pass picks what needs fine detail. Grasps, releases and the first and last frames are always seen at full resolution.
Accounts for moving cameras
Argus knows how each camera is mounted, and reports a change in the scene only when a fixed view confirms it.
Recorded wrist speed · flags
Checks its own answers
Labels are checked against recorded motion and physics, with deterministic checks for sped-up and swapped streams.
Flags the errors breaking training
We audited nine open-source datasets and found potentially misleading errors in as many as half of a dataset's episodes.
- 3,546
- episodes audited across nine public robotics datasets
- 50%
- of the most affected dataset's episodes had an issue that would mislead a model
- 1 in 5
- MolmoAct2 episodes carried an instruction that describes a different task
| Issue | OpenAoE | MolmoAct2 | RealOmin | Galaxea | FastUMI-100K | Gen-HumanEgo | Egocentric-100K | HABIT | ABC-130k |
|---|---|---|---|---|---|---|---|---|---|
| Instruction doesn't match the footage | 50% | 20% | 12% | 20% | 7.8% | 10% | · | 2.9% | 2.2% |
| Recorded motion disagrees with the video | · | 0.2% | 6.4% | 0.4% | 6.0% | · | · | 1.6% | · |
| Recording plays faster than real time | · | 23% | · | · | · | · | · | · | · |
| Goal undone before the recording ends | · | 1.9% | 1.8% | 10% | · | · | · | · | · |
| Head camera taken off | · | · | · | · | · | · | 8.9% | · | · |
| Long idle stretches | · | 0.4% | 4.6% | 0.9% | 0.1% | · | · | · | · |
Work with Argus
We help our partners annotate and audit their datasets using Argus. Tell us about yours.
Related research
Argus: A Dense Annotator for Robotics Data
Introducing Argus, an open source dataset-and-format-agnostic harness for the densest annotation of robot learning data.
Read the articleWe Looked at the Data
We found recurring quality problems during an audit of popular public robotics datasets, and are releasing a manifest of the results.
Read the article