We Looked at the Data

It's well-known that robotics data is messy. For the data to be suitable for training, several unstable input streams must be well-formed and perfectly in sync. Many different data corruption errors, ranging from very obvious to very subtle, can silently damage training runs and raise the noise floor. Without clean underlying data, strong model architectures can fail to outperform weaker, but more noise-resistant ones.

We looked at several major open-source UMI and teleoperated bimanual 6-DOF arm + 1-DOF gripper datasets to diagnose and correct issues. We're releasing per-episode manifests for these datasets:

• IPEC-COMMUNITY/FastUMI_100k_lerobot
• allenai/MolmoAct2-BimanualYAM-Dataset
• genrobot2025/10Kh-RealOmin-OpenData
• XDOF/ABC-130k

We also inspected and cite examples from:

• WorldEngineAI/WEB-Dataset
• simple-world-lab/HiFi-UMI-2K
• agibot-world/AgiBotWorld-Beta
• RealSourceData/RealSource-World
• BitRobot/HIW-500-LeRobot
• tencent/Hy-Embodied-0.5-VLA-Data
• x-humanoid-robomind/RoboMIND
• RogersPyke/Galaxea-Open-World-Dataset_10K_20260123

Preprocessing and Unified Episode Schema

To make the process of diagnosis and cleaning tractable over many varying datasets, we adapt every dataset into a unified “episode” struct. We do so by writing dataset-specific adapters which are cognisant of each dataset's quirks (e.g. the number of cameras and their types, video resolutions, the structure and file format of the dataset, etc). For our purposes, we also downscale and center-crop all cameras to a 224p square. Some datasets contain stereo egocentric cameras or multiple cameras per wrist; in these cases, we drop additional cameras and limit each episode to at most one overhead view and one view per wrist. The episode struct below is illustrative.

episode
├── episode_id: string
├── source_dataset: string
├── task_description: string
├── duration_s: float64
├── num_frames: int64
├── fps: float64
├── cameras[]
│   ├── name: string
│   ├── role: {left_wrist, right_wrist, overhead}
│   ├── video: encoded bytes[num_frames, 224, 224, 3]
│   └── frame_timestamps_ns: int64[num_frames]
└── robot_states
│   ├── states: float32[num_frames, 14]
│   └── state_timestamps_ns: int64[num_frames]
└── robot_actions
    ├── actions_local: float32[num_frames-1, 14]
    ├── actions_global: float32[num_frames-1, 14]
    └── action_timestamps_ns: int64[num_frames-1]

For actions, we convert everything to relative end-effector delta transformations ¹. This requires some care since states cannot be naively subtracted to find deltas; they must be treated as SE(3) group operations.1 In current UMI data, the consensus default for the “action” taken at a frame is the next state achieved. We want our action data to be in terms of frame-to-frame state deltas, consistent with the signals we would send to a robot.

We additionally trim all episodes such that every episode has precisely T states and T-1 actions, and such that every video and action stream in an episode has the same number of states and actions respectively. This is straightforward since each frame has a globally-synced timestamp.

At this normalization phase, we already find many malformed episodes with either non-overlapping video streams, large numbers of missing frames, or extremely short malformed episodes.

Visualization and Filtering

We can now visualize the data and start filtering for issues. Our methodology is essentially the following loop:

1. Establish a prior about how well-formed data should look, either from existing knowledge or spot checks of randomly sampled episodes. For example, “episodes where the gripper speed exceeds some threshold are likely to contain action errors” or “motion seen by the cameras should match motion taken by the end effectors.”
2. Express the prior as an explicit programmatic flag condition.
3. Run our flag condition on all episodes and spot check episodes that fail it.
4. Determine the severity (as bad as a total reject and as good as a false positive) of the flag.

To visualize the data, we use a dashboard that shows synced camera views, action graphs, and 3D reconstructions of end effector trajectories for each episode. We add features to this dashboard as necessary for each new prior and spot check. Our visualizations in this article are derived from this dashboard.

Our priors are biased towards those which would impact world and world action models the most, with de-emphasis on VLA-based or behavior cloning-based policies. In the next section, we use “world modeling” to mean both action-conditioned “world models” and state-conditioned “world action models”. Our process quickly uncovered a lot of data anomalies; we give examples of the most interesting and important ones we found below.

Anomalies

We note that many of the flags we apply require setting an arbitrary threshold. For example, how fast is too fast for an end effector? How little motion in the overhead view is too little before we flag a frozen camera? Because of this, our filtering flags catch a fair number of false positives and may produce some false negatives, depending on the flag. We occasionally see a false positive for the “visual change unexplained by action” flag when an object is dropped or very close to a wrist camera, since a small action causes a large visual change.

We also note that some of our examples are not automatically detectable and/or point to a source of bias rather than an indisputable error, like slow actions and unnecessary handoffs. We give these only as additional examples of situations to look out for in datasets, as we don't explicitly flag them with static checks.

Finally, we note one video issue we don't illustrate above, which is missing frames. A large number of episodes have intermittent gaps in state timesteps or action timesteps, in particular in XDOF/ABC-130k. It also impacts a smaller but significant chunk of genrobot2025/10Kh-RealOmin-OpenData. If only a few frames are missing, as is the case with most of our quarantined ABC-130k and GenRobot episodes with this flag, this issue is relatively benign and can be patched over, but the fix is somewhat opinionated so we only flag the episodes and provide the missing frame indices in our manifests.

Action Processing and Smoothing

To verify our SE(3)-based action conversion, we draw the original ground truth states as a trajectory in 3D space and compare it against the same trajectory written as an SE(3)-cumulative sum of actions. In other words, we convert original states to relative actions and then convert back to absolute states, and we verify that the round trip preserves the original trajectory.

We generally prefer robot actions with small and bounded accelerations, where velocity increases and decreases smoothly. Not only are these actions safer to execute, they're also more natural and easier for policies to predict since the variance of the distribution of actions is much lower. Raw robotics data, especially UMI data (which is not bound by robot arm kinematics) is full of abrupt spikes and jerky motions, so this requires careful post-processing.

To obtain an ideal medium between staying true to the ground truth and smoothing out jerky motions, we apply smoothing in a way that penalizes sharp accelerations while requiring that the SE(3)-based trajectory through space is minimally altered end-to-end. To do this, we minimize the sum of two loss terms that act in opposition: one that penalizes high acceleration and one that penalizes deviation from the global trajectory. This allows us to compute much smoother trajectories while avoiding compounding global error from smoothness operations; they may not be as perfectly accurate as the original state data on a frame-by-frame basis, but have significantly lower variance on short horizons of multiple actions in a chunk.

An example of our smoothing is shown below. It is particularly necessary for episodes with spiky actions since such actions would be strange and difficult to handle both as input (for world modeling) and output (for world action modeling).

Beyond Our Current Static Checks

Our automatic checks catch and filter a large portion of the most severe dataset issues, but data quality issues have a long and infinitely subtle tail. As noted, we expect a fair share of false positives and false negatives due to thresholds that may vary significantly in optimal value for each dataset or task. By checking state and action correlation, we can catch mismatched cameras and actions in episodes, but datasets containing many examples of the same trajectory could slip through this check if each trajectory is similar enough. Other issues like action misalignment or partial-episode disabled grippers can occur on an episode-by-episode basis and are difficult to catch with weaker static checks; the presence of gripper action errors in some datasets warrants a dedicated per-dataset extraction policy that can somewhat deterministically verify that the gripper video is consistent with open/close actions.

As mentioned before, issues like unnecessary handoffs are higher-level opinionated issues and are much harder to check without either a human annotator or a sophisticated VLM-based pipeline (which we are exploring).

As a more general technique, a small (or large) IDM could be fitted to all datasets, or each dataset individually, to identify extremely strange actions which are difficult for a small policy to predict. This would be a smarter version of our state delta vs action delta correlation check. We are exploring this approach internally as it is a promising data quality filter, but do not speculate further on this approach here as it would be significantly more opinionated than our other standard static checks.

Credits

This project and writeup were undertaken by Sambhav Gupta. We thank XK Lu, Jenna Hong, Jannik Schilling, and Kunvar Thaman for edits and suggestions.

For past work, we broadly searched for citations of anomalies like the ones we found, but did not find major examples pointing to the same specific errors. Please reach out if you know of or produced any such work so we can cite and link it here.

If you are interested in a data-related collaboration or have any questions or comments, please contact us at data@pantheon.inc.

Open-source release

We've released a per-episode manifest of our findings for FastUMI_100k_lerobot, MolmoAct2-BimanualYAM-Dataset, 10Kh-RealOmin-OpenData, and ABC-130k. In these manifests, for each episode, we assign a list of flags for the issues we encountered. In the case that the flag is assignable at a frame level, we include a boolean mask. We also include our raw and smoothed frame-to-frame action data.

Dataset citations

• AgiBot World Colosseum Contributors. “AgiBot World Colosseum.” 2024.
• Allshire, Arthur, et al. “Scalable Behavior Cloning with Open Data, Training, and Evaluation.” arXiv, 2026.
• BitRobot, Unitree, and Hugging Face. “HIW-500: Humanoids In-the-Wild Dataset for Robot Learning.” 2026.
• Fang, Haoquan, et al. “MolmoAct2: Action Reasoning Models for Real-world Deployment.” arXiv, 2026.
• Galaxea Team. “Galaxea G0: Open-World Dataset and Dual-System VLA Model.” arXiv, 2025.
• GenRobot. “10Kh-RealOmin-OpenData.” Hugging Face, 2025.
• Hou, Chengkai, et al. “RoboMIND 2.0: A Multimodal, Bimanual Mobile Manipulation Dataset for Generalizable Embodied Intelligence.” arXiv, 2026.
• Liu, Kehui, et al. “FastUMI-100K: Advancing Data-driven Robotic Manipulation with a Large-scale UMI-style Dataset.” arXiv, 2025.
• RealSource. “RealSource World: A Large-Scale Real-World Dual-Arm Manipulation Dataset.” Hugging Face, 2025.
• Simple AI, et al. “HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone.” arXiv, 2026.
• Wu, Kun, et al. “RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation.” Robotics: Science and Systems, 2025.
• World Engine AI. “WEB-Dataset.” Hugging Face, 2026.
• Zhang, He, et al. “Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack.” arXiv, 2026.