150 days after inception, we’re excited to announce a major breakthrough towards solving robot learning: Reinforced Planning (RP-1), a novel method that uses Reinforcement Learning to improve World Model planning.
Robot learning today runs either on Imitation Learning methods, which cannot reason over counterfactuals, or on sampling-based search, which plans too slowly for real-world deployment. RP-1 offers a third option: by reimagining planning as a learned policy over a frozen world model, it achieves higher success rates at a fraction of the compute and time spent.
How does Reinforced Planning Work?
Reinforced Planning (RP-1) is a learned planner which uses a reward-guided mechanism to iteratively improve action plans. Starting with an initial candidate action sequence, it uses a frozen World Model to imagine the plan’s outcome and a learned critic to judge how close that outcome is to the goal. A neural planner then uses this feedback to revise the action sequence, repeating the process over several iterations. Crucially, RP-1 learns how to improve plans from offline imagined rollouts, replacing fixed optimization rules with reusable rules for improving plans.
Before going deeper on how RP-1 performs compared to state-of-the-art alternatives, it helps to understand the limitations of today’s dominant robot learning approaches.
Why Current Robot Learning Approaches Fall Short
Imitation Learning
Today, the predominant approach for robot learning relies on Imitation Learning (IL) techniques (e.g., VLAs, WAMs, ACT policies) where a model is trained to match the actions of an expert demonstrator. Concretely, the policy learns a direct mapping from state to action, observing input and outputting a move with no reasoning step in between. This is simple to train, produces working policies quickly, and, given enough data, yields smooth, natural behavior.
However, the direct state-to-action mapping that makes IL so simple also makes it brittle. Minor changes in lighting can prevent the policy from recognising the state it’s meant to act on, causing double-digit decreases in success rate; real-world variation is effectively unbounded, forcing days or weeks of onsite data collection for every new environment. This is a main reason the field has so many demos and so few deployed robots.
The intuitive solution is to mitigate these limitations by exposing the model to suboptimal trajectories, teaching it to recover from bad states. However, an IL-based model will then learn to imitate the suboptimal actions taken, increasing the failure rate. Worse, every recovery action spawns new states with failure modes of their own. Teach the model how to recover from those, and the problem will repeat itself one step later. Instead of making IL more robust, recovery data spawns a recursive spiral of failure modes that ultimately leads to collapse. As such, this brittleness is inherent to IL.
World Models and Planners
A World Model can be thought of as a physics simulator. Given an agent’s current state and an action, it simulates the resulting next state. A World Model is trained on diverse data to learn how the physical world behaves, enabling it to simulate the outcome of many possible action sequences.
This gives World Models two critical properties. First, a World Model can in theory reason about the consequences of any action, including scenarios it has never encountered before1. Second, World Models can learn from orders of magnitude more data. IL-based approaches replicate expert demonstrations, so each trajectory must be whole and completed by a single expert, sharply constraining the data IL policies can learn from. On the other hand, World Models predict transitions and are capable of stitching together data from a wide variety of sources. This data flexibility allows World Models to learn from tens of millions of hours of data, growing more robust with every additional hour.1Note that some existing approaches described as World Models, like World Action Models (WAMs), don’t actually reason about which action sequence is best. They simulate the entire trajectory a human demonstrator would take in the same scenario and select the closest match. This is Imitation Learning in a wrapper and inherits IL’s limitations.
World Models are the key to unlocking self-directed robot learning. So why haven’t they been deployed yet? Unfortunately, the field is still nascent. Although the best planning algorithms available today do well with constrained control settings (such as autonomous driving), they are poorly matched to the complexity of real-world robotic control. Planners commonly used in continuous environments such as the Cross-Entropy Method (CEM) or Model Predictive Path Integrals (MPPI) are sampling-based search algorithms built on handpicked heuristics. They draw candidate action sequences at random, simulate each through the world model, and keep whichever scores best.
World Models with today’s planners suffer from three limitations that make them undeployable in the real world:
- 01
Fixed-algorithm methods cannot reliably discover good plans.
Relying on hand-crafted search heuristics and luck to discover good plans is workable for very short horizons, but collapses when task horizons lengthen and the search space explodes.
- 02
Fixed-algorithm inference is too slow for the real world.
Current planners take several thousands of steps even on short horizons, which makes existing models at least an order of magnitude too slow for economically feasible deployments.
- 03
Compute scales polynomially with the planning horizon.
Therefore, even short horizon tasks are enormously compute-intensive, and cost rises rapidly with even slightly longer tasks.
What can RP-1 do?
We ran evaluations across three of the most widely used benchmarks in robotics: TwoRoom, OGBench, and Reacher. Within each, we compared RP-1 against the three most popular planning methods used with latent World Models - sampling-based algorithms MPPI and CEM, and the gradient-based optimizer Adam - across two leading open-source world models (LeWM and PLDM) and a range of task horizons. RP-1 beat state-of-the-art (SOTA) latent methods on 48 of 48 head-to-head comparisons.
TwoRoom
TwoRoom is a continuous navigation environment where the agent must reach a goal position in two rooms separated by a wall with a door gap. Easy goals lie in the agent’s own room; hard goals lie across the wall, requiring the planner to find action sequences that pass through the gap.
On short horizon tasks (25 steps to goal), baseline planners perform reasonably well, achieving an average success rate of 82.9%. However, RP-1 is able to outperform, with an average success rate of 99.1%. Crucially, RP-1 remains effective when task horizon increases to 100 steps, succeeding 95.1% of the time, whereas baseline planners collapse to an average success rate of 30.8%.
OGBench cube manipulation
OGBench cube manipulation is a robot-arm benchmark where the agent must grasp a block and place it within 4 cm of a target pose. Unlike navigation, an accurate grasp-transport-place sequence is required, which makes undirected search methods (like CEM) non-viable.
OGBench cube challenged planners more across the board. Overall, baseline planners had a success rate of 38.7%, compared to RP-1’s 74.2%. This split becomes even more dramatic on hard tasks: baseline planners succeeded only 20.2% of the time on the 25-step hard task and 15.6% on the 100-step hard task, while RP-1 holds at 68.2% and 63.0% respectively.
Reacher
Reacher is a 2-link arm-reaching benchmark, where the agent must move its end-effector to a target position. Unlike TwoRoom and OGBench, it has no obstacles and every configuration is reachable from every other, so latent distance is already an adequate proxy for cost-to-go. What Reacher isolates instead is search quality.
On Reacher, the split that matters is tolerance. At loose tolerance (τ=.1), RP-1 has a slim lead of 0.6 percentage points over the best baseline (CEM) and a more commanding margin of 12.9 percentage points over the baseline average. However at tight tolerance (τ=.05), where the plan must land exactly, baseline performance declines (CEM 80.2%, average 61.2%), and RP-1’s lead increases to 24.1%.
Overall, RP-1 proved far more capable than SOTA planning methods across a diverse set of benchmarks. On average, RP-1 exceeded SOTA planners by 31 percentage points, and almost quadrupled SOTA success rates.
Why does RP-1 Outperform?
The decisive margins RP-1 is able to achieve over state-of-the-art planners are due to two innovations: its use of value functions, and its ability to iteratively improve plans.
- 01
RP-1 selects plans with a value function, not latent distance.
Current work on pretrained World Models predominantly scores action sequences according to how close they land to the goal in the World Model’s latent space. This is a fixed, hand-picked proxy that struggles to navigate complexity. For example, if there is an obstacle in between the agent and the goal, existing planners run directly into it rather than taking the long way around. RP-1 takes inspiration from model-free Reinforcement Learning and replaces the latent method with a reward-guided planning mechanism. Rather than judging a plan by how close its predicted endpoint is to a goal representation in latent space, RP-1 scores plans based on temporal (“remembered”) closeness to goal - effectively, how many steps remain before the goal is reached2.2For more detail on our methodology, please refer to our arXiv paper, Reinforced Planning with Latent World Models.
Figure 9. The TwoRoom heatmaps compare how each method judges distance to the goal—latent distance versus RP-1’s learned value function—against true geodesic distance. RP-1’s mapping matches the true distance much more closely. 
Figure 10. Value-based scoring improves every baseline planner, while RP-1 achieves the strongest results with only nine rollouts3.3These results are from the OGBench Cube “hard” evaluations. The heatmaps above compare how each method judges distance to the goal - latent distance versus RP-1's learned value function - against the true geodesic distance on the right. RP-1’s mapping matches the true distance much more closely than the latent distance method conventionally used.
- 02
RP-1 iteratively improves its plans.
Rather than rolling out thousands of candidate plans and simply executing the highest-scoring one, RP-1 proposes one plan, simulates the outcome, evaluates that plan, and improves its next proposal. The result is a planner that gets smarter with every step, enabling RP-1 to find more accurate action sequences with orders of magnitude fewer rollouts2.
Figure 11. Learned refinement. RP-1 converges on a successful trajectory in eight refinement iterations; CEM fails after 30.
The reason RP-1 outperforms the SOTA planning algorithms boils down to The Bitter Lesson. Every part of an AI system should be learned end-to-end, and relying on hand-designed recipes never scales. All World Model planning methods so far involve hand-engineered4 heuristics. We are the first team to build a planner that is learned end-to-end.4Current planners are either hand-designed, distilled from a hand-designed optimizer, or learned only to inform an amortized policy rather than to revise the plan itself.
What does this mean?
Across all three benchmarks, RP-1 demonstrates four consistent advantages:
- 01
RP-1 is far more successful.
RP-1 outperforms every baseline planner in 48 of 48 head-to-head comparisons. On average, RP-1 success rate was 3.9x that of SOTA planners.
- 02
RP-1’s advantage grows with task difficulty.
The difference in RP-1 vs SOTA success rate increased from 3.9x to 4.8x for 100-step tasks only, and to 7.3x for ‘hard’ tasks (OGBench Cube hard, .05 Reacher).
- 03
RP-1 is far faster.
Compared to CEM, RP-1 is 13x faster if we run it on an H200 GPU for one robot arm deployment, and 67x faster if we run it for 50 arm deployments. Accelerating planning rollouts by orders of magnitude is what's needed to move model-based planning into real-time range.
- 04
RP-1 is far more compute-efficient.
It requires 333x fewer World Model queries per decision than Adam and 1000x fewer than MPPI and CEM.
Taken together, these improvements move World Models dramatically closer to deployment, because they kill the two biggest practical bottlenecks: reliability and inference speed. RP-1's success rate is higher across the board, but because it iterates on a learned value function to improve its own plans, the gap is widest on difficult tasks. RP-1’s performance over SOTA widens from 3.9× overall to 4.8× on long-horizon tasks5 and 7.3× on hard tasks6, meaning it is much better equipped to navigate the complexity of deployment. At the same time, RP-1's design cuts the rollouts required per decision by orders of magnitude, collapsing the compute cost that has made World Models impractical to run at scale. A high reliability floor and a low cost per decision are precisely what deployment requires.5Long-horizon tasks refer to 100-step tasks6Hard tasks include OGBench Cube hard and .05 Reacher
What’s Next?
RP-1 massively alleviates the planning bottleneck, delivering strong gains across all evaluations we run, but there is still headroom. When we investigated RP-1’s remaining failure modes, we found that the bottleneck had shifted upstream to the fidelity of the World Model itself.
In some cases, the World Model would hallucinate incorrectly that it had grasped the cube. The planner then finds that the world model's imagined rollout was inaccurate and tries to grasp again, running into the same problem. Thus, the World Model and planner become trapped, looping endlessly on the same doomed reach for the cube.
These flaws are embedded across World Models today. We are now training a new foundation model designed to overcome them, with a native understanding of the fine-grained manipulation that robot control requires.
World Models have the potential to make robots dramatically more capable: instead of learning a narrow behavior for each task, they can reason over possible futures and adapt to new tasks. We believe that RP-1 transforms World Models from passive predictors to engines for effective decision-making, putting solving robot learning within reach.
We’re a research lab in San Francisco working to put robots to economically useful work at scale. If you are excited by this mission, we’d love to hear from you.
Acknowledgments
We thank XK Lu for editing this post.