Do RoboMME Policies Generalize to RoboMME?

RoboMME-Plus: Benchmarking Memory Generalization for Robot Policies

Yinpei Dai1, Hongze Fu1, Yuejiang Liu2, Chelsea Finn2, Nima Fazeli1, Joyce Chai1
1University of Michigan2Stanford University

Memory is essential for long-horizon robotic manipulation, helping robots recall object locations, track task progress, avoid repeating mistakes, and learn new tasks from video demonstrations. RoboMME was designed to evaluate these capabilities. In less than eight months, its best reported success rate has risen from 44.5% to 97.1%, exceeding the reported human baseline of 90.5%.

But how much of this progress carries over when task conditions change? Our pilot studies reveal two gaps: fine-tuned policies struggle beyond their training ranges, while a strong proprietary model transfers better but falters under precise timing, dynamic scenes, occlusion, and limited interaction.

These findings motivate RoboMME-Plus, a more challenging extension that retains RoboMME’s 16 tasks and interfaces while making success harder to achieve. Fine-tuned policies achieving up to 88% success on RoboMME fall below 3% without adaptation, while even the strongest Astra-based policy achieves only 35.3% success. These results highlight the need for memory-augmented policies that generalize to more challenging memory-dependent settings.

Motivation

1. The current state of the RoboMME benchmark

RoboMME evaluates robot memory through 16 tasks across four suites: Counting tests memory for repeated actions and events; Permanence tests tracking objects once they are hidden; Reference tests which objects were indicated earlier; and Imitation tests how demonstrated actions were reproduced. Together, these tasks assess whether policies can use past information to guide current actions.

Since its release, the benchmark has supported steady progress across a variety of memory-augmented policies (Figure 1). As leading methods approach its performance ceiling, a natural question emerges:

Can these policies generalize their memory capabilities beyond their training conditions?

2. Fine-tuned policies: limited generalization

We begin with SimpleMemVLA , the strongest open-source policy on the leaderboard. Its approach to memory modeling is surprisingly simple: fine-tune a Qwen3.5-4B VLM to draw directly on native video history, without dedicated memory design. The VLM predicts subgoals, and their hidden representations guide a DiT-based action expert. After 2,560 H100 GPU-hours of training, the policy achieves 88% overall success, including 100% on the PickXtimes task.

To probe its generalization, we retain the original RoboMME scenes but extend task parameters beyond their training ranges, evaluating each setting over 10 episodes. Figure 2 reveals clear limits. On PickXtimes, the policy succeeds in every episode requiring up to five repetitions but fails when more are required. Its predicted subgoals never advance past “for the fifth time,” matching the largest count in the training data (Figure 2, top).

Similarly, on PickHighlight, it solves 27 of 30 episodes within the training range, where 1–3 cubes are highlighted among 3–6 total cubes. When 4–6 cubes are highlighted among 8–12 total cubes, it solves only 4 of 30 episodes, with predicted subgoals rarely advancing beyond “the third highlighted cube” (Figure 2, middle).

On VideoUnmaskSwap, the robot sees a cube revealed under one of four containers, then watches the containers swap positions before lifting the one that hides it. SimpleMemVLA solves 28 of 30 episodes with 1–3 swaps, as in training, but only 9 of 30 with 4–6 swaps, falling to 1 of 10 with 6 swaps (Figure 2, bottom). In 20 of its 21 failures beyond the training range, it lifts the container standing where the cube was first revealed, as if the swaps had not happened.

These results indicate that the policy’s ability to count and track objects remains bounded by its training data. Despite extensive fine-tuning, it struggles to apply these capabilities beyond training conditions, even with familiar setups. Other fine-tuned policies, including FrameSamp+Modul and PonderPounce , show similar limitations (Figure 2b). Together, these findings suggest that high success within the training distribution may reflect learned task patterns rather than generalizable memory capabilities.

See results for more models

3. Proprietary policies: strong transfer, remaining challenges

Large proprietary models offer a different route. Embodied Astra prompts GPT-6-Astra with the full image history, robot state, and camera calibration to generate reasoning and end-effector waypoints directly. Without task-specific fine-tuning, it achieves 92.1% success on RoboMME, exceeding the reported human baseline of 90.5%.

On the same out-of-range tests, Embodied Astra succeeds in 91 of 100 episodes, showing strong transfer within these tabletop settings. But this capability comes with practical constraints: Astra is unavailable for downstream fine-tuning, and a full RoboMME evaluation costs approximately $300 and takes around 40 hours with a single worker. Its undisclosed model size also limits direct comparison with smaller open-source policies.

Beyond these practical constraints, Astra’s success raises a broader question: How reliable are its capabilities under more demanding manipulation conditions? Our experiments reveal limitations in four settings: (1) time-sensitive tasks, where the robot must act at the right moment based on remembered events; (2) continuous scene changes, where it must track target identities as objects move; (3) occlusion, where it must establish and remember target associations despite partial observations; and (4) interaction-limited tasks, where it must learn from previous attempts with few opportunities for correction. The following studies examine each challenge in detail.

Challenge 1: Time-sensitive tasks

StopCube is a time-sensitive task in RoboMME, where the robot must press a button when a moving cube reaches a target for the k-th time. Success requires both remembering the number of passes and acting at the critical moment. Fine-tuned open-source policies perform well: SimpleMemVLA reaches 92% success, PonderPounce 83%, and T²Mem 70%. However, Astra reaches only 54%.

On inspection, Astra often counts the passes correctly in its reasoning but misses the 0.4–0.8-second window for pressing the button. It sometimes oscillates above the button while waiting, then moves closer as the critical moment approaches, but this strategy can trigger premature presses (Figure 3). These failures highlight the gap between remembering events correctly and acting on them at the right time.

Challenge 2: Continuous scene changes

In VideoRepick, the robot watches a demonstration to identify a cube, then picks it up a specified number of times. In the original condition, the cubes swap positions 0–2 times at the end of the demonstration and then remain stationary.

To test continuous tracking, we instead keep the cubes moving throughout the demonstration. Both conditions use the same initial layouts, with 3–7 identical cubes, running 10 episodes per count under each condition. Astra succeeds in all 50 original-condition episodes but only 25 of 50 with continuous motion, falling to 2 of 10 with N=7 cubes (Figure 4).

VideoPlaceOrder shows a similar pattern. The robot must place a cube on the ring it visited at a requested step of the demonstration. We compare the original condition, with at most one ring swap at the end, against continuous swaps throughout the demonstration. With 3–7 rings and 10 episodes per count under each condition, success falls from 50 of 50 to 27 of 50, including just 3 of 10 with N=7 continuously swapping rings (Figure 5). Placements remain spatially accurate, but target identities are lost.

Challenge 3: Occlusion

Real scenes often obscure task-relevant objects, requiring robots to combine viewpoints and remember earlier observations. In VideoUnmask, the robot watches colored cubes being simultaneously covered by containers, then lifts the containers hiding the requested colors in order. Some containers are empty, and no other cube shares a target’s color. The original task provides an unobstructed front view of the cubes before they are covered. To stress-test Astra under partial observability, we introduce a fence that obscures at least half of each target cube from this view without fully hiding any target.

We compare identical layouts with and without the fence across five container-count ranges, from 6–10 to 26–30, with 2–4 targets per episode and 10 episodes per range. Astra succeeds in all 50 episodes without the fence but only 40 of 50 with it (Figure 6). In nine of the ten failures, it lifts a container that does not hide the requested cube, revealing difficulty in establishing or retaining the correct target–container association under occlusion.

Challenge 4: Interaction-limited tasks

Astra often recovers by correcting its actions repeatedly. In MoveCube, it can replan and push again when a cube deviates from its intended path (Figure 7, left). But can it use a failed attempt to improve the next when continuous correction is unavailable?

We test a variant in which the robot must roll a ball into the cube. Once released, the ball cannot be redirected. We allow two attempts, giving Astra one opportunity to use the first trajectory to adjust its aim or release.

Astra succeeds in only 2 of 10 episodes (Figure 7, right). This result exposes a challenge beyond recalling execution history: using that history to make an effective action adjustment within a limited number of attempts. Both successes come on the second attempt; no first attempt succeeds.

4. Designing RoboMME-Plus

Our out-of-range tests expose generalization limits in fine-tuned policies that Astra largely overcomes, while Section 3 reveals challenges that remain even for Astra. RoboMME-Plus combines these findings into a benchmark for memory-guided behavior under more demanding conditions. It retains RoboMME’s 16 tasks, four suites, observation format, and action spaces, allowing direct evaluation of existing policies, and introduces four types of variation:

  1. Larger counts and more objects. Higher repetition counts, more distractors, and more goal objects test counting and tracking beyond the original training ranges.
  2. Continuous motion and tighter timing. Moving targets, conveyor belts, and swapping containers and rings test tracking during demonstrations and execution. Shorter action windows require timely responses to remembered events.
  3. Partial observability. Barriers obscure task-relevant objects from the front camera, requiring policies to combine front and wrist views with earlier observations.
  4. Limited correction. Some tasks permit only two attempts and no correction after release; others end immediately after errors such as a repeated button press or interaction with the wrong object. These constraints test execution precision and the ability to use feedback across attempts.

Figure 8 compares the original and extended tasks. All tasks have easy, medium, and hard levels except MoveCube, which varies the manipulation method. Easy levels preserve the original mechanics but extend parameters beyond the training ranges (variation 1). Medium and hard levels add increasingly demanding motion, timing, visibility, or correction constraints (variations 2–4). The evaluation contains 800 episodes across 16 tasks, with overall success reflecting combined demands on memory, perception, timing, and control. Episodes also require longer execution: the oracle’s median episode length increases from 241 to 608 steps (2.5×), with the distribution shifting toward longer episodes (Figure 9).

5. Experimental results

We evaluate six methods on all 800 test episodes:

The first five methods are evaluated zero-shot, while one-shot Embodied Astra receives a single in-context example from the RoboMME-Plus training set. We leave few-shot fine-tuning of open-source policies on the 50 RoboMME-Plus demonstrations to future work.

Table 1 reports per-task success rates; Table 2 summarizes overall success, runtime, and API cost. Three findings stand out.

  1. Fine-tuned policies transfer poorly. SimpleMemVLA falls from 88.3% on RoboMME to 2.0% on RoboMME-Plus, FrameSamp+Modul from 44.5% to 2.8%, and PonderPounce from 60.8% to 2.9%. High performance on the original benchmark carries over only minimally without adaptation.
  2. Astra’s transfer has clear limits. Embodied Astra falls from 92.1% on RoboMME to 22.6% on RoboMME-Plus. It succeeds on 59% of easy episodes but only 10% of hard ones, and reaches just 2% on VideoUnmaskSwap. Its strong performance on the initial out-of-range tests does not extend to the full range of challenges.
  3. One example helps, but selectively. One-shot Embodied Astra reaches 35.3% overall, a gain of 12.6 percentage points. The largest improvements occur on SwingXtimes (28% → 98%), ButtonUnmask (34% → 94%), and PickXtimes (56% → 84%), suggesting that an example can clarify an effective strategy. But it does not improve VideoUnmaskSwap, VideoRepick, VideoPlaceOrder, or RouteStick, where tracking or execution remains challenging.

6. Looking forward

RoboMME-Plus shifts the focus from remembering in familiar settings to using memory reliably under more demanding conditions. Our findings point to four promising directions:

We hope RoboMME-Plus helps reveal where current memory-augmented policies fall short and advances robots that can turn remembered experience into reliable action.

Citation

If you find this blog useful, please cite:

@article{dai2026robommeplus,
  title = {{RoboMME-Plus}: Benchmarking Memory Generalization for Robot Policies},
  author = {Dai, Yinpei and Fu, Hongze and Liu, Yuejiang and Finn, Chelsea and Fazeli, Nima and Chai, Joyce},
  journal = {robomme.github.io},
  year = {2026},
  month = {October},
  url = {https://robomme.github.io/blog/}
}

References