About RoboMME-Plus Task Examples Training Data Evaluation RoboMME-Plus Leaderboard

RoboMME-Plus: Benchmarking Memory Generalization for Robot Policies

Yinpei Dai1, Hongze Fu1, Yuejiang Liu2, Chelsea Finn2, Nima Fazeli1, Joyce Chai1

1University of Michigan 2Stanford University

Blog Code Training Data Collection Leaderboard

What is RoboMME-Plus?

RoboMME-Plus tests whether models trained on RoboMME can use their memory skills under more demanding conditions. It retains the 16 tasks across Counting, Permanence, Reference, and Imitation, while varying object counts, motion, occlusion, demonstration histories, and tool usage to introduce new memory-augmented manipulation challenges.

The benchmark supports zero-shot evaluation, without adaptation to RoboMME-Plus data, and few-shot adaptation using a small RoboMME-Plus training set. Both settings use the same 800 fixed test episodes, enabling consistent evaluation of direct transfer and the gains from limited adaptation data.

RoboMME-Plus Task Examples

Loading task examples…

View the RoboMME-Plus leaderboard →

A small training set for adaptation

The training split contains 50 successful episodes: three for each task, except MoveCube, which includes one for each of its five manipulation methods. Training and test seeds are disjoint within each task.

The HDF5 data contains front and wrist camera observations, robot actions, task goals, and episode information. The videos above are previews of these training episodes.

Adaptation protocol

  • Adapt only with the released RoboMME-Plus training data or simulator interactions using the provided training seeds. Keep test episodes exclusively for evaluation.
  • Do not reverse-engineer the task source code to derive solutions or extract privileged information.
  • Keep task-specific prompting minimal; avoid detailed, hand-written solution instructions.

Get the training data on Hugging Face

Browse the RoboMME-Plus collection →

Evaluate your model

Zero-shot transfer

Evaluate a RoboMME-trained model directly on the RoboMME-Plus test split, without using RoboMME-Plus training examples.

Few-shot adaptation

Adapt using the RoboMME-Plus training examples or by interacting with the simulator using the training seeds, then evaluate on the same test split.

Results use two leaderboard tracks: Standard for open-source models and Agentic for methods using proprietary frontier models.

Use the RoboMME simulator setup and the robomme_plus environment builder. All 16 tasks × 50 test episodes support joint_angle, ee_pose, and waypoint actions.

Minimal evaluation loop 
from robomme_plus import BenchmarkEnvBuilder

builder = BenchmarkEnvBuilder(
    "VideoRepick", dataset="test", action_space="joint_angle"
)
env = builder.make_env_for_episode(0)
try:
    obs, info = env.reset()
    task_goal = info["task_goal"][0]
    while True:
        action = policy.predict(obs, task_goal,
                                error_message=info.get("error_message"))
        obs, reward, terminated, truncated, info = env.step(action)
        if terminated or truncated:
            success = info.get("status") == "success"
            break
finally:
    env.close()

Implement policy.predict for your model. Keep the demonstration frames returned by reset. Observation fields are lists; the last item is the current sample.

Code and installation ↗ Submission template ↗ Hugging Face collection ↗ RoboMME-Plus leaderboard →

Citation

If you find RoboMME-Plus useful, please cite:

@article{dai2026robommeplus,
  title = {{RoboMME-Plus}: Benchmarking Memory Generalization for Robot Policies},
  author = {Dai, Yinpei and Fu, Hongze and Liu, Yuejiang and Finn, Chelsea and Fazeli, Nima and Chai, Joyce},
  journal = {robomme.github.io},
  year = {2026},
  month = {October}
}