Benchmarking memory-augmented robotic generalist policies
(Data updated on 10/30/2026)
Standard uses open-source models; Agentic uses proprietary frontier models.
ZS: Zero-shot · FS: Few-shot
Model rollouts are available on Hugging Face.
Params: approximate parameter count. B = billion; M = million.
API Cost: reported API spending in USD for the RoboMME result, excluding local GPU compute. Hover over a value for its scope.
API cost: approximate total API spending across all 800 test episodes; excludes local GPU compute.
N/R: not reported. N/A: not applicable. ≈: approximate.
A benchmark for evaluating memory-augmented robotic policies across 16 tasks in four suites: Counting, Permanence, Reference, and Imitation.
Models are evaluated on 800 test episodes (50 per task). Multiple runs with different random seeds are recommended.
(1) Download the dataset and RoboMME repository, (2) train and evaluate with your policy, (3) submit a PR following the submission instructions.