1. 🌍 Genie Sim

Genie Sim is the simulation platform from AgiBot. It provides developers with a complete toolchain for environment reconstruction, scene generalization, data collection, and automated evaluation. Its core module, Genie Sim Benchmark is a standardized tool dedicated to establishing the most accurate and authoritative evaluation for embodied intelligence.

The platform integrates 3D reconstruction with visual generation to create a high-fidelity simulation environment. It pioneers LLM-driven technology to generate vast simulation scenes and evaluation configurations in minutes. The evaluation system covers 200+ tasks across 100,000+ scenarios to establish a comprehensive capability profile for models. Genie Sim also opens over 10,000 hours of synthetic data including real-world robot operation scenarios.

The platform will significantly accelerate model development, reduce reliance on physical hardware, and empower innovation in embodied intelligence. Simulation assets, dataset, and code are fully open source.

2. ✨ Features

• 🧞 geniesim CLI: one command-line entry point for the entire platform — docker lifecycle, ROS 2 workspace builds, health diagnosis (doctor), pure-Python wheel deploy, asset / scene bootstrap. Packaged as a standalone PEP 517 / PEP 621 wheel.
• 🤖 Agent-ready SKILLs: packaged SKILL.md recipes for benchmarking, inference checks, scene generation, asset search, and teleop — invokable directly by Claude Code / agentic coding agents, so the simulator can be driven without a human first reading the docs.
• ⚡ Genie Sim RT Engine (realtime, interactive): geniesim_ros brings physics, render, and the robot into ROS 2 natively — teleop, MoveIt, and ros2_control close the loop on one shared sim_time. Multiple physics backends (Isaac Sim PhysX, Newton-standalone) selectable per scene, with cloth + soft-body support on the Newton path.
• 🎨 High-Fidelity Sim-Ready Assets: 5,140 validated 3D assets covering five real-world operation fields: retail, industry, catering, home and office. ModelScope.
• 🛰️ 3DGS-based Reconstruction Pipeline: Integrate 3DGS-based reconstruction process with visual generative model to synthesize realistic simulation environment with high-precision meshes. ModelScope.
• 🌐 Genie Sim World: A multimodal spatial world model which generates photorealistic 3D world from diverse input types in minutes. GitHub.
• 💬 LLM-Driven Scene Generation: Natural language-driven generation and generalization which instantly generates diverse simulation scenes through conversational interaction.
• 📦 Large-Scale Synthetic Dataset: Over 10,000 hours open-source synthetic data across 200+ loco-manipulation tasks with multi-sensor streams, alongside multi-dimensional variations.
• 🛠️ Synthetic Data Generation: Efficient toolkit for data collection with error-recovery mechanism, supporting both low-latency teleoperation and automated data programming. ModelScope.
• 🧪 Robust and Diverse Benchmark: Provide 100,000+ simulation scenarios and use LLM to autonomously generate task instructions and evaluation configurations. Discrepancy between simulation and real-world test results is less than 10%.
• 🔍 VLM-based Auto-Evaluation System: Full-spectrum evaluation criteria to provide model’s capability profile covering manipulation skills, cognitive comprehension and task complexity.
• 🚀 Zero-Shot Sim-to-Real Transfer: Model trained with our synthetic data exhibits zero-shot sim-to-real transfer capability with superior task success rate compared to model trained with real data.

3. 🚀 Quick start

3.1 🧞 CLI & 🤖 SKILLs — install, build the container, get a shell

flowchart LR
  pip(["📦 pip installgeniesim_cli(host)"]) ==> docker(["🐳 geniesimdocker up(host)"]) ==> auto(["⚙️ tier-1 peersauto-install(in-container)"]) ==> shell(["💻 geniesimdocker into(host)"])
  classDef step fill:#dbeafe,stroke:#1e40af,color:#1e3a8a,font-weight:bold
  class pip,docker,auto,shell step
  linkStyle 0,1,2 stroke:#1e40af,stroke-width:2px

# === on the host ===
pip install -e source/geniesim_cli/      # standalone CLI wheel
geniesim docker build                    # build the Isaac Sim 5.1 + Jazzy image
geniesim docker up                       # boot the container; the entrypoint
                                         # editable-installs every tier-1 from
                                         # the bind-mounted workspace
geniesim docker into                     # interactive shell inside the container

# === inside the container — sanity-check the stack ===
geniesim status                          # per-distribution health probe
geniesim doctor                          # diagnose & repair (rosdep, env, …)
geniesim bootstrap                       # re-install tier-1 peers + opt-in tier-2

🤖 With an AI assistant — just ask

The repo ships AGENTS.md + per-package skills//SKILL.md as machine-readable signposts. Claude Code, Cursor, etc. follow them on their own — say “run the pnp benchmark against localhost:8999” or “launch the WBC scene with MoveIt” and the agent walks the trail.

📖 Without an AI assistant — follow the signposts

AGENTS.md (root, boot sequence) → source/AGENTS.md (module map) → source//AGENTS.md (architecture + commands) → source//skills//SKILL.md (copy-paste recipes). Full CLI verb table: .agent/geniesim_cli.md.

[!IMPORTANT] 🎨 Tier-1 peers come up automatically; tier-2 peers are opt-in. geniesim docker up’s entrypoint editable-installs every tier-1 peer declared in source/geniesim/pyproject.toml. The asset library geniesim_assets is a tier-1 peer distributed out of band (sourced from HuggingFace / ModelScope), and editable-installed by the entrypoint so every other peer can find it.

Tier-2 peers (geniesim_teleop · geniesim_generator · geniesim_world) have heavy stacks (VR / LLM / CUDA-ML) and are not auto-installed. Install with pip install -e "source/geniesim/[teleop|generator|world|all]" — or run geniesim bootstrap for an interactive re-install of the whole tree.

3.2 🧪 Benchmark — debug locally, compete globally on the AgiBot World Challenge: Open-Session

flowchart LR
  list(["📋 benchmarklist / categories"]) ==> probe(["🔌 check-inference"]) ==> run(["🧪 benchmarkrun · batch"]) ==> submit(["🏆 Open-Sessionleaderboard"])
  classDef step fill:#ede9fe,stroke:#5b21b6,color:#3b0764,font-weight:bold
  class list,probe,run,submit step
  linkStyle 0,1,2 stroke:#5b21b6,stroke-width:2px

The Genie Sim Benchmark is the engine behind the AgiBot World Challenge: Open-Session — and the same tool you use to get ready for it:

• 🏆 It’s a competition. Open-Session ranks embodied-AI policies on four boards (instruction / robust / manip / spatial). The boards and their baseline scores are in the Genie Sim Benchmark Leaderboard below.
• 🧪 Debug locally → submit remotely. Run the benchmark tasks on your own machine to debug your policy against your own inference server, then launch a model evaluation on the official leaderboard with the same server — no blind submissions.
• 🤖 Agent-friendly one-click SKILLs. The whole competition pipeline — download datasets, fetch the π0.5 baseline weights, run an example leaderboard service, submit & track results — ships under source/geniesim_benchmark/skills/agibot-world-challenge/; just ask an AI agent (Claude Code, Cursor, …) to drive them.

Run it locally — cat source/geniesim_benchmark/USAGE.md for how to run a benchmark task on your own machine against your inference server (CLI verbs, common commands, and overrides).

3.3 ⚡ RT Engine — realtime, interactive ROS 2 simulation

flowchart LR
  build(["🔨 ros builddev"]) ==> scene(["🎬 launchscene × launcher"]) ==> sim(["⚡ RT EnginePhysX · Newton · mjwarp"]) ==> moveit(["🦾 MoveIt 2+ WBC RViz"])
  classDef step fill:#dcfce7,stroke:#15803d,color:#14532d,font-weight:bold
  class build,scene,sim,moveit step
  linkStyle 0,1,2 stroke:#15803d,stroke-width:2px

The RT Engine ships its own SKILLs that walk you through every step — workspace build → scene launch → MoveIt + WBC RViz → bringing your own robot → teleop → recording → debugging. Don’t memorize ros2 launch flags — follow the skill:

# inside the container:
geniesim ros build dev
cat source/geniesim_ros/skills/build-workspace/SKILL.md    # `geniesim ros build dev`
cat source/geniesim_ros/skills/launch-scene/SKILL.md       # scene × launcher matrix
cat source/geniesim_ros/skills/moveit-wbc/SKILL.md         # MoveIt 2 + WBC RViz for Genie G2
cat source/geniesim_ros/skills/add-robot/SKILL.md          # bring your own URDF / xacro
cat source/geniesim_ros/skills/*/SKILL.md                  # and more

3.4 🌐 Generate a 3D world — geniesim_world

flowchart LR
  pano(["🖼️ Panorama(.png)"]) ==> da(["🔭 DA360depth"]) ==> sharp(["✨ SHARPGaussians"]) ==> world(["🌍 3D world(PLY · Gaussians)"])
  classDef step fill:#ffedd5,stroke:#c2410c,color:#7c2d12,font-weight:bold
  class pano,da,sharp,world step
  linkStyle 0,1,2 stroke:#c2410c,stroke-width:2px

Turn an equirectangular panorama into an explorable photorealistic 3D world. (Loading the output into the RT Engine is a 🚧 W.I.P. — for now the generated PLY / Gaussians are consumed by geniesim_world’s own tooling.)

cat source/geniesim_world/skills/generate-world/SKILL.md

3.5 🧱 Scene Reconstruction — scene_reconstruction

flowchart LR
  input(["📥 LiDAR and Images"]) ==> align(["📐 Pixel-level Alignment of Point Cloud and Camera Pose"]) ==> gs(["🌈 3DGS"]) ==> world(["🌍 3D world(PLY · Gaussians)"])
  classDef step fill:#e0f2fe,stroke:#0369a1,color:#0c4a6e,font-weight:bold
  class input,align,gs,world step
  linkStyle 0,1,2 stroke:#0369a1,stroke-width:2px

Reconstruct high-fidelity 3D scene assets from real-world LiDAR scans and camera images. The pipeline aligns point clouds with camera poses at pixel level, trains 3D Gaussian Splatting, and produces simulation-ready assets.

cat source/scene_reconstruction/AGENTS.md

See source/scene_reconstruction/README.md for the full pipeline details and AGENTS.md for the per-subsystem agent guides.

3.6 💬 Generate a scene from a prompt — geniesim_generator

flowchart LR
  nl(["💬 NL prompt'an apple on the table'"]) ==> assets(["🔍 searchasset library"]) ==> dsl(["📝 Scene-LanguageLLM_RESULT.py"]) ==> usda(["🏗️ scene.usda+ layout graph"])
  classDef step fill:#ede9fe,stroke:#6d28d9,color:#3b0764,font-weight:bold
  class nl,assets,dsl,usda step
  linkStyle 0,1,2 stroke:#6d28d9,stroke-width:2px

Describe a scene in words; an LLM grounds it against the asset library, writes a Scene-Language DSL program, and the generator compiles it into scene.usda + a relationship graph. Drive it through the Open WebUI agent loop, or have Claude write the DSL directly (no WebUI / no MCP server). Embeddings run via a text-embedding API (no GPU) or a local VL model (GPU) — your pick at deploy time.

Generated scenes are written under source/geniesim_generator/src/benchmark/config/llm_task///. If generated inside the Docker stack, the same bind-mounted files appear at /opt/geniesim_generator/src/benchmark/config/llm_task/// in the container and at the source/... path on the host. Open the resulting scene.usda with Isaac Sim inside the Genie Sim container so its payload paths resolve as generated; if you open it from the host side, adjust the payload paths inside the USDA first.

cat source/geniesim_generator/skills/deploy-generator/SKILL.md    # install + start: MCP stack + Open WebUI, text vs VL embeddings
cat source/geniesim_generator/skills/search-assets/SKILL.md       # query the asset index by concept
cat source/geniesim_generator/skills/generate-scene/SKILL.md      # prompt → LLM_RESULT.py → scene.usda

See source/geniesim_ros/README.md for the engine overview, source/README.md for the full module map (every geniesim_* package with its docs + skills), and AGENTS.md for the per-subsystem agent guides.

3.7 🎮 Teleoperation — geniesim_teleop

flowchart LR
  up(["🐳 docker5.1 up"]) ==> collect(["🎮 autoteleop.shVR collection"]) ==> keep(["💾 keep / discard(y / n)"]) ==> post(["📦 post-process→ HDF5 episodes"])
  classDef step fill:#fce7f3,stroke:#be185d,color:#831843,font-weight:bold
  class up,collect,keep,post step
  linkStyle 0,1,2 stroke:#be185d,stroke-width:2px

Collect demonstration data by teleoperating the robot with a Pico VR headset, then turn the recorded rosbags into HDF5 episodes. Once the image is built and the GUI container is up (geniesim docker5.1 up succeeds), the whole flow is two host-side scripts:

# 1. VR collection — press y (keep) / n (discard) when the collection is done:
./source/geniesim_teleop/scripts/autoteleop.sh

# 2. Post-process the recording into HDF5 episodes:
./source/geniesim_teleop/scripts/autoteleop_post_process.sh 

See source/geniesim_teleop/README.md for the controller button map, Pico setup, and task configuration.

3.8 📥 Data collection — data_collection

flowchart LR
  build(["🔨 docker build +autocollect build"]) ==> list(["📋 autocollect listpick a task"]) ==> run(["📥 autocollect runcuRobo auto-planning"]) ==> ep(["💾 episodesrecording_data/"])
  classDef step fill:#e0f2fe,stroke:#0369a1,color:#0c4a6e,font-weight:bold
  class build,list,run,ep step
  linkStyle 0,1,2 stroke:#0369a1,stroke-width:2px

The automated counterpart to teleoperation: pre-orchestrated tasks are collected unattended using cuRobo trajectory planning — no human in the loop. Install the geniesim client and assets first (see § 3.1).

# 1. Build the GenieSim base image, then the data-collection image:
geniesim docker build         # → registry.agibot.com/genie-sim/geniesim3:latest (base)
geniesim autocollect build    # → registry.agibot.com/genie-sim/geniesim3-data-collection:latest

# 2. Discover tasks, then collect one (headless, unattended):
geniesim autocollect list --robot=g2 
geniesim autocollect run  --headless --standalone

Or let your AI assistant drive the whole run — point it at the skill and it follows the workflow (resolve task → check prereqs → launch → monitor & verify):

cat source/data_collection/skills/run-data-collection/SKILL.md

Episodes land in source/data_collection/recording_data/. See source/data_collection/README.md (and its AGENTS.md / skill) for flags, task configs, and the two-process server/client layout.

4. 📜 Roadmap & Updates

The arc of the platform — what shipped when, and what’s coming next.

flowchart LR
  v2["v2.x · 2025Challenge tasks2× sim speed"]
  v30["v3.0 · Jan 2026Isaac Sim 5.1Genie G2 · 3DGS"]
  v31["v3.1 · Apr 2026Genie Sim WorldRLinf"]
  v32["v3.2 · Jun 2026🧞 CLI · 🤖 SKILLs⚡ RT Engine"]
  next["NextIsaac Sim 6.0More Newton solversBenchmark refactor"]
  v2 --> v30 --> v31 --> v32 --> next

  classDef done fill:#d1f0d8,stroke:#1f6f3d,color:#0b3b1f
  classDef current fill:#cfe2ff,stroke:#084298,color:#084298,stroke-width:2px
  classDef future fill:#fff3cd,stroke:#856404,color:#5c4400
  class v2,v30,v31 done
  class v32 current
  class next future

🎯 Next

• [ ] Upload all assets and dataset on Huggingface
• [ ] More benchmark task suites — deformable / soft-material manipulation, whole-body control (WBC), vision-language-navigation (VLN), and more
• [ ] Support more tasks and larger models for RLinf
• [ ] More Newton solvers — broaden the rigid + deformable solver in the Newton-standalone
• [ ] Refactor geniesim_benchmark into a benchmark layer on top of geniesim_ros (collapses the two parallel stacks — see .agent/geniesim_benchmark.md)
• [ ] Data recorder improvements (today: ros2 bag via the record-episode SKILL)
• [ ] Wire geniesim_world outputs into the RT Engine (load .ply / .gsp Gaussians from a scene_*.yaml)
• [ ] Isaac Sim 6.0

📜 Release log

5. 📚 Documentation

5.1 📖 Documentations

Please refer to these links to install Genie Sim and download assets and dataset:

• User Guide (applies to Genie Sim < 3.2; from 3.2 the canonical workflow is the geniesim CLI + per-package SKILLs — see § 2 ✨ Features and § 3 🚀 Quick start)
• Assets
• Dataset

5.2 🏆 Genie Sim Benchmark Leaderboard

The four tables below are the AgiBot World Challenge: Open-Session boards — instruction, robust, manip, spatial — reporting baseline scores for four reference models. The final GenieSim-Sim2Real table is not a competition board; it quantifies the simulation-to-reality fidelity of the benchmark. Each task is evaluated under a 2×2 design that crosses the training-data source (simulation vs. real) with the evaluation environment (simulation vs. real), yielding the sim-to-sim, real-to-sim, sim-to-real, and real-to-real conditions. The close agreement between corresponding simulated and physical success rates indicates that evaluation in Genie Sim is a faithful proxy for real-world evaluation.

Baseline model repositories:

GenieSim-Manipulation

GenieSim-Spatial

GenieSim-Sim2Real

† Sim Data: 500~1500 episodes of simulation data. Real Data: 500 episodes of real-world data. All models are post-trained from the π0.5 baseline.

5.3 💬 Support

5.4 📄 License and Citation

All the data and code within source/geniesim_* and source/data_collection are under Mozilla Public License 2.0 unless licensed specifically. The source/scene_reconstruction project contains code under multiple licenses, for complete and updated licensing details, please see the LICENSE files

Please consider citing our work either way below if it helps your research.

@misc{yin2026geniesim30,
  title={Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot},
  author={Chenghao Yin and Da Huang and Di Yang and Jichao Wang and Nanshu Zhao and Chen Xu and Wenjun Sun and Linjie Hou and Zhijun Li and Junhui Wu and Zhaobo Liu and Zhen Xiao and Sheng Zhang and Lei Bao and Rui Feng and Zhenquan Pang and Jiayu Li and Qian Wang and Maoqing Yao},
  year={2026},
  eprint={2601.02078},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2601.02078},
}

5.5 🔗 References

1. PDDL Parser (2020). Version 1.1. [Source code]. https://github.com/pucrs-automated-planning/pddl-parser.
2. BDDL. Version 1.x.x [Source code]. https://github.com/StanfordVL/bddl
3. CUROBO [Source code]. https://github.com/NVlabs/curobo
4. Isaac Lab [Source code]. https://github.com/isaac-sim/IsaacLab
5. Omni Gibson [Source code]. https://github.com/StanfordVL/OmniGibson
6. The Scene Language [Source code]. https://github.com/zzyunzhi/scene-language
7. COAL [Source code]. https://github.com/coal-library/coal
8. OCTOMAP [Source code]. https://github.com/OctoMap/octomap
9. PINOCCHIO [Source code]. https://github.com/stack-of-tasks/pinocchio
10. URDFDOM [Source code]. https://github.com/ros/urdfdom
11. LIBCCD [Source code]. https://github.com/danfis/libccd
12. LIBMINIZIP [Source code]. https://github.com/switch-st/libminizip
13. LIBODE [Source code]. https://github.com/markmbaum/libode
14. LIBURING [Source code]. https://github.com/axboe/liburing
15. MuJoCo [Source code]. https://github.com/google-deepmind/mujoco