NVIDIA IsaacLab-Arena Lerobot

[nv-sachdevkartik]

Type / Scope

• Type: Feature
• Scope: envs, processor

Summary / Motivation

This PR introduces NVIDIA IsaacLab-Arena integration into LeRobot, enabling GPU-accelerated simulation environments through the EnvHub infrastructure. IsaacLab-Arena provides high-fidelity physics simulation for large-scale policy evaluation for various tasks. The community is encouraged to create custom environments using the IsaacLab-Arena framework.

The integration allows developers to:

• Train imitation learning policies (PI0, SmolVLA, ACT, Diffusion) with the collected teleop dataset
• Evaluate trained policies in high-fidelity IsaacLab-Arena simulation environments

This implementation follows the EnvHub pattern, loading environments from the Hub repository nvkartik/isaaclab-arena-envs while providing a native environment configuration (IsaaclabArenaEnv) for seamless CLI integration.

Available Environments

The following are example environments built with NVIDIA IsaacLab-Arena:

Preview	Environment	Description
	gr1_microwave	Reach out to the microwave and open it.
	galileo_pnp	Pick objects and place in target location
	g1_locomanip_pnp	Pick up the brown box from the shelf, and place it into the blue bin on the table located at the right of the shelf.
	kitchen_pnp	Kitchen object manipulation tasks
	press_button	Locate and press button

This is just the start, more examples coming soon! 🚀

What Changed

New Files

File	Description
tests/processor/test_arena_processor.py	Comprehensive test suite for IsaacLab-Arena observation processor

Modified Files

File	Description
src/lerobot/envs/configs.py	Added IsaaclabArenaEnv configuration dataclass with IsaacLab-Arena-specific parameters (embodiment, object, environment, camera/state keys)
src/lerobot/envs/factory.py	Added IsaacLab-Arena environment instantiation via Hub loading and IsaaclabArenaProcessorStep creation
src/lerobot/envs/utils.py	Extended preprocess_observation() to handle IsaacLab-Arena's observation format (policy, camera_obs)
src/lerobot/processor/env_processor.py	Added IsaaclabArenaProcessorStep for processing IsaacLab-Arena observations (state concatenation, image normalization/permutation)
src/lerobot/scripts/lerobot_eval.py	Added render_all() support for IsaacLab wrapper video recording
tests/processor/test_pipeline.py	Additional processor pipeline tests

Key Implementation Details

Environment Configuration (IsaaclabArenaEnv):

@EnvConfig.register_subclass("isaaclab_arena")
@dataclass
class IsaaclabArenaEnv(EnvConfig):
    hub_repo_id: str = "nvkartik/isaaclab-arena-envs"
    environment: str = "gr1_microwave"
    embodiment: str = "gr1_pink"
    object: str = "power_drill"
    state_keys: str = "robot_joint_pos"
    camera_keys: str = "robot_pov_cam_rgb"
    # ... additional configuration

Observation Processor (IsaaclabArenaProcessorStep):

• Extracts state from obs["policy"] based on configurable state_keys
• Processes images from obs["camera_obs"] based on camera_keys
• Handles BHWC→BCHW permutation and uint8→float32 normalization
• Concatenates multiple state components into single observation.state vector

Breaking Changes: None

How Was This Tested

Automated Tests

pytest -q tests/processor/test_arena_processor.py

Test coverage includes:

• State extraction and float32 conversion
• Multiple state key concatenation with ordering verification
• Higher-dimensional state flattening
• State key filtering
• Camera BHWC→BCHW permutation
• uint8→normalized float32 image conversion
• float32 image passthrough
• Camera key filtering
• Missing observation section handling
• Empty observation handling
• Default and custom key configuration
• Feature transform passthrough

Manual Verification

Zero-agent environment test:

python test_env_load_arena.py \
    --env.type=isaaclab_arena \
    --env.environment=galileo_pnp \
    --env.embodiment=gr1_pink \
    --env.object=cracker_box \
    --env.num_envs=4 \
    --env.enable_cameras=true \
    --env.seed=1000

Policy evaluation (SmolVLA):

lerobot-eval \
    --policy.path=nvkartik/smolvla-arena-gr1-microwave \
    --env.type=isaaclab_arena \
    --rename_map='{"observation.images.robot_pov_cam_rgb": "observation.images.robot_pov_cam"}' \
    --env.environment=gr1_microwave \
    --env.embodiment=gr1_pink \
    --env.num_envs=1 \
    --env.headless=true \
    --policy.device=cuda \
    --env.enable_cameras=true \
    --env.video=true

Policy evaluation (PI0.5):

TORCH_COMPILE_DISABLE=1 TORCHINDUCTOR_DISABLE=1 lerobot-eval \
    --policy.path=nvkartik/pi05-arena-gr1-microwave \
    --env.type=isaaclab_arena \
    --rename_map='{"observation.images.robot_pov_cam_rgb": "observation.images.robot_pov_cam"}' \
    --env.environment=gr1_microwave \
    --env.embodiment=gr1_pink \
    --env.num_envs=1 \
    --env.headless=true \
    --policy.device=cuda \
    --env.enable_cameras=true

Training (PI0.5):

lerobot-train \
    --policy.type=pi05 \
    --dataset.repo_id nvkartik/Arena-GR1-Manipulation-Task \
    --rename_map '{"observation.images.robot_pov_cam":"observation.images.camera1"}' \
    --policy.empty_cameras=2 \
    --policy.max_state_dim=64 \
    --policy.max_action_dim=64 \
    --batch_size=16 \
    --steps=20000

How to Run Locally (Reviewer)

1. Install Dependencies

sudo apt update && sudo apt install -y libglu1-mesa libxt6

# Create conda environment
conda create -y -n lerobot-arena python=3.11
conda activate lerobot-arena
conda install -y -c conda-forge ffmpeg=7.1.1
pip install uv

# Install IsaacSim 5.1.0
uv pip install "isaacsim[all,extscache]==5.1.0" --extra-index-url https://pypi.nvidia.com
export ACCEPT_EULA=Y
export PRIVACY_CONSENT=Y

# Install IsaacLab 2.3.0
git clone https://github.com/isaac-sim/IsaacLab.git
cd IsaacLab && git checkout v2.3.0 && ./isaaclab.sh -i && cd ..

# Install LeRobot from source
git clone https://github.com/liorbenhorin/lerobot.git
cd lerobot
git checkout isaaclab_arena_lerobot
##  fix transformer version as per issue: #2641
sed -i 's|^pi = \[.*\]|pi = ["transformers @ git+https://github.com/huggingface/transformers.git@dcddb970176382c0fcf4521b0c0e6fc15894dfe0"]|' pyproject.toml
uv pip install -e ".[pi]"
 
# Install NVIDIA IsaacLab-Arena
cd ..
git clone [email protected]:isaac-sim/IsaacLab-Arena.git
cd IsaacLab-Arena
git checkout b4feca387f280e3d2d08ed3c4825f7877ab709f4
uv pip install -e .
uv pip install onnxruntime lightwheel-sdk vuer[all]

# Fix dependencies
pip install numpy==1.26.0 qpsolvers==4.8.1 lxml==4.9.4 packaging==23.2

2. Run Tests

pytest -q tests/processor/test_arena_processor.py -v

3. Run Environment Smoke Test

python -c "
from lerobot.envs.configs import IsaaclabArenaEnv
cfg = IsaaclabArenaEnv(environment='gr1_microwave')
print(f'Config created: {cfg.environment}')
print(f'Hub kwargs: {cfg.hub_kwargs}')
"

Checklist (Required Before Merge)

• Linting/formatting run (pre-commit run -a)
• All tests pass locally (pytest tests/processor/test_arena_processor.py)
• Documentation updated
• CI is green

Reviewer Notes

Focus Areas

1. IsaaclabArenaProcessorStep (src/lerobot/processor/env_processor.py): Core observation processing logic - verify state concatenation ordering and image normalization
2. IsaaclabArenaEnv config (src/lerobot/envs/configs.py): Validate Hub kwargs mapping and feature definitions
3. Factory integration (src/lerobot/envs/factory.py): Ensure proper Hub loading and processor instantiation

Available Resources

Datasets:

• Arena-GR1-Manipulation-Task - GR1 manipulation demos
• Arena-G1-Loco-Manipulation-Task - G1 loco-manipulation demos

Trained Policies:

• pi05-arena-gr1-microwave - PI0.5 policy
• smolvla-arena-gr1-microwave - SmolVLA policy

Environment Repository:

• nvkartik/isaaclab-arena-envs - EnvHub environment

Notes for Community Reviewers

• GPU required for running IsaacLab environments
• NVIDIA driver and CUDA toolkit must be compatible with IsaacSim 5.1.0
• Environment loading requires accepting NVIDIA EULA (export ACCEPT_EULA=Y)

[liorbenhorin]

@imstevenpmwork - For the installation phase - we will try to simplify the process as possible - and include it in the documentation page [to be written] - it will be similar to how it is done for our gr00t implementation.
This is mainly due to to
pip install numpy==1.26.0 qpsolvers==4.8.1 lxml==4.9.4 packaging==23.2
The numpy dependency will be resolved as soon as Isaac Sim 6.0.0 will be available. (it is technically already available, but developers need to build from source, so we prefer to wait)

[jadechoghari]

This might be unecesary.
If "policy" represents the robot state, you can change the IsaaclabArenaProcessorStep to conform to the accepted namings of preprocess_observation which include agent_pos or robot_state.
Same thing for camera_obs. Our naming convenion is OBS_IMAGES.{camera_name} not "{OBS_STR}.camera_obs" for visual observations.

I also see you're already renaming correclty in the env processor

[nv-sachdevkartik]

@jadechoghari : is there a plan to complete this todo as mentioned, this would give flexibility to use custom names?

[nv-sachdevkartik]

@aliberts : it would be great to remove the hardcoded features.

[jadechoghari]

why is this change necessary?

[jadechoghari]

Can we instead introduce a small env.render wrapper for the env? instead of adding a special case branch here

[nv-sachdevkartik]

done, added IsaacLab wrapper based on AsyncVectorEnv, thus removing this code.

[jadechoghari]

Can i switch embodiments in each tasks?. For instance in task A, can i select robot A or robot B

[viiik-inside]

yes

[jadechoghari]

Won't this version of numpy break compatibailtiy with the current numpy version in LeRobot?

[liorbenhorin]

It is not ideal, but so far our tests didn't exhibited any breaking.
We can include a note recommending the developer to create a dedicated conda env for evaluation using this feature, and other conda env for training, in case they encounter any numpy-related warnings

[jadechoghari]

If i don't have a nvidia gpu, i won't be able to run isaac arena?

[liorbenhorin]

Isaac Lab requires NVIDIA GPU - if one does not have one handy we usually point them to Brev

[jadechoghari]

Isn't this what this conditioning branch (envhub) already does?

lerobot/src/lerobot/envs/factory.py

Line 115 in b2ef6ae

if isinstance(cfg, str):

So we can just pass the hub string instead, no? nvkartik/isaaclab-arena-envs

[jadechoghari]

FYI, we will merge this PR soon: https://github.com/huggingface/lerobot/pull/2599/files for the custom kwargs if it works for you