Locomanipulation with Improved IK Control, G1 teleoperation and Locomotion policy #3150
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ed testing - Add NullSpacePostureTask for natural shoulder/waist joint postures during manipulation - Refactor IK with damping, null-space control, and joint optimization - Improve testing framework with JSON-based configurations and stricter tolerances (1mm/1° vs 30mm/10°) - Apply IK improvements across all environments (PickPlace, NutPour, ExhaustPipe) - Add waist-enabled manipulation environments and target_eef_link_names mapping - Disable IK joint limits safety checks so that it always produces a solution
michaellin6
requested review from
ashwinvkNV,
rwiltz,
huihuaNvidia2023 and
peterd-NV
michaellin6
requested review from
jtigue-bdai,
kellyguo11,
Mayankm96 and
pascal-roth
as code owners
michaellin6
changed the title
michaellin6
force-pushed
the
team/locomanipulation-integration
branch
from
ac61052 to
d2e5558
Compare
the locomanipulation environment for G1. Moved policy inference to action term. Adding test files for controllers in each environment.
Adds G1 fixed base teleop support, G1 retargeter with hand rotation fixes, comprehensive unit tests with pytest, and mock device testing.
- Migrate from command manager to using retargeter. - Enhance pink IK with kinemaitcs model accounting for non-pink-controlled joints, and introducing pink task LocalFrameTask that can specify ee goal poses relative to any robot link reference frame. - Generalize observation functions for manipulation and locomanipulation tasks.
- Added locomotion and locomanipulation policies with agile observation - Enhanced IK controller with gravity compensation and tuned gains - Expanded test coverage for ik - Added unit tests for new pink kinematics config class, and new custom pink frame task
michaellin6
force-pushed
the
team/locomanipulation-integration
branch
from
d2e5558 to
6951dbf
Compare
rwiltz
reviewed
Aug 15, 2025
rwiltz
approved these changes
Aug 15, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Contributors: @huihuaNvidia2023, @rwiltz, @life1ess, @peterd-NV, @yanchangNvidia
These changes introduce new environments:
Isaac-PickPlace-Locomanipulation-G1-Abs-v0andIsaac-PickPlace-FixedBaseUpperBodyIK-G1-Abs-v0. These environments, specially the locomanipulation one, feature:Below is a diagram showing the high level controller composition. The environments introduced have the waist controlled by IK, but it is possible to configure it such that waist DOFs are controlled by the lower body policy as well.
This code has been tested through the Mimic pipeline, and the output dataset was used to train a robomimic policy that achieves a success rate of 92% from 50 policy rollouts.
Fixes # (issue)
Some fixes included in these changes:
class:PinkKinematicsConfigurationthat allows for all joints of an articulation to be updated while only a subset are controlled by IK.Type of change
Screenshots
Robomimic policy for PickPlace trained with the Locomanipulation environment. We achieve 92% success rate from 50 rollouts.
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there