[Bug Report] MJCF converter generates lots of warnings and errors

[KyleM73]

Describe the bug

I'm working on a PR #1672 to mirror the UrdfFileCfg importer/spawner for MJCF files. The code is written (see PR) but during testing I noticed that there are a lot of errors logged to the console during conversion (see below). To make sure these errors were not my fault, I tested the convert_mjcf.py script and found that it produces the same errors. I don't see another issue bringing this up.

Steps to reproduce

All MJCF files tested came from mujoco_menagerie. I tested the conversion for the Apptronik Apollo (humanoid), BD Spot (quadruped), and UR5e (arm) and got the same result for all except Apollo, which fails entirely (there are goblins in their MJCF, I fear...). I use the docker workflow. Then

isaaclab -p <path to IsaacLab>/source/standalone/tools/convert_mjcf.py <path to mjcf>.xml <path to desired usd>.usd --headless

The resulting errors are below:

(some non-error output truncated for brevity, e.g. printing out the GPU information)

[INFO] Using python from: /workspace/isaaclab/_isaac_sim/python.sh                                                     
[INFO][AppLauncher]: Loading experience file: /workspace/isaaclab/source/apps/isaaclab.python.headless.kit
Loading user config located at: '/isaac-sim/kit/data/Kit/Isaac-Sim/4.2/user.config.json'
[Info] [carb] Logging to file: /isaac-sim/kit/logs/Kit/Isaac-Sim/4.2/kit_20250114_032057.log
2025-01-14 03:20:57 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
2025-01-14 03:20:58 [570ms] [Warning] [omni.platforminfo.plugin] failed to open the default display.  Can't verify X Ser
ver version.

[...truncated...]

2025-01-14 03:21:01 [3,186ms] [Error] [gpu.foundation.plugin] DriverShaderCacheManager::init() called without a shutdown
()!

[...truncated...]

2025-01-14 03:21:01 [3,761ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::w
indowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2025-01-14 03:21:01 [3,899ms] [Error] [omni.ext._impl.custom_importer] Failed to import python module omni.importer.mjcf
.tests. Error: No module named 'omni.kit.test'. Traceback:
Traceback (most recent call last):
  File "/workspace/isaaclab/_isaac_sim/kit/kernel/py/omni/ext/_impl/custom_importer.py", line 76, in import_module
    return importlib.import_module(name)
  File "/workspace/isaaclab/_isaac_sim/kit/python/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/isaac-sim/extscache/omni.importer.mjcf-1.1.1+106.0.0.lx64.r.cp310/omni/importer/mjcf/tests/__init__.py", line 1
0, in <module>
    import omni.kit.test
ModuleNotFoundError: No module named 'omni.kit.test'

2025-01-14 03:21:01 [3,899ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: 'omni.importer.mjcf.tests' in '/isaac-sim/extscache/omni.importer.mjcf-1.1.1+106.0.0.lx64.r.cp310' failed to load.

At:
  /workspace/isaaclab/_isaac_sim/kit/kernel/py/omni/ext/_impl/_internal.py(213): startup
  /workspace/isaaclab/_isaac_sim/kit/kernel/py/omni/ext/_impl/_internal.py(328): startup_extension
  PythonExtension.cpp::startup()(2): <module>
  /isaac-sim/exts/omni.isaac.core/omni/isaac/core/utils/extensions.py(99): enable_extension
  /workspace/isaaclab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mjcf_converter.py(96): _get_mjcf_import_config
  /workspace/isaaclab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mjcf_converter.py(60): _convert_asset
  /workspace/isaaclab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/asset_converter_base.py(106): __init__
  /workspace/isaaclab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mjcf_converter.py(48): __init__
  /workspace/isaaclab/source/extensions/isaaclab.hcrl/scripts/convert_mjcf.py(100): main
  /workspace/isaaclab/source/extensions/isaaclab.hcrl/scripts/convert_mjcf.py(111): <module>

2025-01-14 03:21:01 [3,899ms] [Error] [omni.ext.plugin] [ext: omni.importer.mjcf-1.1.1] Failed to startup python extension.
2025-01-14 03:21:02 [4,379ms] [Warning] [omni.usd] Warning: in SdfPath at line 81 of /builds/omniverse/usd-ci/USD/pxr/usd/sdf/path.cpp -- Ill-formed SdfPath <>: syntax error

2025-01-14 03:21:02 [4,379ms] [Warning] [omni.usd] Coding Error: in _IsValidPathForCreatingPrim at line 3340 of /builds/omniverse/usd-ci/USD/pxr/usd/usd/stage.cpp -- Path must be an absolute path: <>

MJCF importer output:
Generated USD file: <path>/universal_robots_ur5e/robot.usd
--------------------------------------------------------------------------------

Note that the conversion still appears to complete (not for Apollo).

System Info

• Commit: 8b5fd06 (main as of Jan. 13 2024)
• Isaac Sim Version: 4.2.0-rc.18+release.16044.3b2ed111.gl
• OS: Ubuntu 22.04.3
• GPU: RTX 3080 12Gb
• CUDA: 12.4
• GPU Driver: 550.142
• Headless in Docker workflow

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

• convert_mjcf.py runs without dumping errors and logs to the console
  OR
• it is verified that this is the expected behavior

[RandomOakForest]

Thanks for submitting this. The team will follow up.

[itsmehere]

Same error when training using rsl_rl/train.py on a custom imported URDF asset, loaded using sim_utils.UrdfFileCfg.

2025-02-02 09:57:45 [15,522ms] [Error] [gpu.foundation.plugin] DriverShaderCacheManager::init() called without a shutdown()!

[kellyguo11]

Would you mind validating if the same error still happens on the latest Isaac Lab 2.0 release?

[itsmehere]

I just tried again, and I believe it still does. This error only arises when using a custom urdf asset that I try to import using UrdfFileCfg, and it wasn't happening before the 2.0 release.

Output

(env_isaaclab) mrao@em10:~/frankatask/IsaacLab$ ./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Lift-Cube-Bimanual-UR5-v0 --num_envs=1 --headless
[INFO] Using python from: /home/mrao/miniconda3/envs/env_isaaclab/bin/python
[INFO][AppLauncher]: Loading experience file: /home/mrao/frankatask/IsaacLab/apps/isaaclab.python.headless.kit
[Warning] [simulation_app.simulation_app] Modules: ['omni.kit_app'] were loaded before SimulationApp was started and might not be loaded correctly.
[Warning] [simulation_app.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
Loading user config located at: '/home/mrao/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/omni/data/Kit/Isaac-Sim/4.5/user.config.json'
[Info] [carb] Logging to file: /home/mrao/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/omni/logs/Kit/Isaac-Sim/4.5/kit_20250203_223919.log
2025-02-03 22:39:19 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
2025-02-03 22:39:19 [7ms] [Warning] [omni.ext.plugin] [ext: extensions] Extensions config 'extension.toml' doesn't exist '/home/mrao/frankatask/IsaacLab/source/extensions' or '/home/mrao/frankatask/IsaacLab/source/extensions/config'
2025-02-03 22:39:19 [7ms] [Warning] [omni.ext.plugin] [ext: standalone] Extensions config 'extension.toml' doesn't exist '/home/mrao/frankatask/IsaacLab/source/standalone' or '/home/mrao/frankatask/IsaacLab/source/standalone/config'
2025-02-03 22:39:19 [261ms] [Warning] [omni.usd_config.extension] Enable omni.materialx.libs extension to use MaterialX
2025-02-03 22:39:20 [462ms] [Warning] [omni.platforminfo.plugin] failed to open the default display. Can't verify X Server version.
2025-02-03 22:39:20 [948ms] [Warning] [omni.datastore] OmniHub is inaccessible
2025-02-03 22:39:20 [1,233ms] [Warning] [omni.isaac.dynamic_control] omni.isaac.dynamic_control is deprecated as of Isaac Sim 4.5. No action is needed from end-users.

|---------------------------------------------------------------------------------------------|
| Driver Version: 555.42.02 | Graphics API: Vulkan
|=============================================================================================|
[...]
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA A100-PCIE-40GB | Yes: 0 | | 40960 MB | 10de | 0 |
[...]
|=============================================================================================|
| OS: 24.04 LTS (Noble Numbat) ubuntu, Version: 24.04.1768292455, Kernel: 6.8.0-31-generic
| Processor: Intel(R) Xeon(R) Gold 5220R CPU @ 2.20GHz
| Cores: 48 | Logical Cores: 96
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 515599 | Free Memory: 214569
| Total Page/Swap (MB): 0 | Free Page/Swap: 0
|---------------------------------------------------------------------------------------------|
2025-02-03 22:39:34 [14,921ms] [Warning] [gpu.foundation.plugin] ECC is enabled on physical device 0
2025-02-03 22:39:34 [14,939ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled.
2025-02-03 22:39:34 [14,939ms] [Warning] [gpu.foundation.plugin] Detected IOMMU is enabled. Running CUDA peer-to-peer bandwidth and latency validation.
Cuda failure ../../../source/plugins/carb.cudainterop/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu:681: 'peer access is already enabled'
[INFO]: Parsing configuration from: isaaclab_tasks.manager_based.manipulation.lift.config.bimanual_ur5.joint_pos_env_cfg:BimanualUR5CubeLiftEnvCfg
[INFO]: Parsing configuration from: isaaclab_tasks.manager_based.manipulation.lift.config.bimanual_ur5.agents.rsl_rl_ppo_cfg:BimanualUR5CubeLiftPPORunnerCfg
[INFO] Logging experiment in directory: /home/mrao/frankatask/IsaacLab/logs/rsl_rl/bimanual_ur5_lift
Exact experiment name requested from command line: 2025-02-03_22-39-36
Setting seed: 42
[INFO]: Base environment:
Environment device : cuda:0
Environment seed : 42
Physics step-size : 0.01
Rendering step-size : 0.02
Environment step-size : 0.02
2025-02-03 22:39:36 [16,812ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2025-02-03 22:39:36 [16,812ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2025-02-03 22:39:36 [16,815ms] [Error] [gpu.foundation.plugin] DriverShaderCacheManager::init() called without a shutdown()!

|---------------------------------------------------------------------------------------------|
| Driver Version: 555.42.02 | Graphics API: Vulkan
|=============================================================================================|
[...]
2025-02-03 22:39:42 [23,244ms] [Warning] [gpu.foundation.plugin] ECC is enabled on physical device 0
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
[...]
P2P=Enabled Latency (P2P Writes) Matrix (us)
[...]
2025-02-03 22:39:50 [30,559ms] [Warning] [gpu.foundation.plugin] CUDA peer-to-peer observed bandwidth: 5.9 GB/s.
2025-02-03 22:39:50 [30,559ms] [Warning] [gpu.foundation.plugin] CUDA peer-to-peer observed latency: 36.9 us.
2025-02-03 22:39:50 [30,559ms] [Warning] [gpu.foundation.plugin] Please verify if observed bandwidth and latency are expected.
2025-02-03 22:39:50 [30,560ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled.
2025-02-03 22:39:50 [30,560ms] [Warning] [gpu.foundation.plugin] Detected IOMMU is enabled. Running CUDA peer-to-peer bandwidth and latency validation.
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
[...]

Output End

If I continue to let this run, I get the error we were discussing here. It would also be nice if there was some way to disable the bandwidth tests above, as it takes a significant amount of time to complete if I'm SSH-ed into a server.