How to create collider mesh for gaussian splats

[fortmeier]

Hi,

I created a .usdz GaussianSplatting scene with 3dgrut. My understanding is that this can be imported into IsaacSim and rendered, but there is no way to use it for collision detection for physics simulation. Is there some way to do that? I don't expect there to be something apart from somehow creating a proxy collider mesh from the gaussians with some additional tooling.

Any thoughts on this?

[PeterL-NV]

A .usdz GaussianSplatting scene can indeed be imported and rendered in Isaac Sim, but by default Gaussian splats are visual-only and cannot directly participate in physics-based collision detection. ¹² There is currently no native workflow where the splatting representation itself functions as a physics collider. However, viable workarounds exist to enable collision detection for such scenes.

Recommended Workflow for Colliders

• Visual-Physics Separation: Use the GaussianSplatting .usdz (or USD) as the visual mesh, and separately provide an approximate collider mesh for physics simulation. This is the approach used for most non-manifold or dense surfaces like point clouds and nerf-based assets.
• Manual Collider Mesh Creation: Either manually model a simplified mesh (using a 3D modeling tool or mesh processing software), or use the following tools to generate proxy geometry:
  • Convex hulls
  • Bounding boxes or spheres
  • Custom mesh (low poly, primitive-based, etc.)
• Assigning Colliders in Isaac Sim:
  • Import your GaussianSplatting asset.
  • Import or create the collider mesh as a separate prim.
  • Use the Physics -> Rigid Body tools in Isaac Sim to add a collider component to the proxy mesh, aligning it spatially with the Gaussian model.
  • Optionally, hide or set the collider mesh to be invisible in the renderer but present in the physics world.

Limitations and References

• The collider mesh should be a rough approximation; highly detailed or non-manifold collision geometry will reduce performance and may not be simulation-compatible.
• There is no feature in current Isaac Sim or Omniverse releases that automatically generates a physically meaningful collider from splatting data alone.
• Research projects (like Re3Sim, SplatSim) use auxiliary mesh or particle approximations for physics tasks involving Gaussian splats.¹

So in summary, use a visually accurate Gaussian Splatting .usdz for rendering and a manually modeled or approximation-based collider mesh alongside it for collisions.

Footnotes

1. https://developer.nvidia.com/blog/building-robotic-mental-models-with-nvidia-warp-and-gaussian-splatting/ ↩ ↩²

2. https://insights.evercoast.ai/integrating-4d-gaussian-splats-into-omniverse-and-isaac-sim-50b66011a393 ↩

[nhewitt99]

Hi there! Just adding that at GTC last week, there was a workshop covering how to use fVDB for creating splats and workflows to create collision meshes from these by fusing a TSDF or estimating depth with DLNR. These can then be imported to Isaac like any other mesh (convert to OBJ then import to USD). There's a really helpful notebook here.