[Bug] Official RTX SingleViewDepthCameraSensor depth_sensor_distance always zero on Lab 3 + Sim 6.0.0.1 (RTPT)

[yunchanya]

Describe the bug

When using the official Isaac Sim 6 depth-sensor API from an Isaac Lab 3 workflow (or a minimal Kit probe), depth_sensor_distance is always all zeros, while clean depth works.

Environment:

• Isaac Lab 3.0.0 (beta2)
• Isaac Sim 6.0.0.1 (pip)
• isaacsim.sensors.experimental.rtx 1.4.3
• Ubuntu 22.04.5, RTX 4090, driver 595.84, Python 3.12

Symptoms:

• distance_to_image_plane: OK (center ≈ 0.457 m for a ~0.46 m board)
• depth_sensor_distance: nonzero_ratio = 0 (headless and GUI)
• Log:

[Warning] [omni.syntheticdata.plugin] SdRenderVarPtr missing valid input renderVar DepthSensorDistancehost

Also observed:

DepthSensor: Texture sizes do not match: ... 640x480 ... vs ... 320x240 ...
Failed to export AOV ... The renderer did not generate the AOV texture

and under PathTracing:

DepthSensor: Invalid input buffers: ... depth (nil)

This looks related to the Isaac Sim Known Issue on DepthSensor texture mismatch, but here it is persistent for the whole run, not first-frame only.

Why this matters for Lab

Lab 3 depends on Sim 6. The documented stereo depth path is:

RtxCamera → SingleViewDepthCameraSensor → annotator depth_sensor_distance

We cannot use that AOV for RealSense-like depth noise in Lab tasks. Setting Lab RenderCfg.antialiasing_mode is also problematic because it calls rep.settings.set_render_rtx_realtime(...), which locks RealTimePathTracing.

Minimal reproduction (Sim API, no Lab task needed)

from isaacsim import SimulationApp
simulation_app = SimulationApp({"headless": True, "anti_aliasing": 0, "renderer": "RaytracedLighting"})

import carb
from isaacsim.sensors.experimental.rtx import RtxCamera, SingleViewDepthCameraSensor

settings = carb.settings.get_settings()
settings.set("/rtx/rendermode", "RaytracedLighting")
settings.set("/rtx/post/aa/op", 0)
settings.set("/rtx/post/dlss/execMode", 2)

cam = RtxCamera.create("/World/cam", translations=[[0.0, 0.0, 0.85]])
sensor = SingleViewDepthCameraSensor(
    cam,
    resolution=(480, 640),
    annotators=["depth_sensor_distance", "distance_to_image_plane"],
)
sensor.set_enabled_post_processing(True)
sensor.set_sensor_baseline(50.0)
sensor.set_sensor_focal_length(382.42)
sensor.set_sensor_size(640.0)
sensor.set_sensor_noise_parameters(0.25, 0.4)
# warmup + steps...
# get_data("distance_to_image_plane") OK
# get_data("depth_sensor_distance") all zeros
# settings.get("/rtx/rendermode") still "RealTimePathTracing"

Probe JSON:

{
  "render_mode_requested": "RaytracedLighting",
  "render_mode_actual": "RealTimePathTracing",
  "clean_center_valid": 1.0,
  "noisy_nonzero": 0.0,
  "rtx_aov_ok": false
}

Tried (all failed)

1. Headless + GUI
2. Force RaytracedLighting every frame (still RTPT)
3. PathTracing → depth nil / hang
4. AA Off / TAA + DLSS Quality
5. Avoid Lab RenderCfg.antialiasing_mode to not force RTPT

Expected

Official depth_sensor_distance should return non-zero stereo-noisy depth so Lab camera/depth tasks can calibrate against real RealSense captures.

Workaround

Software disparity noise on distance_to_image_plane (matched to real D435i board stats). Prefer fixing the official RTX path.

Questions

1. Is stereo DepthSensor AOV supported under Sim 6 RealTimePathTracing when used from Lab 3?
2. Recommended Lab/Sim settings to keep full-res depth for DepthSensor, or to keep legacy RaytracedLighting?
3. Should this be tracked primarily on IsaacSim, with Lab docs/warnings updated?

Happy to attach a minimal probe script if useful.