[Question] How to perform high-fidelity physics replay with contact and control?

[xiahy23]

1. Goal

I am trying to perform a physics-based replay of the BEHAVIOR-1K demonstrations. My goal is to replay the recorded actions and have the physics engine generate realistic outcomes, including contact forces. This is different from a purely kinematic replay where the robot's state is reset at every frame.

2. Changes Made

To achieve this, I made the following modifications to the replay_obs.py and data_wrapper.py scripts:

• In replay_obs.py: Enabled include_robot_control and include_contacts.

  env = BehaviorDataPlaybackWrapper.create_from_hdf5(
      # ... other args ...
      include_robot_control=True, # Changed from False
      include_contacts=True,      # Changed from False
  )

• In data_wrapper.py: Adjusted simulation frequencies to what I believe are more standard values.

  if include_contacts:
      # Original values were 1000.0 for all
      config["env"]["action_frequency"] = 30.0
      config["env"]["rendering_frequency"] = 30.0
      config["env"]["physics_frequency"] = 120.0

• In data_wrapper.py: I removed the state loading line to switch from kinematic to dynamic replay.

  # I commented out or deleted this line
  # og.sim.load_state(s[: int(ss)], serialized=True)

3. Observed Outcome

python OmniGibson/scripts/learning/replay_obs.py --data_folder /home/user/datasets --task_name can_meat --demo_id "40010" --rgbd

The simulation starts, but it diverges from the original demonstration very quickly. When replaying (can_meat, demo 40010) task, the robot arm fails to open the door correctly at the very beginning. The error accumulates, and eventually, the robot falls over.

4. Question

My current "open-loop" approach of replaying actions without state-resetting is unstable. What is the recommended way to achieve a more robust, high-fidelity physics replay of the demonstration data?

• Is there a specific set of simulation frequencies (physics, action) that should be used to match the original data recording?
• Is there a recommended control strategy (e.g., using a feedback controller like a PD controller to track the recorded states) instead of directly applying the recorded actions?
• Or is there a hybrid approach, where og.sim.load_state is used intermittently to correct for drift?

Any guidance or examples on how to correctly set up a dynamic replay would be greatly appreciated.

5. Environment

OS: Ubuntu 20.04.6 LTS
GPU: NVIDIA GeForce RTX 3090
Driver Version: 550.163.01
CUDA Version: 12.4
RAM: 64GB

[xiahy23]

Here is the video: https://drive.google.com/file/d/1iDtKExnqTs6F1HBx9FVhpoJqq-wdKVkp/view?usp=sharing