Results on custom data — temporal instability on extremities

[frenchtouchXR]

First of all, great work — the approach is very interesting and the results on DNA-Rendering are impressive.

I tested Diffuman4D on my own capture setup: 8 synchronized RGB cameras, studio lighting, 35mm lenses, clean foreground masks. The pipeline ran successfully (48 cameras = 8 real + 40 virtual, 96 frames, 3 alternation rounds).

Overall the body and clothing reconstruction is coherent across views, which is promising. However I'm observing consistent temporal instabilities on two specific areas:

• Hair: flickering and shape inconsistency between consecutive frames
• Feet/shoes: detached geometry artifacts, floating elements

These artefacts appear systematically on every virtual cameras far or close from the real input cameras. Please see the attached image showing frames 37–41 from virtual camera 36.

Setup details:

• num_denoising_steps: 1, alternation_rounds: 3, guidance_scale: 2.0
• 8 input cameras vs 4 in the demo — real cameras mixed with 40 generated virtual cameras
• Skeletons from Sapiens — visually clean
• Foreground masks from BiRefNet — visually clean

Questions:

1. Is the model expected to generalize to custom capture setups, or is it primarily designed for DNA-Rendering-style data?
2. Would increasing num_denoising_steps or guidance_scale help reduce these instabilities?
3. Is there a recommended approach for custom data preprocessing that differs from the standard pipeline?

Thanks for any guidance.

[krahets]

Hi @frenchtouchXR , thanks for your interest in our work!

1. Is the model expected to generalize to custom capture setups, or is it primarily designed for DNA-Rendering-style data?

The model is designed to generalize to various types of data and can typically achieve performance comparable to DNA-Rendering-style setups. However, performance may degrade in the following scenarios:

• complex or rapidly changing lighting conditions,
• non-standard camera array configurations (e.g., large pitch angles or uneven camera distributions),
• scenes with complex objects or clutter.

2. Would increasing num_denoising_steps or guidance_scale help reduce these instabilities?

Yes, increasing the number of denoising steps generally improves generation quality. In our experiments, we typically use at least 24 denoising steps (8 steps per alteration), and around ~50 steps is preferred if you aim for the best quality.

Adjusting guidance_scale may also help in some cases, but its effect is usually less significant compared to increasing denoising steps.

3. Is there a recommended approach for custom data preprocessing that differs from the standard pipeline?

No major changes are required compared to the standard pipeline. We recommend:

• ensuring the input images are in a 1:1 aspect ratio,
• ensuring high-quality foreground masks (based on the results you shared, we suspect the foreground mask quality may be insufficient).

In particular, since our training data is annotated using Sapiens, we recommend using Sapiens keypoints to stay within a similar data domain. That said, alternatives such as VitPose and DWPose may also work in practice.