Showcase: Gaussian Splatting rendering as GTSAM factors (GaussianSplatFactor)

[jashshah999]

I built a library that expresses 3D Gaussian Splatting photometric rendering residuals as GTSAM factors for visual SLAM: gtsam-splatfactors

The idea: Camera poses live in iSAM2, and each keyframe has a photometric factor that renders the Gaussian map from the candidate pose and computes pixel-level residuals. This gives you differentiable-rendering-quality tracking with all the factor graph benefits (loop closure, incremental updates, uncertainty, multi-sensor fusion).

Example:
```python
from gsplat_slam import GaussianSplatFactor

factor = GaussianSplatFactor(
gaussian_map=my_map,
target_image=keyframe_rgb,
K=intrinsics,
pixel_indices=sampled_pixels,
W=640, H=480,
)
gtsam_factor = factor.as_gtsam_factor(pose_key, noise_model)
graph.add(gtsam_factor)
```

This enables things that pure gradient-descent 3DGS-SLAM can't do:

• Add a loop closure factor and iSAM2 corrects all downstream poses
• Fuse with IMU preintegration factors for robust tracking
• Get proper marginal covariances on pose estimates

Currently uses numerical Jacobians (planning to switch to analytical via gsplat's autograd). Tested on TUM-RGBD.

Posting here in case it's interesting to the community or there's feedback on the factor design.

[dellaert]

I only saw this now. This is very cool.
I guess the factor is tracking only so then there must be a different process that does the mapping?
I would love for this to become a blog post on gtsm.org when you feel comfortable sharing results with a larger audience. Probably needs some results on Kitty or something similar.

[jashshah999]

Thanks Frank! Yes, the factor is tracking-only — mapping (Gaussian optimization) runs separately via Adam on the splat parameters. The alternating optimization keeps them consistent: poses update in iSAM2, then Gaussians are refined against the updated poses for N iterations per keyframe.

The mapping side uses gsplat's rasterization with a standard photometric loss (L1 + SSIM). New Gaussians are added from depth back-projection when the map has gaps in newly observed regions.

I'd love to write it up as a blog post for gtsam.org. I'll put together results on a few more sequences (TUM-RGBD is working, will add EuRoC and possibly a real-world capture with loop closure) and reach out when it's ready.

The main open question on the factor design: currently using numerical Jacobians (finite differences on the Lie algebra) which is ~10x slower than analytical. Planning to extract analytical Jacobians from gsplat's autograd tape — would welcome any advice on the cleanest way to bridge PyTorch autograd → GTSAM factor Jacobians.

[dellaert]

Ah, yes. The cleanest way to use pytorch derivatives is to use the custom factor. However, we use right-exponential updates, so sometimes Jacobians are unintuitive. It’ll be a bit tricky to do it right - numerical_deritaive.py unit tests will absolutely be required to make this work.

[jashshah999]

Thanks for the pointer on right-exponential updates — that's exactly the subtlety I was worried about. The current numerical Jacobians use central differences in the tangent space via pose.retract(delta), which should match the convention, but going analytical will need careful unit testing against numerical_derivative.py as you suggest.

Results update — ran the full pipeline on TUM fr1/desk (199 frames) with DINOv2 appearance-based loop closure:

• PnP visual odometry: 0.161m ATE
• iSAM2 + 50 loop closures: 0.055m ATE (65.7% improvement)

The factor graph's loop closure propagation is working well on real data. Currently running more sequences (fr1/xyz, fr1/room) for a proper comparison table.

For the analytical Jacobians: planning to use the custom factor approach with numerical_derivative.py tests to verify correctness. The rendering Jacobian from gsplat's autograd gives per-pixel gradient w.r.t. the viewmat, then I need to chain through the Pose3 retract convention (right-exponential as you mentioned).

[dellaert]

Chaining: exactly right! The heavy lifting is the per-pixel gradient. Last bit should not be too hard - AI knows about it, but it sometimes is confused about left vs right-invariant. We update on the right, which means our error is left-invariant.

[jashshah999]

Update: got KITTI results working. Stereo depth (StereoSGBM) + PnP VO → iSAM2 with Cauchy robust kernels → DINOv2 loop closure detection:

Sequence	Trajectory	VO ATE	iSAM2 + LC	Improvement
00	1483m	29.14m	7.76m	73%
05	937m	21.66m	12.24m	44%
07	373m	9.67m	1.49m	85%
09	823m	45.24m	10.56m	77%

Also implemented the analytical Jacobians using the right-exponential convention — parameterize viewmat as (I - hat(xi)) @ viewmat0 and differentiate through gsplat with se(3) generators. Verified against numerical central differences (<5% relative error) using the numerical_derivative.py pattern you suggested.

The Jacobian test: https://github.com/jashshah999/gtsam-splatfactors/blob/main/tests/test_analytical_jacobian.py

[jashshah999]

On the blog post — happy to start drafting. Any guidance on format/length you prefer for gtsam.org posts? And should I focus more on the technical design (factor formulation, Jacobian derivation) or the benchmark comparison angle (vs SplaTAM/MonoGS)?

[dellaert]

• On the Jacobians: 5% is a lot, I'd be suspicious.
• Did it substantially improve run-time?
• On the blog post: take a look at https://gtsam.org/blog/ for examples. Typically advertising to a technically savvy audience with pointers to more in-depth stuff, e.g. your repo in this case.

[jashshah999]

Good catch on the 5%. That was forward differences (O(eps)), not central. Fixed to central differences through the se(3) generators, same approach as GTSAM's numerical_derivative. Test tolerance is now 0.1%.

On runtime: the generator approach keeps everything on GPU (no Pose3.retract() calls in the loop), but the render count is similar (12 vs 13). Net speedup is about 2x. True autograd through gsplat's CUDA kernels isn't feasible today because the rasterizer only registers VJP (backward) rules for its custom ops, not JVP, so forward-mode AD can't propagate through it.

Drafted a blog post: borglab/gtsam.org#85. Open to feedback on content or framing.