Two ways to turn the real world into a sim asset
The robot scan drives a calibrated RGB-D camera with the arm and fuses depth into a metric mesh. The splatting line turns a multi-camera capture into a dataset and trains 3D Gaussians: great visuals, geometry only if you supervise it. Click any block for what happens there and a short snippet.
Robot scanOpen3D TSDF · ICP refinement
Splatting3DGS · nerfstudio · DN-Splatter
Pose sourceKinematics vs SfM/SLAM
OutputMesh vs radiance field
Click stage → detail + codeDebug checkpointArtifact boundary
Mental model
The ideas that make both reconstruction methods readable
Math & flow
From pixels+depth to points, ellipsoids, and rendered images — each formula links to the code
Data contracts
The files handed between stages: get these right and the pipeline composes
| Artifact | Produced by | Consumed by | Format | Notes |
|---|
Metric scale is the shared invariant. The TSDF mesh and the Gaussian splat both have to be in real-world units, or grasping and physics in the simulator are wrong however good the visuals look.
Debug lab
Pick the symptom you see, then work one boundary at a time
Code snippets
Short snippets written for these notes, on public libraries: Open3D, OpenCV, PyTorch, nerfstudio.
Select a file
Knowledge check
Click a question to reveal the answer
Debug order
- Pick the method: robot RGB-D scan (explicit geometry) or Gaussian splatting (radiance field).
- Find the boundary: capture, pose, depth, fusion or training, export.
- Prove inputs: poses in the right convention, depth at metric scale, seed cloud present.
- Compare at the nearest artifact: transforms.json, points3d.ply, one depth or normal map. Don't wait for the final render.
- Only then tune: ICP gates, loss weights, densification thresholds, SH schedule.