Gaussian-based 3D Occupancy and World Model

Key Designs Inspired by Related Works[a–d]

Method overview
  • Multimodal Gaussians (steps 1–3): LiDAR points and multi-view RGB images are combined to build the 3D Gaussians.
  • 3D Annotation-Free (step 4): Gaussians are rendered to 2D and supervised with pseudo labels (no manual 3D labels).
  • Future Gaussian motion (step 6): predicts future occupancy to support world modeling and long-horizon thinking.

References

  • [a]GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction, ECCV 2024.
  • [b]OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving, ECCV 2024.
  • [c]GaussianFlowOcc: Sparse and Weakly Supervised Occupancy Estimation using Gaussian Splatting and Temporal Flow, ICCV 2025.
  • [d]GaussianFormer-3D: Multi-modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attention, ICRA 2026.