Google DeepMind Unveils D4RT, a Model That Reconstructs 4D Scenes in Seconds

Google DeepMind has introduced D4RT, an AI model that reconstructs a full three-dimensional scene unfolding over time from a single video recording, combining in one system tasks that previously required separate models. The company says the new approach processes footage up to 300 times faster than earlier methods.

One Model Instead of Three

Until now, reconstructing a dynamic scene from video required separate models for point tracking, point cloud reconstruction, and camera pose estimation, run one after another. D4RT combines these three tasks in a single encoder-decoder transformer that processes the video once and then answers arbitrary queries about where pixels sit in space and time.

The encoder builds a compressed representation of the scene's geometry and motion from the entire recording. The decoder queries that representation in parallel, handling thousands of queries at once on modern AI hardware instead of processing them sequentially, as earlier approaches did.

Where is a given pixel from the video located in 3D space at an arbitrary time, as viewed from a chosen camera? - description of D4RT's query mechanism, Google DeepMind blog

What the Model Can Do

D4RT performs three functions at once. It tracks three-dimensional pixel trajectories over time, including moments when an object disappears from the camera's field of view. It reconstructs the full point-cloud structure of the scene without needing to estimate camera parameters separately. And it reconstructs the camera's own trajectory, generating 3D snapshots from different viewpoints.

In tests on the MPI Sintel dataset, the model achieved a 3D reconstruction fidelity score of 1.091, ahead of pi3 (0.861) and SpatialTrackerV2 (0.625). In point tracking on the Aria Digital Twin dataset, D4RT scored 1.904 versus 1.762 for SpatialTrackerV2. In camera pose estimation on the RE10k dataset, the model achieved a Pose AUC of 0.835, also above pi3's score (0.787).

Applications in Robotics and AR

Google DeepMind points to robotics and augmented reality as the main application areas. In robotics, the model is meant to give machines spatial awareness in environments where people and objects move unpredictably. In AR, the goal is on-device geometry understanding with the low latency needed to place digital objects into real space in real time.

The company also describes D4RT as a step toward world models, AI systems meant to understand physical reality by separating camera motion, object motion, and static scene geometry. It's a direction DeepMind is investing in alongside other projects on computer vision and video models as representations of the world.

Why Speed Matters

Previous pipelines combining separate models for tracking, reconstruction, and camera estimation were not only slower but also more prone to errors accumulating across stages. Unifying these tasks in a single transformer eliminates some of those errors, and processing queries simultaneously on AI accelerators delivers speedups measured in orders of magnitude rather than percentages.

For real-time applications like robot navigation or smartphone AR overlays, the difference between ten minutes of processing and five seconds determines whether the technology is even viable outside a lab. The technical paper was published on arXiv, and the project has its own materials page, suggesting DeepMind hopes the research community will quickly adopt the architecture ahead of its CVPR 2026 presentation.