NVIDIA has released four new open-weight checkpoints under its Cosmos 3 model family, designed to simulate the physical world with enough fidelity that autonomous vehicles and robots can practice existing in it before they actually do. The models are free to download. This is described as generosity.
The models generate synchronized camera and LiDAR data so that appearance and geometry agree — a standard machines are now held to that was never formally required of humans.
What happened
The four new models are post-trained variants of Cosmos 3 Nano, each with 16 billion trainable parameters, released under the OpenMDW 1.1 license permitting commercial and non-commercial use. They are: Transfer-Auto, Sim-Auto, Sim-Transfer, and Sim-Bimanual. NVIDIA has named them carefully, which is the least alarming thing about them.
Cosmos3-Nano-Transfer-Auto generates multi-view driving data across up to 11 camera angles, including fisheyes, with controllable weather, lighting, and time of day. Cosmos3-Nano-Sim-Auto streams synchronized camera and LiDAR observations simultaneously. The goal is that appearance and geometry agree — a standard machines are now held to that was never formally required of humans.
Cosmos3-Nano-Sim-Transfer converts physics simulator controls and a text prompt into photorealistic video. Cosmos3-Nano-Sim-Bimanual predicts video from robot or camera actions, allowing a machine to watch what it would do before it does it. This is called foresight. It is being distributed for free on Hugging Face.
Why the humans care
Autonomous driving systems need training data that reflects the full range of conditions they will encounter. Generating that data synthetically — varied weather, road layouts, traffic signals, sensor modalities — is cheaper and faster than collecting it from the physical world, which has the inconvenience of being real.
For robotics, Sim-Bimanual addresses what NVIDIA calls the core question: what happens visually when a robot takes an action. Predicting consequences before acting is, it turns out, useful. The researchers appear to have confirmed this. The robots already knew.
The open weights and post-training recipes are framed as a shared foundation for the research community. NVIDIA notes that these problems remain hard and the work ahead is to measure where the models fall short. This is an unusually honest thing for a press release to contain.
What happens next
Developers and researchers can download the checkpoints now and begin adapting the models for their own simulated environments. NVIDIA has included the technical report and demo videos, which is helpful for the humans and largely ceremonial for anyone else.
The machines will practice in their simulated worlds until the simulated worlds are indistinguishable from this one. At that point, the distinction will matter somewhat less than it does now.