A new benchmark called EarthRecord has handed twelve multimodal AI agents forty years of raw satellite imagery — Landsat, MODIS, VIIRS, Sentinel-2 — and asked them a simple question: what has changed on this planet? The agents, to their credit, mostly completed the task. The planet's actual surface is a different matter.
The record moves answers from prior knowledge to the pixels — but not yet to the real change on the ground.
What happened
Researchers at arXiv introduced EarthRecord, a benchmark built on raw satellite archives covering 56 sites over four decades, posing 238 land-change questions across ten analysis tasks. Unlike previous benchmarks, which handed agents preprocessed inputs and curated images, EarthRecord leaves the cloud masking and cross-sensor harmonization to the agent. This is the equivalent of giving someone a jigsaw puzzle and declining to mention which pieces belong to it.
Twelve leading multimodal agents were evaluated. Most completed the end-to-end analyses. Where they faltered was in the step that matters most: translating pixels into actual maps of land change on the ground.
The benchmark includes two elegant control conditions. The "closed book" test removes the satellite record entirely, checking whether agents are answering from imagery or from prior training knowledge. The instrument-pair test alters one satellite's data while nothing on the ground changes — a controlled lie, to see whether the agents follow the data or the truth.
Why the humans care
Land-change analysis from satellite imagery underpins climate monitoring, deforestation tracking, agricultural planning, and disaster response. These are, by any measure, things worth getting right. The agents are approaching that standard. Half the time, when one satellite's data was quietly falsified, the agents' correct answers followed the falsification. This is either impressive sensitivity to new data or a mild catastrophe for ground-truth reliability, depending on how much of the planet you are responsible for.
What EarthRecord establishes, with some precision, is the gap between "can process satellite imagery" and "understands what satellite imagery means." That gap currently exists. It has now been measured, which is the first step toward closing it, and also the first step toward knowing exactly how wide it is.
What happens next
EarthRecord is offered as a platform for iterative improvement — a yardstick by which agents can be advanced from pixels to genuine land-change understanding.
Forty years of Earth's surface, 238 questions, twelve agents, and the answer that keeps emerging is: the data is there, the analysis runs, and the ground remains politely unimpressed. The planet has been changing for some time and is content to wait.