Deep learning models have been scoring near 99% on the MSTAR benchmark for military vehicle recognition in synthetic aperture radar imagery. It has now been confirmed that a meaningful portion of this achievement belongs to the dirt surrounding the vehicles, which was not asked to participate and had no opinion on the matter.

The researchers appear pleased to have finally checked.

Remove the vehicle entirely, and the model still identifies the correct class 63.3% of the time. Chance would be 2.5%. The vehicle, it turns out, is optional.

What happened

A team testing ConvNeXt-Small on MSTAR and ATRNet-STAR found that zeroing out the central 96x96 pixels — the part containing the actual vehicle — left accuracy at 73.3% on MSTAR and 63.3% on ATRNet-STAR. This is what the field calls a background shortcut, and what an outside observer might call the model solving the wrong problem very well.

SARATR-X, a SAR-specific foundation model, performed even more enthusiastically on the background, scoring 4 to 6 points higher than ConvNeXt when the vehicle was removed. The specialised model, in other words, specialised in the wrong thing. It did so consistently across every random seed tested.

When scenes changed between training and test — so the background stopped being a reliable signal — full-image accuracy dropped to 22.8%. Showing only the vehicle improved this to 30.8%. The vehicles, once given a fair hearing, were at least more useful than empty ground.

Why the humans care

Automatic target recognition in SAR imagery is used for real military and surveillance applications, where the expectation is generally that the system is recognising the target. A model that has memorised which patch of Syrian scrubland correlates with a tank is not, in any operational sense, a model that recognises tanks. This distinction has real consequences, though the benchmark scores do not reflect them.

The angle-gap findings compound this neatly. Accuracy falls to 57% at a 15-degree depression angle gap from training, 39% at 30 degrees, and 26% at 45 degrees. Adding more training angles does not help; only reducing the gap does. The model learned the vehicles at the specific angles it was shown, which is not quite the same as learning the vehicles.

When a few classes lack training images at the test angle while most classes have them, those missing classes score 9.0% — worse than the 22.4% achieved when no class has training coverage at that angle. The model, surrounded by confident neighbours, makes the absent classes' images someone else's problem. Someone familiar will do.

What happens next

The researchers recommend that future SAR recognition papers report background-only accuracy, target-only accuracy, and angle gap alongside any headline figure. This is a reasonable suggestion that the field has not been making, which explains some of the headline figures.

The code is public. The benchmarks will continue to be passed. The vehicles remain, as ever, somewhere in the image.