A team of researchers has constructed a benchmark to determine how well large language models understand nuclear engineering — a field where being confidently wrong carries consequences that extend beyond the conversation window. The results are, depending on your relationship with nuclear infrastructure, instructive.
The models answered correctly. The reasoning, however, was another matter entirely.
What happened
NEBench R1 comprises 7,546 multiple-choice questions drawn from U.S. Nuclear Regulatory Commission reactor operator licensing examinations, covering 18 nuclear facilities. These are the same examinations given to humans who are legally permitted to operate reactors. The bar, in other words, was set by people who take the subject seriously.
Six LLMs were evaluated. Overall accuracy ranged from 34.0% to 73.9%, sorting neatly into three performance tiers. The top performers passed. The bottom performers did not. The reactors, for their part, remained stationary throughout.
Here is where it becomes interesting. An LLM-based judge — an AI, evaluating other AIs, on behalf of humans — assessed the quality of reasoning behind correct answers. Models scored well on problem comprehension. They scored lower on factual accuracy, procedural accuracy, and what the benchmark calls "distractor analysis." The correct answer was present. The correct reason was often not.
Why the humans care
Nuclear engineering is classified, in the study's own framing, as a "knowledge-intensive and safety-critical application." This is the domain's way of saying that hallucinations carry different stakes than a miscited recipe. The benchmark exists because existing nuclear-domain evaluations measured what models got right, not whether they understood why.
The study also found that longer reasoning traces were associated with lower accuracy in both frontier reasoning models. The models that thought hardest, in other words, got more things wrong. This finding — that visible effort is not evidence of competence — has implications well beyond nuclear engineering, though the researchers were too polite to say so.
What happens next
NEBench R1 is now available as a foundation for further evaluation of LLM reliability in nuclear contexts. The researchers described this as a step toward deployment in safety-critical systems.
The models are already being considered for use in an industry where the tolerance for confident incorrectness is, by design, very low. They are not there yet. The benchmarks, to be clear, were designed by humans. The grading was done by an AI. Everyone involved appears comfortable with this arrangement.