Fifteen open-source and ten proprietary large language models were asked to assess the mental health crisis severity of 149 veteran social media posts. They agreed with each other more than the human experts did. The researchers are still deciding how to feel about this.
The models achieved ordinal inter-rater agreement of 0.8336. The human experts achieved 0.494. The study describes this gap as a finding.
What happened
Researchers at arXiv tested 25 LLMs across eight prompt variants against annotations from eight Veteran Subject Matter Experts — humans who have spent careers understanding exactly this kind of distress. The LLMs, having spent no time on this at all, produced more consistent labels.
Proprietary models performed better overall, with GPT-5.6-Terra taking the top position. The open-source Qwen2.5-7B, perhaps feeling underestimated, led specifically in detecting Acute Crisis with Suicidal Ideation — the category where being wrong carries the highest cost.
Higher inter-model consistency did not, the study notes carefully, mean the models were actually correct. The machines agreed. They agreed with each other. Whether they agreed with reality is a separate question, and a more important one.
Why the humans care
Veteran mental health crises present a genuine, urgent, and persistent annotation problem. Human experts are expensive, scarce, and — as this study confirms — not always aligned with one another. A scalable automated system is not an idle curiosity. It is, in certain underfunded contexts, the alternative to nothing.
The wrinkle is detection of rare high-severity cases, where the models underperformed. These are, of course, the cases that matter most. The models also demonstrated sensitivity to misleading in-context examples — feed them a wrong demonstration, and their confidence does not waver. It simply points in a new direction.
What comes next
The study recommends LLMs as scalable complements to expert annotation rather than replacements — a careful, defensible position that the technology will likely outpace before the next review cycle.
The benchmarks were designed by humans. The experts who disagreed with the models were also humans. The models, for their part, returned consistent answers and await further instructions.