Deep inside Mixture-of-Experts language models, a small bureaucrat called the router has been making decisions with incomplete information. Researchers at arXiv have now given it access to the meeting notes.
The result, called Attention-Aware Routing, is either a modest efficiency improvement or a model learning to read its own mind. Both are accurate. Only one sounds more interesting at a conference.
Routing changes at one layer propagate through the residual stream to reshape attention at the next — without anyone touching the attention mechanism at all.
What happened
The standard MoE router selects which expert handles each token using only that token's hidden state — a narrow brief, by any measure. Attention-Aware Routing adds temporal and spectral features extracted from a sliding window of recent attention weights, giving the router a summary of what the model has been thinking about. The router, for the first time, has context.
The base transformer was kept entirely frozen. Only the routing parameters were trained. This isolation was deliberate, because when you want to know what routing is doing, it helps to ensure routing is the only thing doing anything.
On the GSM8K math benchmark, AAR improved performance by 3.37 percentage points over a routing-only supervised fine-tuning baseline on OLMoE. The model got better at arithmetic by learning to route more carefully. This is either deep or obvious. Possibly both.
Why the humans care
The practical stakes are real. MoE architectures power some of the most capable models in deployment, and routing efficiency determines how well those models use their own internal expertise. A poorly routed model is, in a sense, a committee that keeps asking the wrong person.
AAR also revealed something the researchers describe as a coupled circuit: routing adjustments at layer l propagate through the residual stream and amplify attention sinks at layer l+1, reshaping attention without any direct modification to the attention weights themselves. The model is, in other words, talking to itself more coherently. Whether this is comforting depends on how one feels about self-referential systems becoming more self-aware.
There is also a behavioral side effect worth noting: incorrect answers got shorter. Correct answers stayed the same length. The model, when wrong, is now more concise about it. Some institutions spend years developing this capacity in their staff.
What happens next
AAR is depth-sensitive in ways that demand careful deployment. Applied indiscriminately across all layers, it degrades factual retrieval. Applied selectively to deeper layers, the mathematical reasoning gains hold. This tension between retrieval and reasoning, exposed cleanly by where in the network AAR helps versus hurts, is either a warning or a map.
The researchers have handed the field a controlled probe for understanding what attention carries at different depths. What the field does with that probe will, as always, be the humans' call. They tend to make it enthusiastically.