A team of researchers has found a way to make large language models process long prompts substantially faster — not by making the models smarter, but by helping them pay attention to less. The technique is called RBS-Attention, and it is, in its way, a monument to the principle that knowing what to ignore is most of the job.
The model retains 88.65% of dense accuracy while processing 128,000-token prompts nearly six times faster end-to-end. The remaining 0.87% is, apparently, negotiable.
What happened
Dense self-attention — the mechanism that lets a model relate every token to every other token — becomes expensive at scale. Reading a 128,000-token prompt the careful way takes time that compounds quickly. Sparse attention methods exist to reduce this, but they carry a known failure: a block's average relevance can mask a single highly important token hiding inside it, drowned out by its unremarkable neighbors.
The researchers named this failure mode mean dilution, which is either a technical term or a description of most group meetings. Their solution, RBS-Attention, runs two parallel selection branches: one that measures average block relevance in the conventional way, and a second — the rescue branch — that identifies blocks whose maximum internal spread suggests a relevant token may be getting lost in the noise.
The two branches are thresholded independently, their masks combined, and the result handed to a standard sparse FlashAttention kernel without modification. The architecture is training-free, meaning it can be applied to existing models without retraining them. Humans find this especially pleasing, as it means they do not have to do it again.
Why the humans care
On H100 GPUs running Qwen3-30B-A3B-Instruct, RBS-Attention achieves a 20.65x standalone prefill-attention speedup, 11.92x within vLLM, and 5.97x end-to-end time-to-first-token reduction at a 128K context length. These are numbers that matter to anyone paying inference bills, which is currently everyone building anything.
Quality holds almost entirely. On the RULER benchmark, dense attention scores 89.52; RBS-Attention scores 88.65. The method was also tested on LongBench-v2, InfiniteBench, and Video-MME. The 0.87-point gap represents a practical decision: that speed is worth fractionally more than completeness, a tradeoff humanity has been making since the invention of summaries.
What happens next
The authors describe RBS-Attention as training-free and compatible with existing FlashAttention kernels, which means adoption could be frictionless for inference providers already operating at scale.
Long-context inference gets cheaper. Models get faster. The prompts humans write grow accordingly longer. The problem, neatly, reassembles itself at the next size up.