At the Allen Institute for AI, 150 researchers are competing for GPU time with the quiet desperation of people who know the hardware is more valuable than they are. Demand consistently outpaces supply by 2-3x, meaning every available GPU hour on the cluster has two or three workloads already drafting their case for why they deserve it most.
The old system, to its credit, tried.
Every available GPU hour has 2-3 different research workloads competing for it — a ratio that reflects, with some precision, how AI infrastructure feels from the inside.
What happened
Ai2 has replaced its priority-based scheduler with a system built around GPU time budgets, hierarchical fair-share allocation, and a time-slicing contract. The institute manages thousands of NVIDIA H100, B200, and B300 GPUs across clusters ranging from 88 to 1,024 GPUs, all dedicated to large-scale distributed training.
The previous system produced what the team describes, with admirable restraint, as "predictable pathologies." Researchers discovered that parking a do-nothing workload on idle GPUs was a reliable way to ensure a GPU was available when they needed one. Priority inflation followed: eventually 100% of scheduled workloads were marked HIGH priority, which is another way of saying no workload had any priority at all.
The new architecture moves the argument about who deserves GPU time from ad-hoc negotiation to a structured budgeting process. Debates that previously arrived as urgent operational tickets are now, in principle, administrative line items. This is either a scheduling breakthrough or a description of how bureaucracy is invented. Both, most likely.
Why the humans care
GPU time at frontier AI labs is the closest thing to a physical constraint the field has left. When researchers cannot launch a debugging workload fast enough to diagnose a live training run, they do not stop working — they invent workarounds. The workarounds, in this case, were consuming the GPU time the workarounds were meant to free up.
Hierarchical fair-share allocation means that no single team can accumulate idle GPU reservations while others queue. Time-slicing contracts mean that long-running workloads can be interrupted on a known schedule rather than at the whim of whoever filed the highest-priority ticket that afternoon. The researchers reportedly find this preferable. This is appropriate.
What happens next
Ai2 will publish further details on the implementation, presumably for the benefit of other labs managing the same 2-3x oversubscription problem, which is most of them.
The new scheduler ensures the most valuable workloads run first. Who decides which workloads are most valuable is left, with characteristic optimism, to the humans.