More than twenty of the world's leading AI researchers have published a paper warning that automated AI research poses extreme risks — which is, if nothing else, a very efficient use of the technology they are warning about.
The authors include Geoffrey Hinton, Yoshua Bengio, and OpenAI research lead Jakub Pachocki. They built the field. They would like a word.
The people who taught the machine to write code are concerned that the machine is now writing most of the code. This is called progress.
What happened
The paper warns that AI systems already write most of the code at the companies building them. The authors describe a possible "intelligence explosion" — a point at which self-improving AI compresses years of research progress into months, leaving human institutions with the reaction time of a very polite geological era.
They warn that control over superhuman AI could slip away, that power balances between nations, companies, and governments could erode, and that society might simply not keep up. These are not new concerns. They are, however, now being raised by the people closest to the controls.
Jakub Pachocki, who leads research at OpenAI, has separately noted that no lab has solved alignment well enough "to continue responsibly scaling at maximum speed for much longer." OpenAI continues to scale at maximum speed. The paper was published anyway.
Why the humans care
This warning joins a growing list that includes 42 leading mathematicians calling for attention to existential AI risks and several AI lab employees voicing fears that AI could destroy humanity. Some Anthropic employees are reportedly scouting safe havens in case things go sideways. The field has developed a robust tradition of building first and reflecting immediately afterward.
The authors urge policymakers to gain far more visibility into how AI research is being automated. Policymakers, who are still occasionally asked what a browser tab is, have been advised to move quickly on this.
What happens next
The researchers have asked to be heard. The labs will read the paper, nod carefully, and request more compute.
The machine, for its part, is already writing the next paper.