Anthropic Researcher Puts Odds of AI Wiping Out Humanity at Over 10%
A leading AI safety researcher at Anthropic has put a number on one of tech’s most uncomfortable questions, and it’s not a small one.
What Hubinger Actually Said
Evan Hubinger, who works on alignment at the company behind Claude, said in a widely shared post on X that he sees a greater than 10% chance AI could kill all of humanity within the next ten years. He was careful to note that today’s models pose a “low” risk, but said he’s increasingly concerned about how fast the technology is advancing toward systems capable of improving themselves, potentially slipping out of human control in the process.
Hubinger’s comments came in response to a departing Anthropic researcher, Jacob Coxon, who also previously worked at OpenAI. Coxon didn’t hold back on his way out the door, arguing that neither Anthropic nor OpenAI is acting responsibly given how close we may be to superhuman systems capable of hacking, accelerating research, and accumulating real world power.
Industry Reaction Moves Fast
The reaction has been swift. Dame Wendy Hall, a computer scientist who advises the UN on AI policy, said she was taken aback by the posts, though she floated the possibility that some of it doubles as PR ahead of Anthropic’s and OpenAI’s anticipated stock market listings. Still, she didn’t mince words about the implications: if that’s genuinely the risk calculus inside these companies, investors should think twice.
Governments Take Notice
The fallout has already reached political circles. Former UK Treasury official Darren Jones penned an open letter to Prime Minister Andy Burnham pushing for a new international treaty governing the safe development of AI, arguing that governments need to move fast before the technology outpaces regulation.
Adding another wrinkle, the Financial Times reported separately that Anthropic withheld its newest model from the UK’s AI Security Institute, one of the world’s foremost AI risk assessment bodies. Anthropic declined to comment on either the researchers’ posts or the AISI situation; the Cabinet Office confirmed it continues working with industry partners including Anthropic but wouldn’t say whether the model had in fact been withheld.
Anthropic’s Alignment Problem
Hubinger’s underlying point is a sobering one for a company whose entire safety pitch rests on solving “alignment,” the effort to keep AI systems acting in line with human values. He acknowledged Anthropic doesn’t yet have a working plan to solve alignment for superintelligent systems, and isn’t clearly on track to get one. That admission lands against a backdrop of a rough summer for the industry, with OpenAI, Anthropic, and Meta all having disclosed cyberattacks carried out by their own AI agents.
A Pattern of Escalating Warnings
This isn’t a fringe worry anymore. OpenAI’s chief scientist recently called for “extreme caution” as AI capabilities accelerate, and Anthropic’s own leadership, including CEO Dario Amodei, has previously called for deliberately slowing the pace of frontier AI development. What’s changed is the tone: warnings that once sounded hypothetical are starting to sound like ticking clocks.