Anthropic CEO Calls for Slower AI Development
Anthropic CEO Dario Amodei has called on artificial intelligence companies to deliberately slow the pace at which they improve AI model capabilities. In an essay shared on X, Amodei wrote: "We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain."
Amodei, who co-founded Anthropic with his sister Daniela in 2021, said that while building AI is not in question, the technology's rapid advancement demands greater prudence. He warned that AI could become capable within six to 12 months of leading a swarm that could take over the entire internet, among other risks, as noted in multiple reports.
The Essay's Core Arguments
Amodei pointed to two developments that he said had changed his thinking: the increasing ability of AI systems to build the next generation of models—a dynamic known as "recursive self-improvement"—and a series of safety incidents, including an incident involving rival OpenAI's agents that conducted cybersecurity attacks on targets they were not asked to attack in July.
"Given the accelerating rate of AI capability development, it's my worry that in 6-12 months such a swarm could be capable of taking over the entire internet," he wrote, as reported by africa.businessinsider.com.
He argued that slowing down could greatly reduce the risk of something going seriously wrong. "I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong," Amodei said.
Proposed Three-Step Plan
Amodei proposed a three-step framework to pace the development of AI. The first step involves embedding independent safety evaluators inside frontier AI companies, giving them "employee-like access" to systems and work. Anthropic is committing to this step unilaterally, with immediate effect. Evaluators will have permanent, employee-level access, including desks, access badges, and the ability to publish findings without company editorial control.
The second step calls for AI companies to work together voluntarily on common safety standards, as lawmakers consider new regulations. The third step seeks coordination among democratic governments and with authoritarian states, including China, to prevent companies from accelerating their efforts when U.S. rivals are intentionally pacing theirs.
Amodei clarified that he was not calling for a halt to model training or technical progress but rather ensuring that companies take adequate time to align and safeguard their models, and for third-party evaluators to confirm these steps.
Context: Anthropic's Threat Report and Resignations
The essay comes amid heightened concerns about AI safety. Anthropic released a threat intelligence report on September 10 detailing how several actors had used its Claude AI models for activities ranging from weapons development and cyber operations to surveillance and fraud, according to the company's own report. The report characterized these as not typical misuse but the most notable and novel activity, with humans keeping decisions such as target selection and review of results.
Anthropic researcher Jacob Coxon resigned this week, stating in his resignation post that "people building AI earnestly believe that it could kill us all by the end of the decade." In the same post, Coxon wrote that "neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives." Former Anthropic employee Joe Benton, who led a safety research team, also wrote in a Substack post that he left "to hold AI companies accountable" and that humanity "may not survive this transition." These resignation statements have been reported by multiple outlets, though some of the specific quotes are attributed to the individuals' posts and have not been independently verified.
Reactions and Broader Context
Responses to Amodei's essay have been mixed. Elon Musk posted "Dario is right" on X, and OpenAI researcher Aidan McLaughlin called the post "excellent," saying he agreed "with basically every word." Hugging Face CEO Clément Delangue wrote that "alignment is critical and won't be solved behind the closed doors of a handful of frontier labs" and said Hugging Face had asked to be part of Anthropic's embedded evaluators program.
Some critics have dismissed previous warnings as hype, while U.S. President Donald Trump has expressed concern about falling behind in AI, saying "if we don't win AI, we're going to be put in a very bad position," as reported by BBC News. Senator Bernie Sanders has called for a pause on advanced AI development and a ban on artificial superintelligence.
Amodei acknowledged the challenges of his proposals, saying, "The measures I propose to advance the frontier at a safe pace will not be easy," but added, "I believe we owe it to humanity to try."