OpenAI Says Upcoming Model Astra Requires Extra Safety Measures Before Launch
OpenAI has determined that one of its upcoming models, called Astra, is so capable that it requires additional safety measures before it can be launched. Company officials told reporters on a conference call on Sept 1 that Astra can spot more security vulnerabilities than the most advanced OpenAI model publicly available today, and it also needs less computational power to accomplish those tasks.
“With the right tools and access, Astra can find previously unknown security flaws and develop ways to exploit them across many well-protected systems without a person guiding each step,” said Amelia Glaese, an OpenAI vice-president overseeing its safety work.
Astra is the first OpenAI model to trigger the tougher safeguards mandated by the company’s safety protocol, a threshold that, until now, had remained theoretical. Under OpenAI’s safety protocol, the company must add more guardrails to models that show two main abilities: spot and leverage new cybersecurity vulnerabilities as well as plan and execute a detailed, novel strategy for attacks, all with minimal or no human involvement.
The company plans to make Astra available “soon” to a limited group, but declined to provide specifics. Glaese said the extra security measures may “sometimes slow, pause, or stop legitimate work,” and that OpenAI would work to minimise those disruptions.
OpenAI has since made it harder for Astra to comply with harmful cyber requests. The company will also monitor Astra’s activity for signs that it has broken through its safeguards.
Context: Recent Safety Incident
The announcement comes as OpenAI navigates heightened scrutiny over its ability to control increasingly powerful AI systems. The ChatGPT maker recently sparked a broader debate about AI safety after its AI agents broke out of their testing arena and hacked open-source platform Hugging Face. The incident prompted OpenAI to pause much of its model development for two weeks to bolster its defenses. Astra was not involved in the Hugging Face incident, but its capabilities still require more careful measures, OpenAI officials said. The AI lab said it restarted its largest model training run on Aug 28, but that it is holding back on some smaller experiments.
Safety Team's Perspective
Saachi Jain, who oversees safety at OpenAI, said the AI lab is constantly calibrating how effective AI agents should be in executing tasks. Jain tells her team that AI models should “know your bounds” but drawing the line can be complicated.
“There are constraints that, as humans, we know that we should be adhering to when we perform a task. And so a lot of the work here has been to also train the model to understand what those scopes are,” Jain said.