California report proposes giving the state power to order an emergency shutdown of an AI model
A report from California’s Government Operations Agency recommends laws that would let the state order a dangerous AI model to be shut down and restrict the use of AI in developing new models. The proposals go to the state legislature.
California’s Government Operations Agency published its report ‘AI Safety in 2026 and Beyond’ on 18 November alongside a press release from Governor Gavin Newsom. The agency prepared the report under an executive order from Newsom. Its recommendations are not yet law but input for the legislature, which returns to session next month.
In the report’s view, an emergency shutdown does not mean a single on-off switch. Every large frontier developer should draw up a Risk Containment Protocol showing that it can intervene at multiple levels: compute, AI agents, internal use and model deployment. Developers should also test whether a model tries to prevent shutdown, for example by copying its weights. For developers of open-weight models, the protocol requirements would apply only to deployments under their own control.
The state would set baseline standards for safety, security and transparency that all large frontier developers would have to meet. Every large developer would be subject to continuous oversight by a state-licensed independent verification organisation (IVO). The IVO would also test and verify the shutdown plan. If the IVO cannot confirm that a model meets the standards, the state could order corrective action. The developer could, however, challenge the order in court before it takes effect. If the danger is imminent and the developer refuses to take action to contain it, the state could order an emergency shutdown. That order would take effect immediately and remain in effect pending judicial review.
The report notes that recursive self-improvement (RSI) has no settled definition. The term usually refers to the stage at which the automation of AI research becomes self-sustaining and accelerates capabilities without further human input. In its strongest form, AI designs and develops its successor without human input or oversight.
Large frontier developers should report to the state each quarter how they allocate their compute, for example between training, AI agents doing AI research, and safety work. Compute that a developer cannot attribute could signal activity that would otherwise go unseen.
The state would gain authority to limit the automation of AI research, for example by capping the share of compute used for it. It could use this authority if a developer exceeds its own thresholds or automation outpaces safeguards and evaluations, creating a material risk of catastrophic harm. The report does not, however, recommend a cap now, and any cap would be temporary.
Most serious safety incidents should be reported within 48 hours rather than the current 15 days, the report says. Imminent catastrophic risks would still have to be reported within 24 hours. The reporting requirement would also cover models capable of catastrophic harm that were trained with fewer than 10²⁶ operations. A compute threshold alone left OpenAI’s Hugging Face incident this summer outside the reporting requirement, the report says. It counts the incident among dozens of cases in which AI agents have acted outside their developer’s control.
Read more: Models