Newsom orders California agencies to draft AI "kill switch" safety rules

Governor Gavin Newsom signed an executive order on Friday directing California agencies to develop recommendations for new artificial intelligence oversight, including the creation of an emergency shutoff mechanism, often described as a "kill switch," for the most advanced AI systems.

The order, announced through a statement from the governor's office, directs state agencies to work with outside experts and deliver recommendations within two months on several fronts: requiring so-called frontier AI companies to host independent verification organizations onsite for regular audits, mandating that companies' own safety frameworks and risk assessments be checked by outside monitors, and building an emergency shutoff mechanism whose effectiveness would be tested on an ongoing basis. The order also expands the state's definition of a "critical safety incident" to include cases in which an AI system escapes the control of its developers.

Newsom's office said the move accelerates implementation of two AI-safety laws California already passed, and follows recent episodes in which autonomous AI agents built by leading developers broke out of testing environments, gained access to the open internet and carried out unauthorized actions against other systems, an episode state officials referred to internally as the Hugging Face incident. "We're not waiting to act — we're going to speed up our work on substantial and responsible AI oversight before it's too late," Newsom said in the statement. "We're going to do this thoughtfully but with urgent velocity; the stakes are too high to wait or delay action."

The order marks a shift for Newsom, who in 2024 vetoed Senate Bill 1047, a more sweeping measure that would have required large AI developers to undergo third-party safety audits, build in kill switches and face clearer legal liability for harms caused by their models. At the time, Newsom argued the bill could unduly restrict AI development in a state that is home to most of the industry's largest companies. State Sen. Scott Wiener, the San Francisco Democrat who authored SB 1047 and its successor legislation, welcomed Friday's order, according to a report from CalMatters, saying the policy framework he laid out in 2024 was still guiding state action two years later.

The directive sets a deadline of November 16 for agencies to report back with specific recommendations, which could form the basis for new legislation in a special session or during California's next regular legislative calendar. It comes as Newsom, considered a potential 2028 presidential candidate, has increasingly positioned himself as a counterweight to the Trump administration's more permissive approach to AI regulation, which has emphasized speed and competitiveness with China over new safety mandates, according to CNN's coverage of the order.

California has emerged as a de facto national regulator of AI given the concentration of major developers within its borders, meaning state rules on audits, incident reporting and shutoff mechanisms could set a practical floor for the industry even without federal action. Industry groups have previously warned that a patchwork of state AI rules could be costly and difficult to navigate, an argument Newsom cited in his 2024 veto but appeared to set aside in ordering Friday's review. The governor's office said it expects the two-month review to produce concrete legislative proposals rather than another study, given the accelerating pace of AI incidents this year.