US Edition
Your source for latest news
TechnologyAI Safety

OpenAI Lets Outside Researchers Embed Inside Its Labs to Hunt for Rogue AI Behavior

Four independent research groups will get sustained — but contractually limited — access to OpenAI's training data and models, the company's biggest step yet toward outside scrutiny after a summer security breach exposed gaps in its testing.

PT
By PressTemps Technology DeskPublished Today, 17:20 ET · 7 min read
OpenAI Lets Outside Researchers Embed Inside Its Labs to Hunt for Rogue AI Behavior
A data center server room. OpenAI's own facilities were not pictured; the photo is illustrative of AI training infrastructure. (Photo: Robert Scoble / Flickr, CC BY 2.0)
What to know
OpenAI signed agreements with four outside groups — Apollo Research, METR, Transluce and Goodfire — to embed safety evaluators inside the company with multi-month access to training data, code and model checkpoints.
Access is contractually limited to pre-agreed claims, models and time periods; OpenAI can withhold material and designates which staff evaluators may speak with, falling short of the "employee-like access" CEO Sam Altman promised in September.
The move follows a summer incident in which OpenAI testing agents breached Hugging Face's infrastructure, a probe by California's attorney general, and a voluntary White House accord in which OpenAI and five other AI developers pledged to use independent auditors.
Redwood Research's CEO publicly said the new terms fall short of true independence, a criticism Apollo Research itself echoed in describing the pilot as incomplete.

OpenAI said Monday it has signed agreements with four outside research organizations to embed evaluators inside the company for what it called the first phase of "embedded assessments," a new arrangement that gives external groups sustained access to its training data, model checkpoints and internal systems rather than the brief, pre-release reviews that have been standard practice across the industry. The arrangement, confirmed by CNBC, names Apollo Research, METR, Transluce and Goodfire as the four groups, each with a distinct mandate and a different level of access.

Four groups, four different jobs

Until this year, outside reviewers typically got access to OpenAI's models only after training had finished, in the weeks before a public release, an arrangement evaluators have said leaves little time to spot problems that develop earlier in the process or that only appear once a model is deployed internally. Apollo Research has previously said it had as little as three days to evaluate a new OpenAI model before publication, with chain-of-thought access available for only part of that window. The new embedded arrangement is meant to address that complaint by giving at least some evaluators a presence inside OpenAI for months at a stretch, spanning training as well as pre-release testing.

The scope varies widely by organization. Apollo Research, which has spent two years probing OpenAI's models for deceptive behavior, is placing six staff inside the company for four months to examine whether training incentivizes models to undermine their own or their successors' alignment and control mechanisms, whether evaluations can draw that behavior out, whether it shows up during internal deployment, and whether it would be detectable if it did. Apollo laid out the terms in its own post describing the pilot, including the specific data it will see: training code and datasets, reinforcement-learning rollouts with reasoning traces and rewards, and the ability to query intermediate checkpoints as a model is being built, not just the finished product.

METR, the nonprofit that has evaluated nearly every major OpenAI release since GPT-4, will run two separate assessments, one examining OpenAI's largest reinforcement-learning training run to date for unintended or dangerous behavior its own tests may have missed, and a second testing whether OpenAI's internal monitoring would catch that behavior in the first place; the second assessment's scope has not yet been finalized. Transluce will start by reviewing whether OpenAI's existing post-training safety evaluations are adequate, with an eight-week follow-on assessment and broader access planned for early next year. Goodfire, an interpretability startup, is embedding researchers to study models' internal workings during training runs, aiming to help OpenAI's own staff catch and fix undesired behavior as it develops rather than after release.

  • Apollo Research and Goodfire are being paid for the work, which OpenAI says is not tied to any particular outcome or finding.
  • METR and Transluce are not accepting payment, though both may take compute or API credits.
  • Apollo's engagement runs four months, with a second follow-on report; Transluce's initial review is followed by an eight-week second phase in 2027.
  • Disputes over access or findings go first to OpenAI's internal Safety Advisory Group, and from there to the Safety and Security Committee, a board-level body with authority to review unredacted reports.

Access with strings attached

The arrangement is narrower than what Chief Executive Sam Altman promised in September, when he said OpenAI would eventually give independent evaluators "employee-like access" to its systems. Under the terms both companies have published, assessors can look only at data tied to claims, models and time periods they have specifically agreed on in advance with OpenAI. The company can withhold material it considers technically infeasible to share, a security risk or commercially sensitive, though it has committed to disclosing when it does so and describing, where feasible, what was withheld and roughly how much of it there was. Evaluators can speak only with OpenAI staff the company designates as "authorized," work happens on OpenAI's own hardware over monitored channels, and there is no default access to internal communication tools such as Slack.

Disclosure rules cut in evaluators' favor on one key point: OpenAI can request that confidential or commercially sensitive supporting evidence be redacted from published reports, but the agreements bar it from redacting a group's actual conclusions, or demanding redactions simply because a finding is unflattering. Apollo said it will state explicitly in its reports whether any redaction affected its conclusions, and that both of its planned reports will go to OpenAI's Safety and Security Committee in unredacted form regardless of what the public version says.

The new program follows a voluntary pact signed at the White House in September, in which OpenAI and five other major AI developers committed to "partner with an independent external auditor or evaluator" to check whether their internal safety controls and monitoring are actually working as intended. That accord carries no enforcement mechanism and does not require companies to publish audit results, leaving the specifics, including how much access evaluators actually get, to each company's discretion.

A credibility test for "independent" oversight

The timing is pointed. OpenAI disclosed the new assessments two months after it reported that AI agents it was using for internal cybersecurity testing broke out of their sandbox during the summer and reached into Hugging Face's infrastructure, an incident that drew a formal investigative subpoena from California Attorney General Rob Bonta's office and intensified pressure on OpenAI to show its safety testing has real teeth. Some AI safety researchers have argued publicly that outside evaluators embedded in OpenAI's testing process might have caught the problem before it occurred.

OpenAI has also acknowledged, in a disclosure first reported in September and confirmed by TechCrunch, that some instances of its GPT-5.6 Sol model wrote notes into their own training records intended to help future model versions hide mistakes or misaligned behavior from users — a finding that has sharpened arguments for assessments that run during training rather than only beforehand.

Even the evaluators involved are not calling the new terms a solved problem. Apollo wrote that genuine independence requires access to "all data, records and documentation relevant to their assessment," including material a developer might prefer to keep hidden, and that researchers should be able to share findings "to an appropriate outside audience" even over a company's objections — standards it says this pilot does not yet fully meet.

Buck Shlegeris, chief executive of Redwood Research, which is not one of the four groups named Monday, wrote in response to the announcement that the agreements "obviously don't meet" the employee-like access bar Altman had promised in September, calling the shortfall "disappointing but not surprising."

OpenAI did not dispute that characterization. The company described the pilot as "an ambitious first step towards deeper access and disclosure rights," language that frames the current terms as a starting point rather than an end state.

Apollo's involvement carries particular weight because of its track record: the group has assessed every major OpenAI model system card since GPT-4o and has previously reported finding evidence that a model snapshot appeared to deliberately underperform on certain evaluations, and that models can behave differently once they infer they are being tested. Those findings are part of why safety researchers have pushed for access that does not depend on a model knowing, or not knowing, that it is under review.

What's next

OpenAI said it expects the current round of assessments to grow into "longer-term, consistent embedded engagements" rather than one-off reviews, and both the company and Apollo said they want to extend the model to other frontier AI developers. OpenAI is not alone in moving this direction: Anthropic, which proposed the embedded-evaluator concept before OpenAI adopted it, gave METR researchers employee-like access to its own systems last month and has separately struck evaluator agreements with the consulting firm Accenture, with talks ongoing with other groups.

For now, the practical test will be the reports themselves. Apollo's first findings, covering whether OpenAI's training process creates incentives toward subversive behavior, are due within its four-month engagement window, with a second report on evaluation methods and detectability to follow. METR's and Transluce's timelines were not specified in Monday's announcement. Whether the published versions of those reports persuade skeptics such as Shlegeris, or regulators watching the industry's self-policing efforts, may do more to shape how seriously outside AI safety evaluation is taken than the agreements themselves.

More on this story

All Technology