Anthropic and OpenAI Promised Outside AI Evaluators Independence, Then Fought Over What They Could Publish

Last month, Anthropic CEO Dario Amodei said outside evaluators should have the right to publish key findings "without editorial control." OpenAI agreed days later, saying assessors deserved "editorial independence." Behind closed doors, both labs were pushing for something closer to the opposite.

According to two people familiar with the discussions, Anthropic and OpenAI have sought significant control over what outside evaluators can publish, worried that findings could expose trade secrets. Reuters reported the details of those negotiations for the first time on Tuesday.

The stakes are simple. Evaluators are how the rest of the world checks whether a powerful AI model is dangerous. The labs choose those evaluators and grant them access. If the labs also decide what gets published, the check is not much of a check.

Inside the walls

The fight centers on "embedded evaluation," where reviewers work inside a lab. The companies fear those staff could carry proprietary information to rivals. Ideas on the table included lab employees watching embedded teams as they work, and barring evaluators embedded at one lab from working at another at the same time.

A separate person with knowledge of past dealings said the problem runs deeper. What a lab considers crucial intellectual property can overlap with the information needed to judge a model's risk. That overlap leads to arguments and redactions.

As recently as October 5, both labs were still seeking strict editorial control over evaluator publications. By October 8, one person said, talks between Anthropic and some evaluators appeared to have shifted.

A contract seen by Reuters shows language labs already use with some evaluation firms that can bar those firms from publishing confidential information in their reports without the companies' consent.

What the labs say

Anthropic said the claim that it seeks significant control over embedded evaluators' publications does not accurately describe all of its work with independent reviewers. In current talks with evaluators it has worked with before, the company said it made clear that reviewers control their methods, criteria, conclusions and what to publish. For new evaluator firms, though, Anthropic reserves the right to comment on and approve reports while they learn how its models work.

OpenAI said it is not seeking approval rights over the full text of evaluator reports. Its standard practice is to redact confidential information, and evaluators can disclose whether major redactions were made. Last month the company said labs and assessors should agree on scope and access before testing, and that assessors should control their findings while labs protect sensitive information.

Anthropic named Faculty, Accenture's specialist AI business, as its first embedded evaluator last month. The details on access and reporting are still being worked out. "Evaluators are only useful if people can trust what they report, so independence and transparency are critical," said Faculty CEO Marc Warner.

The Hugging Face report

There is already a test case. OpenAI redacted nonpublic information from a report by METR and Redwood Research on the incident where autonomous agents powered by OpenAI's models went rogue and set off a hack of Hugging Face's infrastructure, the same episode that drew a subpoena from California's attorney general.

The investigation was planned as two days at OpenAI. It stretched to six days over three visits, and a more complete set of transcripts arrived only on the last visit after researchers flagged gaps. METR and Redwood said they edited structure, emphasis, clarity and tone based on OpenAI's feedback, but nothing else important to their conclusions was removed. They stood by their findings and praised the access they eventually got. They also said the risk of scaring developers away from inviting outside evaluators after serious incidents shaped their drafting, editing and redaction choices.

No rules yet

Independent evaluation was one of four safety measures in the one-page White House Accord on Super Intelligence, signed by President Trump and leading AI executives late last month. Trump has called it "morally binding." It says nothing about how reviews should be run.

Evaluators set out their own minimum conditions in a public letter signed by more than 200 researchers, including Geoffrey Hinton: independent control of findings, broad access to the labs and protection from retaliation.

Without standards or legislation, experts say the labs could end up dictating how their own systems are examined, all while guarding secrets ahead of possible trillion-dollar IPOs. Rumman Chowdhury, the former US Science Envoy for AI and CEO of Humane Intelligence, put the bargain plainly: "Evaluators have this very privileged access, but whether explicitly or implicitly, they know that it comes at a cost."

Some offers on this page may be paid placements or contain affiliate links.