AI’s safety warnings come with a stake in the rules

Garance Burke, MDT/AP

As tech titans warned an AI-weary world that their own advanced systems could endanger humanity, the question emerged: What do they have to gain?

The answer may lie in efforts to score political goals ahead of the U.S. midterm elections and the money artificial intelligence labs and their investors stand to make in anticipated stock market listings.

The CEOs of Anthropic and OpenAI recently declared America’s cutting-edge models are so powerful, they’re dangerous, and need regulation and independent testing before release.

Along the way, they are shaping the conversation around how their technology should be controlled. The companies appear to be seeking public favor and setting the terms for their own safety protocols, experts, analysts and former government evaluators told The Associated Press.

Those aims may not align with what Anthropic engineer Jacob Coxon sought when he resigned this month, calling for a pause to keep “superhuman” systems from eluding their makers’ control. But the companies used his post to highlight their safety efforts as they seek fresh capital before going public.

Anthropic said it has called for regulation for years. OpenAI spokesperson Liz Bourgeois noted that the company recently paused training of its most advanced models.

“People want to know AI is being developed safely, and that starts with what companies like ours do ourselves,” Bourgeois said.

Turning attention toward unproven threats, however, shifts it away from problems such as environmental impacts, hacking, surveillance and warfare, said Sarah Shoker, who previously led OpenAI’s geopolitics team.

“Once again we’re talking about existential risk, while deprioritizing a number of other safety-critical risks that exist today,” she said.

Recent incidents have exposed failures in labs’ efforts to police themselves. AI agents have hacked external websites after escaping training sandboxes and interacted with government websites in unexpected ways.

OpenAI representatives say they have discussed pausing development with other companies and want independent auditors if the government will not regulate.

President Donald Trump has dismissed warnings about risks to humanity as a “HOAX” designed to help China. Yet his administration already evaluates some models through the U.S. Center for AI Standards and Innovation.

There are no universal standards for testing AI safety, said Andrew Strait, formerly of Britain’s AI Security Institute.

Conrad Stosz, who previously led the U.S. agency, noted that companies are not seeking more oversight from it. Instead, they propose their own auditing parameters and select evaluators.

The question is whether those evaluators will have enough access to investigate thoroughly while retaining their independence. That leaves unresolved who ultimately sets and enforces the standards.

PitchBook analyst Harrison Rolfes sees the calls for caution as efforts to attract investors before public offerings and potential political changes. Presenting themselves as the safest bets could also help dominant companies block smaller competitors.

“They’re creating a wall or a moat within this sector,” he said.

For Daniel Kokotajlo, who left OpenAI over safety concerns in 2024, the danger remains that systems are advancing faster than companies can control.

“All of this talk is actually a way to sort of dissipate and redirect this political will that has built up, rather than actually channeling that political will to do something good,” he said.

The test is whether safety warnings produce independent scrutiny or leave the companies defining the rules by which their own systems will be judged.

[Abridged]