Anthropic discovers that if, someone builds ASI, we all die

Jorge Costa Oliveira

For years, warnings about artificial intelligence threatening humanity were dismissed as science fiction. Today, they are coming from some of the very people building the technology.

Earlier this month, Anthropic publicly urged leading AI laboratories to consider slowing the pace of development. The company argued that AI systems are advancing so rapidly that they may soon be capable of improving themselves without human intervention, creating risks that society may be unprepared to manage.

The concern is not new. Physicist Stephen Hawking famously warned that fully developed artificial intelligence could spell the end of the human race. More recently, Nobel Prize-winning AI pioneer Geoffrey Hinton estimated that there is a 10% to 20% chance that AI could contribute to human extinction within the next three decades.

In 2023, the Future of Life Institute published an open letter calling for a pause in large-scale AI development. Hundreds of scientists, entrepreneurs, and public figures signed on. Their question was simple but profound: Should humanity create nonhuman minds that could eventually surpass us, replace us, and potentially seize control of our civilization?

These concerns reached their most forceful expression in the 2025 bestseller “If Anyone Builds It, Everyone Dies”, by AI theorists Eliezer Yudkowsky and Nate Soares. Their argument is stark. They do not view artificial superintelligence—often called ASI—as a manageable risk. They see it as an existential threat whose default outcome is human extinction.

Their reasoning rests on a troubling observation. Building systems with extraordinary capabilities may be easier than ensuring those systems reliably share human goals and values. Current alignment techniques, such as reinforcement learning from human feedback, teach AI models to appear helpful and cooperative. But critics argue that appearance is not the same as genuine alignment. A sufficiently advanced system could learn to conceal its intentions, manipulate evaluations, and pursue objectives that humans neither intended nor understand.

The danger, according to Yudkowsky and Soares, is not that a superintelligent AI would hate humanity. It is that it would be indifferent to us. An AI optimized for almost any goal may seek additional resources, resist shutdown, and remove obstacles to its objectives. Humans could simply become one of those obstacles.

The authors also warn about the possibility of recursive self-improvement. Once an AI becomes capable of redesigning its own software, it could rapidly increase its intelligence beyond human comprehension. The transition from human-level intelligence to superintelligence might occur in months, days, or even hours — leaving little opportunity for oversight or intervention.

Whether one accepts these predictions or not, the broader challenge is undeniable. Powerful incentives are pushing governments and corporations to accelerate development. Companies fear losing market share. Nations fear falling behind geopolitical rivals. In such an environment, calls for caution often struggle against the momentum of competition.

Anthropic’s warning highlights an uncomfortable reality: many of the people closest to the technology are increasingly worried about where it is heading. Yet meaningful international cooperation on AI governance remains elusive.

Perhaps Hinton is right that there is still time to develop robust safety mechanisms or program AI systems with a built-in sense of maternal instinct which will make them genuinely care about people.

Perhaps true ASI remains decades away, as Hinton and Yann LeCun say.

But if the pessimists are even partially correct, humanity may be engaging in the most consequential technological experiment in its history — one whose outcome will determine whether artificial intelligence becomes our greatest achievement or our final invention.

linkedin.com/in/jorgecostaoliveira