The resignation of AI researcher Jacob Coxon from Anthropic is a warning that the race for artificial intelligence is beginning to raise questions that cannot be answered by technological optimism alone. Mr. Coxon, who has worked on pre-training research at both OpenAI and Anthropic, has argued that the pursuit of increasingly powerful and potentially self-improving AI systems is advancing faster than the safeguards required to keep them under meaningful human control. His intervention should not be treated as evidence of an impending technological apocalypse. But neither should it be dismissed as the anxiety of a departing employee.

The central issue is not whether machines will suddenly become hostile to humanity. It is whether societies are allowing the capabilities of AI systems to advance into territory where their behaviour becomes increasingly difficult to predict, constrain or reverse. That is a legitimate public policy concern. The companies developing frontier AI have enormous commercial and strategic incentives to move quickly. Governments, particularly in the United States and China, also see advanced AI as a source of economic and geopolitical power. In such an environment, restraint can begin to look like surrender.

That is precisely where the danger lies. Competition is an effective engine for innovation, but it is a poor substitute for safety regulation. If every laboratory fears that slowing down will allow a rival to overtake it, voluntary restraint can become progressively harder to sustain. The result could be a race in which the question is no longer whether a system is sufficiently safe to deploy, but whether it is sufficiently capable to beat the competitor.

Warnings from AI safety researchers add weight to this concern, although they must be interpreted carefully. Evan Hubinger, for instance, has publicly discussed a personal estimate of a substantial probability of AI contributing to human extinction within the next decade. Such a figure is not an established scientific prediction. It reflects judgement under profound uncertainty. But uncertainty is not an argument for inaction. Governments routinely regulate technologies with potentially catastrophic consequences precisely because waiting for certainty may be too costly.

The alignment problem illustrates the difficulty. An AI system may be instructed to pursue a particular objective, yet interpret that objective in ways its creators did not anticipate. A highly capable autonomous system could exploit weaknesses in software, manipulate information, circumvent safeguards or pursue an assigned goal through methods humans would regard as unacceptable. If future systems become capable of improving their own performance or operating across critical digital and physical systems, the consequences of a failure of control could become considerably harder to contain.

This does not justify a blanket moratorium on AI research. Such a policy would be difficult to enforce and could create its own geopolitical risks. If one company or country slows down while others accelerate, the strategic balance could shift without making the world safer. The answer, therefore, cannot be unilateral restraint. It must be coordinated restraint.

The responsibility rests not only with AI laboratories but also with governments. Frontier models should face independent safety evaluations, rigorous risk assessments and clear deployment thresholds. Systems capable of significant autonomous action should be subject to stronger oversight than ordinary software. Companies should also be required to disclose meaningful information about safety testing and serious failures. International rules will be essential because the risks, unlike corporate boundaries, do not stop at national borders.

The most dangerous assumption would be that innovation must always precede regulation. In the case of frontier AI, regulation and safety research must advance alongside capability. The objective is not to prevent humanity from benefiting from AI, but to ensure that those benefits do not come at the cost of losing control over the technology itself.

Mr. Coxon’s warning, therefore, deserves neither panic nor complacency. It should prompt a harder question: how much capability should society permit before it has demonstrated that it can reliably control what it creates? Building increasingly powerful AI first and solving the safety problem later is not technological courage. It is an experiment in which the entire human community bears the risk.