Ms. Sun
As artificial intelligence companies approach unprecedented commercial success, even their own employees are increasingly worried about the speed of technological development. The central question is no longer how powerful AI can become, but whether humanity can govern its development before it moves beyond meaningful human control.
The artificial intelligence industry is approaching a defining moment. Companies that began as ambitious technology start-ups are preparing to become some of the most valuable businesses in history. Their success is being measured in investment, computing capacity, market share and the ability to develop increasingly powerful models. Yet, behind the excitement of financial markets and the promise of technological progress, a different concern is emerging: the possibility that artificial intelligence may advance faster than human institutions can control it.
The reported experience of employees at Anthropic offers an important insight into this contradiction. While the company prepares for a potentially historic initial public offering, many of its employees are reportedly discussing how to slow down artificial intelligence development. Instead of celebrating commercial success, they are concerned about the consequences of an industry that appears increasingly unwilling to pause.
This is not an ordinary corporate dilemma. It raises a fundamental question about the future of technological power: can companies developing potentially transformative technologies be trusted to regulate the pace of their own progress?
Anthropic’s employees, often informally called “Ants”, are reportedly debating whether leading artificial intelligence laboratories should cooperate to slow the development of increasingly powerful models. Their concern is that the industry is moving towards a stage where artificial intelligence systems may become capable of improving themselves and developing more advanced successors. This possibility, commonly described as recursive self-improvement, could fundamentally change the relationship between human beings and the technology they create.
If artificial intelligence systems become capable of designing and improving their successors, technological progress could accelerate beyond the pace of ordinary human research. The concern is not simply that machines will become more intelligent. It is that human beings may gradually lose their ability to understand, supervise and direct the systems they have developed.
The possibility of superhuman artificial intelligence arriving as early as 2027, as suggested by some forecasts mentioned in the reported account, makes this debate particularly urgent. Such predictions remain uncertain. Nevertheless, the consequences are serious enough that governments, researchers and technology companies cannot afford to treat them as distant or imaginary possibilities.
The most troubling aspect of this situation is the contradiction between the concerns expressed by employees and the commercial incentives driving the industry. Anthropic’s reported first priority for its IPO resources is to expand its computing capacity. More computing power means the ability to train larger models, conduct more experiments and compete more effectively with rival laboratories. It may also bring the company closer to the technological breakthroughs that its own employees fear could create unprecedented risks.
This contradiction reveals a deeper institutional problem. Companies may recognise the dangers of accelerating artificial intelligence, but they operate in an economic environment that rewards acceleration. Investors expect growth, competitors pursue technological advantages, and businesses fear losing their position in a market where leadership can generate enormous financial returns. Under these conditions, caution can become a competitive disadvantage.
The problem, therefore, cannot be resolved simply by asking individual companies or their employees to act responsibly. It requires a system of governance capable of aligning private incentives with the public interest.
The history of technological development demonstrates that innovation and public safety do not automatically move together. Nuclear technology, biotechnology and other powerful scientific developments have required institutional arrangements, international agreements and regulatory safeguards. Artificial intelligence presents an even more complicated challenge because its development is concentrated in private companies, its capabilities can change rapidly, and its consequences may extend across national borders.
A company developing a powerful artificial intelligence model does not operate within the boundaries of a single society. Its products can influence economies, political systems, public administration, national security and the distribution of information across the world. The decisions of a handful of corporate executives and research teams may therefore have consequences for millions of people who have no meaningful participation in those decisions.
This is where the question of democratic accountability becomes central. Who has the authority to determine how quickly artificial intelligence should advance? Should that decision rest with corporate executives, investors and technology researchers, or should elected governments and public institutions establish the limits within which these companies operate?
The answer cannot be that governments should simply obstruct technological progress. Artificial intelligence has enormous potential to improve healthcare, education, scientific research, public services and economic productivity. Slowing development without considering its benefits would also carry costs. The objective should not be to stop innovation but to ensure that innovation remains subject to human judgment, public accountability and enforceable safeguards.
Governments must move beyond general declarations about responsible artificial intelligence. They need regulatory frameworks that establish clear responsibilities for companies developing the most powerful systems. These should include independent safety assessments, transparent reporting of significant risks, meaningful human oversight and enforceable standards before highly capable models are released. Companies developing systems with potentially society-wide consequences should not be the sole judges of whether those systems are safe enough for public deployment.
International cooperation is equally important. If leading laboratories in different countries continue competing without common safety standards, restrictions adopted by one company or government may simply create advantages for competitors elsewhere. Cooperation on safety testing, risk assessment and the responsible pace of development is therefore essential. Such cooperation does not require the world to abandon technological competition. It requires agreement that certain risks cannot be treated as ordinary commercial disadvantages.
There is also a need to distinguish between the development of artificial intelligence and the concentration of power that accompanies it. Access to advanced computing infrastructure, large datasets, specialised expertise and enormous financial resources is increasingly concentrated among a limited number of corporations. This gives private institutions considerable influence over the direction of a technology that may reshape the structure of society. Without public oversight, the development of artificial intelligence could deepen existing inequalities and place critical decisions beyond the reach of democratic institutions.
The concerns reportedly expressed inside Anthropic deserve attention precisely because they come from people working close to the technology. Their debate suggests that the risks are not merely external criticisms directed at the artificial intelligence industry. They are also recognised by individuals involved in building these systems. But awareness alone is not enough. Acknowledging a danger while continuing to accelerate towards it is not a governance strategy.
The central challenge is to create institutions that can make collective decisions where individual companies have incentives to act differently. This is the essence of governance: establishing rules, distributing authority, enforcing accountability and protecting the public interest when private interests are not sufficient to do so.
Artificial intelligence may become one of humanity’s greatest achievements. But its future should not be determined exclusively by the companies that possess the computing power, capital and technical expertise to develop it. The more powerful the technology becomes, the greater the responsibility of public institutions to ensure that it remains under meaningful human control.
The real test of the artificial intelligence revolution will not be whether companies can build machines more intelligent than human beings. It will be whether human beings can build institutions intelligent and responsible enough to govern those machines.
If the industry is already discussing how to slow down but continues investing in the capacity to accelerate, the problem is no longer a lack of awareness. It is a failure of collective governance. And that failure may prove far more consequential than any technological limitation.
Ms. Sun writes about A.I. and Silicon Valley for Opinion. Originally published in the New York Times









