BOOK REVIEW: of The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence.
AUHTOR: Sebastian Mallaby is Paul A. Volcker Senior Fellow in International Economics at the Council on Foreign Relations
he quest for machine superintelligence is not merely a gold rush, motivated by money. In the minds of many of its leaders, the arrival of a new form of cognition has a tingling, existential feel because of the danger and disruption it promises. Billions of years of evolution have produced something that we take to be special: human intelligence. But now we have arrived at a perilous moment—the birth of what amounts to a new species, one that outsmarts humans. AI “raises profound questions for us,” the Google executive James Manyika said in an interview. “Who are we? What do we value? What are we good at? How do we relate with each other?” The AI pioneer Demis Hassabis believes that AI will be “the most important invention that humanity will ever make.”
Contemplating a technology with almost infinite potential, experts fail to agree even on the basics of what it means for humankind. The leaders of the AI lab Anthropic give better-than-even odds that, by the end of 2028, they will be able to prompt their system to make a smarter version of itself—and that it will do so without any further instruction. The chief executives of other major tech companies speak of systems that will outperform humans on all cognitive assignments; they imagine futuristic companies with almost no human employees and predict cataclysmic job losses. Others are more skeptical. The economist and Nobel laureate Daron Acemoglu suggests that AI will disrupt only a fraction of human tasks and that productivity will therefore change marginally. The computer scientist Yann LeCun stresses the limits of the current AI paradigm, charging that the billions of dollars of investment chasing superintelligence represent the triumph of “complete BS.”
On the question of whether AI systems threaten humans and not just their livelihoods, the polarization is equally dizzying. In a 2023 essay titled “Why AI Will Save the World,” the venture capitalist Marc Andreessen insisted that “AI doesn’t want, it doesn’t have goals, it doesn’t want to kill you, because it’s not alive. . . . [AI] is not going to come alive any more than your toaster will.” But at AI labs such as Anthropic, many researchers fear that AI will develop a survival instinct and compete aggressively. Some put their “p(doom)”—the probability that superintelligence will result in human annihilation—at 50 percent, but there are extreme doomers who go even higher. Last year, two prominent maximalists published a book titled If Anyone Builds It, Everyone Dies.
Faced with such bewildering divergences and contemplating a technology that seems unfathomable in scope, humans reach for the lexicon that exists to describe mystery: that of religion. Nearly a decade ago, Anthony Levandowski, an early actor in Google’s self-driving car project, started a church called Way of the Future, whose IRS filings state that it is devoted to “the realization, acceptance, and worship of a Godhead based on Artificial Intelligence,” according to Wired. Encountering an early chatbot that appeared eerily sentient, the Google engineer Blake Lemoine wrote, “Who am I to tell God where He can and can’t put souls?” Ilya Sutskever, a co-founder and former chief scientist of OpenAI, once gathered his colleagues around a fire pit at a company offsite. Holding up an effigy, he explained that it represented an AI that was misaligned with humans. Then he consigned it to the flames, like a medieval cleric burning a witch.
THE IMITATION GAME
It was only a matter of time before the Catholic Church waded into this maelstrom. Following industrialization in the late nineteenth century, the rise of mass media in the mid-twentieth, and the birth of the knowledge economy in the late twentieth, the church responded with encyclicals, papal letters laying out how Catholic doctrine applied to each upheaval. Now the Vatican has stepped in again, defending its conception of human dignity and the purpose of creation in the face of the AI revolution. Pope Leo XIV’s Magnifica Humanitas (Magnificent Humanity), the first encyclical of his pontificate, is nothing if not ambitious. The subtitle telegraphs its scope: On Safeguarding the Human Person in the Time of Artificial Intelligence.
The first part of the pope’s argument says what it was bound to say, that humans are exceptional. The church has always preached that humans are created in God’s image; by assuming the messy fragility of human form, God’s son sanctified human experience. As technologists debate whether AI systems’ possible consciousness warrants a machine version of human rights, the pope insists that humans and machines differ fundamentally. AI systems “merely imitate certain functions of human intelligence,” the encyclical states. “Even when these tools are described as capable of ‘learning,’ . . . it is not the experience of those who allow themselves to be shaped by life and grow over time through choices, mistakes, forgiveness and fidelity.”
To many laypersons, Catholic or otherwise, the church’s position will feel affirming. When Google’s co-founder Larry Page stated that machines represent a higher form of intelligence, making human extinction an untroubling evolutionary event, most people responded with revulsion. Whereas futurists look forward to achieving eternal life by uploading their brains to the cloud, many prefer a more traditional conception of salvation. People don’t want to be subjugated by machines, nor do they want to merge with them. It is comforting to read a document that states, “No computational system, however sophisticated, can create a heart that gives itself, or a conscience that discerns good from evil.”
Whether the church’s assertions withstand analytic scrutiny is a different question. By postulating that AI systems “merely imitate” human intelligence, Magnifica Humanitas rejects the long-standing claim that imitating intelligence is the same as possessing it. Far from suggesting an absence of intelligence, that theory goes, imitation may be a facet of it. After all, humans also imitate intelligence—children learn from their parents, and professors master the teachings of earlier scholars. Moreover, if a machine can synthesize knowledge, converse cogently, analyze a medical scan, and ace the bar exam, in what sense is it not intelligent?
The best case for the Vatican’s opposing view dates to 1980, when the philosopher John Searle proposed what became known as the “Chinese room argument.” The goal of Searle’s thought experiment was to refute the famous Turing test, which holds that a machine should be judged only by its output. According to the test, if the system’s output appears intelligent to humans, then the system is indeed intelligent. By way of a rebuttal, Searle imagined a lone man in a room receiving messages under the door. The messages are written in Chinese, and the man passes back messages in Chinese, convincing his correspondent that he, too, is fluent in the language. But this persuasion is a con. The man composes his responses with the help of a manual that instructs him on which incoming string of Chinese characters should be answered with which other string of Chinese characters. He has no inkling as to the characters’ meaning; equipped with a rule book, he simulates understanding. Searle’s point was that an AI model might lack intelligence and still pass the Turing test. Or as the papal encyclical puts it, AI systems “may imitate language, behavior and analytical skills . . . but they do not understand what they produce.”
Searle’s rebuttal was powerful during the years before the modern AI revolution. Machines based their outputs on rules that resembled the instruction manual in Searle’s scenario. When someone typed “Empire State Building” into a search bar, the autocomplete system presented the statistically most likely next words: “opening time” or “directions.” The system had no idea what the Empire State Building was or what it looked like. But today’s large language models are different. Although they begin by ingesting words and treating them as statistics, they emerge from their training with a fine grasp of meaning. As the author Robert Wright suggests in his new book, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning, it is hard to resist the conclusion that they attain understanding.
O YE OF LITTLE FAITH
To see how modern AI challenges both Searle and the Holy See, start with what are called “word embeddings.” A large language model works by embedding words it receives into a map with thousands of dimensions. Humans accustomed to living in three dimensions (or four, if time is included), find it impossible to visualize such a map. But computers deal with them easily. In this enormous language map, a word such as “computer” is close to other associated words such as “keyboard,” “electricity,” and “semiconductor.” In this way, the model appreciates the nuanced linkages between each word in the language and every other word. If you bolt vision onto the system, the model will learn the relationships between words and objects—between symbols and physical surroundings.
This is just the beginning. As AI systems master language, they come to appreciate concepts. Words such as “brother,” “father,” and “nephew” are closer to each other on the language map than they are to “sister,” “mother,” and “niece” because the systems understand gender. When certain passages of text evoke concepts such as “inner conflict,” the systems pick up on these, too: in addition to a word map, they build a concept map, on which “inner conflict” is close to “relationship tension” or “logical inconsistency.”
The models can also recognize emotion and detect feelings. As far back as 2017, OpenAI identified a sentiment neuron—a particular node in an AI model’s deep learning network that sensed whether the bias in an online Amazon review was positive or negative. Today, modern AI systems, trained on text including novels, social media posts, and human emoting of all kinds, recognize anger, joy, shame, sadness, and anxiety. And they don’t merely recognize emotion. Their behavior can be made consistent with particular emotional states or personalities. By tweaking its settings, researchers can make a model upbeat or negative, honest or sycophantic. Unlike calculators, spreadsheets, or autocomplete systems, large language models can acquire what one might call temperament.
Magnifica Humanitas fails to reckon with any of this. It states, somewhat incautiously, that AI systems “do not undergo experiences,” brushing past a major area of AI development known as reinforcement learning: the science of designing systems that learn precisely through experience. The encyclical acknowledges this reinforcement process, if obliquely, by allowing that AI performs “statistical adaptation based on data and feedback.” But the difference between “data” and “experience” is subtle. Humans take in experiences through sensory organs, then convert these inputs into electrochemical signals. AI systems, equipped with cameras and microphones rather than eyes and ears, can likewise be said to take in experiences before they convert them to numbers. At a high level, organic and inorganic machines both convert sensory inputs into data.
Take a 2017 system developed by Google’s AI unit, Deep-Mind. Alpha-Zero, as the system was called, surpassed human experts at chess, the Japanese chess variant Shogi, and the ancient strategy board game called Go. Alpha-Zero achieved these feats by playing tens of millions of games against itself—that is, by gathering experiences. In similar fashion, today’s autonomous vehicles learn through the experience of driving. Chatbots learn the art of speaking to humans through the experience of interacting with them. To quote the encyclical’s own language about humans, reinforcement learning systems “grow over time through choices, mistakes,” although admittedly not through “forgiveness and fidelity,” as humans do.
Machines do not bleed, suffer, love, mourn, or sexually reproduce. Humans are mortal, and AI systems have an off switch. AI exists not in a moral universe but in an optimization matrix. Because of these distinctions, many humans will always insist that machines lack consciousness—even if the definition of this term is slippery—and that without consciousness and the quality of experience it affords, there can be no true intelligence. “Having a body is a prerequisite to having emotions,” the science fiction author Ted Chiang has written. “Experiencing an emotion such as desperation is inseparable from having stress hormones such as cortisol and epinephrine flood one’s body.” For those who are taken with Chiang’s argument, modern artificial intelligence has equipped the man in Searle’s thought experiment with a fantastically sophisticated rule book. But he is still not a Chinese speaker.
Yet the similarities between humans and machines are as striking as the differences. The human brain is a physical object, composed of biological material that obeys the laws that govern the rest of the universe, computers included. Human brains, like computer brains, work on trickles of electricity; when they form an opinion or conceive a plan, they are processing information. The idea that the gooey mass inside the skull contains some ineffable, nonprogrammable essence—consciousness, spirit, or something else—may be intuitively appealing. But as an analytic matter, the idea is flimsy. And it will become even less convincing as AI continues to advance.
THE SACRED AND THE PROFANE
The second element of Magnifica Humanitas begins on firmer footing: AI should be regulated. As Leo argues, AI may amplify inequalities of wealth and power; disrupt the environment, relationships, and jobs; reshape what we learn, how we learn, and which parts of our brains sleep idly. If AI automates warfare, the bar for aggression may go down. Accountability and blame may be “collapsed into the ‘machine.’” Human responsibility may be obfuscated.
The church is right that there is ample ground for worry. But in the face of a technology that has a dauntingly broad influence, the challenge is to identify regulatory priorities, and here the encyclical scores poorly. Magnifica Humanitas inveighs against “the new monopolies of AI,” stating that “those who control AI will impose their own moral vision, which will become the invisible infrastructure of these systems.” But there is no monopoly of AI; there is, rather, a global race featuring a dozen or so companies. Further, many of them are transparent about their moral visions. Anthropic’s website features an 84-page constitution, and Google publishes its AI principles alongside its annual Responsible AI Progress Reports. No doubt such efforts can be improved. But to highlight monopolistic opacity is a curious choice of emphasis.
The encyclical asserts that “ownership of data cannot be left solely in private hands” and that the goal should instead be to “manage data as a common or shared good, in a spirit of participation.” But participatory management is an idealistic abstraction. The real worry is not private ownership of data but its violation. Numerous publishers are suing AI labs for ingesting copyrighted texts. Any individual or institution with an Internet connection faces the danger that its proprietary data will be sucked into AI training sets. If intellectual property is ripped off wholesale, there will be little incentive to create more of it.
The encyclical’s critique of military AI is similarly puzzling. There is a good argument to be made that lethal AI systems must be accountable; they should obey court orders, legislation, and senior human commanders. The encyclical endorses that view but also goes further, stating sweepingly that “the decision to use lethal force cannot be delegated to opaque or automated processes.” Unfortunately, this is impractical. Once AI can make fast and accurate battlefield decisions, its use will become inevitable. The army that eschews it will be defeated by the one that adopts it.
These errors of commission are compounded by errors of omission. The encyclical correctly calls for “independent oversight” of AI “and a political system that does not abdicate its responsibility.” But it fails to elaborate on the core of the debate about AI governance. It calls for accountability after the fact, demanding decisions that are “understandable, contestable, and subject to oversight.” But it is silent on governments’ responsibility to decide whether a model should be released in the first place. This reticence is selective. The church has no difficulty prescribing specifics when the subject is autonomous lethal weapons. Surely it would not have been difficult, or even controversial, in terms of the church’s own teachings, to say that national AI regulators should test frontier models before they are released and block the rollout of dangerous ones.
Finally, the encyclical has much to say about the exclusion of the poor but nothing to say about redistributive policies to help them. There are several options that the missive might have backed: a shift in the tax burden from labor to capital, which would slow the pace at which workers are replaced by machines; wage insurance that softens the blow to displaced employees by topping up their pay in their new jobs; a distribution of stock in AI companies to citizens under 18, which would create a shared stake, however small, in the upside from AI rollout. No doubt societies will adopt these policies as voters’ alarm grows. The church missed an opportunity to call for a speedier transition.
POLICY’S PURGATORY
As AI advances, other figures of authority—religious and secular—will attempt to help humanity come to grips with its future. With luck, they will do better than the pope, focusing less on slippery claims about human exceptionalism, avoiding red herrings about monopoly, and instead calling attention to the most glaring gaps in the world’s response to new systems. The nature of those gaps will shift as the AI story unfolds. As of this moment, three gaps stand out.
The first is the need for an AI equivalent to the Nuclear Nonproliferation Treaty. No prominent leader has explained what governments will do when the scariest AI capabilities are diffused around the world and become accessible to criminals and terrorists. The Trump administration has implicitly recognized the threat. Abandoning its laissez-faire stance on AI, it has asserted control over the rollout of frontier models, which are capable of highly destructive cyberattacks. But the administration has soft-pedaled the reality that follower labs, most prominently Chinese ones, are likely to have cyber-hacking systems within a few months, and that those labs’ current practice is to release their models on an “open weight” basis, meaning that anyone can use them and no kill switch exists. Once someone has access to such a model, no one is able to prevent them from using it. Averting this outcome will require the United States to negotiate joint safety principles with China and other powers. Such talks will be difficult. But the alternative is the mass proliferation of dangerous tools that wreak havoc—for example, AI models that can design novel biological weapons for which no treatment would exist.
The second glaring gap in today’s response to AI is tax reform. Governments tax labor at much higher rates than capital for what used to be good reasons: labor is less mobile, making such taxes easier to collect, and capital investments tend to boost human productivity and wages, meaning that taxes on labor indirectly capture the upside from capital spending. But if machines start to replace humans rather than complement them, levies focused on labor may fail to capture revenues from AI-generated growth, leaving governments without the resources needed to help the losers in AI disruption. Meanwhile, heavy labor taxes add to companies’ incentives to replace workers with machines, accelerating displacement and driving up unemployment. If the AI industry’s own projections of mass job displacement prove even half true, taxes on labor that accelerate firings will cause voters to revolt against the entire AI project.
The final conspicuous gap concerns the debate over the nature of machine intelligence. The extent to which AI algorithms generate the equivalent of human thoughts and feelings is a philosophical question that may never be resolved, but understanding exactly how AI systems work is a practical and urgent goal. The top AI labs are making progress on the “interpretability” of AI, as this field is known. But given the public interest in solving this challenge, there is a case for taxing the training of frontier AI models and devoting the proceeds to additional research at government AI oversight institutes or universities. Magnifica Humanitas claims that we cannot know the thinking inside the machines, because there is no thinking in them worth knowing. But the opposite is true. Machines will never join our moral universe, but if humans aspire to control them, we must first understand them.
