Eighty years ago, responsible and courageous nuclear scientists involved in the Manhattan Project warned policymakers of the dangers their invention posed to international peace. Influential scientists like Robert Oppenheimer used their prestige and connections in Washington to promote proposals aimed at reigning in the genie they had let out of the bottle. Their efforts were ultimately unsuccessful, as we know now. But some were willing to pay a high price for their engagement to avert a nuclear arms race.

Pope Leo’s recent invitation to Anthropic co-founder Chris Olah to publicly respond to his encyclical, Magnifica Humanitas: On Safeguarding the Human Person in the Time of Artificial Intelligence, could have been an Oppenheimer moment. Olah, who has worked with the handful of companies driving AI forward, could have used the high-profile 25 May event at the Vatican to acknowledge private sector responsibility and outline ways to tackle the existential risks AI poses.

Yet, Olah missed the mark. His rejoinder to the Pope indicated that AI companies like Anthropic will likely not be of help in tackling the dangers the technology poses to humans, the environment, democracy, and international peace.

Even worse, Olah demonstrated that any meaningful regulation will have to be imposed on the private sector. Governance of this technology has to rest with governments and international organisations. Those who want to reduce risks cannot count on help from the likes of Olah.

Outsourcing damage control

The Pope’s Encyclical Letter is a comprehensive and – considering that it is coming from the head of a predominantly male autocracy – surprisingly radical and deeply political analysis of all that is wrong with AI. The Pope does not mince his words. More importantly, he puts the spotlight on those who he believes hold primary responsibility for ensuring that the technology is used for good: the people who run and are working for AI companies.

“Today”, Leo argues, “the main drivers of development are private, often transnational, parties that are endowed with resources and the capacity to intervene that surpass those of many Governments”.

In his response, Olah evades that responsibility by pointing to “incentives” (money, geopolitics, ambition, pride) that sometimes “conflict with doing the right thing”. Rather than outlining how powerful companies can overcome these restrictions, he suggests that it falls to “people outside the incentives” to insist that companies do the right thing. Even more irresponsible, Olah suggests the Catholic Church pick up the pieces of the damage done by AI companies.

Here are two examples.

The Pope describes how AI companies are creating a new form of colonialism which “appropriates data, transforming personal lives into exploitable information. Entire regions, especially those marked by structural fragility and limited geopolitical relevance, are currently subjected to a new mindset of extraction: that of health data, epidemiological profiles, genetic maps and demographic information”.

On the question of ensuring that AI’s benefits are distributed globally, Olah’s response amounts to a shrug: “We do not have a mechanism for this. It is an unsolved problem, and it is the kind of problem the Church has historically refused to let the world ignore”. You know, like the missionaries who helped to feed and heal the colonised and disenfranchised in the 19th century.

The Pope also warns of the psychological damage AI can inflict, especially on young people. Here, he puts the onus for fixing this problem on those who do the harm, namely the private sector: “When business models thrive on human weakness, the person is treated as a means rather than as an end; those who design or finance such systems bear a moral responsibility that cannot be ignored”.

Olah’s response: “These are not questions a lab can answer but they are questions traditions like yours have carried for millennia, and we need you to keep carrying them into this new moment in history”.

Over to you, Church!

Don’t mention the war

Most telling, however, is what Olah chose not to address. The Pope also warns that AI is reshaping warfare in ways that make conflict more impersonal and less constrained by human judgement. He cautions that “the growing ease with which autonomous weapons systems can be deployed makes war more ‘feasible’ and less subject to human control”. To address these risks, he proposes three “criteria for discernment”: personal responsibility; a “moral timeframe” that prioritizes ethical reflection over speed and efficiency; and effective means of identifying and protecting civilians. He also describes how the risk of nuclear war has grown.

It would have been easy for Olah to pick up this thread by highlighting how Anthropic refused to bend the knee, when the Department of War demanded that the company throw overboard all principles regarding human control in warfare (to be fair, the Pope wrote 42,000 words to make his case, while Olah had only seven minutes to respond).

Yet Olah chose to say nothing about how AI companies intend to address the risks the technology poses to international peace and security. Maybe this is because Anthropic, despite its high-profile dispute with the Pentagon, has still actively supported the US war against Iran? Or did he evade a response because Anthropic is desperately trying to get its foot back into the door of business with the Pentagon?

Either way, Olah displayed the typical attitude of irresponsible scientists who disassociate themselves from the consequences of their own work.

Disarming AI

If the co-founder of an AI company that paints itself as being more responsible than the rest of the pack does not rise to challenge of accepting responsibility for the human, political, and economic damage the technology is causing, other actors need to step up to the plate.

The Pope seems to have a hunch that not much help may be forthcoming from the likes of Olah. As is appropriate for a religious leader, the Pope makes a plea for dialogue (motivated possibly also by a desire to secure a seat for the church at a table where regulations are debated). But he also urges governments to regulate AI.

“When such power is concentrated in the hands of a few, it tends to become opaque and evade public oversight, increasing the risk of distorted forms of development that give rise to new dependencies, exclusions, manipulations and inequalities”, the Pope warns. “What is needed is a more active political involvement that is capable of slowing things down when everything is accelerating, and of protecting the opportunities for communities still to be able to participate and ask questions”.

What we need to do, he argues, is to disarm AI by “discrediting the assumption that technical power automatically confers the right to govern”.

He is right here, too. Governments, and particularly legislators, need to step in. They should treat AI just like any other new technology that needs to be regulated for the common good.

“Nothing in the world of AI is immaterial or magical”, the Pope dryly concludes. Over to you, governments!

The European Leadership Network itself as an institution holds no formal policy positions. The opinions articulated above represent the views of the authors rather than the European Leadership Network or its members. The ELN aims to encourage debates that will help develop Europe’s capacity to address the pressing foreign, defence, and security policy challenges of our time, to further its charitable purposes.

Image credit: Wikimedia Commons