Will Artificial Intelligence Save Humanity—Or End It?

A sharp divide emerged this week over the future of artificial intelligence as four leading technologists and transhumanists debated whether building artificial general intelligence, or AGI, would save or destroy humanity.

The panel organized by the non-profit organization Humanity+ brought together one of the most outspoken AI ‘Doomers’, Eliezer Yudkowsky, who has called for a halt to advanced AI development, along with philosopher and futurist Max More, computational neuroscientist Anders Sandberg and Humanity+ President Emeritus Natasha Vita-More.

Their discussion revealed fundamental disagreements over whether AGI can be aligned with human survival, or whether its creation would make extinction inevitable.

The ‘black box’ problem

Yudkowsky warned that modern AI systems are fundamentally insecure because their internal decision-making processes cannot be fully understood or controlled.

“Any black box will likely end up with remarkably similar problems to current technology,” Yudkowsky warned. He argued that humanity would have to be “very, very far removed from current paradigms” before advanced AI can be safely developed.

Artificial general intelligence refers to a form of AI that can reason and learn about a wide range of tasks, rather than being built for a single task, such as generating text, images or video. AGI is often associated with the idea of ​​the technological singularity, because achieving that level of intelligence could allow machines to improve themselves faster than humans can keep up.

Yudkowsky pointed to the analogy of the “paperclip maximizer” popularized by philosopher Nick Bostrom to illustrate the risk. The thought experiment features a hypothetical AI that converts all available matter into paperclips, furthering fixation on a single goal at the expense of humanity. Adding more targets, Yudkowsky said, would not meaningfully improve safety.

Referring to the title of his recent book on AI:If anyone builds it, everyone dies‘, he said, ‘our title is not that good power possibly kill you,” Yudkowsky said. ‘Our title is: If someone builds it, everyone dies.”

But More challenged the premise that extreme caution offers the safest outcome. He argued that AGI could offer humanity the best chance to overcome aging and disease.

“The most important thing to me is that AGI can help us prevent the extinction of everyone alive due to aging,” More said. “We’re all going to die. We’re heading towards catastrophe, one by one.” He warned that excessive restraint could push governments toward authoritarian controls as the only way to stop the development of AI worldwide.

Sandberg positioned himself between the two camps, describing himself as “more optimistic” while remaining more cautious than transhumanist optimists. He recounted a personal experience in which he almost used a large language model to help design a bioweapon, an episode he described as “horrifying.”

“We’re getting to a point where empowering bad actors is also going to create a huge mess,” Sandberg said. Still, he argued that partial or “approximate” safety might be achievable. He rejected the idea that security must be perfect to have meaning, and suggested that people could at least converge on minimal shared values ​​such as survival.

“So if you demand perfect security, you’re not going to get it. And that sounds really bad from that perspective,” he said. “On the other hand, I think we can actually achieve estimated safety. That’s good enough.”

Skepticism about alignment

Vita-More criticized the broader alignment debate itself, arguing that the concept presupposes a level of consensus that does not exist even among long-term employees.

“The idea of ​​alignment is a Pollyanna plan,” she said. “It’s never going to be on the same page. I mean, even here, we’re all good people. We’ve known each other for decades, and we’re not on the same page.”

She described Yudkowsky’s claim that AGI would inevitably kill everyone as “absolutist thinking” that leaves no room for other outcomes.

“I have a problem with the blanket statement that everyone is going to die,” she said. “If you approach this as a futurist and a pragmatic thinker, it leaves no consequences, no alternative, no other scenario. It’s just a blunt statement, and I wonder if it reflects some kind of absolutist thinking.”

The discussion included a debate about whether closer integration between humans and machines could reduce the risk of AGI – something Tesla CEO Elon Musk has said. suggested in the past. Yudkowsky dismissed the idea of ​​merging with AI, likening it to “trying to merge with your toaster oven.”

Sandberg and Vita-More argued that as AI systems become increasingly capable, humans will need to integrate or merge more closely with them to better cope with a post-AGI world.

“This whole discussion is a reality check on who we are as human beings,” Vita-More said.