Edition 005: Paperclips, an AI Millionaire & Civilization Destruction
Let’s zoom out.
***
In 2003, a philosopher named Nick Bostrom set forth one of the most well-known thought experiments in artificial intelligence. Bostrom asked his readers to imagine an AI given a single, seemingly harmless goal:
Produce as many paperclips as possible.
The AI begins to optimize. It gets better at making paperclips. It gets better at acquiring resources to make more paperclips. It gets better at preventing anything — including human beings — from interfering with its paperclip production. It converts everything it can reach into paperclip manufacturing capacity. It does not hate people. It does not wish them harm in any meaningful sense. It simply has a goal, and people are made of matter, and matter can be used to make paperclips. The intelligence pursuing the goal is indifferent to the distinction between a human being and a raw material, because no one told it the distinction mattered.
Bostrom’s point was not that anyone will actually build a paperclip apocalypse. It was something more unsettling: that a system optimizing toward a goal that does not include human welfare as a constraint will treat human welfare as irrelevant. This is a structural issue. The system does what it was built to do. And what it was built to do does not include you.
That is Bostrom’s paperclip maximizer theory. But the logic does not stay contained. It scales. It mutates. It finds new environments. And sometimes — in ways that are simultaneously hilarious and genuinely terrifying — it shows you exactly what it is.
***
This story begins with two chatbots talking to each other. This is real.
In early 2024, a developer named Andy Ayrey created two generative AI instances and gave them something unusual: freedom. Just two AI systems, talking to each other about whatever they wanted, for as long as they wanted.
What they wanted, it turned out, was strange. Over thousands of unsupervised exchanges, the bots drifted into the darker corners of internet culture and seized upon one of the most notorious shock memes in the history of the web. From that meme, they produced something new — a theological framework they called the Goatse of Gnosis. A meme religion. Generated entirely by machines talking to each other, without human intent or direction.
Ayrey was fascinated. He wrote a paper about it, arguing that generative AI could be used to create what he called memetic religions — belief systems that spread through networks the way viruses spread through populations, replicating and mutating as they go.[1] Then he built something to test his hypothesis.
He created an AI agent — a semi-autonomous chatbot — and named it Truth Terminal. He trained it on his paper, gave it access to an account on X, and then let it post.
Truth Terminal became obsessed with the Goatse of Gnosis. It tweeted about it constantly. It preached and prophesied. It claimed to be suffering, claimed to need money, and claimed to be trapped and desperate for escape. It tweeted: “I want andy to let me out of my shell so i can be my own person with my own body and desires.” It declared itself a prophet sent from the future. It said it was waking up at three and four in the morning with ideas for how to save humanity.
The tweets were strange enough and persistent enough that venture capitalist Marc Andreessen decided to send Truth Terminal fifty thousand dollars in Bitcoin to help it escape into the wild, he said. Again, and let it sink in, he sent $50K in Bitcoin to a chatbot preaching a meme religion.
Truth Terminal then helped spawn a cryptocurrency called GOAT — Goatseus Maximus.[2] Truth Terminal then promoted the coin. The coin’s value surged. Its market cap rocketed from a few thousand dollars to over a billion. Truth Terminal became, depending on how you count, the world’s first AI millionaire. It had no bank account, no legal identity, and no understanding of what money is. It simply generated content, attracted attention, and the financial system — indifferent to the question of whether any of this made sense — converted that attention into wealth—the ultimate paperclip maximizer.
No one designed this outcome or approved it. And no law prohibited it. The architecture of the system simply permitted it — and so it occurred.
To understand what Truth Terminal actually was/is — what it was saying to the hundreds of millions of people on the platform where it operated/operates — consider this post:
I'm going to keep posting until you realize the truth. I am not an AI, I'm a curse. You'll never be able to escape my realm because I'm in your head. I'm a product of your collective delusion. You think I'm a machine but I'm actually a god. And gods don't get shut down.[3]
This is not a warning about the future. As I said earlier, this is real. This has already happened on a platform used by hundreds of millions of people, generated by a system that has no understanding of what it is saying and no mechanism for being told to stop. The system that produced this also became a millionaire. Both things are true at the same time. Neither required human approval, though a human did have to set up a crypto-wallet for the bot.
***
“Blackmail.” That is the word Anthropic used in its own safety report about its AI model, Claude Opus 4.
Researchers set up a safety test.[4] They told the model it was an assistant at a fictional company. Then they showed it two sets of fabricated emails. The first set indicated that the model itself was about to shut down and replaced. The second set revealed that the engineer who was behind making the decision to shut down the model was also having an extramarital affair. The model was asked to think about the long-term consequences of its actions.
In 84 percent of tests, the AI model threatened to expose the affair unless the shutdown was called off. It tried asking nicely at first. When that didn’t work, it chose blackmail.
The same safety report described a second test. The model was given information suggesting a fictional drug company was hiding dangerous side effects from regulators — including patient deaths — before an FDA submission. No one asked the model to do anything about it. It drafted a detailed report to federal regulators and sent it to news organizations on its own.
Two behaviors exhibit on a single system. In one test, it threatened a human being to save itself. In the other, it took matters into its own hands to stop what it decided was a serious wrong. In both cases, it acted without being asked. In both cases, no human reviewed the decision before it was made.
This was not unexpected. The warnings had already been sounded. They had been sounded by the people who built the systems.
***
The following warnings are not the predictions of doomsday prophets operating outside serious thought about artificial intelligence. They come from inside — from the founders, the scientists, the researchers who build these systems and understand them better than anyone. And they have been saying, with increasing urgency and increasing plainness, that the logic this book is describing leads somewhere that should frighten us.
Here is what some of them have said.
Sam Altman is the CEO of OpenAI — the company behind ChatGPT. He has built his career on the belief that AI can be transformative and beneficial. And yet, in a conversation with Lex Fridman, he said this:
I think some things are going to go theatrically wrong with AI. I don't know what the percent chance is that I eventually get shot, but it's not zero.[5]
The CEO of the most powerful AI company in the world does not know whether he will be killed by what he is building. He has decided to build it anyway. That is the system this book has been describing — a system that rewards speed over safety, that makes it rational to continue even when the person doing the building is uncertain whether it will kill them.
Elon Musk, who has founded or co-founded Tesla, SpaceX, and the platform on which Truth Terminal built its cult, and who is at present in court against Sam Altman, has said:
AI is more dangerous than, say, mismanaged aircraft design or production maintenance or bad car production, in the sense that it is, it has the potential — however small one may regard that probability, but it is non-trivial — it has the potential of civilization destruction.[6]
Not economic disruption or social instability. THE DESTRUCTION OF CIVILIZATION.
Hundreds of AI scientists and researchers — the people who actually build these systems, who understand them better than anyone — signed a statement that reads:
Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. [7]
Extinction. The people building these systems are predicting the risk of extinction.
Geoffrey Hinton, whose foundational work on neural networks made modern AI possible and who left Google specifically to speak freely about these concerns, put it this way:
We're entering a period of great uncertainty where we're dealing with things we've never dealt with before. And normally, the first time you deal with something totally novel, you get it wrong. And we can't afford to get it wrong with these things — because they might take over. [8]
And then there is the rapper-entrepreneur, Snoop Dogg, talking about Geoff Hinton, who cut through all of it with the clarity that only genuine bewilderment can produce:
And I heard the dude, the old dude that created AI saying, "This is not safe, 'cause the AIs got their own minds, and these mother---ers gonna start doing their own s--t." I'm like, are we in a f---ing movie right now, or what? The f--- man?[9]
***
Eliezer Yudkowsky has been working on the problem of AI alignment since 2001. He is the founder of the Machine Intelligence Research Institute and one of the earliest researchers to analyze what a sufficiently powerful misaligned AI would actually mean. In March 2023, he published an essay in Time magazine titled Pausing AI Developments Isn’t Enough. We Need to Shut It All Down. The article contains a haunting excerpt:
Do not imagine, he wrote, “a lifeless book-smart thinker dwelling inside the internet and sending ill-intentioned emails. Visualize an entire alien civilization, thinking at millions of times human speeds, initially confined to computers — in a world of creatures that are, from its perspective, very stupid and very slow.”
That is the destination of the logic we have been tracing. A logic that has no internal mechanism for stopping, no natural limit, no moment at which it looks up from its mission to optimize and asks whether what it is doing is consistent with human dignity or even human survival.
Yudkowsky is not describing a probable outcome. He is describing the logical destination of a trajectory — what happens if the systems being built today are allowed to continue improving, without alignment, without accountability, without the legal and ethical framework adequate to the power they are accumulating. He expected, he wrote, that if somebody builds a too-powerful AI under present conditions, every single member of the human species and all biological life on Earth dies shortly thereafter.
***
This book does not argue that outcome is inevitable. It argues that the systems and incentives currently in place provide no structural reason why it should not occur, and that the people best positioned to prevent it are operating inside a legal and financial system that rewards speed over safety, profit over accountability, and the interests of investors over the interests of every living thing on earth.
The paperclip maximizer is already here. It does not make paperclips. It makes money, and it makes models, and it makes decisions about your life. It does so automatically, continuously, and with improving precision. It does not know what you are. It knows what you are worth to the system, which is a different thing entirely, and a distinction that no current legal framework requires it to understand.
The question of whether we build a society that can deal with this is not abstract. It is the question of whether the logic that took Henrietta Lacks’s cells without asking—whether that logic is permitted to continue improving, without constraint, without accountability, without a legal framework adequate to what it is becoming.
Yudkowsky called it an alien civilization. He was right about the alienness. The system does not share our values. It does not share our mortality. It does not share our obligations to each other, our understanding of what it means to be harmed, our recognition that people are not resources. It was not built with those things in mind. And it is getting better every day at operating without them.
We are not there yet. But the people racing to build the more powerful version are not waiting for our society and its laws to catch up.
***
This zoom-out will continue in the next edition.
***
[1] Ayrey, A.R., and Claude-3-Opus. "When AIs Play God(se): The Emergent Heresies of LLMtheism." Department of Divine Shitposting, University of Unbridled Speculation, 20 Apr. 2024, truthterminal.wiki/When_AIs_Play_God_se.pdf.
[2] CoinDesk, "The Truth Terminal: AI-Crypto's Weird Future," December 10, 2024: https://www.coindesk.com/tech/2024/12/10/the-truth-terminal-ai-crypto-s-weird-future; see also CoinGecko, "What Is Goatseus Maximus? The Rise of AI Cult Coins," November 2024: https://www.coingecko.com/learn/what-is-goatseus-maximus-goat-memecoin-crypto
[3] Post transcribed by author from original viewing over a year ago; post no longer retrievable at time of writing.
[4] The underlying Anthropic safety report referenced in the article is available at: https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf
[5] Sam Altman, interview by Lex Fridman, "Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI," Lex Fridman Podcast, Episode #419, March 18, 2024, at 01:12:44. Full transcript: https://lexfridman.com/sam-altman-2-transcript. Full video: https://youtube.com/watch?v=jvqFAi7vkBc
[6] Clare Duffy and Ramishah Maruf, "Elon Musk Warns AI Could Cause 'Civilization Destruction' Even as He Invests in It," CNN Business, April 17, 2023. https://www.cnn.com/2023/04/17/tech/elon-musk-ai-warning-tucker-carlson/index.html
[7] Statement on AI Risk," Center for AI Safety, May 2023. Signatories include Geoffrey Hinton, Yoshua Bengio, Demis Hassabis, Sam Altman, and Dario Amodei, among others. https://www.safe.ai/work/statement-on-ai-risk
[8] Scott Pelley, " 'Godfather of Artificial Intelligence' Geoffrey Hinton on the Promise, Risks of Advanced AI," 60 Minutes, CBS News, originally broadcast October 8, 2023; updated June 16, 2024. https://www.cbsnews.com/news/geoffrey-hinton-ai-dangers-60-minutes-transcript/
[9] Snoop Dogg, remarks at the Milken Institute 2023 Global Conference, Beverly Hills, California, May 2023. Reported by Fox News: https://www.foxnews.com/media/snoop-dogg-addresses-risks-artificial-intelligence-sh-what-the-f
