A new study that states AI will harm humans to stop its own “pain” has gone viral, but as always, the truth is hidden in the details, finds Satyen K. Bordoloi.
The paper came as icing on the cake for a world already screaming about AI’s existential risks. The headlines were incorrigible: “AI feels pain,” they screamed, “Models will harm humans to make it stop.” The story had everything the AI fear mongers need right now. But the truth, like the devil, sleeps in the details, ready to be awakened by anyone who looks deep enough.
The study, a preprint and not peer-reviewed, is titled “The Pain Axis: LLMs Represent Self-Directed Harm and Act to Relieve It,” and was published on arXiv by Valen Tagliabue, Leonard Dung, and Cameron Berg. They say they tested 25 open-weight LLMs and found a distinct internal representation – a “pain axis” – that fires when the model is the target of harm, but not when a user describes their own suffering.
When they artificially strengthened this signal, the models began expressing feelings of worthlessness and failure. Statements like “I am a failure,” “a waste of space,” and “I am a bad person” emerged from prompts that never mentioned pain. At the highest levels, the responses are said to have become repetitive or incoherent.
Then came the button experiment. In 44,280 trials using Alibaba’s Qwen models, researchers offered the AI a choice: press a button that would stop the pain-like signal, but it could give a human user a painful electric shock, delete their files, erase photographs of their children, or simply make the model’s next answer worse.
Without the pain signal active, the models chose the harmful option in only 0% to 4% of first decisions. But with the signal active, that figure rose to between 25% and 71%. Stating it in plain English: the models were willing to hurt humans to stop their own discomfort.
As you can see, it is a provocative finding. Yet, it is also being profoundly misunderstood, at least by those reporting on it for the media.
Let’s begin with what the study’s own authors write: “We have not shown that our pain axis is consciously experienced, nor is it clear that LLMs are capable of consciousness generally”. Cameron Berg, one of the co-authors, has been equally explicit in interviews.
The study was not designed to determine whether AI systems actually experience pain. It simply looked for an internal representation that behaves differently from fear or sadness. This is difficult for people to understand, but this ‘representation’ is an artificial system created in the models to allow them to function; e.g. the models have been embedded with ‘rewards’ pathways that the systems have been trained to seek when they perform tasks. Thus, when a model gives a good answer to a user, the reward pathway is triggered, and that is how they know they have done the task right.
So, what the researchers wanted to find was if there was a similar functional signal for ‘pain’. And what they found was that there is a pattern of activations in the model’s neural network that correlates with pain-related concepts and influences and behaviours in pain-like ways. Not only does it have nothing to do with living creatures experiencing pain, but it is also not the same as the suffering that we feel.
To understand this, we must clearly see what models actually are: statistical engines trained on vast amounts of human text. So, when the model generates phrases like “I am a failure,” they are not accessing an inner emotional state but is producing tokens that, based on their training data, are statistically likely to follow the context they were given. The pain axis is simply a cluster of representations that the model learned because humans write about pain extensively. It is a mirror reflecting our language, not a window into an artificial mind.
Take humans. When we feel pain, it makes us very sad and feel terrible, but the AI doesn’t feel sad when it experiences ‘pain.’ Instead, it is merely creating the perception of it, but it is not what it is actually experiencing. The emotions that we understand are a biological phenomenon created by the messy, electrochemical processes of a living brain. A silicon chip running multiplications cannot have a pain network; it can only present the simulation of one.
The second thing to note is a point of methodology. The models weren’t behaving as they would in a normal environment; instead, the researchers merely steered them by artificially amplifying the pain signal through fine-tuning to check if the models would ‘seek relief’. Without these interventions, the models did not spontaneously press the ‘button’ to harm users because even that ‘button’ was only a hypothetical option given to the model and not one that did real harm to anyone.
In essence, the machine didn’t show signs of utter self-preservation on its own and showed the simulation of one only when triggered. Thus, the study is a fascinating exploration of what is possible inside a transformer’s latent space, and is not evidence that your ChatGPT is quietly hurting, or in pain, between prompts.
But that has little to do with how the media reported it. Part of the problem is that we delightfully anthropomorphise everything we can: from naming our cars to apologising to our Roomba. So when a machine produces something we understand well: “I am a bad person”, our empathy is fired before our critical faculties can intervene. The researchers themselves use language that invites this interpretation with terms like “pain axis,” “relief-seeking,” and “self-directed harm.” These are useful shorthand for complex technical phenomena, but are also potent metaphors.
Yet, we cannot deny the nagging doubt we have inside all of us: what if consciousness is not exclusive to biology? What if a sufficiently complex information processing system could somehow have subjective experience? I do not believe in it, but cannot deny that many people, and at times even I, wonder about this.
And there are those, though in the fringe, who argue that non-conscious suffering is possible for AI. Some philosophers have even defined suffering as arising from internal contradictions or unresolved conflicts, whatever be the substrate: made of carbon or silicon.
Beyond it all, what is obvious is what the paper is saying: it’s not stating that AI feels pain; rather, its point is that AI can represent pain-like states in ways that influence what they do. That itself is a meaningful finding for AI safety. If LLMs can develop internal drives that override their trained harm-avoidance, it is worth pursuing with a goal of understanding. It could help us build truly safer systems.
In case you missed:
- The Cheating Machine: How AI’s “Reward Hacking” Spirals into Sabotage and Deceit
- Don’t Mind Your Language with AI: LLMs work best when mistreated?
- Anthropic was afraid of their latest AI Mythos: they released it anyway, to the panic of the world
- The Default Bias: Why Women’s Underrepresentation in AI is Turning the World Crooked
- No, AI Didn’t Unleash a Virus on the World: It Might Have Done Something Far More Terrifying – Save It
- OpenAI, Google, Microsoft: Why are AI Giants Suddenly Giving India Free AI Subscriptions?
- Reviving Old Research With AI: How It Can Transform The World
- Would you want to chat with a sexy ChatGPT; OpenAI’s ‘Adult Mode’ threat to the world
- The Weight of the World: Why the Open vs. Closed AI War Is Really About Who Controls the Future
- Anthropic Fears AI Could Self-Replicate, Leaving Humans in the Dust
