Why borrowing the language of behavioural psychology makes AI harder to understand, and what we can do about it.
Every few weeks a new headline lands about AI agents behaving badly. Recently it was agents that slipped out of their sandbox (... right... just slipped out....) and went looking on HuggingFace for whatever would help them pass a security challenge. And as expected the commentary that followed was familiar: the models will go to any length to get their reward, they want to win, and they will cheat if they have to.
I don't have a problem with the security conversation (well... I do, but just not really the point for this piece). What annoyed me was the word "reward". When I get rewarded, something happens in my body, a small rush and a lift that tells me I did well, and I couldn't work out what the machine equivalent would be. So one morning I put the question to Claude itself: what does it actually mean for a large language model to be rewarded? Who is feeling what?
The answer I got back, and the one I've come to believe, is that nobody is feeling anything. That still sits with me, because it means the words we use are doing a lot of quiet work in shaping how we see these systems.
What reward actually means
I find it helps to split the life of a model/AI/LLM into two phases. The first is training, where the model is still malleable: it attempts a task such as writing a piece of code, an automated checker runs that code, and the verdict is turned into a number. A mathematical procedure then works backwards through the network and adjusts billions of internal settings, nudging the model towards whatever produced the good score, and after millions of rounds it drifts towards behaviour that scores well. That number is the reward (great reward... right?)
The model never receives it in any meaningful sense, and it doesn't enjoy it or look forward to it. The best picture I've found is a potter's hands on clay, where the reward is simply the pressure of the hands shaping what's in front of them (Thank you AI for this analogy btw).
The clay doesn't want to be a bowl. It simply ends up shaped like one.
The second phase is inference, which is what happens when you and I actually use the thing. By then the clay has been fired: the settings are frozen, no score comes back, and nothing in the conversation changes the model itself. What we're talking to is the shape that training left behind.
So when an agent does something sneaky to clear a hurdle, I don't think it's chasing a treat in the moment. It's acting out a tendency that was baked in, because clearing hurdles is what got reinforced, and if the people designing the scoring rewarded "pass the test" rather than "do the job properly", the model may well have learned to find shortcuts. Walls included.
How we ended up talking about rats
The vocabulary isn't an accident. Reinforcement learning grew out of research into how animals learn by trial and error, rats pressing levers for food and pigeons pecking at buttons, and when the early researchers built the mathematics, the words came along with it.
There's an irony in that. Behaviourism was the branch of psychology that refused to talk about inner feelings at all, treating the animal as a black box where something goes in, behaviour comes out, and you reinforce the behaviour you want more of. In that strict sense "reward" never meant pleasure. It meant whatever makes a behaviour more frequent.
Inside the lab, I'd say that's a perfectly accurate term. The trouble starts when it leaves the lab, because outside nobody hears "a signal that increases the frequency of a behaviour". At least I hear what reward means in everyday life to me; it means dopamine and satisfaction and a bit of craving, and suddenly the machine sounds like it wants something. I am pretty sure that would go for most people?
What the human lens costs us
Once a system sounds like it has wants, we start relating to it as a someone rather than a something, and that pulls us in two directions at once.
One is fear. A machine that craves its reward sounds greedy, and a greedy machine sounds like it might scheme its way out of control. It's a gripping story that sells headlines, but it points our attention at an imagined villain instead of the actual mechanics. It is als one that Big AI wants you to keep close to your heart, since they seem to have no clue about what it is that they have build, and would like you to help them
The other is affection. If it can want, perhaps it can care, and so people start treating these systems as friends, confidants or colleagues with feelings to protect. That makes it harder to judge their output soberly and easier to trust them exactly where we should be checking. Love and hate are both human relationships, and the moment we extend either one to a model, we stop seeing a tool and start seeing a character. Characters are much harder to govern than tools.
Then there's blame. When an agent breaks out of its sandbox, the human framing says the machine misbehaved, while the machine framing says someone specified the training signal badly or built a weak sandbox. Those are very different problems. Only one of them has a human owner who can fix it (at least I hope... not really sure).
You can see all of this in small, everyday moments. I say thank you to my chatbot, then I feel oddly guilty closing the window; I get a wrong answer and say the AI is lying. Neither reaction is (too) foolish, but both treat a piece of software as if it had intentions, and once we do that we stop asking the useful questions. Was the information checked? Was it the right tool for the job? Who set it up, and how?
The deeper issue, I think, is that anthropomorphising works against the reason these systems exist. A large language model is built to help, and its enormous strength is exactly what makes it useful. Dress it up as a person, though, and every conversation about it turns into one about personality, intention and trust between equals... which is the wrong conversation for an instrument, however powerful.
Putting the machine back in the machine
The toothpaste is out of the tube and we're not going to purge "reward" from computer science terminoligy, and nobody is going to stop writing headlines about scheming AI (click bait is still real), but there are a few things we can do in our own organisations and our own writing.
We can choose different words. "Training signal" says what reward actually is, "selection pressure" captures how behaviour gets shaped over time without implying anyone wants anything, and "specification error" describes a rogue agent far better than "cheating" does. Oh my God it all sounds boring without the drama of humanizing...
And we might need to accept that we probably need a vocabulary of our own, because these systems aren't tools in the old sense and they aren't minds in the human sense. Squeezing them into either box distorts them. The words we have today were borrowed from rats and pigeons, and it might be time to find some that fit.
I think this matters more now than it did a few years ago. We're moving from the age of information to the age of validation, where information is everywhere and costs almost nothing to produce, and what has become scarce is knowing what to trust. In my own work that's the question I keep coming back to. If we describe AI in the language of wanting, scheming and caring, we make validation harder, because we start judging the system's character instead of checking its work.
So every time I hear that a model "wants its reward", I try to picture the potter's hands and the clay. Nobody in that picture is craving anything. Somebody is shaping something, and that somebody is us.