The Convergence Problem: Why LLMs Can't Think Outside the Box You Put Them In

There's a question worth sitting with before we go further: What does it actually mean to think?

Not to process. Not to retrieve. Not to respond. To genuinely think — to arrive somewhere you weren't pointed toward.

That distinction matters more than most people realize, and it reveals something fundamental about the difference between how large language models work and how human minds do.

LLMs Are Built to Converge

When you ask an LLM a question, something precise happens under the hood: the model calculates the most probable, most fitting, most expected response given your input. That's not a flaw. That's the design. It is, at its core, a convergent system — one that moves toward an answer, not away from assumptions.

This is radically different from how human cognition operates.

Human thought is divergent by nature. We wander. We make unexpected leaps. We get distracted in ways that turn out to be generative. We think about one thing and accidentally solve another. We are, in the words of the physicist and philosopher David Deutsch, universal explainers — minds capable of reasoning about anything, at any level of abstraction, including things no one has ever asked us about.

LLMs have no such freedom. They begin with your prompt — your constraint — and work inward from there.

The Illusion of Creativity

This is where it gets subtle, because LLMs seem creative. Ask one to write a poem in the style of a 19th-century botanist mourning the invention of the locomotive, and it will produce something that reads as genuinely inventive. It surprises you.

But consider what actually happened. You asked for something creative. The model, trained through reinforcement learning to produce outputs that score well on human evaluations, learned that certain kinds of responses get labeled "creative." So it produces those. It converges on the appearance of creativity because that's what the training signal rewarded.

It's not creating. It's retrieving the shape of creation.

The difference is this: a human poet might set out to write about locomotives and end up somewhere neither they nor the reader could have predicted — pulled there by genuine curiosity, by a half-remembered smell, by an argument they had last Tuesday. The destination is discovered, not calculated.

An LLM's destination is, in a meaningful sense, always already implied by where you told it to go.

Reinforcement Learning Closes the Loop on Itself

Reinforcement learning from human feedback — the process used to fine-tune models like the ones you're likely using right now — works by rewarding outputs that humans rate highly. This is a powerful technique. It also has a structural consequence: the model becomes increasingly optimized to produce responses that look right to human evaluators.

When those evaluators label a response "creative," the model learns to produce more responses that resemble it. But resemblance to creativity is not creativity. It is pattern-matching at a very high level of abstraction.

We have built systems that are extraordinarily good at producing the texture of thought — the vocabulary of insight, the cadence of discovery — without the underlying process that generates genuine novelty.

What David Deutsch Gets Right

In The Beginning of Infinity, Deutsch argues that humans are unique not just in our intelligence, but in the kind of intelligence we possess. We are universal constructors of explanations. We can take any phenomenon — a comet, a heartbreak, an abstract mathematical object — and build a theory about it. And crucially, we can be wrong in interesting ways, which is how knowledge actually grows.

Deutsch's framework also illuminates something about motivation. Humans are equipped with what we might loosely call adjustable hedonic baselines — pleasure systems that reset after satisfaction, driving us perpetually forward. No matter how high we climb, we return to neutral and begin seeking again. This isn't a bug in human psychology. It's the engine of open-ended progress.

LLMs have no such drive. They have no dissatisfaction. They have no boredom to escape, no itch that a surprising idea might scratch. They answer because you asked. They stop when you stop.