We Built Something We Don't Understand

The assumption going in was that we would teach the models. That knowledge would flow from us into them, and they would become more capable as a result.

That's not quite what happened.

In using LLMs, we've started to understand how they work. We are now understanding through using them. Through noticing what they do and don't do. Through being surprised. Through watching the gap between what we expected and what we got.

We built something and now we're reverse engineering it by living with it.

We Know the Pieces. We Don't Know the Whole.

The basics are understood. Tokens, weights, attention, training on prediction. That's the mechanism. But the emergent behavior — the things these models can do that nobody explicitly trained them for — that part is not fully explained.

And the reason it's not explained is that we can't reason about scale the way we need to.

This isn't unique to LLMs. It's a pattern that shows up everywhere.

Gravity is a simple rule. Mass attracts mass. The equation is short. But run that rule at the scale of the universe and you get galaxies, black holes, the orbital resonance of moons, the slow inspiral of dying stars. Nobody derives a galaxy from the equation. The complexity lives in the scale, not the rule.

Stock markets work the same way. Every participant follows simple logic: respond to information, manage risk, buy and sell. None of them are coordinating to produce the market. And yet the market has its own behavior, its own patterns, its own crashes and bubbles that no single participant intended or can fully predict.

LLMs are in this same category. The rules are known. The behavior at scale is not something we can read from the rules.

Scaling down the LLMs for study doesn't resolve this fully

From what we can see in testing, the structure that emerges in the weights or LLMs changes with scale. The type of information, their reasoning, and the structure all change when we scale up.

The blog post I found to be the most shocking was the frankenmodel that was slapped together and still functioned. In spite of slicing up the model, it still worked.

"The astounding thing about Goliath wasn't that it was a huge leap in performance, it was that the damn thing functioned at all. To this day, I still don't understand why this didn't raise more eyebrows." - David Noel Ng

Read about it here: https://dnhkng.github.io/posts/rys/

Poking Black Boxes

We don't fully understand what we built. The emergent behavior at this scale is genuinely beyond what we can reason about with data.

We have to rely on intuition for much of our observations until we can test our hypothesis with stronger computers and better reasoning.

Until then, we can poke the box, observe what happens, and use that to reason about them.