The unreasonable effectiveness of irrational intelligence

AI only started working once it stopped reasoning like a scientist and started learning like a tradition.

The Jewish people were washing their hands thousands of years before germ theory. No-one had seen a germ, or could explain why ritually pouring water over your hands before a meal would keep you alive, it was just the practice. Each person did it because The Scriptures said so. Yet, the people who did it tended to fare better.

By the standards of a certain kind of rationalist thinker, that’s superstition… belief without proof. And in the 60s and 70s, that same kind of rationalist community tried to build artificial intelligence in their own image: intelligence as a logical, rule based system where everything could be proven. As we all know, they failed.

And in a surprising twist, a few decades later, now we’re building AI systems that actually work, and we’re doing so using a technique which is… almost the opposite.

Consider how a neural network learns. You start with a basically random function and let it guess. Then you check the guess against the world. When the model does well, the weights stay. When the model does badly, the weights evolve. We do this a few trillion times in “pre-training”, and eventually a really capable model pops out.

There is very little about this process which could be described as a “rational”, and a hell of a lot about this which can be described as… traditional. Nothing follows from a principle… a neural network retains a belief if-and-only-if it survives in the future world that neural network interacts within. A modern AI agent doesn’t accept or deny a claim because it’s rational… it retains any belief which helps it maximize reward.

So I want to compare two ways of knowing, and I want to be fair to both, because I think the popular ranking of them is backwards.

The first is science. A scientist forms a hypothesis, tests it, and keeps that belief if the data holds. The second is tradition… religion, custom, and the accumulated do-this-not-that of a people. The usual story says science is the rational one and tradition is the leftover, surviving with out-dated ways of living based on habit and authority.

But if we look a little closer, the difference isn’t quite rational-vs-irrational. It’s more like… how long the test runs, and against what.

A scientific study tests a narrow claim, fast, on not much data, and gives you an answer this year. A tradition tests a whole way of living, slowly, against everything at once… famine, plague, war, raising children, staying married, staying sane… and it only gives you an answer in the form of who's still around in a thousand years. Science is fast and cheap. Tradition is slow and unbelievably expensive. But tradition is tested end-to-end, against survival itself, and science is tested on a slice. And a thing tested on a slice can overfit to the slice.

Here's what I mean by overfit, with a real example.

For years a fair amount of social science said, fairly clearly, that parents are less happy than non-parents (example). If you took that at face value, the lesson was obvious, and plenty of people drew it: kids are a tax on your wellbeing, opt out. Which is a strange conclusion for a species to reach, given that a society that fully believed it would quietly select itself out of existence.

Fortunately, researchers also find more nuance to these findings. Control for various factors, and look at a more life-holistic scale, and other factors like loneliness and family start to reverse this conclusion.

Tradition, meanwhile, had a much blunter and much older answer, which was: have the kids. Tradition couldn't tell you why in the language of a regression. But it had been running the experiment, end-to-end, for a very long time, and the people who took its advice are the reason anyone's around to publish the study.

This is the failure mode of rationalism. A rationalist walks up to an inherited practice, asks for a proof, gets none, and concludes there's nothing there. But no proof is not the same as no evidence. The evidence is just spread across everyone who ran the experiment before and lived. A rationalist sees a fence with no plaque explaining it and tears it down… and only later finds out what it was holding back.

Now back to AI, because this is exactly the shape of the thing we built.

In 2016, in the middle of a game of Go against one of the best players alive, AlphaGo played a move. Move 37. It’s widely reported that no strong human player would have played it. And quite famously during the match, the commentators thought it was a mistake. It broke received wisdom about how that part of the board is supposed to work. But AlphaGo won. The system "knew" something the experts didn't, but it could not have justified that move in the language of Go theory, because it didn't get there through theory. It got there the way a tradition gets anywhere: a staggering number of trials, keep what survives, drop what doesn't, until the surviving instinct is wiser than anything logic could have proven from first principles.

That's what backpropagation and gradient descent are. They aren’t a logic engine that happens to run on tensors… a selection mechanism. A belief stays in the weights exactly as long as it lowers the loss, and not one step longer. The model can't show you its axioms. It just knows things, the way a culture knows things, because the not-knowing got selected out. Logic shows up inside it only as a tool that paid off… and you can fine-tune that tool right back out if it stops paying.

So here's where I land, and I'll try to be fair to the other side while I do it, because I don't think tradition wins clean. Science is genuinely better at some things: it's fast, it needs almost no data, it can give you an answer in a year that tradition would take a millennium to confirm. That's not nothing… that's most of the modern world. But it pays for that speed by testing on a slice, and a slice can lie to you about the whole. Tradition is the opposite trade. Brutally slow, brutally expensive, paid for in actual lives… but tested against everything, all the way through, so it generalizes in a way no single study can.

The AI we built is on tradition's side of that trade. It gets smart by giving up on proof and doing the old thing instead: keep what survives, trust the weight of trials you never personally saw.

It's worth noticing who has been running this exact procedure the longest. Not for a few training runs… for a couple thousand years. The oldest surviving institutions have been keeping what works and dropping what doesn't, and they are still here. And now those institutions are engaging in how AI should be shaped in society.

Rationalists, athiests, and everyone else who finds solace in logic and reasoning should feel uneasy. They were promised a mind that reasons from first principles to the truth. What they got thinks like the thing they probably spent their whole lives talking down… a mind that knows more than it can prove, and got there by surviving.

But just like the Jewish people washing their hands… AI knows more.