Random doesn't look random

Your brain runs a pattern detector that never sleeps and never asks permission. It is the reason you can read a friend's handwriting and hear your name across a loud room. It is also the reason you will look at a scatter of dots and feel, with real conviction, that something arranged them.

The trouble is that randomness is lumpy. Genuine chance produces clusters, streaks, runs and coincidences at a rate that feels wrong to almost everybody, including people who have studied it for years. Meanwhile the thing we picture when we say "random" is something closer to evenly spread, well shuffled, nicely mixed. That picture is not random at all. It is the signature of a process that is actively avoiding repetition, which is to say, a designed one.

Below are six demonstrations. Each one runs live in your browser, on numbers generated at the moment you press the button. Nothing is pre-baked and there is no trick in the code. The only trick is the one your visual cortex is going to play on you.

A twenty track playlist, five songs each from four artists. The first button shuffles it honestly, a Fisher-Yates shuffle with every one of the 2.4 quintillion orderings equally likely. Watch how often the same artist plays twice in a row.

The iPod was the first place most people met a true shuffle, and Apple immediately started fielding complaints that it was broken, because it kept playing the same artist back to back. In 2005 they shipped Smart Shuffle, a slider that spread artists apart, and Steve Jobs described the fix as making it "less random to make it feel more random". Spotify hit the identical wall a decade later. Their shuffle was a textbook Fisher-Yates, users kept insisting it was rigged, and in 2014 an engineer named Lukáš Poláček replaced it with an algorithm borrowed from image dithering, one that deliberately spaces each artist's songs down the running order. That is the shuffle you listen to now.

The complaints were not stupid. They were accurate observations of genuine randomness, wrongly read as evidence of design. In this playlist any two neighbouring slots have about a one in five chance of matching, and there are nineteen such pairs, so an honest shuffle averages four back-to-back repeats, produces at least one in ninety nine shuffles out of a hundred, and plays the same artist three or more times in a row in well over a third of them. Every one of those events was heard as a bug. Two of the largest software companies on earth looked at the gap between randomness and the feeling of randomness, concluded the feeling could not be educated away, and shipped the feeling instead. Nobody has complained since.

Tap heads and tails sixty times, from your own head, no coin and no dice. While you type, the page will try to predict each press before you make it. Against a genuine coin nothing on earth can hold above fifty per cent. Against a person it usually can, and it will show you its score as it goes. When you finish, your full sequence gets compared against twenty thousand simulated runs of a real coin.

Two things give people away almost every time. The first is the switch rate. A fair coin changes its mind on about half of all consecutive flips. People inventing a sequence switch closer to six times in ten, because staying on heads feels like it is breaking the rules, and every extra head feels like it owes a tail back.

The second is the longest run. In sixty flips of a real coin you should expect to see five or six of the same face in a row, and a run of seven or eight is entirely ordinary. Almost nobody writes that down voluntarily. A hand written run of six looks so much like a mistake that people delete it before they finish typing.

The predictor exploits both. It is nothing clever, just a running count of what you did after each short pattern of presses, which is enough because people repeat themselves without noticing. The moment you start consciously fighting it, you become more predictable, not less, because now you are following an anti-strategy and it learns that too. If you want to beat it honestly, there is exactly one way: use a real coin. That is the entire point of this page in one game.

Here is a strip of five hundred fair coin flips. The longest run is marked. Redraw it as many times as you like and you will never get a strip without a conspicuous streak somewhere in it.

In 1985 Gilovich, Vallone and Tversky went through the shooting records of the Philadelphia 76ers looking for the hot hand, the widely held belief that a player who has just scored is more likely to score again. They found nothing. A hit after a hit was no more likely than a hit after a miss. The finding became one of the most cited examples of humans inventing structure inside noise, and it stayed that way for thirty years.

Then in 2015 Joshua Miller and Adam Sanjurjo noticed something about the method. If you take a finite sequence of coin flips, find every flip that followed a heads, and average the proportion of those that were themselves heads, you do not get a half. You get less than a half. The bias is a property of the counting procedure, not of the coin. It is small, it is completely counterintuitive, and it is large enough to have hidden a real hot hand effect in the original data.

Sit with that for a moment. Three of the most careful researchers in the field built a landmark result on a measurement that quietly tilts downwards, and the error survived peer review, thirty years of citation and a place in every undergraduate reading list. The lesson is not that the original team were careless. It is that intuition about randomness fails even when you are actively hunting for the ways intuition about randomness fails.

Below is a county. Five thousand households, spread evenly, and one illness that strikes nine homes in a thousand. Every household carries exactly the same risk. There is no pollution, no transmitter mast, no contaminated water and no cause of any kind beyond chance. Press the button and watch the local paper write itself.

Run several years in a row and watch where the circle lands. The hotspot wanders the county, because it was never a place, only the densest patch of that particular year's noise. A real cause stays put. Noise moves house annually.

The circle is drawn once the cases are on the map, which is what makes it dishonest. Anywhere you look, given enough places to look, something is denser than average. Pick the densest spot afterwards and you can always produce a number that sounds alarming, because you chose where to point the microscope by looking at where the answer already was. Statisticians call this the Texas sharpshooter, after the man who fires at a barn and then paints the target around the tightest group of holes.

This is not an abstract concern. American state health departments spent decades investigating reported cancer clusters, thousands of them, and confirmed a specific local environmental cause in a vanishingly small number of cases. Atul Gawande wrote about the pattern in 1999. The investigations were expensive, they were slow, and the families who raised the alarm were not being irrational. They were doing exactly what a working pattern detector does. The problem is that a working pattern detector, pointed at a country, will find hot spots in a country where nothing at all is happening.

The suspicion has a long history, and so does the honest way out of it. During 1944 south London was hit by V-1 flying bombs, and it became common knowledge that the strikes were clustering on particular districts, which implied precision guidance or informers on the ground. In 1946 an actuary named R. D. Clarke settled the question by doing the opposite of the circle above: he fixed a grid of 576 equal squares over the area first, counted the 537 hits into them, and compared the counts against a Poisson process, the mathematics of things landing entirely at random. The fit was almost exact. The clusters were real, the targeting was not, and the whole difference between his method and the wandering circle is that he decided where to look before looking.

Take a large collection of real world numbers spanning several orders of magnitude, and look only at the leading digit. You might expect each digit from one to nine to turn up about eleven per cent of the time. Instead about thirty per cent of them begin with a one, and barely five per cent begin with a nine. Choose a source and see.

The effect falls out of scale invariance. If a quantity grows by multiplication rather than addition, it spends longer passing through the stretch between one and two than it does crossing from nine to ten, because that first stretch is a doubling and the last one is an eleven per cent nudge. Anything that compounds inherits the pattern, which covers populations, river lengths, share prices, invoice totals and street addresses. Notice that the uniform source in the list above does not follow the curve at all. The law needs data that spans orders of magnitude, and it is misapplied constantly by people who skip that part.

Forensic accountants use this. Numbers that a human invented to fill a gap in a ledger tend to be spread far too evenly across the leading digits, because the inventor is doing the same thing you did with the coin flips: producing what randomness feels like rather than what it is. It is a screening tool rather than proof, and it has been stretched well past its evidential weight in some high profile election disputes, but as a first pass over a set of books it is genuinely useful.

The classic. Drag the slider and watch the probability that at least two people in the room were born on the same day of the year.

Twenty three people gets you past even odds, which feels absurd until you count the pairs rather than the people. Twenty three people form two hundred and fifty three pairs, and every one of those pairs is a fresh chance to match. The intuition fails because we instinctively ask how likely someone is to share our own birthday, which is a question about twenty two pairs and has a completely different answer.

Diaconis and Mosteller gave this its general form. If something has a one in a million chance of happening to a given person on a given day, then in a country of sixty eight million people it happens to sixty eight people every day, roughly twenty five thousand times a year. Every one of those people will experience it as a miracle, tell their friends, and be entirely correct that it was staggeringly unlikely. It was also completely inevitable that it happened to somebody.