A recap on complexity
In January 1885 a young farmer in Jericho, Vermont, called Wilson Bentley fixed a camera to a microscope and took the first photograph of a single snowflake. He spent every winter for the rest of his life doing the same, catching flakes on a black board and photographing them before they melted, until he had more than 5,000. He never found two that were alike. And no-one ever will.
Water vapour in the cloud freezes onto the ice and dust, and wherever the flake sticks out a little it catches yet more vapour, so that bit grows faster and sticks out further. A bump becomes an arm, and the arm sprouts branches of its own, so the shape a flake already has decides the shape it grows into next (network scientists call this kind of rich-get-richer growth preferential attachment). A feedback loop like this takes the smallest differences and magnifies them.
Feedback like this is a classic non-linear interaction seen in complex systems. Lay twenty-four snowflakes on top of each other and the details that made each flake its own blur away, so the average looks like a snowflake that has started to melt. Each snowflake has a lowish entropy, but since each one is unique, their average has a high entropy.
A diamond is the opposite. Its carbon atoms organise themselves into exactly the same pattern every time, so its average is a perfect diamond and knowing one tells you everything about the next. Water is at the other end of the spectrum, high entropy and indistinguishable every time.
Yet the snowflakes’ average isn’t a total shapeless blur either: it still has six arms and a six-sided outline, and every one of Bentley’s photographs is instantly recognisable as a snowflake.
What is self-organisation?
Snowflakes are recognisable because, for all their differences, they share a great deal, and they share it in two quite different ways.
The first is inside each flake. Every snowflake has six arms, and each arm is symmetrical down its middle, so the same pattern repeats twelve times around the flake and one twelfth of it is enough to draw the whole thing. The arms match because of the shape of water and the electromagnetic forces which propagate across the whole surface.
A snowflake always organises itself to have six arms. The system does that. There are no external forces, no fairy or god crafting each one, just the snowflake’s existing shape and electromagnetic forces. This is what defines the “self” in self-organisation.
But snowflakes also self-organise a second way. In the 1930s the physicist Ukichiro Nakaya grew the first artificial snowflakes in a cold room in Hokkaido, and found that their shape depends on the temperature and humidity they grow in. Just below freezing they grow as thin six-sided plates, at around minus 5 degrees they grow as columns and needles, and at around minus 15 degrees they come back as plates or as branching stars.
No two plates are alike, but any plate is far more like another plate than like a column or a star. Their chaotic system is attracted to a few totally discrete outcomes, and only a few, not an infinite array of different shapes.
This is why people find self-organisation (sometimes called emergence) so magical. A murmuration is always identifiable as a murmuration, yet each one is constantly unique. It’s a recognition of nature finding a balance in a way that feels alive. Each part gets enough freedom to be unique, to evolve and to adapt, while the whole still keeps a literal sense of self, remaining a murmuration rather than just a bunch of birds.
These attractor patterns are seen all the time, from Turing patterns, where a single algorithm stabilises into either leopard spots or zebra stripes, to the shape of galaxies, and they are the core reason why these systems are so interesting.
Given enough time, high entropy systems like burnt toast will occasionally create images of Jesus. But these systems create low entropy, identifiably unique patterns nearly every time.
How does it affect decisions?
Say you’re in charge of preparing the UK for climate change. You have a fixed budget and a long list of things it could pay for. Whatever you build will take decades, so you commission a climate model.
Most runs come back as you’d expect, with the UK’s daily temperatures 7 to 9 degrees warmer than today. Some runs show something else. The UK is only as mild as it is because a system of Atlantic currents carries warm water up from the tropics, and without it London, which is about as far north as Calgary, would have Canadian winters. In these runs fresh water from melting Greenland ice slows the currents until they stop circulating altogether, and the UK ends up 7 to 10 degrees colder than today.
This is a tough situation. You now have to split your budget between two very different scenarios (like the train example). But in a way this is good news. Like the spots and stripes, the runs fall into two clear groups, as you only have to plan for two scenarios.
Things could be much worse, because your model could have come back with this instead:
Here we see that when results self-organise it dramatically changes our decision making ability when it comes to complex systems. We measure this using a metric we call Cohesion.
The difference between the two forecasts isn’t how much the runs disagree, it’s whether their disagreements fall into a pattern.
How do we measure it?
Incoherence tells us whether the results differ. What we want to identify is whether those differences form any sort of pattern. We do this not by comparing them to an average, but instead by comparing them to each other.
Think back to the average of the snowflakes. It looked like none of them, but a good deal of the pattern survived the averaging, because every flake’s distribution overlaps with every other’s: wherever one flake has an arm, so do all the rest. That overlap is one way a system can be organised.
Turing’s runs, like the kinds of snowflake, show the other way. A striped run and a spotted run are so different that their distributions hardly overlap at all, and averaging them would give you neither a zebra nor a leopard. Yet every striped run overlaps closely with the other striped runs, and every spotted run with the other spotted ones, so the runs fall into a few scenarios that each hang together. The first climate forecast is organised in the same way, while in the second the runs neither overlap nor gather into scenarios. Cohesion tracks both kinds of organisation: either the instances overlap with each other, or they gather into scenarios whose instances do.
Cohesion measures how much a few observations reduce your uncertainty about a run, which is its entropy. A system whose instances fall into recognisable patterns gives itself away quickly, because a little evidence tells you which pattern you’re looking at, while one where every instance does its own thing gives you nothing. The paper covers the exact details.
Cohesion measures how much a little evidence tells you about the whole.
The link with Bayesian statistics
In the guide to complexity we saw that frequentist statistics puts every run together and reads off the average, and that Incoherence tells you when that average can’t be trusted. Bayesian statistics takes a different approach. Rather than averaging the runs, it treats each one as a possible future, starts with a belief about which future you’re heading towards, and updates that belief as evidence arrives.
This is what we can do with the climate forecast. When we have high Cohesion, waiting for a few measurements quickly rules out future states. Each observation gives you a lot of information about the future state. This is where Bayesian statistics shines. However, when you have low Cohesion, you just can’t rely on these techniques as confidently.
Bayesian statistics will tell you how to update your beliefs once the evidence has come in, but it won’t tell you beforehand whether that evidence is worth waiting for. So Incoherence tells you how much you can rely on frequentist statistics, while Cohesion tells you how much you can rely on Bayesian statistics.
Cohesion tells you how much you can expect to learn from Bayesian statistics.