A recap on complexity
In January 1885 a young farmer in Jericho, Vermont, called Wilson Bentley fixed a camera to a microscope and took the first photograph of a single snowflake. He spent every winter for the rest of his life doing the same, catching flakes on a black board and photographing them before they melted, until he had more than 5,000. He never found two that were alike. And no-one ever will.
Water vapour in clouds freezes onto dust and existing ice, forming a flake. Wherever the flake sticks out it catches more vapour, so that bit grows faster and sticks out further. A bump becomes an arm, and the arm sprouts branches. The shape of a snowflake decides how it grows, which of course is its shape. A compounding feedback loop like this takes the smallest differences and magnifies them.
Feedback like this is a classic non-linear interaction seen in complex systems. Broadly there are systems like diamonds, where the carbon atoms organise themselves into exactly the same pattern every time. This means that if you were to average every diamond together, you’d also get another perfect low-entropy diamond. Then you’ve got systems like water at the other extreme, where all the atoms slosh about with no particular pattern. But what’s interesting is that when you average those systems together you also get another high-entropy pattern of water sloshing around.
However, if you lay twenty-four snowflakes on top of each other the details that made each flake unique begin to blur away. The “average snowflake” looks like one that’s started to melt. Each individual snowflake is crisp and has a lowish entropy, while their average has a much higher entropy. We see this difference as a hallmark of a complex system.
The question we’re exploring today is: even though every single one of Bentley’s photographs is unique, why are they all instantly recognisable as a snowflake? And how do we quantify that recognisableness?
What is self-organisation?
Snowflakes are identifiable because they share a great deal.
The first is inside each flake: every one has six arms, and each arm is symmetrical down its middle. So the same pattern repeats twelve times around the flake. There are six arms because V-shaped water molecules lock together into a hexagonal lattice, and the arms match because the electromagnetic forces propagate across the whole surface.
The thing to note here is a snowflake always organises itself to have six arms. The system itself does that. There are no external forces, no fairy or god crafting each one, just the snowflake’s existing shape and physics working together. This is what defines the “self” in self-organisation.
However, snowflakes also self-organise a second way. In the 1930s the physicist Ukichiro Nakaya grew the first artificial snowflakes in a cold room in Hokkaido, and found that their shape depends on the temperature and humidity they grow in. Just below freezing they grow as thin six-sided plates, at around minus 5 degrees they grow as columns and needles, and at around minus 15 degrees they come back as the iconic stars.
No two plates are alike, but any plate is far more like another plate than like a column or a star. Their chaotic system is attracted to a few totally discrete outcomes. But only a few, not an infinite array of different shapes.
This is why people find self-organisation (sometimes called emergence) so magical. A murmuration is always identifiable as a murmuration, yet each one is constantly unique. It’s a recognition of nature finding a balance in a way that feels alive. Each part gets enough freedom to evolve and to adapt, while the whole still keeps a literal sense of self, remaining a murmuration rather than just a bunch of birds.
Turing patterns are another example, where a single algorithm stabilises into either leopard spots or zebra stripes. We see it in galaxy formation. We see it when systems form forests or prairies.
Given enough time, high entropy systems like monkeys on typewriters will always create something meaningful, like burnt toast depicting the face of Jesus. These are inevitable statistical anomalies. What’s extraordinary about self-organising systems is that they create low entropy, identifiably unique patterns nearly every time.
Why do we need a measure?
Say you’re in charge of preparing the UK for climate change. You have a fixed budget and a long list of things it could pay for. Whatever you build will take decades, so you commission a climate model.
Most runs come back as you’d expect, with the UK’s daily temperatures 7 to 9 degrees warmer than today. Some runs show something else. The UK is only as mild as it is because a system of Atlantic currents carries warm water up from the tropics, and without it London (which is about as far north as Calgary) would have Canadian winters. In these runs fresh water from melting Greenland ice slows the currents until they stop circulating altogether, and the UK ends up 7 to 10 degrees colder than today.
This is a tough situation. You now have to split your budget between two very different scenarios (like the train example). But in a way this is good news. Like the spots and stripes, the runs fall into two clear groups, as you only have to plan for two scenarios.
Things could be much worse, because your model could have come back with this instead:
Here we see that when results self-organise it dramatically changes our decision making ability when it comes to complex systems.
Knowing how self-organised a system is dramatically alters how we make decisions about it.
How do we measure it?
Incoherence tells us whether the runs differ. What we want to measure here is whether those differences follow a pattern.
One way to do this is to take each run of the climate model and plot its average temperature as a dot on a map. We then draw a line between every pair of dots, and on the right we pile up how long those lines are. If all the runs are alike, the dots sit close together and every line is short. If the runs fall into two clusters, you get short lines inside each cluster and long lines between them. We call these intra-cluster and inter-cluster lines. If there are no clusters at all, the lines come in every length from short to long.
The trick is to notice that the bars go from spiky to flat as the runs become less organised. When the lines come in only a few lengths, you can pick any pair of runs and make a good guess at how far apart they are. Being able to predict something well is what low entropy means. So if you measure the entropy of the line lengths, you are measuring how self-organised the system is.
It doesn’t matter how far apart the clusters are, since that’s what Incoherence measures. What matters is this: does every run clearly belong to a cluster?
This works when a system organises itself into separate clusters, like the plates, columns and stars we saw above. It also works when there’s organisation within a system, like the six arms of a snowflake. That kind of order limits how different two runs can be, so the line lengths bunch together.
This is a simplified picture. In practice we don’t reduce a run to a dot or an average. We compare the whole spread of temperatures in each pair of runs, and we estimate the entropy directly rather than by counting lines into bars. The idea is the same, and the paper has the full details.
Systems like diamonds and water have a Cohesion of 1, because they turn out the same every time. For the same reason they also have an Incoherence of 0. Measuring self-organisation through Cohesion only makes sense when a system is complex, and therefore incoherent.
The link with Bayesian statistics
In the guide to complexity we saw that frequentist statistics puts every run together and reads off the average, and that Incoherence tells you when that average can’t be trusted. Bayesian statistics takes a different approach. Rather than averaging the runs, it treats each one as a possible future, starts with a belief about which future you’re heading towards, and updates that belief as evidence arrives.
This is what we can do with the climate forecast. When we have high Cohesion, waiting for a few measurements quickly rules out future states. Each observation gives you a lot of information about the future state. This is where Bayesian statistics shines. However, when you have low Cohesion, you just can’t rely on these techniques as confidently.
Say that ten years from now you take three readings of the UK’s daily temperature, and they come back at 14, 18 and 21 degrees. In the first forecast those readings rule out the whole colder group at once. In the second they rule out only the six coldest runs, and the ones left still stretch from 6 to 22 degrees.
Bayesian statistics will tell you how to update your beliefs once the evidence has come in, but it won’t tell you beforehand whether that evidence is worth waiting for. So Incoherence tells you how much you can rely on frequentist statistics, while Cohesion tells you how much you can rely on Bayesian statistics.
Cohesion tells you how much you can expect to learn from Bayesian statistics.