Putting a Number on It
Why I cannot stop measuring things, and what measuring does to the things
I have been trying to work out why I am obsessed with measurement.
Not in the way an accountant is, or an engineer, though I envy both. In the way a person is obsessed with a question they cannot put down. Somewhere in my twenties I stopped being able to look at anything, a claim, a feeling, a product, a treatment, a friendship, without a small voice asking how you would know. How would you know it works? How would you know it changed? How would you know you were wrong?
For a long time I assumed this was a personality defect, and it may still be. But I have come to think it is also the most useful question I own, and I want to try to explain why it feels that way from the inside, and then to be honest about where it breaks.
Start with the version of science I fell in love with, which is barely science at all.
Picture Leonardo in his notebook. He watches water move around a rock in a stream, and he draws it. He notices that the swirls repeat. He wonders whether the same shapes would appear in air, or in blood, or in hair. He builds something to test the idea, or drops something, or cuts something open, and he looks again. Then he writes down a rule about the world.
Observe. Guess. Try. Observe again. Conclude.
I find this loop almost unbearably beautiful, and I think most people who love science are secretly in love with this version of it, the one where a single careful person and a single careful eye are enough. It is the version you can do in a kitchen. It is the version where reality answers when you ask it politely.
It is also, we now know, not enough.
The problem is not the loop. The problem is the eye.
The finding that ruined the kitchen version for me is a sugar pill.
Give a person a pill that does nothing, a compressed disc of sugar, and tell them it is medicine. For a surprising range of outcomes, they will report feeling better. Pain in particular. Nausea. Fatigue. Some of the effect is expectation, some is the ritual of being cared for, some is the ordinary fact that most conditions fluctuate and people seek treatment when they feel worst. But a chunk of it is the belief itself, doing something in the body that the pill did not.
I want to be careful here, because the placebo effect is one of those findings that gets larger every time it is retold. When researchers have pooled trials that compared placebo against no treatment at all, the effect on objective outcomes, the things a blood test or a scale can see, is small or absent. It shows up most reliably in what people report feeling. That is still remarkable. It means that the honest, careful, well-meaning observation "I took the pill and I feel better" is not evidence that the pill works. It is evidence that you took a pill.
Leonardo's eye is compromised by Leonardo's hope.
So we built something to protect the observation from the observer. Take a large group. Split them at random, so that whatever else is true of them is true of both halves. Give one half the drug and the other the sugar. Let nobody, not the patient and not the doctor handing over the pill, know which is which. Measure the outcome the same way in both groups. Then ask whether the drug did better than the sugar by more than chance alone would produce.
The randomised, double-blind, placebo-controlled trial is the most boring sentence in medicine and I think it is one of the great inventions of our species, up there with writing. It is the machine that stopped us selling snake oil forever, or at least made it harder. And look at what it actually is: an admission that no eye is good enough, followed by a procedure for measuring anyway.
That admission is the whole thing. It is where measurement begins.
It is also where improvement begins, which is the part I care about most and the part people skip. If you cannot measure what you are doing, you cannot find out that it is not working. You cannot course-correct, because there is no signal telling you which way you drifted. You cannot even locate the mistake afterwards, because there is no record of what you did and what happened next. Every field that has ever got better at anything, medicine, aviation, agriculture, the training of athletes, got better by measuring itself and being unhappy with the result. Evidence-based practice is only the formal name for a refusal to fool yourself about whether you are helping. I love this stuff more than is probably reasonable, and that is the reason: measurement is how you earn the right to be wrong on purpose.
I came to this through psychology, which is the discipline of measuring what you cannot touch.
Nobody has ever seen an attitude. Nobody has weighed intelligence, or held anxiety up to the light. And yet these things move the world, so psychology spent a century building instruments for them: scales, tests, inventories, the whole apparatus of psychometrics. I did not appreciate that apparatus when I was being taught it. Reliability coefficients felt like plumbing. It took years, and a great deal of building things I had to measure later, before I understood that the plumbing was the point.
The idea that finally landed is almost embarrassingly simple. Every measurement is made of two parts: the thing you were trying to measure, and error. The score on the page is the true value plus whatever noise the instrument, the moment, the mood and the method added. You never see the two parts separately. You only ever see their sum.
Once you take that seriously, the question "what did we measure" turns into a pair of questions. How much of this number is signal? And what would it take to find out?
This is where my affection for statistics comes from, late and a little sheepish. Statistics is, underneath, the study of the error in numbers: how big it probably is, which direction it probably leans, and how confident you are entitled to be about what is left. A statistics textbook I keep returning to, by Russell Poldrack, describes what measurement lets us do in four verbs: describe reality, decide what to do, predict what comes next, and explain why. I read that as a list of everything I want from being alive.
Poldrack, as it happens, endorsed me to post a statistics methodology paper on arXiv, at a time when I understood the paper less completely than I do now. I have written elsewhere about that gap. I mention it here because it is the same gap in a different coat: a number I could produce before I could fully defend it. Measurement does not exempt the measurer.
Now widen the frame a little.
A measurement is a type of observation. That is all it is: a way of noticing, disciplined enough to be repeated and compared. And observation begins with the senses, which is where the frame starts to feel small.
You are seeing a sliver. The electromagnetic spectrum runs from radio waves longer than a building to gamma rays shorter than an atom, and the human eye responds to a band roughly one octave wide in the middle of it. Everything else, the infrared glow of your own body, the ultraviolet patterns on a flower that a bee steers by, the radio chatter of the galaxy, is happening in the same room and you cannot see any of it. Your hearing stops where a dog's continues. Your nose, compared to almost any other mammal's, is a rumour of an organ.
I find it genuinely hard to hold this in mind for more than a few seconds. Not the fact of it, which is standard, but the feeling of it: that the room I am sitting in is thick with signal I have no organ for, and always has been.
Instruments are how we cheat. A thermometer is a sense we did not evolve. A Geiger counter is an ear for a sound that would kill us. A questionnaire, if you build it well, is a sense for a thing that has no physical location at all. When I started thinking of measurement this way, as the deliberate construction of senses we were not born with, the obsession made sense to me for the first time. Every good instrument is a small extension of the human perimeter. Every good measurement is a report from slightly outside the sliver.
That image carried me a long way. Then it started cheating.
A sense does not change what it senses. My eye does not make the stream swirl differently. But measurement, it turns out, can.
Go all the way down. At the quantum scale, the act of measuring a system is an interaction with it, and the interaction leaves it in a different state than it would otherwise have been in. This is not, despite what a great deal of popular writing implies, about a mind watching. It is about a detector touching. But the lesson survives the correction: at the smallest scale we have access to, there is no such thing as looking without touching.
Now come all the way up. Study a workplace, a classroom, a person, and the people in it will know they are being studied, and knowing changes them. The famous version of this is the Hawthorne effect, named for a series of factory experiments in the 1920s in which output supposedly rose whatever the researchers changed, because the workers knew they were being watched.
I believed that story for years. Then I found out that when economists went back to the original Hawthorne data, the effect was faint, inconsistent, and partly an artefact of when the measurements happened to be taken. The canonical example of measurement changing behaviour turned out to be, at least in part, a measurement error.
I laughed out loud when I read that, and then I sat with it for a while, because it is the essay in miniature. The observation was compromised by the observer's hope, again. And the fix was the same as before: go back and measure the measurement.
The effect the story pointed at is still real; it just needed better evidence. People do behave differently when observed. And there is a stranger version of it one level up, which economists call Goodhart's law: when a measure becomes a target, it stops being a good measure. Tell a hospital it will be judged on waiting times and the waiting times will improve, sometimes by treating people faster and sometimes by redefining what counts as waiting. The number moves. The thing the number was for may not.
So the sense metaphor fails, and the failure is informative. A sense receives. A measurement participates. From the electron to the emergency department, the act of putting a number on something can move the something.
If measuring changes the measured, how do you ever know its true state?
I used to think this was a question with an answer I had not yet found. I now think it is a question that changes the job.
There is an actual discipline devoted to measurement, called metrology, which I stumbled on late and with some embarrassment, given how long I had been circling it. Metrologists are the people who decide what a kilogram is. Until 2019, a kilogram was a specific lump of platinum alloy kept under three glass domes in a vault outside Paris, and the world's mass was defined by comparison with it. It was drifting. Nobody knew whether the lump was getting lighter or the copies were getting heavier, because there was nothing more fundamental to check against. So in 2019 the kilogram was redefined in terms of a constant of nature, the Planck constant, and the lump became a historical object.
I do not know a single metrologist. I would like to. I imagine them as people who are unusually calm about being wrong by a known amount, because what strikes me about the discipline, beyond the absurd precision, is the humility built into the practice. A measurement, in metrology, is never reported as a number. It is reported as a number together with its uncertainty. The uncertainty is half the result. A weight of 1.000 kilograms with an uncertainty of ten grams and a weight of 1.000 kilograms with an uncertainty of a microgram are different facts about the world, and the second one cost more to obtain.
That, I think, is the better model, and it took me an essay to get to it.
The kitchen version says: observe, and you will know. The trial version says: observe carefully enough, and you will know despite yourself. The metrology version says something quieter. You will not know. You will have a number, an honest estimate of how wrong it might be, and an honest account of what your measuring did to the thing you measured. Then you decide anyway, with the error bars in your hand instead of hidden in your confidence.
I find that oddly freeing. The point of measurement, it turns out, was calibrated doubt all along.
Which brings me back to the small voice, the one that asks how you would know.
I think the voice was asking for the error bars. When someone tells me a treatment works, or a product is the best available, or a decision was the right one, or a feeling is true, the voice is not demanding a randomised trial before dinner. It is asking a smaller, more portable question: how much of this is signal, how would you find out, and what has claiming it already done to the thing you claimed?
Those questions are domain-agnostic in a way I find slightly vertiginous. They apply to drug trials and to marriages. They apply to research papers I submitted before I fully understood them. They apply to the productivity dashboards that quietly turn people into the numbers they are judged by. They apply to the question of whether I am, in fact, obsessed with measurement, or merely obsessed with the feeling of control that a number provides, which is a possibility I have not ruled out and probably cannot.
One last thing, the one I cannot stop noticing since I noticed it. Look up from this screen at any object in the room. You have already estimated its size. You did it before you decided to. The distance to the door, the weight of the cup, the likely temperature of the coffee, the probability that the person walking past is someone you know: your nervous system is a measuring instrument that never switches off, and it reports its results to you without error bars, as if they were facts.
So the question was never whether to measure. You are measuring right now. The question is whether you know how wrong you might be, and what your looking is doing to the thing you are looking at.
We do not pay nearly enough attention to that, and I include myself. I am still learning what my own error bars are.
This essay grew out of Kareem Soliman's long argument with himself about why numbers matter, written in collaboration with an AI that helped him check which of the stories he had been telling about measurement survived being measured. The uncertainty in the result is his.