How Sure Are You
The third gap: between how right you feel and how right you are
I find it fascinating, and a little alarming, that we gravitate towards confident people.
Not competent people. Confident ones. The person in the meeting who has eliminated their own uncertainty, who leaves no room for doubt in their voice or their posture or their plan. We call them charismatic. We call them leaders. We choose them to run things, and when we are choosing between a candidate who says "here is what we should do" and a candidate who says "here are three options and, honestly, I don't know which is best," we know before the sentence is finished which one we are going to pick.
I know this because I have been both candidates, and I have watched which one got hired.
I have written before about two gaps: the one between what I can build and what I understand, and the one between what I can measure and what I know. This is the third, and I suspect it sits underneath the other two. It is the gap between how sure I feel and how right I am. Everything I want to say here is a way of asking why those two things are so poorly connected, and why we keep pretending they are the same.
Start with the part that is not a matter of opinion.
For about seventy years, psychologists have been running a very rude experiment. Take a domain where experts make predictions: whether a patient will relapse, whether a parolee will reoffend, whether a student will finish their degree. Give the expert everything they normally use, the file, the interview, the years of intuition. Then take a very simple statistical rule, a handful of variables with weights attached, and let it make the same prediction. Compare.
The rule wins. Not always, and not by miracles, but reliably, across dozens of domains, in study after study since Paul Meehl first collected the evidence in the 1950s. The expert has more information. The expert has more experience. The expert loses, or at best draws, to an equation you could fit on an index card.
A generation later, Philip Tetlock spent two decades collecting forecasts from political and economic experts, thousands of predictions about elections, wars and markets, and scored them against what actually happened. The famous line from that work is that the average expert did about as well as a dart-throwing chimpanzee. The less famous and more interesting finding is who did better. The forecasters who beat the field were not the ones with the most expertise. They were the ones who thought in probabilities, who started from base rates, who updated in small steps when new information arrived, and who were comfortable saying "sixty percent" out loud in a room full of people who wanted to hear "yes."
I read those studies the way some people read scripture, and I want to be honest about why. They flatter me. I am a natural fence-sitter, and here was evidence that fence-sitters see further. So let me take the flattery seriously for a moment and then put it down.
The flattering reading gets one thing wrong straight away: the confident person is not stupid. Confidence is what a fast, cheap, usually-right cognitive system feels like from the inside. Psychologists have a name for us, cognitive misers: we spend as little thought as we can get away with, reach conclusions quickly, and take the shortcut whenever the shortcut delivers roughly the right answer. Most of the time it does. If a shortcut is right ninety-nine times in a hundred, a mind that insists on the long way every time is not wiser. It is slower, and slowness has costs.
There is a story about why we are built this way, and I like it, so I should tell you it is a story. The story says that thinking is expensive, and that our ancestors could not afford to spend cognition on a decision when a predator might need all of it in the next ten seconds, so the mind that conserved its effort outbred the mind that deliberated. It is a plausible story. It is also exactly the kind of story that is easy to tell and hard to test, and evolutionary explanations for cognitive habits have a long record of being more satisfying than they are supported. I hold it lightly. The miserliness is well documented. The reason for it is a hypothesis wearing a good suit.
The shortcuts miss, though, and the misses are documented rather than hypothesised. We have catalogued them. Confirmation bias: I go looking for evidence that I was right. Primacy and recency: the first thing I heard and the last thing I heard both weigh more than they should. The fundamental attribution error: your lateness is your character, mine is the traffic. There are dozens of these, and they are not random noise. They are predictable. You can name the direction I will be wrong in before I am wrong in it.
Predictable, in theory, means avoidable.
Wrong again.
This is the part I find most humbling. Learning about a bias does very little to protect you from it. People who can explain the bias blind spot, the tendency to see biases in others more readily than in yourself, still exhibit the bias blind spot. Knowledge of the flaw and freedom from the flaw are different things, and the distance between them is roughly the distance between reading about swimming and swimming. I have known the names of my biases for more than a decade and I am about as biased as I was. What has improved is that I can now tell you, afterwards, which one I just committed.
None of us is exempt, and this is the point about the confident person: they are exactly as prone to error as the rest of us. What they have eliminated is the signal that would have told them to check.
So why do we trust our own thinking?
I do not have a good answer. I have a description. Most people, most of the time, treat the contents of their own head as evidence. A thought shows up and we assume it earned its place. Why else would it be there? We rarely apply to our own conclusions the scepticism we would apply to a stranger's, even though our conclusions were produced by the same cheap, fast, shortcut-taking machinery we have just finished cataloguing the faults of.
I think part of it is that doubt is expensive twice. It costs effort in the moment, and it costs something afterwards, when you look back at a decision made under uncertainty and have to live with the fact that you knew it was uncertain when you made it. Confidence protects you from regret in advance. If I was sure, I could not have done otherwise. If I was unsure and chose anyway, I am exposed.
And part of it is that confidence works. This is the uncomfortable half of the essay and I am not going to skip it.
Fake it till you make it. I have heard that advice, I have applied it, and it has worked for me more than once. I have walked into rooms where I had no idea what I was doing and acted as if I did, and the room went along, and sometimes the acting turned into doing. I am confessing a strategy rather than a flaw, because it pays. For a while I thought it paid because acting confident somehow made me more competent. It pays because of how the room responds.
There is a line of research on this that I found after the fact, which is its own small lesson. When people display confidence in a group, they are given more status by the group, and, here is the part that stings, they keep that status even when their confidence turns out to have been unjustified. The group rewards the display. It does not, or not much, punish the miss. A false sense of confidence, seen this way, is a rational response to a social environment that pays for it.
So the conditioning is real, and it is not only in our heads. We have built rooms that reward the elimination of doubt. Then we are surprised that the people who rise in those rooms have eliminated their doubt.
I want to pause here on the flip side, because it is the thing I actually find dumbfounding.
The number of times I have had absolutely no idea what I was doing and stumbled into something genuinely good, and then had no idea what to do next, is embarrassing. I did not fake confidence into those. I fell into them. I look back at some of the most consequential turns in my life and I cannot construct the story where I chose them, only the story where I was standing in the right place when they happened, indecisive on one side and falsely confident on the other, and the two somehow cancelled into luck.
I do not think that is an argument for anything. I think it is a fact about how much of the outcome was never mine to decide, and I think a person who has looked at that fact honestly finds it a little harder to believe the confident story about anyone, including themselves.
So what is the appropriate response? I have been circling it for two thousand words and it is time to say what I think.
The obvious answers are both wrong, and they are wrong in the same way. One says: be more confident, you have a right to be, doubt is weakness. The other says: your thinking is riddled with bias, distrust it. The first ignores the misses. The second ignores that the shortcuts are usually right, and that a person who distrusts everything they think has not become wiser, only paralysed.
The mistake they share is treating confidence as a dial. Turn it up, turn it down, find the right setting. Confidence is not a dial. Or rather, it is, but the dial is not attached to anything that matters. It is a volume knob. You can turn a song up or down and it stays the same song, right or wrong. Loudness tells you nothing about whether the tune is correct. And most of the advice we give each other about uncertainty is advice about volume.
The thing that matters is a different axis entirely, and the forecasting research has a word for it. Calibration. A calibrated person is one whose seventy percent means seventy percent: when they say they are seventy percent sure, they are right about seventy percent of the time. Not more, which would mean they are underconfident and leaving accuracy on the table. Not less, which is the usual failure. Calibration is the connection between the feeling and the frequency, and it says nothing about whether you should feel sure.
Once you see it this way, the confident person and the doubter fail the same test. Neither of them knows how often they are right when they feel the way they feel. One has the knob turned up, one has it turned down, and neither has ever checked the tuning.
This is also where the volume metaphor stops working, and I want to say so rather than quietly change the subject. A song is right or wrong independent of the knob. A decision under uncertainty is not; how sure you are changes what you do, and what you do changes what happens. Calibration is more than a private virtue, more than getting your internal numbers right. It is a claim you make on other people, every time you speak with confidence you have not earned. The metaphor got me to the axis and then it has to be put down.
Calibration has a practice, and the practice is unglamorous, which is probably why it works.
You say a number. Before the decision, not after. Not "I think this will work" but "I think there is a sixty percent chance this will work." Then you find out. Then you look at all your sixties together and ask whether they came true about sixty percent of the time. If they came true ninety percent of the time, you were sandbagging. If forty, you were doing what most of us do. Very few people keep this ledger, and very few organisations do either; the ones that do tend to be the ones with money riding on the answer, which tells you something about how the rest of us are paid. The forecasters who beat the experts had done this, thousands of times, and their feelings of sureness had slowly been dragged into contact with the world.
It is humbling to try, mostly because of how rarely the number is anything other than uncomfortable. Sixty percent is an admission. Ninety percent is a commitment you will have to live up to. "I don't know" is the hardest of all, because it is the one the room punishes.
And this, I think, is what it comes down to. Under the statistics and the biases and the forecasting tournaments there is a small, ordinary act: being able to say "I don't know" in a room that wants an answer.
I do not have children, and there are days I do not envy the people who do, because I imagine the number of times a parent is permitted to say "I don't know" is a fraction of the number of times I am. Children expect answers. Their curiosity is endless and their tolerance for a shrug is low, and a parent learns, presumably, to produce confidence on demand, because the alternative is a small person losing faith in the only authority they have. I sometimes think that is where most of us learned it. Not from the boardroom. From being four, and asking, and being answered.
I will end with a decision made under uncertainty, since that is what the essay claimed to be about, and it would be strange to write three thousand words on calibration without offering a number.
I think this essay is right about most of what it is confident about. The prediction studies are solid; the bias literature is solid; the status finding is replicated, though less thoroughly than I would like. The evolutionary story I have told you is a story. The claim that we learned confidence at four is a guess I find charming, which is a warning sign. If I had to put a number on the whole piece being broadly correct, I would say seventy percent, and I would want you to hold me to it.
Seventy percent is the size of the gap I can see between how sure I feel and how right I am. It is the most honest thing I know how to say about a set of beliefs I have to act on anyway. I am publishing it. I am not sure. Those two facts are supposed to be able to coexist, and the entire argument of this essay is that we have built a world in which they mostly cannot.
Maybe it is time we examined the decisions we are making under uncertainty a little more closely, and got honest with ourselves about how we know what we know, and how much of it we can lean on. I am not sure we will. Call it forty percent.
This essay came out of Kareem Soliman's long-running argument with his own confidence, written in collaboration with an AI whose job included telling him which of his favourite claims about experts, evolution and confidence the evidence would not carry. The seventy percent is his.