← PerspectivesKareem Soliman · 9 min read

What would change my mind?

Where could I be wrong?

I have tried to make that question habitual, almost to the point of religiously questioning my own conclusions. The difficulty is that a question can become a ritual long before it becomes a useful method. It is possible to ask where you could be wrong, feel suitably rigorous for having asked, and carry on believing exactly what you believed before.

I want the question to cost me something more than a sentence.

What are the conditions under which the thing I have just thought (or worse, said out loud) would be false? What observation would surprise me? If I genuinely wanted to discover what was happening, how would I design an experiment whose answer I could rely on?

This is what drew me towards research design and the relationships between variables. I wanted a way to put an interpretation in contact with something that could resist it.

The desire sounds straightforward until you try to specify what would count as resistance.


Suppose a team starts using a new tool and completes more work the following month. This is a hypothetical example, with an appealingly convenient result.

The tool helped. That is one explanation.

Perhaps the new work was easier. Perhaps the team was also given clearer instructions. Perhaps the previous month was unusually bad. Or perhaps completion has become easier to record because work that used to be checked has quietly stopped being checked.

A second month of higher output might be encouraging. It does not automatically distinguish these explanations. If the easier work continues, so can the apparent benefit.

This is an important irritation in research: collecting more observations and collecting more informative observations are different activities. I can become increasingly precise about a relationship I have still misunderstood.

The experiment I want would arrange things so that the competing accounts have different consequences. If the tool makes the difference, what should happen when comparable work is performed with and without it? How will we decide what counts as comparable? If speed rises because checking falls, what would we need to inspect alongside completion?

Every answer introduces another condition. The question becomes more demanding just as it becomes more useful.

At some point, my original wish for an experiment that would definitively settle the matter needs revising. Random assignment can improve what a comparison supports, but it cannot make an unrepresentative task become representative. A carefully chosen outcome can still miss a consequence that matters. An effect in this team need not survive a different team using the same tool.

I can design a stronger test. I cannot design away the need to understand what was tested.


There is another complication waiting inside the apparently solid parts of the example: the measurements themselves.

“Work completed” looks like an observation. But completed according to which rule? At what quality? With how much rework transferred to someone else? Before an analysis begins, there has already been a series of decisions about what the number will be allowed to stand for.

I find the same problem fascinating in less interpretive settings. Even where a quantity has a clear definition, the observations can be imperfect. Instruments vary. Conditions drift. Measurement uncertainty can become entangled with a relationship we attribute to the world.

Probability Cloud Regression grew into one way of investigating this. The statistical idea is to represent uncertainty around observations when estimating a relationship, rather than asking every measured value to behave as though it were known exactly. It belongs to the broader family of errors-in-variables methods.

The word cloud is useful if it reminds us that a point on a graph may conceal a range of plausible values. It becomes misleading if it suggests that every uncertainty can be described by a convenient probability distribution. We still need grounds for how wide that cloud is, what shape it takes, and whether uncertainties in different observations are related.

An elegant representation of uncertainty can itself be confidently wrong.

In a public preprint using ABIDE-I data, I examined the relationship between IQ and head motion across 935 participants at 19 scanning sites. The ordinary regression slope was 4.67 times the estimate produced under the study's measurement-error model. But the uncertainty assumptions matter, including a site-level proxy for response-side uncertainty. The pooled predictor performed worse on held-out sites than predicting each test site's mean. The study does not establish downstream benefits for brain-imaging analysis, and it remains a preprint. The ABIDE-I analysis.

What interests me about that result is the question it makes harder to avoid: how much of the relationship we think we are reading belongs to the process that produced the measurements?

The same observations can support a different estimate when the measurement assumptions change. That does not give me permission to choose whichever estimate I prefer. It gives me an obligation to examine those assumptions and show how the conclusion depends on them.


Rasad is the wider research system around this work. Its interest extends to what an initial model leaves unexplained.

A residual is the difference between an observation and what the model predicted. That difference can contain random variation. It can also contain structure: observations from one site consistently falling to one side, a curve left behind by a straight-line model, or a pattern that appears only under a particular condition.

Calling something error does not make it uninformative. Nor does finding a pattern establish its explanation.

This is why I think of variance forensics as an investigation. A recurring discrepancy provides somewhere to look. It might point towards a measurement problem, a missing variable or a model that has misunderstood the relationship. Those possibilities can demand quite different responses.

Correcting the wrong one could make the analysis look cleaner while making its account of the world worse.

The possibility of bringing several measurement modalities together adds another layer. EEG and fMRI, for example, provide different kinds of observations of brain activity. The interesting question is what they allow us to constrain together, and what their disagreement might reveal.

Agreement helps only to the extent that we understand how it was produced. Two readings can agree because they are informative. They can also share a source of distortion. Triangulation deserves more than the comforting arithmetic of two instruments agreeing with one another.

This is the aspiration behind finding structure in the noise. It carries no promise that every unexplained variation contains a discovery. Sometimes a more complicated model is an elaborate way to describe an accident. Sometimes the simpler model should win.

The research has to leave that outcome available.


One simple account of inquiry has us explore first and exploit afterwards. We look around, find something useful, and begin using it.

It is a reasonable starting picture. I am interested in whether it becomes too rigid when it turns searching and committing into two separate states that must occur in sequence.

We can act on an explanation while keeping alternatives under examination. We can concentrate our effort without closing the question entirely. The balance can change repeatedly as conditions change. Perhaps some forms of inquiry are better described as movement across a spectrum, with exploration and exploitation at opposite ends, than as a single switch between them.

That is a representation worth investigating. It is not, by itself, evidence that an oscillating controller makes better decisions.

Earlier exploratory work helped make those questions concrete for me. It also made the empirical obligation harder to evade. In one internal evaluation, a fixed baseline performed better than the adaptive approach. A satisfying account of exploration did not produce the hoped-for advantage under that test.

The result need not settle every question about adaptive inquiry. It does remove the right to present that test as confirmation.

Aurora carries a related interest in what happens when the possibilities themselves change. An inquiry can produce a hypothesis that was not in the original list. A useful explanation can split into two when a new distinction becomes visible. Bayesian methods already have ways to compare and expand models; my dissatisfaction is with treating optimal reasoning inside a specified model as a sufficient account of the whole activity.

Choosing how to describe the problem is part of the problem.

I want to know when explicitly managing that changing set of possibilities improves a decision enough to justify the additional work. The answer is allowed to be “not here”. Otherwise I have built the conclusion into the question.


This is where the work connects back to firāsa for me.

An interpretation often begins before it can be fully articulated. A pattern attracts attention. Something about an explanation feels incomplete. A relationship between ideas becomes visible before I know whether it will survive careful examination.

I do not want to eliminate that movement. It gives inquiry somewhere to begin.

Measurement asks what happens next. What observations support the reading? What alternatives remain? What would distinguish them? How much of the apparent pattern depends on the instrument we used to see it?

Firasa's code-quality research asks versions of those questions about software. A piece of code can evoke an experienced engineering judgement whose reasons are worth making inspectable. ConstructLens asks whether psychological measurement can retain more of a person's own language without pretending that richer language automatically produces a better measure. The ambition in both cases needs evidence that the instrument measures something useful under the conditions in which it will be used.

Lemma brings the obligation back to my own understanding. A regression can be a geometric projection, a statistical estimate and a Python computation. Following the bridges between those views gives me different ways to find out what I have merely recognised and what I can actually explain.

That matters when AI helps me build things. A working analysis can make it very tempting to accept the explanation that arrived beside it.

These questions connect the projects without requiring them to share one method.


There is a danger in making uncertainty central to an identity: I could become very good at explaining why another decision should wait.

Something still has to happen. Time is finite. A measurement takes effort, sometimes from a person who has better uses for it. Another study can reduce one uncertainty while leaving the decision unchanged.

The rigour I want is proportionate to the consequences. If an error would be difficult to reverse or could seriously affect someone, more evidence may be necessary before acting. For a small, reversible choice, a provisional decision and a way to learn from it may be enough.

Neither posture is universally more scientific. The question is what the evidence needs to do here.

I want measures useful enough to justify their burden. That means I cannot judge an instrument only by how much information it produces. I also have to ask what changed because the information was available.

Where could I be wrong?

If the answer cannot alter what I look for, how I interpret it or what I do next, I have probably performed doubt rather than used it.

Before the next analysis, I want to be able to finish a smaller sentence:

If I observed this, I would have to reconsider.


This essay was composed in collaboration with AI from Kareem Soliman's research questions, project records and account of his relationship with uncertainty. The AI helped connect the arguments and challenge claims, including claims about AI-assisted research. Kareem remains responsible for understanding the evidence, owning the judgements and deciding whether the essay is ready to publish.

Continue the enquiry

The Reading Before the Verdict ↗Putting a Number on It ↗I Can Build Things I Don’t Understand ↗How Sure Are You ↗What the Eyes Don't Tell You ↗The Organisation Reads You Back ↗