← PerspectivesKareem Soliman · 18 min readAlso on Medium ↗

I Can Build Things I Don’t Understand

What happens when what you can make outruns what you can understand?

I have a growing collection of things I have built that I do not fully understand.

That sentence is more embarrassing to write than I expected.

Some of them are research experiments. Some are software systems. Some are conceptual frameworks whose individual components I can explain perfectly well until somebody asks how all of them interact, at which point there is a small but detectable decline in the structural integrity of my face.

The strange part is that I am not pretending otherwise. I know when I am standing at the edge of my comprehension; I can usually feel the ground change. There is a moment when understanding gives way to recognition. I know what the words mean. I remember why the pieces are there. I can follow the explanation when it is given to me. But could I reconstruct it myself? Could I defend every load-bearing decision? Could I tell if something subtly important were wrong?

Sometimes the honest answer is no.

And increasingly, I can build anyway.

That is new. Or new enough that I do not think we have absorbed what it means. For most of intellectual history, the frontier of what you could create sat fairly close to the frontier of what you knew how to do. You could collaborate with people who knew things you did not. You could use instruments you did not manufacture. You could rely on centuries of accumulated knowledge without re-deriving mechanics every time you crossed a bridge. We have never required total understanding before action, and civilisation would have stalled at the first bridge if we had.

Still, something about this feels different.

I can now sit in front of a machine, describe the rough shape of an idea that exceeds my technical ability, argue with it for a while, ask stupid questions without embarrassment, ask slightly less stupid questions because of the answers to the first ones, change direction completely, and emerge several hours later holding something I could not have produced alone.

This is exhilarating. It is also, if I am being honest, dangerously compatible with my personality.

I love the edge of understanding. There is a dopamine hit in getting there. A field I barely know appears on the horizon and some part of my brain responds the way a Labrador responds to somebody opening the cupboard with the tennis balls in it. We must go there immediately. Fundamentals can follow. Documentation can follow. The years of disciplined apprenticeship undertaken by people with substantially better attention spans than mine can, apparently, also follow.

This may be ADHD. It may be intellectual laziness. It may be the temperament of a generalist. It may be a useful form of interdisciplinary exploration that traditional intellectual structures are poorly designed to accommodate. I have considered the possibility that it is all four, which is inconvenient, because I would very much prefer an explanation in which everything annoying about me turns out to have been a misunderstood competitive advantage.

AI is exceptionally good at supplying those explanations if you ask nicely enough.

So I have learned to distrust them. Not because the machine is plotting to inflate my self-esteem. Usually it is doing something much less interesting: responding to the framing I supplied. Researchers call this tendency sycophancy and have measured it in the systems I use every day. If I say, perhaps my unusual way of connecting distant disciplines gives me an advantage, there is an excellent chance that a sufficiently agreeable model will discover several paragraphs of reasons why this is visionary interdisciplinary cognition. If instead I say, perhaps I am skipping the work required to earn a seat in fields I have not paid my dues to, it can make a compelling case for that too.

Both stories contain something true. That is the problem.

The higher-resolution version is rarely as flattering as the first story or as damning as the second. Maybe I genuinely am good at connecting ideas across domains. Maybe I also prefer discovery to consolidation. Maybe the same cognitive style that helps me find interesting things makes it hard to sit still long enough to understand exactly what I found. AI dramatically accelerates the first process. It does nothing automatic about the second.

I am beginning to suspect that this is my real problem. Not an ideas gap. A comprehension gap.

And once I gave it a name, I started seeing it everywhere.


Imagine that your best friend became the smartest person in the world. Not slightly smarter than you. Not annoying-at-trivia-night smarter. Absurdly smarter. They understand physics, history, biology, economics, philosophy, software engineering, psychology, law, and whichever inexplicable sequence of decisions led you to your current romantic situation.

What would you do with them? Would you hang out? Watch television together? Ask how their day was? Or would you turn the friendship into the most aggressive tutoring relationship in human history?

You would have questions. At least I would. I would become unbearable. Why does time exist? What would an alien civilisation find most peculiar about us? Which of my beliefs are embarrassingly wrong? Could you look at this research design? Could you look at this other research design? Can you explain tensor calculus without saying "tensor" until absolutely necessary?

At some point my omniscient friend would move house without telling me.

AI does not. It sits there. Waiting.

The thought experiment is imperfect, obviously. Today's systems are not universally reliable super-geniuses. Their capabilities are uneven; they reason impressively in one moment and misunderstand something embarrassingly simple in the next, and treating them as infallible authorities would be a serious mistake. Still, the analogy exposes something. When access to cognitive capability increases dramatically, curiosity starts to matter more. Two people can sit in front of the same system and extract radically different value from it because one asks three questions and the other asks three hundred.

For a while I found this enormously exciting. Intelligence was becoming abundant, and curiosity would be one of the constraints that remained. I have a lot of curiosity. This seemed like good news for me.

Then I noticed the seventeen research folders piling up behind me.

Curiosity was not the bottleneck any more. Understanding was. You can ask three hundred questions. You can receive three hundred excellent answers. You can build seventeen things from them. And then discover, rather awkwardly, that you cannot explain number four without reopening the chat.

There is a well-studied cognitive phenomenon uncomfortably close to this. People believe they understand how everyday things work in far more detail than they do. Ask somebody whether they understand a zipper, a toilet or a bicycle and they feel reasonably confident. Ask them to write out the mechanism and the confidence meets reality quite quickly. Leonid Rozenblit and Frank Keil documented this two decades ago and called it the illusion of explanatory depth. The part I love is almost offensively mundane: the illusion is punctured by the attempt to explain. There is no mystical test of wisdom. Just: okay, explain it.

That is increasingly the rule I try to impose on myself. After AI helps me construct something, I ask it what I would need to understand to claim real comprehension. Then I ask for those things to be explained. Then I explain them back. Then I ask it to find the holes. This is less exciting than generating another idea, which is probably why it is useful.

And yet there is a problem hiding underneath the practice.

Suppose the AI is wrong.

If I ask the same system that generated the work to teach me why the work is correct, I have built something epistemically suspicious: the defendant is giving evidence, acting as expert witness, and running the jury orientation program. So I push back. I ask for counterarguments. I use other models. I go looking for original sources and try to work out which claims can actually be checked by a person who is not the machine.

This matters because people do not become immune to deference simply because the authority is made of silicon. Decades of human factors research on decision-support systems, in aviation and medicine especially, describe automation bias: under some conditions people over-rely on automated recommendations and reduce the checking they would otherwise do, and workload, trust and task complexity all shape how strong the effect is. The literature is careful to say "under some conditions." I am careful to notice that my conditions, tired and out of my depth, are the ones it warns about.

Which leads to a slightly uncomfortable possibility. The better these systems become, the harder it may be to remember that disagreement is still allowed. If a model is wrong half the time, scepticism requires no courage at all. If it is right ninety-nine times, the hundredth disagreement feels different. And if you are outside your own field, you may not know there is anything to disagree with.

The danger is not merely that AI will tell us false things. It is that sufficiently reliable intelligence may slowly train us out of noticing when our own judgement has left the loop.

That is not an argument against using it. It is an argument for learning how to stay intellectually alive while doing so.


For now, experts are a powerful defence.

If an AI writes questionable code, a good software engineer may see it. If it proposes a dubious statistical method, a statistician may notice. Expertise gives people something I badly want: the ability to know when an answer merely looks like an answer.

This makes me jealous of specialists. There are things they know without remembering that they know them. Years of pattern recognition have compressed themselves into intuition, and they notice the thing I do not know enough to notice. I can ask AI what I am missing. They can feel that something is wrong before they could say why. There is intellectual sediment underneath expertise, and you cannot prompt your way into all of it. That should make anyone using AI outside their training extremely humble. It does me, at least on the good days.

But now widen the frame.

What happens if AI capability keeps improving? Not tomorrow, and not according to some breathless countdown. Just take the direction seriously for a moment. More capable systems, then more capable systems after those. At some point, certain outputs may become hard for even highly trained humans to evaluate directly. This is not science fiction; it is already a working problem in AI safety research, usually discussed under the name scalable oversight, and one proposed family of approaches uses AI systems to help humans evaluate other AI systems precisely because direct human evaluation may not always be enough.

I began with a mildly embarrassing problem about unfinished research folders. I am building things I do not fully understand. Follow the direction a few steps and the sentence changes its subject without changing its shape. Experts are evaluating things they do not fully understand. Institutions are making decisions based on systems whose reasoning cannot be independently reconstructed by the people responsible for them. And eventually, perhaps, a civilisation depends on cognitive processes that no human can fully verify.

We are very far from knowing whether that last situation will occur. I find the direction exhilarating and frightening in roughly equal measure.

Part of the reassurance is that we have met greater intelligence before. Children depend on adults. Patients depend on surgeons. Citizens depend on experts. Students depend on teachers. None of those relationships requires the less knowledgeable person to reproduce the cognition of the more knowledgeable one. We have always lived partly through trust.

Except there is a difference, and it took me a while to see it.

When I rely on a surgeon, the surgeon belongs to a human epistemic community. Other surgeons can challenge her. Her training is legible to other humans. Her claims can be contested inside institutions built around roughly human cognitive limits. Now imagine an intelligence whose relevant reasoning sits genuinely beyond those limits. Perhaps we respond by building a second AI to interpret the first: a less powerful, more comprehensible system that sits between us and the frontier model and acts as translator. That seems reasonable. Until you ask who verifies the translator. So perhaps another system verifies it. And another watches that one. At which point we have reinvented epistemic turtles all the way down, or possibly up.

The image becomes slightly ridiculous, which is probably healthy, because this subject otherwise disappears into the kind of prose where every second sentence contains the words "humanity stands at the threshold."

Underneath the joke there is a serious question, and it is the one I keep returning to.

How much understanding do we need in order to retain sovereignty?

I keep reaching for the image of a steering wheel. Perhaps humans do not need to understand the engine. Most drivers do not. We need to know where we are going, understand the controls, recognise obvious failure, and keep the ability to stop. That is reassuring for about a paragraph.

Then the metaphor starts cheating. A car does not negotiate the destination with you. It does not persuade you that the cliff is actually a road. It does not produce an elegant argument for why the brake pedal should be temporarily redefined. The better analogy might not be a vehicle at all, and I do not yet have the replacement.

A model helps until it does not. Then you have to notice where it stopped helping. The noticing may be most of the game.


I used to think I had a clean rule for all of this.

If I do not understand it, I cannot put my name on it.

It has the great virtue of being simple. It has the great flaw of being impossible. I do not understand how almost anything I use actually works. I cannot explain the semiconductor physics inside my computer. I cannot reproduce every statistical routine inside the research software I rely on from first principles. I could not build the network carrying these words to you if you gave me a warehouse, a decade and an unusually patient electrical engineer. Civilisation works because understanding is distributed and nobody understands the whole stack.

So total comprehension cannot be the standard. My beautiful simple rule lasted approximately three paragraphs, which I am choosing to count as progress.

The question becomes: what kind of understanding actually matters?

I think I am slowly arriving at something more demanding and less impossible. If I put my name on a piece of work, I should understand its load-bearing elements. I should know what the central claims are and why I believe them. I should know which assumptions would materially change the conclusion. I should understand the method well enough to recognise its important limitations. I should know where evidence ends and speculation begins. I should be able to explain why the competing interpretations were rejected, or admit that they were not.

And most of all, I should be able to defend the work without outsourcing my judgement back to the same intelligence that helped me create it.

None of this is specific to one person at a desk. A team, a journal, a regulator can ask the same questions of a system it is about to rely on, and as the scale goes up the questions get harder rather than different.

That does not mean knowing everything. It means knowing where my ignorance matters. That feels different: more realistic, and more uncomfortable, because this standard does not let me hide behind the quality of the output. Something can be brilliant and still not be mine to publish yet.

That hurts. There is a mountain of work on my computer that I am excited about. Some of it may be useful. Some may be wrong. Some may contain genuinely interesting results wrapped in methodological decisions I have not yet earned the right to defend. The tempting option is to keep creating. New ideas are clean; old ones develop administrative requirements. The frontier is exciting, and closing the comprehension gap involves reading documentation. There are very few heroic films about somebody finally understanding their dependency graph.

But there is an ethical difference between producing something and standing behind it. AI makes production cheap. Responsibility remains stubbornly expensive. I do not think that is a bug.


There is another possibility I have been reluctant to confront, because I would like it too much.

Maybe expertise itself is changing.

We tend to picture expertise as depth inside a defined body of knowledge, and for good reason. Depth matters. Years matter. Practice matters. Tacit knowledge matters; that sediment I was jealous of a few paragraphs ago is real. But what happens when cognitive tools can supply increasingly large amounts of missing depth on demand? Perhaps some forms of future expertise will involve holding less information by heart and becoming exceptionally good at identifying what must be known, asking for it, interrogating the answer, integrating across domains, detecting hidden assumptions, recognising when verification is required, and knowing where one's own comprehension turns unreliable.

That would not eliminate specialists. It might make them more valuable, because the faster everyone can generate plausible work, the more it matters to recognise what merely looks plausible. The people most entitled to be irritated by a flood of plausible generalist work are precisely the ones now needed to sort it. But it might also make room for a different kind of generalist: a person who knows how to move intelligently through what they do not know.

I would obviously like this to be true. Which is exactly why I do not trust myself to conclude it too enthusiastically. Maybe this is a genuine intellectual transition. Maybe I am constructing an elaborate philosophical defence for not reading the manual.

Both stories contain something true. Again. The uncertainty does not stop me acting. It changes how carefully I act.

We tend to talk about uncertainty as though it were standing between us and the decision: collect information, uncertainty falls, eventually the decision becomes safe. Sometimes it works like that. But many important decisions do not: the clock runs out, or the paper is due, or somebody simply has to choose. Eventually you have to press publish. Perhaps maturity is not the ability to eliminate uncertainty before acting. Perhaps it is the ability to know which uncertainty you are carrying with you when you act anyway.

I realise this sounds like a sophisticated philosophical framework for sitting on the fence. I am comfortable with that. The fence has excellent visibility.


There is a strange inversion happening in my relationship with these machines.

At first I valued the answers. Now I increasingly value being shown what I do not understand, and those are not the same service. An answer closes something. A good question opens it again at higher resolution. What assumption am I making? What would an expert challenge here? Which part of this conclusion depends on something I cannot currently explain? What have I mistaken for understanding because the explanation sounded familiar? Where am I accepting your answer because I lack the confidence to disagree?

These questions slow me down. I resent them. They may also be the most valuable thing the machine gives me. As AI becomes more capable, perhaps one of the most useful things it can do is help humans resist becoming passive in its presence. The asymmetry stays. What changes is that we can locate the edge of our own understanding precisely enough to walk toward it on purpose.

There is something almost hopeful in that.

Imagine intelligence continuing to expand around us, moving the horizon rather than replacing it. Every answer exposes another question. Every instrument reveals a smaller scale, every telescope a larger one. Every model lets us see patterns we could not see before, while reminding us that seeing a pattern and understanding the world are different things. For a moment that can make a person feel very small, and I think that is good for us.

There are trillions of microorganisms living in and on the human body, participating in systems vastly larger than any bacterium could conceptualise. I sometimes wonder, carefully, because this is a thought experiment rather than a claim, whether humans could occupy some analogous position. Not literally bacteria in a cosmic gut; that would be an unfortunate direction for theology. But structurally. Could there be levels of organisation so far beyond our cognitive horizon that asking us to comprehend the whole would be like asking a microorganism to understand a mortgage?

I have no idea. That is the point. For a second the ceiling moves and the room becomes larger.

Then we come back down. A desk. A computer. A folder full of unfinished research. One person trying to understand what he has helped create.

I still want to explore. I still want to move across disciplines faster than I probably should, and ask the absurd questions, and I still suspect there are useful things hiding in the gaps between established fields, and I remain arrogant enough, or hopeful enough, to think I might occasionally find one. But I am trying to add another discipline to the curiosity. Return. Find the part you skipped. Explain it without the machine. Fail, find out why, and try again.

There may always be a gap between what I can reach with intelligence outside myself and what I can hold inside my own head. Maybe that gap grows. Maybe all of ours will. If so, the answer cannot be to stop reaching, and it cannot be to pretend that reaching is the same thing as understanding. We will have to learn to live somewhere between the two.

I do not know what that relationship should look like yet. But I think I know what I want from it: to use intelligence I do not possess without pretending that borrowed capability has become personal understanding, and to stay curious enough to cross the boundary of what I know while staying disciplined enough to come back for the pieces of myself I left behind.

Perhaps that is all closing the comprehension gap ever means. Not reaching the point where nothing lies beyond you. Just knowing, a little more precisely each time, where you end.

And where the questions begin.

This essay emerged from Kareem Soliman's ideas, lived reflections and uncertainty, composed in collaboration with an AI that helped him explore a subject partly about the dangers of outsourcing too much understanding to AI. The load-bearing claims are his, and he has done the explaining back.

Continue the enquiry

The Reading Before the Verdict ↗What Would Change My Mind? ↗Putting a Number on It ↗How Sure Are You ↗What the Eyes Don't Tell You ↗The Organisation Reads You Back ↗