Capability Is Not Permission

There is a meeting that happens in every AI company and it is almost never minuted.

The demo goes well. Better than the last one. Someone senior watches the system handle a case that would have broken it three months ago, and asks the obvious question: could it just do this on its own now? The room considers. The engineers, who know exactly how much better the model has got, say yes. They are telling the truth. The system can.

Nobody in that room is reckless. Nobody is cutting corners. And yet something has just happened that will not appear in any document. The authority granted to the system expanded, and the argument supporting that authority did not.

That gap is the most under-examined risk in applied machine learning, and it has nothing to do with how intelligent our machines become.

The Comfortable Ceiling

Before getting to the gap, it is worth disposing of the argument that lets people avoid looking at it.

The reassuring story goes like this. An ant cannot conceive of the third dimension. Humans may be equally blind to some higher geometry of thought. Whatever we build inherits our blindness, and so there is a ceiling on machine intelligence, and the ceiling is us. It is a beautiful argument. It is wrong in every part, and the way it fails is instructive.

The ant is not blind. Desert ants of the genus Cataglyphis forage across undulating terrain and come home on a vector computed from ground distance, the horizontal projection of a path that went up and down hills. To produce that projection, the ant must measure the slope of each segment and fold it into its odometer, which it does with roughly constant precision up to forty-five degrees of inclination (Wohlgemuth, Ronacher and Wehner, Nature, 2001). The experiments do not show that the ant holds a human concept of space. They show something better. The animal takes in the third dimension, computes with it, and throws it away, because a horizontal vector is what homing requires and the vertical component is noise.

The ant is performing dimensionality reduction. It is not the prisoner in the metaphor. It is the engineer.

The human half of the argument fails just as quickly. You cannot picture four spatial dimensions. Nobody can. But visualisation is not reasoning, and anyone who trains models manipulates embeddings of thousands of coordinates without ever forming a sensory image of one. Formalism is not decoration applied to intuition. It is the machinery by which thought leaves intuition behind.

The logic fails hardest. It assumes a creator cannot produce capacities it lacks, and no such principle exists. Evolution had no theory of cognition before it produced cognitive organisms. A compiler emits machine code no engineer would write by hand. Search, optimisation and composition routinely outrun the unaided competence of whoever set them running. This does not prove intelligence can grow without limit. It proves the narrower thing that matters: the builder's own understanding is not the natural upper bound of the artefact.

And the serious philosophical version offers even less comfort than the folk one. Cognitive closure, the claim that some truths lie permanently beyond human minds (McGinn, 1989; Nagel, 1974), does not cap machine intelligence. It implies the opposite. If truths exist that we cannot reach, then a system reaching them would be superior to us in precisely the sense the ceiling argument was invoked to deny.

The philosophy hands us no ceiling. It hands us a different problem, and the room in that meeting is standing right in the middle of it.

Behaviour is cheap to produce and expensive to justify. Everything that follows is a consequence of that asymmetry.

Five Things We Call Intelligence

The word intelligence conceals at least five properties that travel together in humans and come apart in machines. Naming them separately is the first step, because most arguments about AI safety are conducted with the five collapsed into one, which is why they go nowhere.

Capability: can the system perform the task at all?

Reliability: how consistently, and under exactly which conditions?

Legibility: can the people affected inspect, understand or contest what it did?

Self-limitation: can the system, or the workflow around it, notice when its operating assumptions have stopped holding?

Assurance: is there enough evidence, control and accountability to permit the use being proposed?

A model can have the first and none of the rest. It can produce a fluent explanation that bears no relation to the process that generated its output, which gives you legibility that is worse than none. A perfectly transparent classifier can be lethal inside a workflow that encourages automation bias. An opaque one can be entirely safe if its role is narrow, its errors are visible and its consequences can be undone.

None of this is unique to machine learning. No surgeon has verified the metallurgy of their instruments, the compiler in the imaging stack, the sterility chain or the electrical installation. Civilisation runs on decompositions. An implementation becomes an interface. An interface becomes a contract. A body of evidence becomes a certificate. An organisation becomes a set of accountable roles.

Every one of those reductions discards information, and each is safe only if what was discarded did not matter for the decision being made.

That is what compression means here. Not smaller files. A representation small enough to inspect that preserves the properties the decision actually rests on.

Compression Relocates Dependence

This is the part that is routinely missed, and it changes what you go looking for.

A benchmark compresses thousands of observations into a number, and the number is useful only if the evaluated population preserved the differences that will matter in the field. An interface hides an implementation, and the hiding is safe only if the contract captured every behaviour that can reach a system-level property. A safety invariant reduces uncountable executions to a statement that must hold, and it works only if the invariant is both true and sufficient.

In each case the complexity did not disappear. The dependence moved onto the choice of what to keep.

Which means the size of the mechanism tells you almost nothing. A billion-parameter model can be governable if its behaviour holds inside a narrow, well-characterised envelope. A three-hundred-line program can be ungovernable if small changes produce long, path-dependent consequences that nobody detects. The variable is never the amount of machinery. It is whether the grounds for trust can be made smaller and more inspectable than the range of behaviour being authorised, without compressing away the risks.

The same model, therefore, may be justified as an offline research tool and unjustified in the clinic. Justified as a recommender and not as a decision-maker. Justified under one acquisition protocol and not across every hospital with superficially similar equipment. Justified for a reversible administrative action and not for an irreversible clinical one.

There is no category called "safe AI" that exists independently of the authority being delegated. Safety is a relation between a system, a claim, a user and a consequence.

The Seduction of Formal Proof

Mathematics appears to have solved this problem, and understanding why it has not is worth the detour.

Thomas Hales proved the Kepler conjecture in 1998 by a computer-assisted exhaustion of cases that conventional peer review could not properly digest; the referees eventually declared themselves ninety-nine per cent certain, which is an extraordinary thing for a referee to have to say. The Flyspeck project then spent sixteen years producing a formal derivation, completed in 2014, checked in HOL Light and Isabelle. Millions of primitive inferences nobody will ever read, verified against a kernel of ten inference rules and three axioms.

That is one of the great compression technologies in human history. It severs the size of the argument from the size of the trusted base.

But the proof is not trustworthy merely because the kernel is small. Trust still depends on whether the formal theorem matches the intended claim, whether the definitions encode the intended objects, and whether translations between systems are sound. In his own retrospective, Hales notes that a classification theorem proved in Isabelle is translated by hand into HOL Light, where the translated statement is accepted as an unproved postulate.

Read that carefully, because it is the lesson rather than the flaw.

The formal machinery did not eliminate human stitching. It made the stitching visible.

That is the transferable idea, and it is not the one people usually take. The goal was never to remove trust. It was to concentrate trust into a boundary narrow enough that a person can look at it and own it.

And formal proof achieves that for one reason above all others. There is a theorem. A statement of what correctness is, against which the derivation can be checked.

The Theorem We Do Not Have

A clinical perception system has specifications, and they are real ones. Intended purpose, target population, acquisition protocol, defined users, thresholds, contraindications, risk controls. Much of the surrounding machinery admits formal treatment: preprocessing constraints, state transitions, access control, alarm logic, update mechanisms, fallback behaviour.

What it does not have is a semantic specification of correct behaviour across the deployment environment. Nobody can enumerate the correct output for every clinically possible observation, including every rare presentation, every equipment fault, every workflow deviation and every interaction between them. Nobody in this field knows how to write down what correctly identifies this instrument means as a formal object.

So we train against samples, labels and proxy losses. We validate on selected distributions. And then the system meets a world whose relevant variation was never enumerable in advance.

That is not a defect in empirical science. It is what empirical knowledge is.

The failure begins the moment performance over a sampled distribution is quietly promoted into justification over an open environment. A test set estimates behaviour under conditions the test set represents. It cannot, by itself, tell you where those conditions end.

"We tested it extensively" is evidence.

It is not an assurance case. An assurance case connects the evidence to the authority: which assumptions carry the connection, which hazards remain, how departures will be detected, who intervenes, and what happens when an assumption turns out to be false.

What Medicine Actually Did

Someone always raises the drugs, and they are right to.

General anaesthesia was clinically indispensable long before its molecular action was understood, and parts of that story remain contested. Plenty of approved compounds work through chains of biological effect that nobody has fully traced. If pharmacology never required a mechanistic account, why should a model?

It should not, and the demand for mechanism was always the wrong demand. But look at what medicine did instead, because it is the whole game.

Medicine does not trust a drug because it understands it. It trusts a drug because it has organised the uncertainty: dosage, indication, contraindication, controlled trials, manufacturing controls, adverse-event reporting, pharmacovigilance running for decades after approval. The mechanism stayed dark. The envelope got built anyway.

So the real question is not whether machine learning deserves an exemption from explanation. It is which of medicine's instruments for bounding uncertainty transfer, which break, and what has to be invented.

Several break in interesting ways. Software replicates identical decision logic across a thousand sites in a week, so a single defect produces correlated failure at a speed no molecule can match. Performance can depend on acquisition conditions that change without anything altering at the interface. Ground truth often arrives late, or never, so degradation stays silent. Models get updated, changing their failure modes while the product function looks unchanged.

And one failure has no pharmacological analogue at all.

The Model Writes Its Own Evidence

Once a decision system enters a workflow, it changes the process that generates its future data.

A triage model changes which patients get attention. A diagnostic model changes which tests are ordered. A surgical guidance system changes where the operator looks, what they verify, and therefore which of its own errors remain observable at all.

The model is no longer predicting inside a distribution. It is participating in the production of that distribution.

The consequences are worse than drift, and they are specific. If clinicians intervene whenever the system looks uncertain, weak autonomous performance is never measured, because the failures were caught before they became data. If clinicians over-trust confident outputs, the contradictory evidence that would have exposed a mistake is never gathered. If the system changes case allocation, comparison against historical data stops estimating the same process. And if its outputs influence the labels later used for retraining, then apparent improvement partly reflects the model teaching the data to agree with itself.

Success hides failure. The better the tool appears to perform, the fewer of its errors survive long enough to become evidence.

A molecule does not reorganise the clinic around itself. Your model does. Which is why the unit of safety is never the model. It is the coupled system of model, users, workflow, data generation and institution, and any assurance argument scoped to the model alone is measuring the wrong object.

The Capability-Permission Error

Now back to that meeting.

The error is not that anyone lied about the model. The error is that performance was treated as evidence about scope, and it is not. Performance tells you what happened under evaluated conditions. It tells you nothing, by itself, about where those conditions end, whether the system will notice the boundary, whether users will catch the failure, how the error propagates, how many people are exposed, whether the outcome can be contested, or whether it can be undone.

Capability and permission rise together in this industry as though the first entailed the second. It does not. And once you see the two as separate axes, the discipline becomes almost embarrassingly simple to state.

Permission must track three things, not one:

The strength of the assurance case.
The authority delegated to the system.
The severity, detectability and reversibility of failure.

Which gives you an actual ladder, and every rung has a price.

Authority granted	What it means	What it costs you in assurance
Offline	Research, retrospective analysis, no patient touched	Reproducibility, data provenance, honest reporting
Advisory	Output shown; human decides independently	Subgroup performance, calibration, known failure modes, evidence the human is not anchored
Workflow-embedded	Output shapes what the human sees or does next	All of the above, plus automation-bias testing, detectability of silent error, monitoring of the coupled system
Autonomous, reversible	System acts; the action can be undone	All of the above, plus abstention behaviour, drift detection with a latency shorter than the harm, defined rollback
Autonomous, irreversible	System acts; the action cannot be undone	All of the above, plus a specification of correct behaviour, or a physical or procedural constraint that makes the missing specification unnecessary

Read the last row again. The requirement for irreversible autonomy is a specification we have already established we do not have. That is not a counsel of despair. It is a statement that irreversible autonomous authority must be bought with engineered constraint, not with benchmark scores, and if you cannot buy it, you do not get it.

A weak assurance case may be entirely adequate for a reversible recommendation under mandatory expert review. The same evidence is worthless for an irreversible act.

When capability grows faster than assurance, authority must stop growing. That is not hostility to innovation. It is the difference between evidence that a system can act and evidence that it should be allowed to.

The Regulator Already Understood This

The FDA's guidance on predetermined change control plans for AI-enabled devices asks a manufacturer to describe the modifications it intends to make, the methods by which they will be developed and validated, and an assessment of their impact. Where an authorised plan covers a change, it can ship without a separate marketing submission. The legal authority for device change-control plans comes from statute; the AI-specific document is non-binding guidance setting out the agency's current thinking.

The procedure is not the interesting part. The shape of the permission is.

The regulator does not authorise a model. It authorises an envelope: what may change, how change will be controlled, how it will be validated, how its effects will be assessed. It cannot inspect every future model state, so it demands a smaller argument intended to hold across a declared class of them.

That is a compressed assurance case, written into regulatory practice by people who had to solve the problem rather than describe it. The related international work on good machine learning practice points the same way, emphasising the performance of the human-AI team and post-deployment monitoring, which is to say: the device is a sociotechnical system, not a model.

Regulators arrived here first. That should be uncomfortable for anyone who believes this industry is ahead of its overseers.

When the Explanation Will Not Fold

Sooner or later every hard system poses the same question, and there is no clean way to answer it.

Have we failed to find the right abstraction, or is the behaviour resistant to the kind of abstraction the decision requires?

From the inside, the two look identical. Both feel like confusion. Both produce failed explanations. Both invite a conclusion nobody has earned. Declare incompressibility too early and opacity becomes a licence to stop thinking; the heavens looked irreducibly complicated until Kepler replaced epicycles with ellipses. Assume indefinitely that a satisfying account is one research programme away and opacity becomes a promissory note that ships anyway.

There are probes. Causal intervention, representation analysis, formalisation attempts, adversarial search, scaling experiments. None of them can prove that no better abstraction exists.

So stop pretending the judgement is technical, and make it institutional. Record which forms of abstraction were attempted, which safety-relevant claims remain unsupported, whether further interpretability work is expected to change the deployment decision, which compensating controls carry the residual burden, who owns the judgement, and when it must be revisited.

Uncertainty about understanding is still uncertainty. Give it evidence, an owner, review criteria and an expiry date, exactly as you would any other risk on the register.

The Receipt

An assurance case is not a story generated after the model has spoken. It is a chain of claims whose dependencies can be inspected, and every link is a place where it can break.

The system was designed for this purpose. The evidence supports this population and this environment. Acquisition remains inside the evaluated range. Known hazards have controls. The system abstains or escalates outside its limits. The human role is specified and tested rather than assumed. Updates are governed. Failures are observable. Authority is bounded. Consequences are reversible where they can be. Responsibility has a name attached.

Each of those compresses a body of evidence, and each can be false. A serious case says where demonstration ends and judgement begins, separates what was measured from what was assumed, and states the conditions under which its own argument stops holding.

It does not prove that failure is impossible. It proves that permission was not granted vaguely.

Permission Is the Scarce Resource

There may be truths we cannot reach, reasoning we cannot follow, systems whose inner life never compresses into anything a person finds satisfying. None of that establishes a ceiling on what can be built. It establishes a constraint on what can be responsibly authorised.

We can generate through search what we cannot derive through insight. We can verify through machinery what we cannot replay in consciousness. We can coordinate through institutions what no single mind can hold. But every step away from direct understanding creates a debt, and the debt is paid in specification, evidence, containment, monitoring, reversibility and named accountability. Where the payment is good, opacity is not disqualifying. Where it is not, performance is not a substitute, however good the demo was.

Go back to the room. The system had got better; everyone could see it. What nobody asked was the only question that mattered: what changed in our argument?

Nothing had. The model improved. The justification stood still. And the authority moved anyway.

Capability can scale faster than justification.

Permission must not.