The AI Model Is Not the Safety System

Working Is a Claim About a Sample

A surgical tray arrives for inspection. Its bill of materials matches the trays the system was validated on, with the same instruments in the same quantities. But it is a newer generation of the tray, several instruments sit in different slots, and the system has never seen this layout.

Is the system working? Every metric we report was measured on the old layouts. Those numbers may be excellent, and none of them is evidence about this tray.

Nobody can give a complete mechanistic account of why a vision model identifies a clamp correctly, and we ship these models anyway. Opacity is not the problem. The problem is that we validate the wrong thing.

We validate models, but what reaches a patient is a decision. The model's accuracy is one input to that decision, alongside the gate it passes, the checks around it and the person who signs off. My argument is that the decision path, not the model, is the unit we should be validating.

Right for the Wrong Reason

The best-documented case comes from radiology. Zech and colleagues trained a pneumonia detector on chest X-rays from two hospital systems.¹ It scored an AUC of 0.931 on its internal test set and 0.815 at an external hospital.

The interesting number is a third one. A baseline that ignored the images entirely, ranking cases only by each hospital's pneumonia prevalence, scored 0.861 on the same internal test set. A separate network identified which hospital an image came from with over 99.9% accuracy at both training sites, partly from the metal markers technicians place on the patient.

Site alone got most of the way to the model's score. Geirhos and colleagues call this shortcut learning: a network finds the cheapest feature that separates its training labels, and that feature holds until the world moves.² A model that works for the wrong reason was never working. It was waiting.

Our Shortcut Was the Tray Layout

We have met the same pattern in surgical instrument inspection. Trays are revised across generations, and the bill of materials can stay identical while the layout changes: a clamp moves two slots, a retractor swaps sides. A validation set built from one generation shares its layout, so position quietly becomes evidence of identity, and held-out accuracy cannot see it.

When a new generation reached us, escalations spiked. Correct instruments in unfamiliar positions were flagged as misplaced or missing. The system failed in the safe direction, and the cost landed on the technicians who had to clear those flags.

The other direction is the one that keeps me careful. A system that treated position as identity would have given the expected label to whatever sat in the expected slot and reported a confident match. Neither failure shows up in a model-level metric, because the test set never contained the variable that mattered.

The obvious safeguards would not have caught it either. A BOM check sees the same instrument list, and a count sees the same number of items. Both are built around the BOM, and a generation change is invisible to the BOM.

What Testing Cannot Find

Behavioural evidence is still the most useful thing we have: error broken out by site, camera, instrument family and tray generation, calibration, and characterised failure modes. Most shortcuts can be exposed without opening the network. Mask the background, swap the context, change the layout, and see whether performance holds.

But behavioural evidence covers only the conditions you sampled. Dijkstra made the point about software decades ago: testing shows the presence of bugs, never their absence. Learned models make it worse, because a training set is not a specification. It is a sample from a world that keeps producing cases nobody listed.

Mechanistic understanding does not close the gap. Fully understood code still fails on inputs nobody anticipated, because knowing how a system works and knowing every situation it will meet are different kinds of knowledge. Interpretability can show that a model leans on instrument geometry rather than slot position, which is useful evidence and not a guarantee.

So the honest answer to "how will you know where it fails?" is that you won't. Not fully, not ever. Unknown unknowns stay unknown, and any safety argument that needs them found rests on something it cannot verify.

Validate Decisions, Not Models

If you cannot find every failure, stop asking only where the model fails. Ask what the system does when the model fails in a way nobody predicted. That question has a testable answer, because it lives in the decision path: the prediction, the gate it must pass, the independent checks, the reviewer, and the action that follows.

Safety engineering made this move long before deep learning. ISO 14971 never assumes a component that cannot fail; it asks whether residual risk is acceptable once controls are in place.³ ISO 21448, known as SOTIF, was written for automotive perception whose limits cannot be listed in advance, and treats unknown unsafe scenarios as a region to shrink and tolerate.⁴

Most ML teams still haven't absorbed what follows from this. Test-set metrics are an input to the safety case, not the safety case. A model's AUC tells you how often it ranks correctly; the escape rate of the decision path tells you how often a wrong answer reaches a patient.

This claim can be tested. Hold out trays from one tray generation, then train two models on the same budget: one that saw some trays from that generation and one that saw none. Score both on the same held-out trays, run their predictions through the deployed decision policy, and ask whether the model-level metrics anticipated the change in escalations and false clearances. If they did, my claim is weaker for that system; our tray-generation spike suggests they often won't. The experiment tests generalisation under shift, and whether the barriers catch what the model misses is a separate question that needs its own test.

The failure we observed took the safe path: escalation. The figure shows the hazardous path the same shift could open. The safeguards appear separate, but they share assumptions about the tray and its configuration, so one change can invalidate several at once, which is why their effectiveness has to be measured on the cases where the model fails.

Redundancy Is Not Independence

Reliability engineering puts the point in one line. For an escape path that starts with a hazardous model error of probability p_m, where a barrier is meant to stop that error reaching the patient:

The conditional term is where safety cases go wrong. Teams multiply two headline error rates as if the barrier failed independently of the model, without checking whether it does. A second vision model shares the camera, the lighting and often the labels, while a count and a BOM check share the BOM. This is also only one path: a correct prediction can still become a wrong decision through a configuration error, a workflow bypass or a reviewer mistake, and each path needs its own analysis.

James Reason's Swiss cheese model is usually drawn with holes scattered at random. In the field the holes are lined up by a shared cause: a new tray generation, a replaced camera, a catalogue change. A safeguard earns its place only if it still works when the primary component is wrong, and that has to be measured on the model's failures, not on its average.

The same logic covers change. Swapping a camera, adjusting the lighting or retraining the model can realign the holes overnight, so change control has to cover the assumptions behind the safety case, not just the software version.

Engineering for the Unpredictable

Constrain the world. The cheapest way to shrink the unknown region is to shrink the world the model sees: a fixed camera, controlled lighting, a defined catalogue, known tray generations. A constrained domain is never exhaustively testable, but its variations become nameable, and a named variation can be stratified, tested and monitored. Once tray generation is written down as a variable, it stops being an unknown.

Fail toward escalation, and let the system refuse. Decide which error is dangerous and bias every gate toward the safe one, so an unfamiliar layout produces a flag rather than a guess. Uncertainty estimates, out-of-distribution detection and conformal prediction help, but they are fitted to the same data and share its blind spots, and standard conformal coverage relies on exchangeability, which a new tray generation may invalidate. Refusal is a risk control, not a proof.

Close the loop in the field. Escalations, overrides and near-misses are the cheapest source of unknown unknowns you will ever get, and perturbation tests over layout, pose, glare and wear can go looking for the next one deliberately. Each new failure mechanism becomes a stratum in the test suite and a question for the safety case. An escalation spike is not noise to be tuned away; it is the system reporting that its operating domain has moved.

Every Escalation Is a Bill

Safety essays usually skip this part. Every abstention is paid for by someone: a technician's time, a customer's throughput, and eventually the vendor's margin. When escalations spiked on the new tray generation, the fastest fix was obvious, because loosening the gate makes the flags disappear.

We didn't. We fix escalation volume through onboarding and data, and we do not relax a safety gate simply to reduce it; any change to a gate needs evidence that decision-level risk stays acceptable. We absorbed the cost of the spike, because a gate tuned to keep a dashboard quiet will eventually wave through the dangerous failure.

There is a second reason, and it is about safety rather than principle. A reviewer who clears hundreds of false flags learns that flags are usually wrong, automation bias does the rest, and the human barrier quietly stops working. The risk has not gone away; it has moved from the model into the workflow. High false-escalation volume is not a safe state that merely costs money. It degrades the next barrier in the path, which makes reducing it through better data safety work, not cost-cutting.

What to Measure Instead

If the decision path is the unit of validation, the safety case needs metrics at that level. These four do work that model accuracy cannot, and each should be broken out by site, camera and tray generation.

Metric	What it tells you	Why model metrics miss it
Hazardous false-clearance rate	Incomplete trays cleared, divided by all incomplete trays entering the system; decompose it by auto-cleared, reviewed and overridden paths to see where escapes come from	Accuracy pools every path, while escapes concentrate in a few
Escalation rate and reviewer load	What refusal costs, and how much pressure sits on the human barrier	Model metrics treat abstention as free
Audited reviewer accuracy	How often reviewer decisions match an independent audit, tracked against queue length	It measures the barrier, not the model
Decisions exposed before a shift is caught	How many decisions a changed operating domain affects before detection, by severity	A static test set has no time axis

Read them together. A system that escalates every tray has zero false clearances and no practical value, so false-clearance rate on its own rewards the wrong behaviour.

They also share a ground-truth problem. A reviewed decision leaves an audit trail, but an auto-cleared tray may never be checked independently, and auditing only escalations measures the cases the system already suspected. Decision-level evidence needs independent audits of accepted decisions too, combining representative sampling for rate estimation with targeted sampling to discover rare hazardous failures.

A system cannot establish its own reliability by counting the errors it noticed. The errors that matter most are the ones it let through without question.

None of these is exotic. What is unusual is treating them as the primary evidence and model accuracy as supporting evidence, rather than the other way round.

Generality Without a Boundary

The industry assumes broader models are safer models: more data, more variation, fewer surprises. Breadth can help, because a model with richer representations may lean less on brittle cues like slot position. The danger is what breadth does to the claims made about it.

A narrow model trained on known tray generations comes with an operating domain you can write down. A general visual foundation model invites a wider one, and its scope tends to grow with its capability rather than with its evidence. When a model can plausibly handle anything, there is a temptation to stop defining precisely what it was validated on, and stratified validation across everything it might now be asked to see becomes impossible to fund.

Breadth may also change the shape of failure, though this is a hypothesis, not a result. A narrow model that learned slot position fails loudly on the first new layout. A broad model can lean on many weaker cues at once, so its failures may arrive rarer, later and with more confidence, which makes them harder to catch and not obviously safer.

Generality becomes a safety asset when it ships with a stated operating domain, decision-level evidence and barriers independent of the model. Until then, my prediction is that the first serious failures of general perception models in regulated settings will not happen far outside their claimed capability. They will happen inside it, in conditions nobody thought to test.

Unknown Unknowns Stay Unknown

We already trust systems whose components we cannot fully explain. Nobody can predict a pilot mechanistically, and aviation is safe anyway, because its safety argument was never built on the pilot being right. It was built on procedures, independent checks and relentless incident reporting.

ML deserves the same standard. Demanding full mechanistic understanding sets a bar nobody clears, and accepting test-set accuracy sets one everyone clears. The useful bar sits between them: evidence, at the level of the decision, that a wrong answer is caught before it matters.

Unknown unknowns stay unknown, and safety engineering cannot make them disappear. What it can do is stop our inability to predict the next failure from becoming a dependence on the model being right.

References

Footnotes

1. Zech, J. R. et al. (2018). Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLOS Medicine, 15(11). ↩

2. Geirhos, R. et al. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2, 665–673. ↩

3. ISO 14971:2019, Medical devices: application of risk management to medical devices. ↩

4. ISO 21448:2022, Road vehicles: safety of the intended functionality (SOTIF). ↩