Model Is Not the Product

We had a model in production that worked. Then one day it didn't. Not because the model failed, but because we couldn't find it.

Someone had retrained with new data and deployed to production. The weights file was overwritten. The old configuration was gone. When validation failed on a customer site, we had no way to reproduce the previous state. No way to compare what changed. No way to roll back.

The investigation took three days. The fix took two weeks. The loss of customer trust was immeasurable.

That incident taught me something fundamental: the model is not the product. The product is the entire system that makes inference possible: weights, configurations, thresholds, preprocessing parameters, version metadata, and the infrastructure that ties them together. Lose any piece, and you've lost the product.

The Artefact Management Problem

Machine learning systems are fundamentally different from traditional software. In conventional applications, the source code is the product: version it, and you can reproduce any build. In ML systems, the 'product' is a complex constellation of artefacts: training data, preprocessing logic, model architecture, learned weights, inference parameters, threshold configurations, and runtime dependencies.

The industry keeps relearning this. Most ML models never make it to production, and rarely because the algorithm fails: the engineering infrastructure around them fails first. An entire MLOps industry has grown out of that one realisation: model management is harder than model development.

The core challenges are interconnected:

Reproducibility: Can you recreate the exact inference behaviour from six months ago? Without explicit artefact management, the answer is almost always no.

Traceability: When something goes wrong, can you trace from a specific prediction back to the model version, training data, and configuration that produced it?

Compatibility: Does model version X work with inference engine version Y? Without explicit constraints, the answer is 'maybe, until it doesn't.'

Auditability: For regulated industries (medical devices, financial services, autonomous systems), can you prove to an auditor exactly what was running when?

Design Principles for Production Artefact Management

After that incident, I rebuilt our artefact management system from first principles. These are the design decisions that matter:

1. Immutability

Once an artefact is created, it is never modified. New versions create new artefacts; old versions remain forever accessible. This seems wasteful until you need to reproduce a failure from three months ago.

2. Explicit References

No implicit dependencies. Every artefact required for inference must be explicitly listed with a resolvable URI. If a configuration file references 'model.onnx', that path must be a full URI that resolves to exactly one artefact.

3. Integrity Verification

Every artefact reference includes a cryptographic hash (SHA-256). At load time, the hash is verified. If verification fails, inference fails. No silent corruption, no 'it worked on my machine.'

4. Modular Separation

Different components (classification, registration, preprocessing) are independent modules with independent versioning. Upgrading the classification model shouldn't require touching registration artefacts.

5. Standard Formats

ONNX for neural networks (framework-agnostic inference). JSON for configuration (human-readable, diff-able). Pickle only for learned non-NN artefacts where no standard exists. No proprietary formats that create vendor lock-in.

6. Version Compatibility as Metadata

Explicit min/max version constraints for the inference engine. A model trained for engine v2.3 may not work on v3.0. This must be checked before deployment, not discovered in production.

The Manifest Pattern

The central concept is the model manifest: a single JSON file that serves as the authoritative source of truth for a validated model configuration. Everything required to run inference is referenced from this file. Nothing is implied.

The manifest contains: unique identifiers for the deployment context (client, product, configuration level); URIs and SHA hashes for all model weights; URIs and SHA hashes for all parameter files; inference engine version constraints; and a timestamp that makes every configuration uniquely identifiable.

A model is considered valid only if: all referenced URIs are resolvable, all SHA hashes match, version constraints are satisfied, and all required parameters are present. Failure of any condition invalidates the entire configuration. There is no 'mostly works.'

This pattern enables several critical capabilities:

Atomic deployment: Either the complete configuration deploys successfully, or nothing changes. No partial states.

Instant rollback: Rolling back means pointing to a previous manifest. All artefacts still exist because they're immutable.

Reproducible debugging: Given a manifest and an input, you can reproduce exactly what happened. No 'it works differently now.'

Audit compliance: Every prediction can be traced to a manifest, and every manifest defines exactly what ran.

Hierarchical Storage Structure

For multi-client, multi-product deployments, artefacts are organised hierarchically: client → product → configuration level → timestamp. Each timestamp directory contains a complete, self-contained configuration: all modules, all parameters, the manifest.

The timestamp is monotonically increasing and never reused. This creates a natural audit trail: you can list all configurations for a given product in chronological order, see exactly when each was created, and understand the evolution over time.

Within each configuration, modules are separated: classification artefacts in one directory, registration artefacts in another, preprocessing in a third. Each module has its own parameter file that fully specifies its execution: no cross-module dependencies, no implicit assumptions.

This structure scales. Adding a new client means creating a new top-level directory. Adding a new product configuration means adding under the client. Updating a model means creating a new timestamp. The old configuration remains accessible forever.

The Model That Cannot Disappear

The system above exists so that the opening story can never repeat. Models still fail; that is what production means. But a failure now traces to a manifest, the manifest resolves to artefacts, and rollback is a pointer to the previous timestamp. The investigation that took three days now takes a query.

The model was never the product. The discipline around it is.