From notebook to production: a guided tour of impending complexity¶

Our model has been born. Now it has to travel. The road between a notebook and a deployed, monitored, maintainable system is where a model’s character gets tested, and it is considerably wider than anyone budgets for.

Few teams have built all of it. Each stage retires one particular way of being unable to answer a question later, and the stages that get skipped are reliably the ones whose question nobody has asked yet.

Research and development¶

The pipeline begins in notebooks, where data scientists do science and data in roughly equal measure. The opening run is exploratory analysis, a split into training and held-out test data, then feature engineering, feature selection, model training, and predictions. These steps proceed with the cheerful informality of a process that assumes someone else will deal with reproducibility later.

The order there is doing work, and it is the order people get wrong. The split comes before anything is fitted. Feature engineering then lives inside a pipeline object rather than in loose cells, which sounds like housekeeping and is not. A pipeline fitted on the training split alone holds its imputation values, its scaling parameters and its category encodings as fitted state, so the test data is transformed by them and never contributes to them. Fit the same steps across everything and split afterwards, and the test set has quietly helped build the transformations it is later judged by. The score that comes out will be excellent and will mean nothing.

The other reason for the pipeline object is that it survives the notebook. Whatever transforms the training data has to transform the request that arrives eighteen months later, and has to do it identically. A sequence of cells cannot make that promise. A fitted object serialised alongside the model can.

Knowing whether it works¶

A model that has been trained has not been judged, and judging it badly is a more common failure than training it badly. Accuracy is the default choice and usually the wrong one: where ninety per cent of cases are the same class, a model that always guesses that class is ninety per cent accurate and entirely useless. Precision and recall separate the two ways of being wrong, and which one hurts is a question about the domain rather than the model. A fraud system that misses fraud and a fraud system that freezes honest accounts are both failing. F1 will average the two, and averaging is not the same as deciding which one the business can afford.

The comparison that carries weight is against a baseline somebody is willing to state out loud. Guessing the majority class is useless as a product and indispensable as a yardstick; so is repeating whatever the last decision was, or applying the hand-written rule the business already runs on. A model that cannot beat those has not earned the infrastructure it is about to be given.

Then there is calibration, which is the difference between a model that ranks well and a model whose 0.8 means anything. If a downstream process treats the score as a probability, somebody has to check that scores near 0.8 are right about eighty per cent of the time.

None of it settles the question. A held-out score is a claim about data drawn the way the training data was drawn, and production traffic is not drawn that way, quite.

Packaging the model¶

A model that lives in a notebook can only be run by whoever has the notebook, on the machine where it was last run, in the environment that was current at the time, assuming the data is still where it was. This is not a deployment strategy. It is a hostage situation.

Packaging wraps the trained model in a proper Python package with a version number, a declared dependency set, and an entry point that trains it from scratch. A task runner such as Tox drives the whole sequence from one command: build a clean environment, lint, type-check, run the tests, train the model, save the artefact. No individual check is the point. The point is that the sequence runs the same way for everyone, including whoever joined last week and has not yet been told which three commands are load-bearing.

Runtime configuration belongs in a YAML file, not in Python. Keeping it out of code means the values a non-engineer might reasonably want to change, thresholds, paths, feature lists, can be changed and reviewed without a pull request full of syntax. It also means the configuration can be validated on load, which is where a schema library earns its keep.

The packaged model can then be trained, tested and versioned consistently across machines. It is the stage at which a great many real projects have quietly stopped.

Serving predictions¶

A packaged model produces predictions when someone runs it. A served model produces predictions when something else asks. A web framework turns the classifier into a service that accepts requests and returns predictions over HTTP, and the useful ones validate the request against a declared schema, so malformed input is turned away at the door and never reaches the model code to produce a cryptic error from somewhere deep inside it.

Structured logging becomes worth its cost at the same moment. When something goes wrong in production the question is always what was asked and what came back, and a service that cannot answer it is debugged by guesswork. Logging the inputs, the prediction and the model version costs almost nothing at the time and is the difference between an afternoon and a fortnight later on.

The API is the boundary between the model and everything that wants to use it, and boundaries harden fast. A field named carelessly in an idle afternoon will still be there, still misspelled, in four downstream services three years from now.

Continuous integration and publishing¶

A model package tested only on the developer’s laptop is not tested. Continuous integration runs the full suite on every change, in a clean environment, and publishes the package when a version passes. This makes the test suite the arbiter of what constitutes a working model, rather than whoever last ran it by hand and decided it looked acceptable.

What the suite contains is the interesting part. Unit tests on the transformations, a check that the trained model clears the agreed baseline, and a test that the serialised artefact loads and predicts on a fixed input. That last one looks trivial and catches a failure that is both common and humiliating: a model that trains beautifully and cannot be reloaded.

Publishing to a registry means downstream services pin a specific version and upgrade deliberately. Without it, they discover the model changed when their predictions changed. Versioning model packages the way software packages are versioned looks like bureaucracy right up until the first time somebody needs to know which model produced last Tuesday’s decision.

Containerisation¶

An application running on a developer’s machine has a Python version, a set of installed libraries and an operating system that may not match the environment where it will run in production. Containers narrow the uncertainty by packaging the application together with its Python version, its dependencies and the model package itself.

Building an image and running it as a container produces an artefact that behaves consistently across hosts, which is not quite the same as identically. The image carries userland and not a kernel, so anything sensitive to kernel version stays outside its control, and CPU architecture is a separate trapdoor: an amd64 wheel does not merely misbehave on arm64, it declines to load at all. Within those limits the image that passed testing can be the image that reaches production, provided the deployment pins a digest rather than a tag. A tag is a label somebody can move. A digest is the image.

Images are versioned alongside model packages, so rolling back a bad deployment means fetching a specific image. The alternative is reconstructing an environment from memory, under pressure, with an audience.

Differential testing¶

Differential testing runs two versions of a model over the same inputs and compares what comes back. In continuous integration that means a fixed set of cases and a diff, with a policy for which differences are acceptable. The related practice of shadow deployment does something similar against mirrored live traffic, without serving the candidate’s answers to anyone.

Shadow deployment is the one that closes the gap a held-out score leaves open, because mirrored traffic is the real distribution and a curated case list is not. The CI diff answers a narrower question, whether this version still does what the last one did on the cases somebody thought to write down, and it answers it in seconds for nothing. The two are worth having for different reasons, at very different prices: mirrored traffic means running two models and storing both sets of answers.

The diff usually gets built. The mirrored traffic waits, and what ends the wait is generally an incident.

Deploying to infrastructure¶

The final stage moves the containerised application onto rented infrastructure, at which point the model is running on servers that are nobody’s laptop, reachable over the network, and subject to the concerns that come with the address: monitoring, scaling, cost management, security, and the particular dread of an alert arriving at an inconvenient hour.

Monitoring here means two things that get confused. Service monitoring asks whether the thing is up, fast and answering, and any competent operations team already knows how to do it. Model monitoring asks whether the answers are still any good, which is harder, because the label that would settle it arrives weeks later or never. What can be watched without labels is the shape of the inputs and the distribution of the outputs, on the theory that a model whose traffic has changed shape is worth looking at before the complaints arrive.

A model in production is no longer a research project. It is a service with users, dependencies, and an implicit promise that it will keep behaving as expected while the world around it changes. The pipeline is what makes that promise maintainable, and what turns each update from a small act of hope into something boring.

A note on the Iris repository¶

The Iris repository sets out seven signposted stages, the pipeline minus the evaluation, using the classic Iris dataset as a vehicle: 150 samples across three species, four measurements each, a multi-class problem small enough that the infrastructure shows through instead of the data. That is the argument for teaching with Iris generally. The complexity in an MLOps pipeline comes from the plumbing rather than the feature matrix, and a dataset nobody has to argue about lets the plumbing be the subject.

It is signposts rather than a worked implementation, which makes it a decent map and a poor substitute for walking the road. Both are worth having, and only one of them is any use when the pager goes off.

Our model has reached production intact, which is rarer than it sounds. What happens to it now depends less on the pipeline than on the building it has been delivered into.