Same city, different physics¶

These are the streets of MLOps, and our model has just been turned loose on them. Here good intentions collide with technical debt, best practices are chalk sketches on a pub wall, every deployment is a gamble, and every cloud invoice is a reminder that the gods of finance have a wicked sense of humour.

The neighbourhood the model lands in decides how it lives. It might be raised in a plucky start-up’s garret or a corporate fortress, but what the neighbourhood actually settles is where the boundaries run: between people, between systems, between a signal and whoever might act on it. Work crosses those lines or it does not, and what fails to cross is where the trouble collects. The rest is atmosphere.

Four neighbourhoods, then, and four boundary shapes, and the failure signature follows the shape. In the garret the boundaries are accidental, drawn around each laptop, and work fails by staying invisible. At scale they run between separately owned systems, and the definitions shared across them diverge without anyone noticing. In the hybrid tower the boundary runs through the middle of a single workflow, and signals cross it but land on no one. In the fortress the boundaries are stacked vertically, and the feedback loop fails by growing too long to close.

Small data Ops¶

In the cosy garret of a plucky start-up, data scientists work in blissful obscurity. Laptops hum like cranky street organs, and enthusiasm runs faster than coordination. Each bright-eyed apprentice constructs pipelines, unaware that their neighbour has been building the same model. Weeks pass before anyone notices, and by then the cloud bill has grown to a terrifying size.

The boundary here is accidental: one around each person, drawn by nothing more deliberate than who sits at which laptop. Almost nothing crosses it, because there is nowhere for a crossing to land. The data fits on a single machine, so nothing forces shared infrastructure into existence: no common feature store to make two people notice they want the same feature, no shared pipeline to make two models collide in review. A repository usually exists, but the work happens in notebooks beside it, and a notebook is a complete and private world. The failure signature follows from the geometry: invisibility, with duplication as its most expensive symptom. The team is small enough that everyone assumes shared context, which is precisely why nobody sets up the mechanisms that would create it.

Trouble builds here as gradually as anywhere; the difference is that nobody sees the slope, only the cliff. The duplicate model, the invoice for GPU instances spun up for a weekend experiment and never shut down, the dataset that quietly outgrew the laptop: each arrives as a sudden, fully formed fact, because nothing was being watched and nothing seemed to need watching. A feature disappears because the notebook that computed it was never committed. Retraining takes twice as long as expected because nobody wrote down what the first run actually did. And spirits remain high, because hope is cheap and a bill that size reads, the first time, as evidence of momentum.

Big data Ops¶

Enter the grown-up world. Data engineers and scientists wrestle with sprawling pipelines and distributed systems. Models are trained and deployed, only to be forgotten like dusty experiments in a wizard’s laboratory. One morning, predictions that were flawless yesterday become gibberish today, and midnight debugging commences as engineers attempt to convince The Patrician that penguins do not require loans.

What distinguishes this neighbourhood is not the volume of data but what the volume forces: specialisation. The boundary here runs between separately owned systems, pipeline on one side, model on the other, and what crosses it is the shape and meaning of the data. That interface is rarely written down as a contract anywhere, so it changes the way unwritten things do. The pipeline team renames a column, changes a type, or recomputes a feature with a corrected formula, each reasonable in isolation, and nothing checks whether the interface still means what the model thinks it means. Nothing errors where the change was made. A changed type can turn predictions to gibberish overnight. A changed meaning does something worse, making them quietly wrong for weeks with nothing to announce it. Either way the symptom surfaces downstream, at the customer, which is the worst available place to learn that a field changed meaning three weeks ago.

So the failure pattern is divergence, silent at its source, with symptoms at a distance: the change happens in the pipeline, the damage appears in the model, and the debugging session is mostly an archaeology of upstream commits, looking for which field changed and when. The tooling compounds this. Kafka and Spark dominate discussions even when a single SQL query would suffice, and every added component is another place where a serialisation format, a partitioning scheme, or a default setting can shift underneath the model without anyone owning the consequence. Reproducibility becomes a gamble not through carelessness but through arithmetic: the number of independently versioned, independently deployed parts has exceeded anyone’s ability to hold them all still at once.

Hybrid MLOps¶

The so-called best of both worlds is more like two rival guilds crammed into a single tower, each convinced their magic is superior. The engineers debate Kubernetes while the scientists tweak features, meetings happen in separate rooms, and communication is mostly cryptic commit messages and vague Slack emojis.

It would be easy to read this as the garret at a larger scale, but the boundary sits somewhere else entirely. The garret’s boundaries fall wherever people happen to sit; the tower has a deliberate one, and it runs straight through the middle of a single workflow. What crosses it are the signals: metrics, alerts, handovers of cleaned data. The workflow is one, its ownership split down the middle, and each guild instruments its own half. Dashboards sparkle, metrics are collected, alerts fire. The machinery of observation exists. The trouble is not absent ownership but double ownership: everyone owns the workflow, and nobody owns the crossing. The signals cross the boundary and land nowhere. Alerts ping the wrong channels. The numbers worsen in full view, like fruit going soft in the back of a tavern kitchen, recorded and unactioned, until a furious customer or the regulator shows up.

This is the inverse of the garret’s failure. There, problems are invisible until they are expensive. Here, problems are visible from the start and expensive anyway, because visibility without ownership changes little by itself. The known-but-unowned problem is arguably the most demoralising kind, since everyone can point to the dashboard that recorded it. Cleaned datasets vanish into forgotten transformations for the same reason: the transformations live in one guild’s notebooks, the results in the other’s storage, and nothing records the path between.

Large-scale MLOps¶

At the pinnacle, bureaucracy reigns supreme. A model update requires seventeen approvals, a ritual sacrifice to compliance, and a council meeting held to schedule the next council meeting. Progress is measured not in predictions made but in forms signed and stamps affixed.

The boundary here is vertical: not one line but a stack of them, and each layer adds latency to the loop between noticing something and correcting it. That loop length is the structural pattern. A model can drift, the drift can be detected, the correction can be designed, and the correction can still die in the approval chain, because by the time it clears, the drift has moved on. The deeper mechanism is that accountability attaches to signatures on changes, and seldom to the slow decay of things left alone, so the cheapest signature is the one never given. The model decays legally, with a complete paper trail, which pleases everyone except the model.

The economics follow the same geometry, through an asymmetry: holding a hundred idle GPUs costs money nobody signs for, while releasing them needs a name on a form. The machines hum on, kept alive just in case, and nobody’s incentives point towards initiating that pilgrimage. Legacy systems add their own vertical layers: new ML models pass through ancient monoliths running COBOL and hope, and one wrong step can cascade failures across the city, like a tavern brawl spilling into the guildhall and onward to the Patrician’s palace.

In the end, large-scale MLOps is less about models and more about survival. Success is avoiding catastrophe long enough to write a report that pleases the regulators. Which is not entirely absurd, given the blast radius; it is simply worth being honest that the thing being managed is the loop, not the model.

The streets keep one law¶

Where the lines fall governs what can be seen and by whom, and that shapes which failures a place is good at producing. So the four are not points on a scale but four shapes, and moving a model between them does not remove its troubles. It exchanges them for the local variety.

That is also where the claim could come apart. A garret whose characteristic trouble is divergence rather than invisibility, or a fortress that fails by duplicating work rather than by taking a year to approve a fix, would mean the geometry is not doing the work claimed for it. Boundaries are not the only thing shaping a failure either, since funding, seniority and plain luck all leave marks. The claim is narrower: geometry is the part that stays put when the people change.

None of this appears to be proprietary to machine learning, and none of it is new. Melvin Conway argued in 1968 that organisations designing systems are constrained to produce designs copying their own communication structures. The move here is smaller: keep Conway’s direction and change what is being predicted, from the shape of a system to the shape of its failures. MLOps just happens to be a city where all four districts sit within walking distance.

And wherever the model ends up on these streets, garret or fortress, its worst troubles will still not be the streets. They will be closer to home: the data it was raised on, and the promises made in its name. No neighbourhood issues those, and every neighbourhood inherits them.