The many doors an uninvited guest might try¶

By now the model has enemies, though nobody thought to warn it. A machine learning system is software with a new name and a considerably larger attack surface than anyone mentioned during procurement. A traditional application does what its code says; our model does what its training data implied, its infrastructure permits, and its users have not yet thought to abuse, three quite different things, and each of them a door an uninvited guest might try.

The doors run the length of the model’s life: from the raw data gathered months before its birth, through the weights serialised and shipped across a network, to the endpoint fielding queries at three in the morning from sources that never appeared in any threat model. Guarding one door and leaving the rest is a very good lock on the front and an open window at the back.

The training data¶

Everything begins with data, which means everything can go wrong with data. This is the first door, and the one furthest from anybody watching, since it was propped open long before the model existed. The model is, in a meaningful sense, a compressed representation of its training set. Corrupt the training set and the model is corrupted with it, often invisibly, until the system does something embarrassing in production.

Data poisoning¶

Data poisoning is the art of introducing malicious examples into training data so that the resulting model behaves in ways its operators did not intend. An attacker with write access to a data pipeline, or simply the patience to upload manipulated content to a platform that scrapes public data, can influence model behaviour without ever touching the code.

Clean-label poisoning is a subtler variant where the injected examples are correctly labelled but carefully crafted to shift decision boundaries in specific directions. The poisoned samples pass human review. The model learns the wrong lesson anyway. Spam filters, content moderation systems, and fraud detectors are all plausible targets.

Backdoor attacks¶

A backdoor, sometimes called a trojan, is a hidden behaviour embedded during training that activates only when a specific trigger is present in the input. The model behaves normally on everything else, passing all standard evaluations with the composure of someone who has been rehearsing for exactly this test. Present the trigger and the model does what the attacker intended rather than what its operators intended.

Triggers can be as crude as a particular pixel pattern in an image or as subtle as a specific phrase in a text input. The attack is particularly dangerous when organisations use pre-trained models or third-party datasets, since the backdoor may have been introduced before the model arrived and will not be visible in any inspection of the fine-tuning code.

The model itself¶

Nothing here has to be broken into, only asked. Once trained, the model is an asset that can be interrogated, copied and used as an oracle by anyone able to see its outputs, which is to say by anyone it was built to serve. Those outputs carry information about the training data and about the decision boundaries, and sufficiently motivated adversaries collect it systematically, not by accident.

Model inversion¶

Model inversion attacks use the model’s outputs to reconstruct something resembling its training data. An attacker queries the model repeatedly, watches how confidence scores shift, and works backwards towards an input the model associates strongly with a given label. Facial recognition models have been inverted this way to produce recognisable faces, and medical models trained on patient records carry the same exposure.

The loose version of this claim has done the rounds for years. What comes back is a class representative, an average of what the model associates with that identity, rather than a specific photograph somebody once supplied. That is still a privacy failure, and a recognisable face is still personal data. It is not the recovery of one person’s training record.

Nor is the access requirement quite nothing. The classic version wants confidence scores rather than bare labels, and usually the target label too, so an API returning only its top answer was for a while a real defence. Label-only variants have since eroded that. Where the scores are exposed, the bar is a motivation and an API key.

Membership inference¶

Membership inference is a narrower question than model inversion: was this specific record in the training data? The attack exploits the tendency of models to be more confident on data they have seen before. Comparing a model’s behaviour on a target record against its behaviour on records it certainly has not seen gives an attacker a signal about membership, though a practical attack generally needs shadow models or a decent prior on the data distribution, not curiosity alone.

How reliable that signal is depends on how it gets measured, and for years it was measured badly. Average-case accuracy flatters an attack that is right about the easy cases and useless on any case that would actually name somebody. The question worth asking is how many members an attack can identify while almost never accusing a non-member. Judged that way most of the early attacks are close to worthless, and the ones built after the question was posed properly are not: they name real members at false-positive rates low enough for the finding to carry weight. The correction deflated the old literature and raised the risk at the same time. What is inferred is the presence of a record, not of a person. An individual may sit in a training set through a dozen records, or be absent while a near-duplicate of them is not.

For models trained on sensitive data, this is a privacy violation with legal consequences. A membership inference attack against a model trained on medical records, financial histories, or any dataset subject to data protection law is not merely a technical curiosity. It is the kind of thing that produces regulatory correspondence.

Model extraction¶

Model extraction, also called model stealing, involves querying a model repeatedly in order to train a surrogate that approximates the original. The attacker does not need the weights, the architecture, or any internal access. They need only the outputs, and enough patience to generate a sufficient query set.

A stolen surrogate model can be used to undermine a competitive advantage, to investigate the original model’s behaviour without the operator’s knowledge, or as a stepping stone to other attacks. If the surrogate is a close enough approximation, adversarial examples crafted against it will often transfer to the original.

The pipeline¶

The internal doors go unlocked on the assumption that nobody outside can reach them. The stages carrying a model from notebook to production, the pipeline, are not merely operational complexity. Each stage is a further attack surface, and the handovers between them are where trust boundaries get assumed instead of checked.

CI/CD pipeline compromise¶

The continuous integration system that tests, packages and publishes model artefacts is a privileged environment with access to source code, secrets, model weights and deployment infrastructure. It is also, for many teams, the thing that launches training and signs off the result. An attacker holding CI need modify neither the training data nor the training code, because CI decides which job runs, and whatever comes out carries the build’s credibility into everything downstream.

The vectors themselves are unremarkable, and that is the exposure. Malicious code in a test dependency, permissions nobody tightened, a poisoned registry, secrets sitting in environment variables. A pipeline that resolves its dependencies at build time without pinning them is one substitution away from building something other than what was asked for, and the build log will record a success.

Artefact tampering¶

Between the point where a model is trained and the point where it serves predictions, the model artefact passes through several storage locations: a local filesystem, a model registry, a container image layer, a cloud storage bucket. Each handover is an opportunity for tampering if the artefact is not signed and the signature is not verified at the point of use.

What makes a swapped model worth more to an attacker than swapped code is what nobody looks at. A backdoored function would be caught in review. Nothing reviews a weights file, and the monitoring that might have noticed is watching latency.

Container image vulnerabilities¶

An image freezes the userland packages, the runtime and the dependencies at build time. Vulnerabilities found afterwards accumulate until somebody rebuilds, so production routinely runs images with known holes in them, because the rebuild cadence is slower than the disclosure cadence.

Machine learning images are worse than most. They are enormous, they carry a dependency tree that took somebody a fortnight to make work. The rebuild that would clear the backlog is the thing everybody is avoiding.

Base images pulled by tag rather than digest are a separate exposure. Registries can be set to make tags immutable and to raise an event when one moves. By default they are not, and nobody is watching the events.

Infrastructure misconfiguration¶

A model deployed to cloud infrastructure inherits that infrastructure’s posture. Over-permissive roles, storage left open, unencrypted traffic between services, network exposure nobody trimmed: operational failures, all of them, and all capable of undoing careful modelling without going near the model.

The storage bucket is the one worth naming. Training data lives in object storage, it is large, it gets copied for convenience, and a copy does not inherit the permissions somebody set carefully on the original. A model that was never attacked at all can lose its entire training set this way, which is a poor return on a year of work.

Defining infrastructure as code makes the configuration reviewable, which is a real gain, and gathers the privileged definitions into one repository, which is a real target. A committed secret or an over-scoped service account in that repository outranks the controls downstream of it.

The deployment¶

A model that has survived training, evaluation, and a largely fictional security review must then be deployed, which hangs the whole thing on a public street. The standard vulnerabilities of any networked service arrive at this point, along with several that are particular to models.

Inference endpoints¶

The API through which a model serves predictions is an endpoint with all the usual exposure of a web service: authentication weaknesses, rate limiting failures, injection vulnerabilities in input handling, and the various consequences of running software on infrastructure maintained by humans with other things to do. A model endpoint that accepts serialised inputs deserialises them before processing, which is a well-established source of remote code execution vulnerabilities in Python environments particularly fond of pickle.

Excessive querying of an inference endpoint can exhaust compute resources, inflate cloud costs to spectacular levels, and degrade service for legitimate users. Without rate limiting and anomaly detection on query patterns, the endpoint is also providing free access to model extraction and membership inference attacks at the operator’s expense.

Model serialisation¶

Some serialised model files are executable artefacts, and loading one from an untrusted source is equivalent to running untrusted code. Pickle-based formats are where this bites: a pickled PyTorch checkpoint can execute arbitrary Python on load. This is not a subtle vulnerability. It is documented, widely known, and regularly exploited when organisations share, download, or deploy models without verifying their provenance.

Defaults have tightened. PyTorch 2.6 flipped torch.load to weights_only=True for exactly this reason, breaking a great deal of downstream code on the way, and the hole reopens whenever someone passes weights_only=False to get an old checkpoint working. Safetensors was designed to carry weights and nothing executable, which removes the primitive without removing the problem. The distinction that does the work is between formats storing tensors alone and formats serialising a whole computation graph: a graph can be backdoored in its own structure, with no code executed at load, and that holds for ONNX and its neighbours alike. ONNX parsers have separately carried path traversal flaws. A format that cannot run code still arrives through tooling that can.

What gets a malicious file through is rarely sophistication. It is a file from a public repository with no malware scanning, loaded because it was the one the tutorial linked to. How bad that turns out to be depends on what the format lets it do on the way in, before anybody has decided whether to trust it.

The inputs¶

A model that accepts no inputs is of no use to anybody, so this door stays open on purpose. A deployed model takes what the world sends it and answers. It was trained on a particular distribution of inputs, and everything outside that distribution is territory it has never practised, which is precisely where an adversary will take it.

Adversarial examples¶

Adversarial examples are inputs crafted to cause misclassification. In the digital case the perturbation is a small, deliberate pattern, imperceptible to a human and decisive to the model. The constraint is that the change stays below some threshold of noticeability, which is what makes the results unsettling: two images a person cannot tell apart, classified differently with high confidence.

The physical world relaxes that constraint instead of extending it, and the canonical result accordingly looks nothing like noise. The stop sign that a classifier read as a speed limit sign carried black and white stickers, plainly visible, and deliberately arranged to pass as graffiti. Visible but unremarkable was the design goal: a perturbation nobody thinks to question beats one nobody can see. Printed patches and adversarial clothing work on the same principle, with results that hold up under some viewing angles and lighting conditions and fall apart under others. Autonomous vehicles, surveillance systems and access control are the obvious targets.

Why these exist at all is less settled than it is often made to sound. The comfortable account is that models latch onto statistical correlations rather than features a human would call meaningful. A harder-nosed reading holds that the features being exploited are genuinely predictive, generalise to the test set, and are simply not the ones a human happens to use. Either way the practical consequence is the same: accuracy metrics conceal the brittleness, because standard test sets contain no adversarial inputs.

Prompt injection¶

Prompt injection resembles an adversarial example and is not one. An adversarial example exploits a learned decision boundary, and defending it is a question of training. Injection exploits an architectural fact that no amount of training addresses: instructions and data arrive through the same channel, as text, and nothing separates them. An attacker embeds instructions inside content the model has been asked to process, and the model treats attacker-supplied text as authoritative because it has no basis for doing otherwise.

Direct prompt injection targets the model through the user’s own input. Indirect prompt injection hides malicious instructions in external content the model retrieves or processes, such as a web page, a document, or an email. When a language model is given tools, memory, or the ability to act on external systems, a successful injection can cause it to exfiltrate data, take unintended actions, or produce outputs that serve the attacker rather than the user.

Every serious mitigation is therefore perimeter work: constraining what the model is permitted to do, treating retrieved content as hostile by default, and keeping the blast radius of a successful injection small. None of that is a fix. The work that would be one, teaching a model to hold a boundary it was never built to hold, is still open.

The ecosystem¶

The last doors are in somebody else’s building. A machine learning system’s attack surface does not stop at its own boundary: it runs on to the pre-trained models it builds on, the libraries it uses for training and inference, the datasets assembled from external sources, and the people with access to any part of the pipeline.

Supply chain¶

A compromised pre-trained model on a public repository carries its compromise into everything fine-tuned from it, which is the whole appeal of fine-tuning working in the wrong direction. A vulnerable version of a widely used library reaches whichever systems installed it without pinning, and the number of those is never as small as anyone hopes.

The ML ecosystem relies heavily on public infrastructure: model hubs, dataset repositories, and open-source libraries maintained by small teams with limited security resources. This is a supply chain with a large surface area and inconsistent controls, and the value of compromising a foundational model or widely used dataset is proportional to how many downstream systems depend on it.

Federated learning¶

Federated learning distributes the training process across many participants, which distributes the trust assumptions along with it. A malicious participant can submit poisoned gradient updates designed to introduce backdoors or degrade performance for specific groups. Byzantine behaviour is the broader case, and the term gets misused: it means a participant deviating from the protocol arbitrarily, which needs neither coordination nor numbers. One client sending nonsense qualifies. Collusion between many is a harder variant of the same problem, not the definition of it.

Gradient inversion runs the other way. Updates carry more of the data that produced them than is comfortable, so the party that reconstructs private data from them is normally the aggregating server, since in the usual topology that is the party positioned to see individual contributions. Secure aggregation narrows the view to an aggregate, though a server willing to send different models to different clients can work around it, and who took part and who dropped out still leaks around the edges.

Both protections, and differential privacy alongside them, add complexity and cost. They are frequently omitted from production deployments that nonetheless advertise the privacy benefits of federated learning in their marketing materials, on the strength of the architecture diagram rather than anything running in it.

One day, after the poisoners and the invertors and the prompt injectors have each had their turn, or after none of them bothered and the model simply grew old, it is switched off, and the doors shut. Its endpoint returns a last 404, its weights are archived on a bucket nobody will pay to keep, and a post-mortem is written that everyone agrees to learn from and no one reads. By then the next model is already being born in a notebook, full of promise and two hundred lines of borrowed code, and it has not heard a word of any of this. That is the mercy of the thing. Nobody ever tells them.