AI systems bolted on without governance¶

In Ankh-Morpork the phrase artificial intelligence has become popular with the sort of people whose enthusiasm greatly exceeds their understanding. This is rarely a good combination. It is the sort of enthusiasm that led the Guild of Alchemists to invent the phrase unanticipated outcome.

Every guild and half the guildless are racing to bolt AI into their operations, less because they understand it than because a neighbour announced an AI strategy and nobody wants to be the only one in the city without a transformative automation journey to brag about over a pint of something that probably counts as a health hazard. The result is a new category of civic debt, growing at machine learning speed: models nobody validated, pipelines held together with rope and one elderly Python script that only runs on Thursdays, prompts fed raw user input, and documentation consisting of a single comment reading fix later.

The gold rush¶

The pitch is usually the same: increased productivity, automated customer service, infinite content, competitive advantage, possibly riches beyond imagining. The delivery is more classically Morporkian. A strategy is demanded in month one, a rival announces theirs in month two, something, anything, is deployed by month six, and people describe it as temperamental to avoid saying broken. What ships is whatever worked once in a demo: a snippet unearthed from a dubious scroll, a prompt written by someone still unsure how to spell algorithm, a model chosen from an alchemist newsletter. Integration involves duct tape. Security review consists of someone muttering that it is only text, what could possibly go wrong. Governance is scheduled for later, and later seldom comes.

Deployed like swamp dragons¶

In an ideal world a model would be understood, validated on local data, tested against edge cases, documented, and monitored for drift. Actual deployment can look more like this:

pip install openai
openai.api_key = "sk-please-do-not-tell-anyone"
response = model(prompt)
return response

Testing often consists of five cheerful examples; edge cases are ignored as too gloomy, and bias testing is skipped because it involves thinking about uncomfortable realities. The classic exhibit is a recruitment model trained on past hires that learned the past’s bias and replicated it with machine precision, discovered only when a rejected candidate said so loudly. Two weeks of validation might have prevented the fiasco. The next project tends to skip it anyway.

Pipelines, prompts, and plausible nonsense¶

AI needs clean, relevant, current data; what it often receives is whatever was easiest to grab, half from an ancient API and half from a CSV that someone updates manually before lunch. Personal data gets swept in by accident, bias creeps in through sloppy sampling, and poisoned records can arrive like a prank with very expensive consequences. Prompts, meanwhile, are code, a fact ignored with the enthusiasm of someone who has not yet set their own trousers on fire: developers who would never trust user input elsewhere happily concatenate it into prompts, then look shocked when a user talks the model into revealing its instructions or approving fictional transactions.

And the models hallucinate. This is not a bug but a core behaviour: they produce plausible nonsense, which is awkward when they are deployed for legal, medical, and financial tasks where plausible nonsense is exactly what is not wanted. Code generation is where this becomes structurally interesting, because a hallucinated citation is inert while hallucinated control flow executes. A model can produce code that compiles, passes linting, satisfies a surface reading, and does something different from what was intended; tests written by the same model against the same misunderstanding also pass. The output has not been reviewed. It has been generated.

The quiet rot and the empty chair¶

Models decay while the world moves on, and without monitoring the decay is invisible; dashboards gather dust and a recommendation system can spend years suggesting office snacks to people locked in their homes. Bias does its damage in the same silence. And when the system finally harms someone, the search for a responsible party wanders through baffled staff until responsibility is found to have evaporated: the model cannot answer, the developer will not, the data team thought it was someone else’s problem, and human oversight can turn out to be someone clicking approve. AI systems are software, with all the old vulnerabilities and several new ones. What many receive instead of security is prayer, and sometimes not even that.

The clerk’s brief¶

July 2026: The August wall approaches¶

Most of the AI Act’s remaining rules, including those for high-risk systems and transparency, apply from 2 August 2026, when enforcement begins, according to the Commission’s implementation timeline. The same timeline records that the schedule has already shifted once: under the AI omnibus simplification package, high-risk systems embedded in regulated products now have until August 2028. The clerks note the familiar shape, a deadline announced with fanfare and amended before arrival, and observe that these bolted-on systems will meet the law in whatever state the gold rush left them.

May 2026: The machines acquired identities faster than governance¶

Analysis published by The Hacker News in May 2026 puts the growth of non-human identities, service accounts, API keys and now AI agents, at 44 per cent between 2024 and 2025, with nearly half holding sensitive or privileged access that no lifecycle process reliably retires. Every bolted-on model brings credentials with it. The clerks observe that the agents are multiplying in the manner of swamp dragons, and that nobody appears to be keeping the register.

March 2026: The assistants leak¶

GitGuardian’s secrets sprawl report, published March 2026, finds AI-assisted commits leaking secrets at roughly twice the baseline rate, against a backdrop of 28.65 million new hardcoded secrets reaching public repositories in 2025. The tools that write the code faster also write the mistakes faster, which is the sort of productivity gain the clerks feel deserves closer reading.

February 2026: What the machine writes goes largely unread¶

Black Duck’s research, published February 2026, reports that around 24 per cent of organisations comprehensively evaluate AI-generated code for security, while AI-accelerated development helped double the vulnerabilities found per codebase in a year. Generated code is being treated as a first draft that has already been reviewed. It has not, and the clerks decline to be surprised by what follows.

August 2025: The first obligations landed on the model makers¶

The AI Act’s obligations for providers of general-purpose AI models entered into application on 2 August 2025, along with the requirement for member states to designate national authorities. Accountability began, notably, at the top of the supply chain rather than at the point of deployment, which the clerks consider sensible and also convenient for everyone doing the deploying.

The assay so far¶

The pattern holds steady across the file: capability is adopted at announcement speed, governance at budget speed, and the gap between them is discovered in production, by customers, auditors, or attackers, whoever arrives first. Since August 2025 the law has begun assigning names to the empty chair, with the heavier obligations arriving August 2026 and, for embedded systems, August 2028. The clerks’ standing assessment is that regulation is moving at the speed of committees while deployment moves at the speed of enthusiasm, and that enthusiasm has never once waited.