AI systems bolted on without governance¶

In Ankh Morpork the phrase artificial intelligence has recently become popular with the sort of people whose enthusiasm greatly exceeds their understanding. This is never a good combination. It is the sort of enthusiasm that led the Guild of Alchemists to invent the word unanticipated outcome.

Every guild and half the guildless are racing to bolt AI into their operations. Not because they understand it. Good grief no. More because everyone else is doing it and nobody wants to be the only person in the city who cannot brag about their transformative automation journey over a pint of something that probably counts as a health hazard.

So we now have models nobody has validated. Data pipelines held together with rope, hope, and one elderly Python script that only runs on Thursdays. Prompts that include user input raw and untreated. Documentation that consists of a single comment reading fix later. A new category of civic debt has emerged and it is growing at machine learning speed.

The AI gold rush¶

Every organisation in Ankh Morpork has suddenly decided AI is essential. This is not because they asked the important questions like what problems do we need to solve or how do we do this safely. No. They saw a neighbour announce an AI strategy and panicked.

The pitch is always the same. Increased productivity, automated customer service, infinite content generation, operational transformation, competitive advantage, possibly riches beyond imagining. The usual.

The reality is a bit more… well… Ankh Morporkian. Deploy first, think never. Copy and paste snippets from dubious scrolls. Hope the magic holds. Act surprised when the magic does not hold. Blame the AI. Blame the intern. Blame the moon phase.

The timeline is predictable.

• Month one: We need an AI strategy.

• Month two: The Guild of Merchants announced theirs. Something must be done.

• Month three: Deploy anything. Anything at all.

• Month four: Marketing launches a proudly AI powered initiative that nobody in engineering has heard of.

• Month five: Engineering looks at the announcement with mounting dread.

• Month six: Something is deployed. It works intermittently. People describe it as temperamental to avoid saying broken.

What gets deployed? Whatever managed to work once in a demo. A code sample unearthed from Stack Overflow. A prompt written by a marketing assistant who is still not sure how to spell algorithm. A model chosen because someone saw it mentioned in an alchemist newsletter.

Integration involves duct tape. Security review consists of someone muttering it is only text, what could possibly go wrong. Governance is scheduled for later. Later, of course, never comes.

Models deployed like swamp dragons¶

In an ideal world an AI model would be deployed only after someone understood how it behaved, validated it on local data, tested it against edge cases, documented its limitations, monitored its behaviour, and prepared for drift.

Obviously none of that happens here.

Actual deployment looks like this:

pip install openai
openai.api_key = "sk-please-do-not-tell-anyone"
response = model(prompt)
return response

No testing beyond it worked on my machine. No review. The sort of approach that would cause Sam Vimes to develop a new type of headache.

Predictably everything goes wrong. Training data leaks out through enthusiastic hallucination. Chatbots obediently reveal internal secrets because a customer asked nicely. Bias spreads through systems like malaria in the Shades. Costs spiral because anyone with a mischievous streak discovers that tokens are not free.

All of this is discovered in production by customers, auditors, attackers, and the occasional passing wizard who wished they had not seen it.

The validation gap¶

Models ought to be validated before deployment. People should ask sensible questions like what was it trained on or what sort of errors does it make.

The questions that actually get asked are more… streamlined.

Does it work.

Define work.

It produced text.

Ship it.

Testing consists of a developer trying five cheerful examples. Edge cases are ignored because they are too gloomy. Bias testing is skipped because it involves thinking about uncomfortable realities. The model is declared excellent on the basis of evidence thinner than a troll’s patience.

A recruitment system illustrates this nicely. A model trained on past successful hires. Past hiring was biased. The model learned the bias. The system proceeded to replicate the bias with machine precision. This was discovered only after a rejected candidate loudly pointed it out. Two weeks of proper validation could have prevented the entire fiasco. The actual fallout cost rather more.

Lesson learned? None. The next project will skip validation again.

Data pipelines held together with Python and prayer¶

AI requires data. Clean, relevant, updated data.

What it usually receives is whatever was easiest to grab. Half the pipeline comes from an ancient API. The rest comes from CSVs left on a shared drive that Janet updates manually before lunch.

The script runs nightly except when it does not. Sometimes it fails in dignified silence. Sometimes it drags corrupted records into the model like a drunk dragging home unexpected souvenirs. This is rarely caught before the model begins acting strangely.

Personal data gets swept in by accident. Outdated data sits there like a lump of pre-clacks bureaucracy. Bias creeps in through sloppy sampling. Data leakage contaminates evaluation. Attackers slip poisoned data into the pipeline like a prank with very expensive consequences.

A fraud detection model proudly deployed by the Guild of Moneylenders had been trained only on detected fraud, which meant it never learned about actual fraud. It became frighteningly good at catching low value amateur attempts while leaving professional criminals entirely unbothered. This created a warm illusion of safety. The reality was cold and expensive.

The prompt injection nightmare¶

Prompts are code. A fact ignored with the enthusiasm of someone who has not yet set their own trousers on fire.

Developers who would never trust user input in any other system happily concatenate raw text into prompts. Then they look shocked when users talk the model into revealing system instructions, bypassing internal rules, exposing secrets, or roleplaying as a pirate for no clear reason.

Prompt injection is everywhere. A single mischievous line can convince a model to ignore its guardrails, output confidential data, approve fictional transactions, or generate malware ironically labelled educational.

Some organisations have tried defences. Harder system prompts. Filtering. Validation. Users sail around these in minutes. Proper defences require architectural separation and genuine security layers. Nobody has budget for that because they spent it all on the announcement campaign.

So the real strategy is to hope no one tries anything clever and to add a note in the terms of service that says please behave.

The hallucination factory¶

LLMs hallucinate. This is not a bug. It is a core behaviour. They produce plausible nonsense because that is what they do.

Despite this obvious limitation people deploy them for legal advice, medical explanations, financial guidance, code generation, and other tasks where plausible nonsense is exactly what you do not want.

The results are predictable. Lawyers use hallucinated citations. Medical systems produce invented guidance. Financial tools fabricate regulations. Code generators hallucinate functions that do not exist and practices that nobody should ever use.

Verifying hallucinations is difficult. Humans need to check the output which defeats the point of automation. Another model can check it but the second model hallucinates too. Confidence scores are misleading because the models are very confident about nonsense.

Nevertheless, people forge ahead.

The documentation void¶

Documentation ought to explain what the system is, what it does, what it should never be allowed to do, and how to handle its misbehaviour.

Instead, documentation consists of a comment reading TODO. The person who wrote the system left months ago. The replacement is afraid to touch the code. Nobody knows what data was used for training. Nobody remembers how the prompts evolved. Integrations were glued on with enthusiasm rather than architecture.

When something breaks people gather around the system like wizards around an unexpectedly ticking artefact. They poke it. It makes no sense. Someone applies a patch. It works for now. Everyone agrees to stop asking questions.

Model drift, the quiet rot¶

Models decay. The world changes. The model does not. Without monitoring the decay is slow and invisible.

Most organisations monitor for a month. Then they forget. Dashboards gather dust. Metrics decline. The model becomes steadily less accurate.

One shop’s recommendation system had been running on 2019 data for four years. It missed the pandemic entirely and continued suggesting office snacks during lockdown. Nobody noticed until someone checked and discovered the model had not been retrained since the previous decade.

The bias time bomb¶

Models trained on biased data replicate that bias. This is not speculation. It happens. Hiring systems reject certain groups. Loan systems disadvantage others. Facial recognition misidentifies people. Medical systems reinforce inequity.

Fixing this requires testing across demographics, accessible documentation, and validation before deployment. Most organisations avoid this because it takes time and effort and might reveal problems.

They deploy. They wait. Someone gets harmed. Then they scramble to fix the issue and issue a statement about fairness. Until the next incident.

The accountability vacuum¶

When AI makes a mistake nobody claims responsibility.

The model cannot. The developer will not. The data team did not think it was their problem. Management trusted the experts. Executives claim strategic oversight rather than operational detail. The result is a moral and operational void.

A customer denied an insurance claim by an AI system tries to find a responsible party. The journey takes them through layers of baffled staff only to reach the conclusion that responsibility has evaporated into thin air.

Regulation ought to fix this with named accountability. Instead we get vague human oversight. This oversight typically consists of someone clicking approve without understanding what they are approving.

The security afterthought¶

AI systems are software. They have vulnerabilities. They add new ones. They need security.

What they receive is prayer. And sometimes not even that.

APIs are left open. Credentials are hardcoded. Data is exposed. Models are stored unsecured. Pipelines accept untrusted inputs. Adversaries poison models, extract training data, or rack up bills that would make Vetinari twitch.

Security reviews do not happen because everyone assumes text is harmless and everything will be fine.

Everything does not seem fine to me.