The queue decides¶

Over two years at a teaching hospital in Leuven, prescribers were shown 38,409 alerts about very severe drug interactions and overrode 33,888 of them. The software worked. Every alert fired because a rule matched a prescription, and the rule was right about the interaction it named. What the count records is the point at which a person stopped being able to treat any single alert as information.

Nobody decided that. No meeting resolved that nearly nine in ten severe warnings would be dismissed. The threshold that decided how many alerts a prescriber sees was set somewhere upstream, by people choosing sensitivity, and the consequence arrived at a different desk.

The step before the decision¶

A system does not have to make the final call to settle the outcome. It can decide which cases reach a person, which wait, which are escalated and which are closed without anybody looking. The judgement stays where the organisation chart says it is. The choice set does not.

This is a different complaint from the familiar one about automated decisions. An automated decision has an output somebody can point at, appeal against and in some jurisdictions demand an explanation for. An overridden alert is such an output; the sensitivity setting that produced 38,409 of them is not. A selection produces an ordering, and an ordering is a fact about a list rather than about a person, which is why it is difficult to be told about and harder to contest.

What an overseer is given¶

From 2 August 2026, high-risk systems in the European Union carry an obligation of human oversight, and the Act is unusually specific about what the overseer has to be able to do: to decide “not to use the high-risk AI system or to otherwise disregard, override or reverse the output”, to interrupt it, and to “remain aware of the possible tendency of automatically relying or over-relying on the output”, which the drafters name as automation bias.

Every one of those powers attaches to the output. A prescriber can override an alert, and 33,888 of them did. None of those powers reaches the sensitivity setting that decided how many alerts arrived, and a person cannot disregard a case that was never shown to them. Oversight of the output is not oversight of the ordering, and the second is where the caseload was determined.

Selecting without deciding¶

The Dutch state ran a system called SyRI from 2014, which linked data held by different public bodies and scored it against risk indicators that were never disclosed, generating risk reports on people it rated as presenting a higher likelihood of benefits fraud. It stopped nobody’s payments. It produced a name and a reason to look, and what followed was that somebody went and looked.

On 5 February 2020 the District Court of The Hague found that the SyRI legislation did not meet the requirements of Article 8(2) of the European Convention on Human Rights, and ordered that its use stop (ECLI:NL:RBDHA:2020:865, English text at ECLI:NL:RBDHA:2020:1878). The case is usually filed under privacy, and it belongs there. The part relevant here is what SyRI produced. A selection is not a determination of anybody’s entitlement, which is why it ran for six years without attracting the safeguards that attach to determinations.

Review as the exception path¶

Australia’s income compliance programme compared tax office figures against what welfare recipients had reported, averaged the annual figure across fortnights, and put the onus on the recipient to show they had reported accurately, for periods up to six years earlier. Failing that, a debt notice followed. Between 2016 and 2019, 526,000 people received one. The Social Security Act allows a debt to be raised against income actually received, and an annual figure divided by twenty-six produces what the Solicitor-General’s 2019 advice called mere conjecture.

People appealed, and the tribunal agreed with them. Over those same years the Administrative Appeals Tribunal issued 327 decisions raising issues with income averaging, some stating explicitly that it had no basis in law. The department appealed none of them, which kept every one unpublished, and meant the scope of the negative decisions stayed invisible to the department that owned the policy. The tribunal was hearing thousands of social security cases a year, and the department’s lawyers were instructed to handle them in a “very transactional” way, resolving the case in front of them.

So the exception path worked, one applicant at a time, and reported nothing. The Royal Commission that reported on 7 July 2023 called the scheme “a crude and cruel mechanism, neither fair nor legal”, and recommended that whatever body replaced the tribunal publish first-instance decisions involving significant conclusions of law. The remedy proposed is a publishing requirement, which locates the defect precisely: not in the decisions, which were correct, but in there being nothing to carry them anywhere.

The clerk’s brief¶

August 2026: The oversight duty arrives, attached to the output¶

The obligations for high-risk systems listed in Annex III became applicable on 2 August 2026. Article 14 requires that those systems be designed so that they “can be effectively overseen by natural persons during the period in which they are in use”, and that overseers be enabled to disregard, override or reverse the output, to interrupt the system, and to remain aware of automation bias. The clerks note that the duty is drafted around a system that produces an output for a person to accept or reject, and that a system which produces an ordering has, in that vocabulary, produced nothing for anybody to override.

July 2023: Three hundred and twenty-seven decisions, none of them published¶

Commissioner Catherine Holmes presented the Report of the Royal Commission into the Robodebt Scheme to the Governor-General on 7 July 2023, recording at page xxix of its overview that “Robodebt was a crude and cruel mechanism, neither fair nor legal, and it made many people feel like criminals”. Between 2016 and 2019 the Administrative Appeals Tribunal had issued 327 decisions raising issues with income averaging, some stating explicitly that it had no basis in law, none of which the department appealed and none of which was therefore published. The Commission found the tribunal’s findings had been effectively ignored for want of any mechanism to review them systematically, and its recommendations include publishing first-instance decisions that involve significant conclusions of law. The clerks note what that remedy concedes: the decisions were right, the applicants who reached the tribunal were vindicated, and nothing in the arrangement was built to carry either fact back to the thing issuing the notices.

2022: Eleven point eight per cent¶

A two-year study at University Hospitals Leuven recorded 38,409 very severe drug interaction alerts, of which 33,888, or 88.2 per cent, were overridden, giving an initial acceptance rate of 11.8 per cent. The clerks record this as the cleanest available measurement of the effect, since the alerts were correct, the prescribers were qualified, and the arrangement still converted a safety mechanism into something a person clicks past. Whoever set the sensitivity chose the acceptance rate, some years earlier, without being in the room when it was paid.

February 2020: The system that only pointed¶

On 5 February 2020 the District Court of The Hague ruled that SyRI, in use by the Dutch state since 2014 to link data from different public bodies and generate risk reports on people rated as more likely to be committing benefits fraud, did not meet the requirements of Article 8(2) of the European Convention on Human Rights, and ordered its use to stop (ECLI:NL:RBDHA:2020:865). The risk indicators were never disclosed. The clerks note that SyRI decided nothing about anybody’s entitlement, that this was the ground on which it was defended, and that it operated for six years because pointing at people is not, in the ordinary reading, a decision about them.

The step nobody recorded¶

Each case in this file has an actor who chose an outcome and a record that does not contain them. A sensitivity setting fixed an override rate at Leuven. An averaging rule fixed who would receive a debt notice in Australia. A risk indication fixed who would be visited in the Netherlands. In each, a person retained the authority to judge and received a list they had no part in assembling. The clerks’ standing assessment is that every power in this file points at an output, the override the Act confers and the appeal the tribunal hears, and that the single case where anything reached a selection itself needed a court and six years to do it.