Where the loop closes¶

Powerful and recursive are different properties, and the argument about these systems has been conducted largely in the first vocabulary. A calculator is powerful. A compiler is enormously powerful. Power is not what puts either of them into what the next one is made from.

Recursion begins when the output of a system changes the conditions under which the next system is produced. That is a claim about shape rather than about capability, and it can hold for a modest system and fail for an impressive one.

The loops are not one loop¶

Several are already running, and they are usually discussed as though they were the same phenomenon: material feeding training, code feeding codebases, tests checking generated code, evaluations scoring generated answers, published research feeding the next research, proposed architectures feeding the next model, and generated infrastructure carrying all of it.

Each has a different relationship with the outside world, and the differences are the part the argument mostly skips.

One loop ends, the other closes¶

A model writes prose. A person reads it and decides whether it is any good. The loop terminates at a person, and an error either stops there or gets published and stops at a reader. Something outside the system has the last word, even when it exercises that word badly.

A model writes code. A model writes the tests. A model runs the tests, reads the result, revises the code and runs them again. Nothing leaves the loop. An error the code and the tests share is not visible from inside as an error at all: it is a fixed point, and the loop will defend it, cheerfully, for as many iterations as it is given.

That second shape is neither new nor confined to artificial intelligence.

The loops that closed long ago¶

Compilers are compiled by compilers. Each version is built with the previous one, and the binary doing the building has not itself been produced by a human-written tool for a very long time, whatever the source says. Chip design tools run on hardware that earlier versions of those tools helped lay out. Both loops closed decades ago. Both produced steady gains. Neither produced anything resembling an explosion.

Ken Thompson’s Turing Award lecture, published in 1984, remains the canonical description of what a closed loop can conceal: a compiler altered to insert a backdoor into the login program, and to reinsert both that backdoor and its own alteration when compiling a future compiler, with nothing in the source to show for it. Every inspection point inside the loop reports correct, because every inspection point is inside the loop. Seeing it requires standing somewhere that was not produced by the thing under test, and finding such a place is the whole difficulty.

Closure is not a synonym for improvement¶

A closed loop compounds whatever it is selecting for. Given a cheap and correct scorer, does it sort, does it compile, is it faster on this processor, closure yields steady improvement nobody writes articles about. Given a scorer that is absent, expensive or gameable, the same closure yields volume, drift, or a system that is excellent at the measurement and poor at the thing the measurement stood for.

The direction is not a property of the loop. It is a property of whatever the loop is using to tell better from worse. Which is why the flood of generated material and the prospect of machines improving machines are not two subjects. They are one arrangement at two settings of one variable.

Entering from outside¶

For each loop there are two questions, and they are not the same question. What enters that the loop did not produce, and can any of it return a verdict? A physical measurement of a fabricated part answers both: it comes from outside, and it can say no. So does a failing test written against a requirement nobody inside the loop invented. The real data that was kept answers only the first: it shapes what the next system learns, and it never tells that system an answer was wrong.

Where the second question comes back empty, the result is coherent and unmoored. The same arrangement occurs with no machinery at all, wherever nothing that could contradict is let in: a company that reads only its own dashboards, a literature that cites itself into consensus, an intelligence service that collects what it already believes. The machinery is new. The failure is older than the machinery, and it has never once announced itself from inside.

The clerk’s brief¶

2016: Machines began proposing the machines¶

Zoph and Le’s neural architecture search used a recurrent network to generate descriptions of neural networks, trained by reinforcement learning to maximise the accuracy of the architectures it generated. An early instance of automated system design rather than a closed loop, published, reproduced, and established as a research technique. The paper records the price of the search: 800 networks training on 800 GPUs concurrently. The clerks note that a decade has passed, that the technique works, that it was out of reach for most groups at the time, and that the discipline it belongs to has gone on being advanced mainly by people.

1984: The compiler that could not be read¶

Thompson’s Turing Award lecture described a self-perpetuating alteration surviving in the compiler binary with nothing in the source to show for it, and drew the moral that code one did not create entirely oneself cannot be trusted on inspection. The clerks file this as the earliest clean statement of the problem: a loop can be correct at every point a person is able to check, and wrong as a whole, and no amount of checking inside it will settle the question.

The ordinary condition¶

Recursion is the ordinary condition of mature engineering rather than a coming event, and the loops that closed longest ago are the ones that have compounded most quietly. What separates them from the loops now being assembled is not the sophistication of the machinery but the presence of a check the machinery did not produce. The clerks’ standing assessment is that the useful question about any of these systems is not how capable it is, but what it is still obliged to be told by something outside itself.