Four loops that already run¶

“AI designs AI” is either trivially true or wildly ambitious, depending on which loop is meant. Four are already running. Machines write code that people ship. Machines search a design space against a measurement. Machines propose research, run it and write it up. And machines build the apparatus the other three depend on. What separates these, is not how clever any of them is. It is what it costs to find out whether the output was any good, and that cost decides whether a loop goes anywhere at all.

The parts list¶

Designing one of these is not a single act, and the system is not a single artefact. It is a training corpus, an architecture, a set of hyperparameters, the kernels the arithmetic actually runs on, a preference dataset that shapes how the thing behaves, an evaluation suite, and a stack to serve it from. The useful question is not whether a machine can design an AI system. It is who produces each of those parts today.

Architectures are proposed by machines, and have been for a decade. Hyperparameters, once a matter of a researcher’s intuition and a great many wasted runs, are searched: population based training discovers “a schedule of hyperparameter settings rather than following the generally sub-optimal strategy of trying to find a single fixed set to use for the whole course of training”. The kernels are searched too. Ansor generates tensor programs by sampling a space of possible implementations and ranking them with a learned cost model, reporting gains of up to 3.8 times on one processor family over the methods it replaced. Evaluation is increasingly one model scoring another model’s answers. And the preference data that shapes behaviour can be machine-written: Anthropic’s constitutional method trained a harmless assistant “without any human labels identifying harmful outputs”, where “the only human oversight is provided through a list of rules or principles”.

That leaves the two ends of the list. The corpus is machine-written in growing part and machine-filtered before use, and the filter is itself increasingly a model. And the model that gets served is often not the model that was trained: distillation compresses what a large system learned into “a single model which is much easier to deploy”, so the thing answering in production is frequently a smaller machine built from the output of a larger one. The phi-4 report goes further, describing a model that surpasses its own teacher on science and mathematics questions, which its authors offer as evidence that the technique is more than compression.

So the answer to whether a machine can design an AI system, everything included, is that most of the parts list is already machine-produced and has been for years. It is also, so far, dull. What remains in human hands is smaller and more specific than the argument usually allows: what the system is for, what counts as success, which failures are unacceptable, and whether the thing may ship at all. Every part that got automated had a measurement attached to it. Every part that has not is one where the measurement is somebody’s judgement.

The diff somebody reads¶

A person decides what is being built, a machine writes the code, and the person integrates it. This is now ordinary, unremarkable in most working weeks, and the loop is not closed: somebody reads the diff and owns the result.

Whether that reading actually happens is a separate question, and where it does not, the loop has closed without anybody deciding to close it. That is the cheapest way for a check to disappear. Nobody removes it. It simply stops being performed.

Where the measurement cannot be argued with¶

Machines have been proposing designs and testing them against a measurement for a decade. Neural architecture search had a network generating descriptions of networks in 2016, scored by validation accuracy. AlphaTensor found matrix multiplication algorithms that beat the best known for several sizes, including a result for four by four matrices in modular arithmetic that improved on Strassen’s two-level method for the first time in about fifty years. AlphaDev’s sorting routines went into the LLVM standard C++ library, which is to say into production, where they have stayed.

These are genuine recursive loops and the strongest cases available, so the overstatement attaches itself to them. Matrix multiplication is the operation a training run is mostly made of, so AlphaTensor looks like a machine improving the thing machines are built from. The honest version is narrower: the headline result is in a finite field rather than in the floating-point kernels a training run actually uses, and the practical speedups reported are hardware-specific and modest.

What these loops share is a scorer that cannot be talked round. Does the algorithm produce the correct product, does the routine sort, is it faster on this processor. The measurement is cheap, repeatable, and indifferent to how impressive the proposal sounded. Loops of this kind have compounded for years without producing anything an observer would call an explosion.

Cheap to write, expensive to judge¶

The next loop up has a machine choosing what to investigate, running the investigation, and writing it up. Sakana’s AI Scientist produced entire papers this way, and an independent evaluation went through what came out. Two in five experiments failed outright on coding errors. Established techniques were reported as novel. Citations were thin and mostly old. Some papers contained numerical results that had not come from any experiment. All of it for a few dollars and a few hours of human attention per paper.

The cost of producing a research artefact collapsed. The cost of determining whether the artefact was any good did not move at all, because that cost is peer review, and peer review is scarce and slow. A loop whose output is cheap and whose scorer is expensive does not accelerate discovery. It accelerates submission.

The apparatus nobody reads¶

The fourth loop is the least documented and the least visible from outside. Training pipelines, evaluation harnesses, data cleaning, orchestration, the tooling that decides which experiment runs and what gets recorded about it. Generated code is throughout this layer, because it is quick to produce and the work is unglamorous, and it invites less scrutiny than model code, because attention goes to the number a harness prints rather than to the harness printing it.

A fault in a model is a bad answer somebody can see. A fault in the apparatus is a number that looks fine.

The threshold people reach for is the moment a machine writes the code for a machine, which passed some time ago and changed remarkably little. A better threshold is the point at which nobody can say cleanly which parts of a development cycle were done by a person and which by the machinery. That arrives one layer at a time, and it arrives through the apparatus rather than through the model.

How long it can be left alone¶

The quantity to watch is therefore not how clever a system appears but how long a piece of work it can carry unsupervised, since that sets how much arrives to be checked at once. That measure records what a system can attempt, not what anything else can confirm. METR has been measuring exactly this, as the length of task a system completes at a given success rate, and reports a doubling roughly every seven months across six years of models. The organisation notes on the same page that the figure has since been revised. The measure is confined to software tasks, depends heavily on how tasks were chosen, and the extrapolation is doing more work than the measurement.

The confirming is done by experiments, and experiments occupy hardware for a wall-clock duration that no amount of cleverness compresses.

The clerk’s brief¶

March 2025: A doubling every seven months¶

METR published a time horizon measure, the length of task a model completes with 50 per cent success, and found it doubling approximately every seven months over six years, with the more recent period looking faster. The same page records that the figure is out of date and has been superseded. The clerks note the caveats the authors themselves supply, that the tasks are software tasks and the selection of them moves the result, and observe that a doubling in what can be attempted is not a doubling in what can be verified.

February 2025: Two in five experiments did not run¶

An independent evaluation of Sakana’s AI Scientist found that “42% of experiments failed due to coding errors, while others produced flawed or misleading results”, that the system misclassified established concepts as novel, that manuscripts carried a median of five citations with only five of 34 from 2020 or later, and that some contained hallucinated numerical results. Cost per paper was six to fifteen dollars with three and a half hours of human involvement. The clerks record the ratio as the point: the production of research became almost free while the evaluation of research became no cheaper at all.

June 2023: Machine-written sorting routines entered the standard library¶

DeepMind’s AlphaDev discovered sorting algorithms that were integrated into the LLVM standard C++ sort library, with improvements of up to 70 per cent for sequences of length five and roughly 1.7 per cent for sequences over 250,000 elements. The clerks note that this is a machine improving a routine a great deal of software calls without knowing it, that it went into production and stayed there, and that the reason it could be trusted is that a sorting routine either sorts or does not.

October 2022: Fifty years of arithmetic improved upon¶

AlphaTensor found an algorithm multiplying four by four matrices using 47 multiplications in a finite field, against the 49 of Strassen’s two-level algorithm, along with faster algorithms for many other sizes and thousands of variants per size. The clerks flag the temptation to read this as a machine accelerating its own training, and record that the headline result belongs to modular arithmetic while training runs use floating point.

Where the boundary sits¶

Every genuine case in this file has the same property: a criterion specified outside the loop, and cheap enough to apply that applying it was never the bottleneck. Every disappointing case has the same one in the negative. The clerks’ standing assessment is that the boundary of machine self-improvement currently sits where measurement gets expensive, that this boundary has moved rather slowly, and that a system able to propose ten times as much work in a year when nothing can check ten times as much work produces a backlog rather than a breakthrough.