Dependency hell in software supply chains¶

If you ask the average European technology department what their software is built on, they will puff themselves up, gesture dramatically at their codebase and announce that they have constructed a magnificent digital edifice. If you look under the floorboards, however, you discover that the magnificent edifice is actually teetering on a foundation of assorted crates, barrels and borrowed planks acquired from a thousand strangers. The whole thing looks rather like a rickety warehouse in Ankh Morpork, held together mostly by hope, rust, and a prayer to any gods that have not yet resigned in exasperation.

Europe runs on oceans of open source libraries. This is excellent. It is also a sort of ongoing civic hazard.

Nobody budgets for upkeep. Nobody looks at transitive dependencies until they start smoking. Every suggestion to upgrade something to version 5.12.37 is met with the sort of expression people normally reserve for dentistry performed by a troll.

Every unmaintained module is interest accruing security debt. It grows in the dark like mould in a Morporkian cellar.

The invisible foundation¶

Modern software is not crafted from scratch in the noble tradition of artisan dwarves. It is assembled the way a Morporkian entrepreneur assembles a tavern. Start with a room, add bits until customers stop complaining, and hope nothing collapses during peak hours.

You write perhaps ten thousand lines of your own code. The rest, all two million or so, are the work of strangers scattered across the multiverse. They each had their own priorities, stylistic choices, grudges and peculiar ideas about indentation. Your application rides atop their collective efforts like a tower built on top of another tower built on top of a wobbly pallet someone found by the docks.

A simple web application can easily include:

• Your code. Five thousand lines.

• Direct dependencies. A few dozen.

• Transitive dependencies. Hundreds or even a thousand.

• Total lines of dependency code. Several million.

• Authors. Many. Locations. Everywhere.

• Maintenance status. Mixed in the same way a plate of sausages in a back alley eatery is mixed.

When the dust settles, you wrote about 0.15 percent of your own application. The rest is faith. Faith that someone else’s code does what it is supposed to. Faith that it does not do things it is not supposed to. Faith that the maintainers are both competent and sober. Faith, in short, that the universe is kinder than experience suggests.

That is a great deal of faith.

The open source miracle¶

Open source is brilliant. It is civilisation’s finest communal pot. Everyone tosses something in, everyone eats, nobody starves. Without it, modern software would cost millions, take decades, and still come out looking like a broken wagon with a coat of paint.

So everyone uses it. Governments, banks, telecoms, hospitals. Entire sectors built on communal volunteer labour. The economic model is simple. Consumption is free. Contribution is optional. Maintenance is a volunteer sport.

Everything works wonderfully. Until it does not.

Nobody budgets for updates¶

A typical organisation builds an application, sprinkles dependencies into it like seasoning, launches it, and declares victory. Budgets are allocated for the initial construction. Budgets are not allocated for the fact that software ages like fish.

• Year one. Marvellous.

• Year two. A few vulnerabilities. Nothing dramatic.

• Year three. Testing needed. Nobody has time.

• Year four. Breaking changes loom. Someone mutters that now is not a good moment.

• Year five. The thing is running on dependencies old enough to vote. The known vulnerabilities could fill a report thick enough to fend off a bar brawl. There is no budget to fix it. Everyone pretends this is fine.

Fixing a vulnerability in your own code is expected. Fixing one in someone else’s dependency is also your responsibility, but nobody told Finance, so here we are.

Debt grows. Risk grows. Everyone hopes the auditors do not look too closely.

The transitive dependency nightmare¶

You choose your direct dependencies. The transitive ones choose you.

Install one package and four cousins arrive uninvited. Install fifty packages and you acquire a thousand. Some of these dependencies nest twenty layers deep like an extended Morporkian family, all of whom have opinions about dinner.

When a vulnerability appears in Package E, buried under layers A to D, nobody even remembers what E is. The scanner complains. Developers ask the Four Questions of Despair.

What is it? Why is it here? What depends on it? What will explode if we upgrade it?

The answer is usually everything.

Given three choices, most teams pick the traditional one. Accept the vulnerability, write a soothing note in the risk register, and move on.

Multiply by hundreds of dependencies and you have a daily lottery of misery.

The upgrade cascade problem¶

A manager says upgrade library X. A developer hears prepare for suffering.

Library X version 3.0 is vulnerable. Version 3.5 fixes it. Lovely. Unfortunately the application runs on Framework Y version 2.0 which only supports library X version 2.x. To upgrade X, you must upgrade Y. Y introduces breaking changes. Those breaking changes break half your other libraries. Before long, a one line upgrade has transformed into a seven week expedition up a mountain of broken builds.

The business weighs security against shiny new features. The shiny wins. Vulnerabilities remain.

This is normal. Depressing, but normal.

The abandoned maintainer problem¶

Open source is carried on the shoulders of volunteers. These volunteers have day jobs, families and, sometimes, volcanic tempers. They age. They burn out. They vanish mysteriously.

When a maintainer disappears, the package does not scream. It simply goes quiet. Issues multiply. Vulnerabilities sit unattended like suspicious packages. Eventually a security announcement forces someone to look and discover the package has not been touched since the Year of the Echidna.

Your options range from bad to worse.

Use the abandoned package. Adopt a fork. Rewrite your code. Or maintain it yourself and inherit a thousand unseen users.

Left pad’s brief career as the brick that brought half the internet down is still whispered about in dark taverns.

The version pinning dilemma¶

You can pin versions to ensure consistent builds. This is marvellously stable until the pinned version becomes a fossil with a critical vulnerability.

You can allow version ranges so that new patches flow in automatically. This ensures you will occasionally be woken at 2 in the morning because production is on fire.

Teams pin versions and promise to update them regularly. These promises age faster than milk.

The dependency confusion attack¶

Attackers have learned that if you cannot breach the gate, you poison the well instead.

By publishing a malicious package with the same name as an internal company package, attackers can trick the package manager into fetching the wrong version. Some of the largest companies on the planet have fallen for this.

Fixing it requires configuration effort. A surprising number of organisations have not invested that effort. Most remain quietly vulnerable, hoping nobody notices.

Hope is not a strategy, but it is cheap.

The malicious package problem¶

Publishing to npm or PyPI requires nothing more than a keyboard and a slight disregard for consequences. This is splendid for creativity but equally splendid for crime.

Typosquatting catches sloppy typists. Account compromise turns benign packages into malware delivery services. Hijacking abandoned packages is a popular pastime for wrongdoers with patience.

Scanners catch known threats. They do not catch clever ones. They do not catch subtle ones. They certainly do not catch ones that only misbehave on Sundays when the moon is waning.

Trust is extended liberally in the open source world. Revocation is difficult. The security model resembles a marketplace with no guards and very enthusiastic pickpockets.

The maintainer burnout cycle¶

The more successful a package is, the faster its maintainer burns out. A few volunteers support entire industries. Heartbleed and Log4Shell were not caused by incompetence. They were caused by overworked volunteers doing heroic tasks for no pay while the world built skyscrapers on their foundations.

Everyone agrees the model is unsustainable. Everyone waits for someone else to fix it.

The corporate exploitation¶

Corporations extract enormous value from open source. The return flow is less generous. Some companies contribute back. Much of the sector simply consumes.

Critical infrastructure rests on packages maintained by exhausted volunteers. Businesses save millions. Maintainers get a mug and perhaps a pleasant thank you email if the stars align.

The imbalance continues because there is no rule that says you must put a coin in the jar after raiding the communal pantry.

The security scanning theatre¶

Security scanners are marvellous at generating reports. They are less marvellous at generating solutions.

A typical report lists hundreds of vulnerabilities. Teams panic briefly and then give up. After enough cycles, the report becomes a background hum, like the river Ankh. Ever present. Largely ignored.

Tools rarely know whether a vulnerability is exploitable in your context. They do not know if you use the affected function or if the vulnerability is effectively theoretical. They provide no prioritisation and often include false positives for extra seasoning.

The result is a ritual. Scan. Panic. Ignore. Repeat.

The upgrade avalanche¶

Every dependency has its own saga of changelogs, migrations, and hidden pitfalls. Upgrading even one can take weeks. Multiply by fifty and you have an avalanche of postponed tasks and a development team that has sworn never to attend another planning meeting.

The framework lock in problem¶

Frameworks give speed and structure. They also create a dependency chain so tangled it might qualify as a magical curse.

A vulnerability in a framework dependency cannot be fixed until the framework maintainers say so. When they do finally update, the upgrade comes with breaking changes that require major rework. Migrating off a dead framework is so expensive that many teams never do it. They simply accept the growing risk and hope the roof holds for one more winter.

The JavaScript special circle of hell¶

The JavaScript ecosystem has invented the micro package culture. Why write a function locally when you can download a package maintained by eleven people across four continents. Installing anything in npm drags in a parade of dependencies of such size and enthusiasm that entire civilisations could be lost in the lower levels.

npm audit runs. It finds problems. Its suggested solutions break things. Not following its advice leaves you vulnerable. There is no winning. There is only differing flavours of defeat.

The Python and Java ecosystems have their own charming nightmares¶

Python’s naming confusion, unmaintained packages, C dependencies and version compatibility snarls have trapped many an unsuspecting engineer.

Java’s enterprise environment values stability above sanity. Applications run for decades. Dependencies rot. Upgrades are rare and traumatic. Maven dependency graphs resemble archaeological layers. Every excavation reveals something that should probably have been left buried.

Outcome¶

If dependency hell had a map, it would look suspiciously like Ankh Morpork. Everything built on everything else, everything leaking into everything else, and absolutely no guarantee that the bricks at the bottom are still where you left them.

And yet here we appear to be. Still building. Still shipping. Still pretending that nothing is on fire.