left-pad and the Fragility of Your Dependency Graph

Last Tuesday, eleven lines of JavaScript disappeared from the npm registry and builds around the world stopped. Babel wouldn't install. React Native projects wouldn't build. CI pipelines at companies that had never heard of the package in question went red at more or less the same time.

The package was called left-pad. Its job was to pad the left side of a string. If you shipped JavaScript last week, there's a decent chance your build depended on it and you found out the hard way. The incident has been treated mostly as a joke, but the warning in it deserves more attention than the joke.

What actually happened

A developer named Azer Koçulu maintained a few hundred small open-source packages on npm, among them a library called kik. The messaging company Kik wanted the name, npm sided with the company after a trademark dispute, and Koçulu, objecting to the decision, unpublished all of his packages from the registry. Roughly 273 modules vanished at once.

Most were obscure. left-pad was not. It sat deep in the dependency graphs of some of the most-installed packages in the JavaScript ecosystem, including Babel. Once it was gone, every fresh npm install that transitively needed it failed. Developers who had never typed the words "left-pad" watched their deploys fail with a 404 from the registry.

npm restored the package within hours, an unprecedented "un-unpublish", and has said it will change its unpublish policy so that one person can't remove a heavily depended-on module this easily again. That's welcome, but it addresses one trigger, not the underlying condition.

Eleven lines

For the record, here is approximately the entire package:

function leftpad(str, len, ch) {
  str = String(str);
  var i = -1;
  if (!ch && ch !== 0) ch = ' ';
  len = len - str.length;
  while (++i < len) {
    str = ch + str;
  }
  return str;
}

Nothing you couldn't rewrite in the time it takes to read this paragraph. Its absence still halted build pipelines at serious companies, because nobody chooses their transitive dependencies. You choose ten packages; they choose a few hundred more for you.

Lesson one: know your dependency graph

The uncomfortable question left-pad poses is whether you could list what your build actually depends on. Not your package.json, but the whole graph. A typical modern JavaScript project pulls in hundreds of transitive packages, each maintained by a stranger.

You don't need to audit every line. You do need to know the shape of your exposure:

• Run npm ls occasionally and read the output. The depth usually surprises people.
• Question tiny dependencies. A one-function package saves a trivial amount of typing and adds a permanent external liability.
• Prefer fewer, better-maintained packages over a pile of micro-modules. Every entry in the graph can break your build by being unpublished, by shipping a bad release or by being compromised.

This isn't an argument against open source or npm; the productivity is real. It's an argument for treating dependencies as code you now operate, written by people with no obligations to you.

Lesson two: a 404 shouldn't stop a deploy

The second failure last Tuesday was architectural. Thousands of pipelines were set up so that the registry being unable to serve one file meant nothing could ship. That's an availability dependency on a third party, taken on silently, with no fallback.

Defenses, in rough order of effort:

• Pin exact versions. Semver ranges (^4.2.0) mean every install can resolve differently. npm shrinkwrap locks the entire tree, every transitive package at an exact version, so builds are at least reproducible:

npm shrinkwrap
git add npm-shrinkwrap.json

• Cache or mirror the registry. A local npm mirror or caching proxy (Sinopia is a popular self-hosted option) means packages you've installed once stay installable even if the registry, or one package on it, goes away.
• Vendor what you can't afford to lose. For critical dependencies, checking them into your repository or an internal registry is unfashionable but dependable. Your deploy should depend on your infrastructure, not on the continued goodwill of every maintainer in your graph.

Shrinkwrap alone wouldn't have saved you last week, since a pinned version of a package that no longer exists still 404s. That's why the caching and vendoring layers matter.

Lesson three: your build pipeline is production infrastructure

If a failure in a system prevents you from shipping, treat that system as production. An outage in the npm registry, your CI server, your git host or your artifact storage doesn't take your site down, but it takes down your ability to fix your site. The worst time to discover your deploy pipeline is broken is mid-incident, when you're trying to push the fix.

Yet most teams monitor their website closely and their pipeline not at all. Builds fail into a dashboard nobody has open, and a broken nightly deploy goes unnoticed until someone asks why Friday's feature isn't live. The pipeline should have checks, and its failures should reach a human.

Monitoring the machinery, not just the site

CompleteStatus can watch this layer alongside your public site. Point an HTTP monitor at your CI server, your internal registry mirror and your staging environment, with the same external checks, keyword assertions and alerts you'd put on production. When something like left-pad happens again, you'll hear about the failing pipeline from an alert at 9:04 rather than from a confused developer at 11:30.

Create a free account and put monitors on the systems you deploy with, not just the ones you deploy to.