The S3 Outage — Why Your Status Page Can't Live on Your Own Infrastructure

Last Tuesday afternoon, Amazon S3's us-east-1 region went down for roughly four hours, and we all got a live demonstration of how much of the web is quietly built on one storage service in northern Virginia. Images vanished from major sites, file uploads failed everywhere at once, smart-home devices went dumb, and entire applications — ones whose owners would have sworn they "don't really use AWS" — simply stopped working.

But the detail everyone will still be telling at conferences in five years is this: for a good chunk of the outage, AWS's own status dashboard showed green checkmarks. Not because anyone was hiding anything — because the red and yellow status icons were themselves hosted on S3. The status page for the outage was a casualty of the outage. AWS ended up announcing the problem on Twitter.

What happened, in Amazon's own words

Amazon's postmortem is admirably direct. An engineer on the S3 billing subsystem, debugging a slowdown, ran a command from an established playbook intended to remove a small number of servers from one subsystem. One parameter was mistyped. The command removed a much larger set of servers than intended — including servers supporting S3's index and placement subsystems, the machinery that knows where every object lives and where new ones should go.

Those subsystems required a full restart. And because S3 in us-east-1 had grown enormously over the years without those particular systems ever being fully restarted, the restart-and-verify process took hours, not minutes.

Note what this story is not: it's not a villain, a hack, or exotic hardware failure. It's a typo, executed with legitimate credentials, inside a documented procedure — the most human outage imaginable. Amazon's fixes (tooling that limits how fast capacity can be removed, and refactoring the slow-to-restart subsystems) are process fixes, because it was a process failure. If your own runbooks let one mistyped argument remove unbounded capacity, you're one bad Tuesday away from the same story.

The dashboard that couldn't turn red

The status page failure is the lesson worth engraving somewhere. A status page has exactly one job: keep working when everything else doesn't. That means it must share as little fate with your production systems as you can arrange:

• Different infrastructure, ideally a different provider. If your app runs on AWS, a status page on S3 — or behind anything in the same region — is a status page that fails in sympathy. Host it somewhere whose bad day is unlikely to be the same bad day.
• Different DNS. If your nameservers are down, status.yourapp.com on the same nameservers is down too. A separate DNS provider (or at least a well-cached, long-TTL record) keeps the front door reachable.
• No dependencies on your stack. The status page shouldn't need your database, your API, your asset pipeline or your single-sign-on to render. Static, boring and self-contained is exactly right.
• Updatable from a phone on coffee-shop wifi. During a serious incident, your VPN, your deploy pipeline or your office network may be part of the problem. If posting an update requires production access, you've built a diary, not a status page.

The test is simple: unplug production in your head, then ask whether customers can still find out what's going on. Amazon — with more infrastructure than nearly anyone on Earth — failed that test in public last week. It's worth five minutes to check whether you'd pass it.

Do you know what quietly depends on S3?

The second surprise of the day was how many teams discovered dependencies they didn't know they had. The app was "up" — and half its features weren't. It's worth literally grepping for:

grep -rn --include="*.php" --include="*.js" --include="*.yml" \
  -e "s3.amazonaws.com" -e "s3-us-" -e "AWS_BUCKET" .

The usual suspects, from that day's casualty list:

• User uploads and avatars — the classic. Pages render, every image is a broken icon, every upload form errors.
• Static assets and downloads — JS bundles, PDFs, installers, podcast audio served from a bucket.
• Background jobs — exports, imports, report generation and backup jobs that read or write S3 and started failing silently.
• Third parties' dependencies — your support widget, payment provider or analytics script may lean on S3 even if you don't. Their dependency is transitively yours.

You don't need to eliminate these dependencies — S3 is still extraordinarily reliable — but you should be able to list them, and each one should fail visibly and gracefully rather than mysteriously.

"Down" versus "degraded" — monitor the difference

Here's the monitoring lesson hiding inside the outage: for most affected sites, the homepage returned 200 OK the entire time. A naive uptime check saw nothing wrong — while signups failed because avatar upload errored, checkouts failed because invoice PDFs couldn't be written, and dashboards rendered blank.

Hard-down is the easy case; any monitor catches it. Degradation is where real revenue quietly leaks, and catching it means checking features, not just the front door:

• Keyword checks — assert the page contains what a working page contains, so a 200-with-error-message counts as down.
• Transaction-critical endpoints — monitor the API route behind checkout or upload, not just the marketing page in front of it.
• The dependency itself — a tiny probe object fetched from your bucket tells you your storage path works, minutes before support tickets do.

The multi-region reality check

The reflexive post-outage advice — "just go multi-region!" — deserves skepticism. True multi-region operation is expensive, architecturally invasive and easy to get subtly wrong; for most small teams it's the wrong first dollar to spend. The honest middle ground:

• Know your blast radius. Which features die if us-east-1 does? Deciding "we accept four hours of broken uploads per multi-year event" is legitimate — deciding it in advance is the part that matters.
• Degrade on purpose. Cache what you can, queue writes for retry, show a friendly "uploads are temporarily unavailable" instead of a stack trace.
• Put the cheap redundancy where it counts — which brings us back to the status page, the one component where independence costs almost nothing and pays for itself in a single incident.

Be the first to know, and have somewhere to say it

The teams that came out of last Tuesday looking good shared two traits: they knew about the problem before their customers told them, and they had a working place to acknowledge it. Both are cheap to arrange and miserable to improvise.

CompleteStatus handles the first half — external uptime and keyword checks that catch degradation, not just hard-down, with alerts by email, Slack or webhook the moment a check fails — and our checks run from infrastructure independent of yours, which is the whole point. For the second half, the status pages guide covers setting up a page that doesn't share fate with your stack. We wrote last week about inheriting your CDN's bugs; this week the internet learned the same lesson about storage. Create a free account and find out about the next us-east-1 afternoon from your monitor, not from Twitter.