The Facebook Outage — What BGP and DNS Taught Everyone This Week

On Monday, Facebook disappeared from the internet. Not "was slow," not "some users affected": the domain stopped resolving, along with Instagram and WhatsApp, for roughly six hours. Billions of users cut off, because of a change made by the company itself. Facebook's engineering team has now published its own account, and the outline is striking: during routine maintenance, a command intended to assess global backbone capacity instead took down the connections between Facebook's data centers, and in the cascade Facebook's BGP routes to its own DNS servers were withdrawn from the internet.

If you run any kind of online service, this week's outage is one of the most instructive incidents in years, not because you operate at Facebook's scale but because you don't. The failure modes are universal; only the blast radius was exceptional.

What actually happened, briefly

Two acronyms carried the whole story:

• BGP (Border Gateway Protocol) is how networks announce to the rest of the internet "these IP addresses live here, route traffic to me." When Facebook's DNS servers detected they'd lost connection to the data centers, they responded, by design, by withdrawing their BGP announcements. The internet's routing tables simply forgot where Facebook's DNS servers were.
• DNS then finished the job. With the authoritative nameservers unreachable, facebook.com, instagram.com and whatsapp.com stopped resolving everywhere on Earth. To every resolver, every browser, every app, the domains effectively ceased to exist.

The worst part came next: Facebook's own engineers couldn't reach the internal tools needed to fix the problem, because those tools lived on the same infrastructure that had just vanished. Engineers had to be sent to the data centers to work on the hardware directly, and there are widespread reports that even badge access to some buildings was affected. The remediation took hours partly because the people fixing it were locked out, digitally and, reportedly, in some cases physically.

Lesson 1: don't put all your eggs in one infrastructure basket

Facebook runs its own DNS, its own backbone, its own everything, and one bad command took it all down together because it was all one system. Most teams have the opposite failure: everything on one cloud account, one DNS provider, one hosting box, one person's credentials.

You don't need Facebook's budget to apply the principle:

• Separate your DNS from your hosting where practical, so a hosting-level failure doesn't take out your ability to repoint traffic.
• Keep your status page on completely separate infrastructure: different host, different DNS, ideally a different provider entirely. A status page that shares fate with the thing it reports on is decoration. When everything is down, the one page that isn't is all users and support staff have.
• Ask the fate-sharing question in reviews. For each critical component: "if X is down, can I still see, reach, and fix Y?" Monday's answer at Facebook was no, several layers deep.

Lesson 2: external monitoring tells you what users actually see

Inside Facebook, the servers themselves were presumably running fine; the internet just couldn't reach them. This is the defining blind spot of internal monitoring: it measures your infrastructure from a vantage point your users don't have.

External monitoring, meaning checks that originate from the public internet and resolve your DNS the way a stranger's laptop would, is the view that matches reality. It would have said immediately: the domain does not resolve; nothing else matters right now. For a normal team the equivalent scenario is mundane and common: your registrar suspends a domain over a billing hiccup, a nameserver change goes wrong, an expired domain auto-parks. Server dashboards stay green through all of it. Only an outside-in check notices, and DNS-level failures deserve their own checks, not just HTTP ones, because knowing which layer failed is half the diagnosis.

Lesson 3: DNS deserves respect and redundancy

DNS is the layer everyone forgets because it almost always works. Monday is what it looks like when it doesn't. Practical hygiene for teams of any size:

• Use more than one nameserver, on more than one network. Most managed DNS providers do this by default; verify yours actually does, rather than listing four names that resolve to the same infrastructure.
• Know your TTLs. Long TTLs (hours) mean caches shield you briefly during a DNS outage but also mean emergency changes propagate slowly. Know what yours are set to before the incident.
• Monitor your DNS records themselves. Unexpected changes to A, MX or NS records are an early-warning signal for misconfigurations and hijacks alike.

Lesson 4: out-of-band access to your own systems

The detail everyone will remember from this outage is engineers locked out of the tools (and reportedly the buildings) needed to fix it. Scale it down and it's a familiar story: the runbook is in a wiki that's down, the SSH bastion resolves via the DNS that's broken, the admin panel is behind the SSO that's offline.

The fixes are unglamorous and cheap:

• Keep copies of critical runbooks somewhere that shares nothing with production, even a printed sheet or an offline doc.
• Maintain a break-glass access path: a direct IP for at least one management host, provider console credentials stored outside your own SSO, a list of provider support phone numbers that doesn't live on the infrastructure it's for.
• Test the path. An untested emergency access route is a hope, not a plan.

Lesson 5: hours of downtime happen to everyone

The last lesson is humility. Facebook employs some of the best infrastructure engineers in the industry, and it was hard down for six hours, with lost revenue estimated in the tens of millions of dollars. If it can happen there, it can happen to you. The goal isn't zero incidents; it's knowing within a minute, understanding which layer broke, and having a fixing path that doesn't depend on the broken thing.

That first part is what CompleteStatus does: uptime, DNS and SSL checks run from outside your infrastructure, so when your domain stops resolving or your site stops answering, you hear about it from your monitoring, not from Twitter. Alerts go to email, Slack and webhooks the moment an outage is confirmed, and because the monitoring lives on infrastructure that isn't yours, it keeps working when yours doesn't. If you don't have outside-in monitoring yet, this week is the argument for it. Set it up free in a few minutes.