1.35 Terabits Per Second: What the GitHub DDoS Teaches Everyone Else

Last Wednesday, February 28th, GitHub absorbed the largest DDoS attack publicly recorded at the time, with traffic peaking at 1.35 terabits per second, well above the 2016 Mirai attacks on Dyn and OVH. GitHub was back within about ten minutes. There was no botnet of hacked cameras this time: just spoofed UDP packets and thousands of misconfigured servers doing the attacker's work.

The record has already been broken. Reports this week describe an attack of around 1.7 Tbps against a US service provider using the same technique. It's worth understanding even if you'll never see a terabit of traffic, because the lessons apply at any size.

How memcached amplification works

The technique is amplification, an old idea aimed at a newly discovered and very effective target: memcached.

Memcached is an in-memory cache that backends use to take load off databases. It was designed for trusted internal networks, so it has no authentication, and until very recently many builds listened on UDP port 11211 by default. Thousands of servers ended up exposing that port to the internet, mostly without their operators knowing.

The attack combines two properties:

• UDP doesn't verify the sender. There's no handshake as in TCP. A packet claims a source address and the receiver believes it. The attacker sends memcached a small request with the victim's IP forged as the source, and memcached sends its response to the victim.
• The response dwarfs the request. The attacker first stores a large value in the open cache, then requests it repeatedly with tiny packets. Reported amplification factors run as high as about 51,000x, far beyond DNS or NTP reflection.

Multiply that across thousands of exposed servers and a modest amount of attacker bandwidth becomes a terabit flood. GitHub says the traffic came from over a thousand autonomous systems and tens of thousands of endpoints, none of them malicious, all of them misconfigured.

Why GitHub was back in ten minutes

GitHub's postmortem is transparent, and the timeline is the lesson:

1. 17:21 UTC: inbound traffic spikes and network monitoring flags the anomaly.
2. 17:26 UTC: GitHub routes its traffic through its DDoS mitigation provider, which filters the flood and passes clean traffic through.
3. About 17:30 UTC: the attack subsides and service is substantially restored. User-visible impact was roughly ten minutes.

None of that was improvised. The mitigation contract existed, the rerouting procedure was known, monitoring caught the attack as it started, and the failover decision took minutes because nobody was debating options mid-incident. A 1.35 Tbps attack became a short outage.

Lesson one: don't expose UDP services you didn't mean to

The supply side of this attack is thousands of memcached servers that should never have been reachable from the internet. The fix costs nothing:

# Is your memcached listening on a public interface?
netstat -ulnp | grep 11211

# In /etc/memcached.conf: bind to localhost and disable UDP entirely
# -l 127.0.0.1
# -U 0

Firewall it anyway, as a second layer:

iptables -A INPUT -p udp --dport 11211 -j DROP
iptables -A INPUT -p tcp --dport 11211 -s 10.0.0.0/8 -j ACCEPT

The same audit applies to every internal service: Redis, Elasticsearch, MongoDB, NTP, SSDP. If it was designed for a trusted network, it shouldn't face the internet. Scan your own address space from outside your network before someone else does.

Lesson two: know your mitigation path before the attack

Once a volumetric flood is saturating your link, you can't fix it from inside. Dropping packets at your own firewall doesn't help when the pipe upstream of you is full. Mitigation means someone with much more capacity (a scrubbing service, a CDN, your upstream provider) absorbing the traffic in front of you.

So the preparation happens in advance:

• Have the relationship in place. Whether it's a CDN with DDoS protection, a scrubbing service or your host's built-in mitigation, know what you have, what it covers and what triggers it. Researching options during an attack costs hours in a fight measured in minutes.
• Know the failover procedure. GitHub's reroute was a practiced move. If yours involves a support ticket and a DNS change on a record with a 24-hour TTL, check your TTLs now.
• Know who decides. Mid-attack is no time to work out who has authority to move traffic. Write the runbook and name the people.
• Right-size it. A small business doesn't need GitHub's setup. Sitting behind a reputable CDN with basic DDoS absorption covers most of what a small site will ever face. Having nothing and no plan is the mistake.

Lesson three: monitoring tells you whether mitigation is working

GitHub's timeline starts with detection and ends with verification. Monitoring told them the attack had begun, and monitoring confirmed that the mitigation was passing clean traffic afterward.

During a DDoS, external monitoring answers the questions that matter minute by minute: are we reachable, from where, and how degraded? Mitigation isn't binary. It can absorb most of the flood while real users still see timeouts, and internal dashboards showing healthy servers won't reveal that your upstream link is saturated. Only a check from the public internet sees what your users see. We made a similar point in January about Meltdown and Spectre: the view from outside is the one to trust.

Be the ten-minute story

The GitHub attack is a security story with a good ending, and the ending was written weeks earlier in a mitigation contract, a rehearsed runbook and monitoring that flagged the flood quickly. Whether the next attack on you is a ten-minute story or a ten-hour one is decided now.

CompleteStatus gives you the outside-in view: uptime checks from the public internet, failure confirmation before alerting, response-time history that shows degradation as well as downtime, and alerts by email, Slack or webhook when reachability changes. Create a free account.