SolarWinds Orion: When the Monitoring Tool Is the Way In

This one is uncomfortable for us to write about, because the software at the centre of it is a monitoring tool.

Over the last two weeks a picture has emerged of one of the most serious supply-chain compromises anyone has seen. Attackers got into the build process for SolarWinds Orion, a widely used network and infrastructure monitoring platform, and inserted a backdoor into legitimate, digitally signed updates. Those updates were then installed by SolarWinds customers through the normal update process. From there, in some organisations, the attackers moved on to email, cloud accounts and internal systems.

The story is still developing daily, so a word on what follows: it reflects public reporting and vendor advisories as of this week, and some of it will be refined or corrected as investigations continue.

What we know so far

• December 8: FireEye disclosed that it had been breached by a highly sophisticated attacker, who had stolen some of its internal red team tools.
• December 13: FireEye published details of how the attackers got in: a trojanized version of SolarWinds Orion, which it named SUNBURST. The same day, SolarWinds published a security advisory, and CISA issued an emergency directive ordering US federal agencies to disconnect or power down affected Orion installations.
• Affected versions: SolarWinds says the malicious code was in Orion Platform builds released between March and June 2020 (the advisory lists 2019.4 HF 5, 2020.2 with no hotfix, and 2020.2 HF 1). Fixed builds have been released, and the advisory has been updated several times since.
• Reach: In a filing with the SEC, SolarWinds said that up to around 18,000 customers may have installed an affected version. That's the number that downloaded it, not the number the attackers actually went further with, which reporting suggests is much smaller. The organisations publicly reported as affected so far include several US government departments, and the list is still growing.

According to FireEye's analysis, the backdoor was careful. It waited up to around two weeks after installation before doing anything, disguised its network traffic as Orion's own telemetry, and used DNS lookups against a domain (avsvmcloud[.]com) to decide which victims were worth pursuing. That domain has since been taken over by Microsoft and industry partners, which is helping identify infected systems and, according to FireEye, can act as a kill switch in some circumstances.

Who did it is being widely attributed in the press, but formal attribution is for governments to make, and for defenders the more useful questions are elsewhere.

Why monitoring software is such a good target

Think about what a typical on-premises infrastructure monitoring server has:

• Network reach into everything. It needs to talk to every server, switch, firewall and database it monitors, so it's allowed through firewalls that nothing else is.
• Credentials for everything. SNMP community strings, SSH keys, Windows domain accounts, database logins, cloud API keys. Often with more privilege than strictly necessary, because it was easier to get the checks working that way.
• Trusted traffic. Security teams expect the monitoring server to be noisy. It connects to lots of things constantly; that's its job.
• Automatic updates from a vendor. Signed by a certificate everyone trusts.

That combination makes it one of the most valuable machines on a network to compromise, and until this month it's rarely been threat-modelled as such. This isn't unique to SolarWinds. The same is true of backup software, remote management tools, configuration management servers, CI systems and patch management platforms. Anything that exists to touch everything can be used to reach everything.

If you run Orion

Follow SolarWinds' advisory and CISA's guidance, which are being updated as more is learned. Beyond patching, the working assumption in public guidance is that if you ran an affected version you should look for signs of follow-on activity, not just remove the backdoor:

• Check whether affected servers made DNS requests to the domain above, using DNS logs, firewall logs or passive DNS.
• Review what credentials the Orion server held, and rotate them. Start with the most privileged.
• Look for unusual authentication activity, particularly around identity systems and cloud tenants, since several reports describe attackers moving there.

If you don't: lessons for every tool with the keys

Most readers won't have run Orion. The lessons still apply to whatever you use to monitor, manage or deploy your infrastructure.

Least privilege for monitoring accounts. A monitoring check that reads CPU usage doesn't need a domain admin account. Use read-only SNMP, dedicated service accounts, and database users that can run a health query and nothing else. It's tedious to set up and it massively reduces what a compromised monitoring server can do.

Restrict egress from infrastructure tools. Your monitoring server needs to reach the systems it watches. Does it need to reach the entire internet? Probably not. Allow update servers and alerting endpoints, and log or block the rest. SUNBURST needed to talk to the outside world; an egress allowlist makes that much harder and much more visible.

Log DNS. DNS lookups were how this backdoor checked in. If you don't keep DNS query logs from your servers, you couldn't answer "did we talk to that domain?" this week. That's worth fixing whatever the next incident turns out to be.

Signed doesn't mean safe. Code signing proves who shipped something, not that it's what they meant to ship. It remains worth checking, but it's evidence of provenance, not a guarantee. Staging updates for infrastructure tools (a few days on a less critical instance before rolling everywhere) won't catch something this patient, but it catches a lot of other problems.

Keep an inventory of credentials. If you had to rotate every secret your monitoring system holds by Friday, could you list them? Most teams can't. Build that list now while it's a project, not an emergency.

We wrote about dependency risk in 2016 after left-pad broke builds across the JavaScript ecosystem. That was an accident with a single package. This is a deliberate attack on the build pipeline of a trusted vendor. The underlying question is the same: what are you running that you didn't write, and what can it do?

Where outside-in monitoring fits

It's worth being straightforward about architecture here, since it is a fair question to ask of any vendor right now. External checks, the kind that load your website or check your certificate from the internet, need no credentials and no network access inside your environment. They see what the public sees. That limits what they can monitor, but it also limits what they could expose: there's nothing inside to pivot to.

Internal monitoring with agents and credentials still has a place, because some things can only be seen from inside. The point is to give each tool only the access its job requires, and to treat every system that holds your keys as a high-value target, because this month showed that attackers already do. CompleteStatus does the outside-in kind. Whatever you use, it is worth asking every vendor in this space how they protect their own build pipeline.