Posted on 2020-12-17 by: Axel Kloth
I want to comment on the so-called SolarWinds hack as there is a lot of misinformation out there, and it is not due to the fact that reporters intentionally mislead, but due to the complexity of the case. I would like to simplify it as much as possible without making it false to explain what happened here.
I am also not going to speculate on who did it, and why this particular route was chosen.
For one reason or another, penetrating a single server, a single data center or a single cloud provider was not enough for the attackers. They wanted full control over the IT infrastructure in their target countries.
This was either impossible to do with traditional methods, or too time-consuming.
To understand this attack and its scope, it is important to know that data centers have grown so huge that they require their own set of tools to just deploy servers, prepare them for customer (or in cloud speak, “tenant”) use, administer them, check on their health status, and maintain them as well as to take them down if they need replacement. These OAM&P (Operation, Administration, Maintenance and Provisioning) tools – in this case Orion – are complex all by themselves, and they are usually built the same way that all other Internet backbone tools including Linux are built: with a collection of a very large number of pieces of source code, some of them proprietary and in-house, and others open source.
That is where this attack originated. The attackers hijacked some of these pieces of software and added code to them that constituted a so-called “backdoor” once their code was included in a new “build” of the OAM&P software. In other words, once the new tool was built (compiled and linked), their code allowed them to use the backdoor to access the servers through the OAM&P mechanism. That is very hard to detect as the tool does what it was intended to do, but for people who were not supposed to access it. The only way to detect an attack like this is by using behavioral monitoring. In other words, if I use the tools to conduct OAM&P and that is my profile of use, and then someone with illegitimate credentials uses the same mechanism (IP addresses, port numbers, APIs and more) to conduct surveillance and to access net user data, then the profile is different and should (and can) be detected as illegitimate and be flagged as a breach.
However, it took SolarWinds and FireEye as well as Microsoft six months to detect this breach, and effectively end it. I rarely praise Microsoft, but in this case they did exceptional work once this breach was suspected. Microsoft then followed through with stopping the attack and helping affected users to preserve evidence as much as possible to submit this to law enforcement. FireEye – while itself being a victim – assisted Microsoft and others in evidence collection and analysis of the logs of victim machines.
The challenge that all victim organizations are facing now is that it is unclear if the attackers managed to install backdoors in the servers themselves, not only in the OAM&P servers in the network management centers (NMCs). The number of servers in the NMCs is limited, and worst case they can be physically replaced. However, it is not clear if the breach went so far as to compromise the servers that deal with customer or tenant data. In other words, at this point in time we cannot trust in the integrity of the servers in the affected organizations – and those include the US Treasury and the administrators of the US stockpile of nuclear weapons.
Only after a thorough review of all software on the servers that might have been affected can we be certain that no additional backdoors were installed. That is a monumental task as likely hundreds of thousands of servers must now be assumed to be compromised.
