Episode 28| When the Well is Poisoned: The Devastation of Supply Chain Attacks

Supply chain attacks are on the increase, with attackers abusing the trust we place in vendors and software. Why are these attacks increasingly emerging and what can companies do about them? Jyrki Huhta, senior security consultant at F-Secure, joins Episode 28 of Cyber Security Sauna to share his thoughts on these devastating attacks and why “trust but verify” should be the motto for preventing them.  

Listen, or read on for the transcript. And don’t forget to subscribe, rate and review!

ALL EPISODES    |    FOLLOW US ON TWITTER

Janne: Welcome to the show, Jyrki. 

Jyrki: Thanks. Nice to be here. 

We remember how tons of companies doing business in Ukraine got hit through M.E. Docs with NotPetya, and British Airways customer payment data was compromised through a web shop component they used. And back in 2013, Target was hit through their HVAC provider. I assume these are the kinds of things we mean when we talk about supply chain attacks. 

Yes. I suppose those are the most famous ones. Target was hit famously through the heat and ventilation manufacturing company. And it’s actually funny. It was a small company that only had, I think, 125 people working for them, which is tiny compared to Target, which is huge. So that’s a very good example of their scale. The target will be the smaller supplier and the actual target will be the huge vendor responsible for billions of dollars.  

Whereas the other things you mentioned, M.E. Docs, we can safely say, was driven by national interests. It was a Ukrainian accounting company that got hit by the M.E. Docs implant and it was implanted with NotPetya. So just the basic ransomware, and it caused a lot of damage. I think the damage cost like 10 billion or something, according to some estimates.  

And the final example you mentioned, that British Airways Magecart, was a good example as well because that targeted the website. So the website was hacked and replaced with something that would collect the finance information or the credit card information of their users. So it just basically stole their brand for a second, so their end users would get hit by the attack. So those are actually good examples, because Target was attacked because they wanted to get the point of sales terminals. NotPetya was attacked because somebody wanted to show Ukraine around and cause damage; and British Airways was attacked because they wanted to exploit their end users, their clients. 

Right. These are different kinds of supply chains too. Like in the case of Target, it was the HVAC system that they bought, like something that big trucks drive to your yard and deliver. And for example, in the case of British Airways, it was a software component that their website used. 

Exactly. And I think in the Target attack, they originally thought that Fazio Mechanical, the company, had more access to the system than they did. I had read somewhere that they only had access to the billing information and some smaller – so it was more like a supplier. Whereas like you mentioned, those other things, the M.E.Docs was more like a software supply chain attack, because it targeted the application, the Ukrainian accounting software, that was infected. And the British Airways was a website. So it’s a web application. So those are good examples in a way because they are different types of things you can have.  

There was also the famous case of the hardware implants lately, with people suspecting that the iPhones or other hardware were implanted with Chinese components. So that’s kind of one example as well. There’s also been cases where there’s been a physical thing. There was a case I remember that a warehouse got drilled through the roof and they stole medical stuff, or something like that. So there can be different kinds of things. But I suppose the most famous ones are kind of like the software supply chain attacks, where you try to infect a software and then there’s the supplier chain attack where the supplier has access to your system. 

So what are the steps of a supply chain attack? What happens? 

I think the first thing that people should consider is that the attackers will always try to find the easiest way in. So if they recognize that their attack target has a weak point, they will use it. And in some cases the weak point is the supply chain. So if they recognize that the supply chain is vulnerable or it is the most critical place in the system, then that will be the target of the attack first. And that is kind of like the first step. But the second step would be to find ways to actually gain the necessary access to the supply chain. And then the third step would be to start doing the actions on objective on the actual target of the attack. 

Right. Why do these things happen? Why are companies getting hit through their providers and suppliers? 

I think the biggest reason is that a lot of the bigger companies are investing and have been more aware of the risks involved and they have been investing more in security, and therefore they have become the more difficult target to compromise. And like I said, when the attackers will try to find the easiest way in, a lot of the time the easiest way in would be maybe the less sophisticated supplier. So for example in the financial sector, it might be very hard to get to the bank systems through their own network, but it might be possible to compromise their systems by an existing component that is vulnerable. It might be a software component, or it might be that some outsourcing company might have less security that can be compromised more easily. And in some cases it might be even a national thing. It might be very hard to get into military systems, but if they know some software component that is used by that military, then it might be more accessible to actually compromise that software component directly and get access that way. So it’s kind of like Island hopping in a way. 

Sure. 

The other option is that sometimes the suppliers have more access. They have more authority into the client system than the client itself. For example, a lot of the time outsourcing companies will control most of the infrastructure of the client. So in that case it might not even make sense to directly attack the client since they don’t have the access – they might not even have domain admin rights to their own system. So in that case the only target that they can actually attack would be the outsourcing partner. 

That makes sense. What about from the defender’s point of view? Is a supply chain attack more likely to do a lot of damage or less likely to do a lot of damage? How would you rate that? 

I would rate that supply chain attacks are very devastating. I think in my history, I’ve only seen a few cases that have been as damaging as supply chain attacks, because they will exploit the trust that you already have between the vendor and yourself. And a lot of the time that trust is kind of like, it’s 100% – people trust their vendors completely. 

Sure. 

And there’s no way to actually have good technical means to enforce that trust. You will have contracts, you will have liabilities, you might have SLAs, stuff like that, but you don’t have clear visibility into their actual systems. You don’t have detection capability inside your outsourcing company. So you will trust them. And that is very dangerous because if that trust is misused, then it can have devastating effects. 

So who’s doing these supply chain attacks? Is it always – like some of the examples we talked about are nation state actors, is it always people like that? Can it be criminals? 

There’s been kind of this general idea in the media that the most sophisticated attacks are carried out by nation states or nation-sponsored groups. But I think that is changing rapidly. I think all the criminal, all the illegal groups will start using a similar kind of approach. So it is not limited to nation state groups anymore. And nowadays it’s even hard to say what group is strictly nation-sponsored or nation state. The attribution is becoming more and more difficult when there’s all sorts of mixed groups working and using similar kinds of techniques. And supply chain attacks is such a general approach and it utilizes such a different variety of techniques and it’s hard to generalize that yes, this will be only done by a nation state. I think it will be a technique that will be utilized by everyone, basically, who is trying to find a way in. 

Do you have some specific examples of supply chain attacks? Something that really stuck to you as an example of something new, something we hadn’t seen before, something that sort of broke the mold? 

I wish I could tell more about my actual experiences with supply chain attacks, but the first time that I saw a real bad supply chain attack it was, it was like…it was heart-stopping. When I realized that yes, the attacker is coming back inside by using the supplier, the outsourcing company. And that is kind of devastating once you realize that nothing that you do inside your own network will stop it. 

I don’t know, as an avid military historian, it seems to me like this is like World War II where the Germans, instead of facing the French Maginot Line, just routed through Belgium. 

Yeah, exactly. The metaphor I’ve been using is that a lot of the time security will focus on the immune system and good hygiene, and you will try to avoid contaminated places so you won’t get infected. But supply chain attacks are like the water that you drink. It’s like the well is poisoned or you’re eating poisoned food. So it’s like there’s nothing to stop it. You’re just ingesting the bad stuff. 

I know because of your work, you can’t go into specifics, but are there any war stories you could sort of anonymize for us and sort of in a general way talk about a little bit about your personal experiences in supply chain attacks? 

It’s good that there are public cases available, that some companies have been straightforward about their experiences. And there has been public information available, for example, from the advanced persistent threat type of actors. So we already know that for example, in the energy sector there have been targeted attacks that have been using outsourcing companies. And there have also been cases where outsourcing companies like Wipro have admitted that their systems have been hacked. And there are even detailed explanations, for example, from the Shadow Hammer group that most famously used the Asus update facility, they infected that.  

But I think that the more cute story is about how they infected the gaming company, I think it was in Korea, some kind of shoot ‘em up game. And I think that was interesting because I read somewhere that they infected the Microsoft Visual Studio that the game developers used, and that got infected and that got transferred into the actual end product of the game. So the game was infected. I think the game was even called infection or something like that.  

But I think those are good examples because those are the types of things that would happen behind closed doors as well, in those kinds of places that we can’t even discuss about. So it is certainly possible that your outsourcing company is compromised and their systems will be used against you. So if they are the administrators of your network, it is possible that the attackers will get access to that information and then will use it against you. 

Some of these more recent things, the Shadow Hammer one using the Asus updater thing, that seemed to be very targeted. The attackers were looking at specific MAC addresses, so they knew who they were hitting. But the Wipro case, for example, strikes me as a sort of opportunistic attack. Like somebody was in a position to compromise Wipro to get at their customers and sort of almost didn’t care who the customers are. Presumably they all have money or something worth stealing and we’ll just see what it is when we get there. Is that correct? 

Yeah, that’s a good point. I mean since there are different kinds of attacks nowadays, and like you said, some of them will be very targeted. I think the Shadow Hammer attack, they were targeting like 600 MAC addresses of the 600,000. And that’s the case that we know. The Shadow Hammer group supposedly is very sophisticated in their supply chain attacks. And what we’ve seen from the operation, they will try to use methods that will target a very limited amount of targets to reduce the chances of discovery and to make it more effective.  

So those kinds of attacks are very dangerous because they will only manifest themselves once they are discovered. And they are impossible to actually start investigating until you first detect them. And I would assume that they might even have access to systems that we don’t know. So it’s good to keep in mind that the things that we know are not the only things that they have done. So that’s a very interesting thing.  

But then like you mentioned, there are cases where the attacker is not even interested in the client itself. And there have been cases where supply chain attacks have been used to just install crypto miners, especially the time when crypto miners were more economically interesting. And so it might be that the target is just to access a large amount of resources like computing power. And it might be that the target is the brand itself.  

For example, in the British Airways case where you will get access to the brand, you will get access to their huge amount of end users, their clients. And you will misuse that trust. There’ve been also cases where they”ve been misusing the email systems. So even if you can send emails on behalf of a very known brand that’s very useful, it can be used by attackers in a number of ways. So it all depends on what the attackers want. And sometimes it might even be that the attackers will use one thing and then they will sell that information to the next group who will use it for another thing. So it’s complicated. 

Yeah, certainly we know of cases where simply access to an organization is being traded in the dark market. Like somebody might compromise a supplier who supplies many different companies and then just resell that foothold, identify these companies, identify the assets they’re in and then sell the access onwards to more targeted attackers who will then come in specifically to harm that organization. 

Yeah, exactly. And that’s kind of interesting as well, that there’s a lot of room for economics in those situations. So there are A groups and B groups, even inside one nation state actor and they might have different objectives, inside once they get a good foothold. And is also interesting to see how sophisticated those channels are that they are using nowadays to control the system. Once they get inside, they will set up all sorts of backup routes and accesses to the system so that once the first one is discovered, they will still have enough foothold in the system to get back to that position of power to get backed into dominance, power inside the network. 

Right. But from the defender point of view, this also leaves you in a very difficult position because you’re looking at the techniques and tactics and procedures of the attacker and you’re thinking, you know, this is just an opportunistic attack hitting me. But actually the bad guys coming in right behind that attack might be a very different kind of beast entirely. 

Yeah. And it’s not even on unheard of. We’ve seen cases where there’s been more than one APT inside the network. So for example, the Countercept guys at MWR have been investigating a case and they’ve been hired there to find and remove a certain APT. And once they’ve been doing their case, they found another. And they have to focus their efforts that, “Well, okay, this is dormant, they are not doing anything. So we will remove the first one first and then we will focus on the second one.” 

It’s hacker party and everyone came. 

Yeah, exactly. And that’s crazy, I think. But that’s just the way it is nowadays.  

Absolutely. But in addition to there being multiple attackers, that companies these days have dozens or even hundreds of suppliers. So how do you make sure that all those vendors are following proper cybersecurity practices? 

That is a very good question and I would even be so pessimistic to say that it is not possible to be 100% sure. So you have to have levels of defense, you can’t rely on them being safe. Of course you can do things to vet your suppliers. That goes both for the applications that you use, the libraries that your developers are using, the platforms that they are using to create their applications, your outsourcing companies, all kinds of supply chains, you can try to make sure that you’re choosing the best players. 

So what does that look like? Is that a mountain of papers or, I don’t know, mandatory red team testing? 

Yeah, I think it is better to have more than one way. There should be a good process for it. But I think the most important thing is that you shouldn’t rely strictly on contracts and paper, like you said. If you’re using an outsourcing company that is responsible for the crown jewels of your network, you should be very sure that you can trust them.  

And I think the only way that you can have trust is to have a good red teaming exercise for example on that. And basically when you approach your outsourcing company, you will go “How can you ensure us that your security is all right?” If they respond to you with PowerPoints and showing their certifications, then I’d be very hesitant. But if they show me that “Well, we are using the best red teams available and we are doing it annually, this is the latest report. We had these kinds of things and we are fixing them,” that would make me more trusting. I would trust them more.  

And I think it’s important to have that sort of third party view and have that verification that they are not trivially exploitable. Especially if they are responsible for critical functions inside the company, like having administrative access and having access to sensitive information or sensitive systems. 

Yeah, I mean a red team exercise or any kind of attack simulation, if it’s done properly, it’s a pressure test for the company. Like how would you fare under an actual attack? So to you, that seems like a good metric of how would that actually play out? 

Yes. And also, lately we’ve seen cases where the managed security providers of the company have been targeted. It’s kinda crazy, I think, when people buy SOC operations or outsourced security services that they don’t include a red team exercise. Because I don’t see any other way that that can be measured. How can you measure the quality, the success of that operation unless you test it? 

Right. But you’re also talking about sort of trust but verify. So we’re trying to establish the security position of our vendors and suppliers, but also, just as companies these days are thinking about insider threats, sort of monitor those suppliers as well. 

Yes. Trust and verify. I think that’s a good motto from this podcast. I think it is important to have that trust. I mean only choose the type of vendors and the type of software development cycles that you can trust. That’s very important. But verify is also very important. So you need to have the capability to detect if something is going wrong, and you need to have enough visibility into your most critical supply chains. And that might be a little difficult because one of the reasons that you are outsourcing is because you’re trying to save money. 

Absolutely. 

So if you have to do part of that operation yourself and you can’t trust them, then you will need to spend resources of your own. And that might be very difficult for some clients. 

Now, attack simulation will make sense if you’re talking about the supplier as a company that is selling you software, something like that. But what about if we’re talking about things that you use in your software, like libraries or dependencies like that, what can you do there to stay safe? 

Yes, that’s a very good question. Like we discussed, there are different types of threats that we need to address here. And these software supply chain attacks, I think are more difficult in a way because the cycles, the sprints that the developers are using nowadays are becoming more and more hectic. You have to do things more and more quickly, and that leaves less and less time to actually go through the quality of your own software. And it’s almost impossible to go through all the libraries that you use. You just have to depend on the third party components to be safe, whether it’s your operating system or your web application framework or the library that you use.  

There are some things that can be done. For example, if you’re using well-known platforms, well-known libraries, well-known applications, those are generally more safe because they have invested more money in their security. For example, if they’re using bug bounties, that’s a good sign because then they are at least putting some money behind their words that if someone will find vulnerabilities, we will pay them.  

We can look at media and how they respond to things. Like if there’s been a vulnerability and the vendor goes, “Well that’s not good” and are very ignorant about the threat, that’s probably a bad sign. But if they have good incident response mechanisms and they will respond with good transparency and with good mechanisms to remediate the problems, then they would probably be a safer partner. But I do understand that sometimes it’s very hard to do these decisions when there is so much pressure to deliver the products. 

The traditional method of securing your software integrity and functionality was to just pick a version of the library and then sort of lock that down and only use that one. Ignore all the updates and changes from then on. Is that something that is still feasible or do we just have to sort of develop ways of staying up to date as our dependencies change? 

I think that is very interesting, especially from the IoT world, that there’s been a lot of discussion that IoT devices are difficult because it’s hard to patch them. And once there is a vulnerability that touches an IoT device, that’s very dangerous because it’s sometimes unfeasible to patch them. So I think it is important nowadays to only rely on products that have the capability to issue fixes. And that goes for smaller things like IoT devices.  

But that also goes for bigger things, because I’ve seen very critical systems, like in finance or even military, where there are systems that can’t be fixed or can’t be patched because they have so much legacy that it’s just too dangerous to issue the patch. It might break them. And that is dangerous because then you can’t have a fix. Then you need to have all sorts of workarounds, gluing stuff together to make the fix work.  

But that’s also relevant for supply chain attacks as well, because supply chain attacks might work through the patches. Your patching cycle might be compromised. So it also introduces dangerous things. So it’s hard. It needs with, we just need to have, we need to be aware of the risks. And that’s kinda like the main thing, I think. 

So is there a way for a company to detect that a supply chain attack has already happened? 

I think that is a very important question because I think all companies should have that final capability and have that level of defense where they can actually detect the actions on objective. So when the attacker is inside your network and he’s reaching your crown jewels, you should have the detection capability to detect it. And you should have the appropriate means to respond as well.  

So if you detect an attacker who is already so deep inside your network and has already such good reach, the response should be appropriate. So you can’t just rely that you will re-install the infected machine or you will add to the IOC signatures to your fourth generation firewall or something like that. That’s not enough. You need to have the capability to do proper forensics and to have the clear view what you’re facing against, and find those threat actors inside your network, and construct a plan and how you will evade them, how you will evict them. 

You’re talking about actions on objective, sort of attacker behavior. So you’re saying that if you have proper detection methods, you will catch all threats, whether they’re external or malicious insiders or come through your supply chains. 

There should be mechanisms for that. I don’t think that everything can be detected, but if there is enough detection capability, you might detect one mistake that they make. And once you detect that, then you need to have the mechanisms to find the root cause, find the smoking gun inside the network. So I don’t think you can have 100% visibility to anything. But you need to invest in detection capability that will detect the final steps of the attacker.  

So you need to be very clear on what are your crown jewels, and how can you access them, and how can you create detection mechanisms that will detect that they have been trying to approach them. They have been trying to search for them, or they’ve been trying to access them. 

Right. Now, CISOs of companies have tons of things to worry about. If you were a CISO of a major company, where on your list of concerns would supply chain attacks be? 

I think currently, especially if you’re working in the finance or manufacturing or electric field, those are the type of segments that I would be very worried about my supply chains. 

Why those in particular? 

Because they have been targeted more than anyone else, especially lately, like in 2018 and 2019. But of course if I’m using outsourcing services, if I am putting a lot of faith in certain outsourcing companies or if my products with my applications rely on certain components, I would put a lot of effort to make sure that those are not compromised and I have the necessary processes available that if something happens, I know who to call and I know what to do. 

Yeah, you’re absolutely right. It’s a risk calculation, sort of how crucial are these dependencies? How much am I reliant on this vendor or whatever they’re providing, how deep is their access into my systems? And that then informs how you should approach that threat. 

Exactly. I think supply chains are critical nowadays in almost any company. Like we discussed earlier, that if someone can infect my well, that will destroy the whole village. 

Yeah. So you want to make sure that the parties you’re trusting are worthy of that trust. 

Exactly. So you don’t want to extend your trust just blindly. You need to have ways to ensure that the trust is merited. 

Trust but verify. 

Trust, but verify. 

Well, on that note, I want to thank you for being on the show today. Thank you. 

Thank you. 

That was our show for today. I hope you enjoyed it. Make sure you subscribe to the podcast and you can reach us with questions and comments on Twitter @CyberSauna. Thanks for listening.