The remotely exploitable Log4j zero day vulnerability disclosed just a few days ago has been called one of the most serious vulnerabilities to date. So what is it all about, and what does it mean for organizations? How is it being exploited? What are the risks, and what can you do if you’re waiting for a patch? F-Secure CISO Erka Koivunen joins episode 62 of Cyber Security Sauna to break down the issue, and explains why this vulnerability should be a wakeup call for security practitioners and developers.
Listen, or read on for the transcript. And don’t forget to subscribe, rate and review!
ALL EPISODES | FOLLOW US ON TWITTER
Erka Koivunen, F-Secure CISO
Janne: Late last week and over the past weekend, something called the Log4j zero-day vulnerability popped into the radar screens for a lot of us. What is it, Erka?
Erka: It’s a remotely exploitable vulnerability in a tool which is widely used to create event logs and to manipulate that event log information. It is something of an opaque piece of internet infrastructure that end users don’t usually see nor interact with. And yet this vulnerability actually enables everybody on the internet to submit attack code for the backend components to then detonate. This is a beautiful vulnerability.
Can you give me the for-dummies version? Like what is this vulnerability and what does it do?
This attempt at explaining this vulnerability actually struck a nerve with me, so I’m trying to recap that. It was based on a discussion in Finnish. So I hope I’m able to do justice to the example.
So, imagine you have an office building where guests can come in from the front door. And you have a front desk officer whose job it is to write down each and every visitor. They give the front desk officer who they are, what company they represent, who is going to be their host. And that front desk officer diligently writes down everything. And that’s the job that they are expected to do.
And then one of the visitors then strikes up a conversation. And the front desk officer has been programmed to dutifully fulfill each and every wish that the visitor has. So not only will that visitor be provided with an excellent cup of espresso, but also the contents of the CEO’s safe, because that’s the type of trigger that the front desk officer was waiting all day long to be used.
Exactly. So where is it located? Where can you find this?
It is typically in the backend components of web applications and portals, and everything that is basically collecting telemetry information. To illustrate, during the weekend people were fooling around changing the names of their iPhones or Tesla cars to match with the attack strings or attack payloads, which caused Apple servers to launch the kind of executables and call back to an address of the attacker’s choosing. Tesla servers did the exact same thing.
Yeah, okay. So while the Apache web server itself is a fairly…this is what it does, this sort of framework that we’re talking about exists in various other applications as well.
Indeed. This has nothing to do with the Apache web server as such. This log tool has been incorporated to a number of other data processing workflows as well. Even F-Secure is using that.
Okay. So remote code execution is always bad. Basically, at that stage the only limiting factor is the attacker’s imagination, but are there any sort of mitigating –
And courtesy.
Sorry?
And the attacker’s courtesy.
Absolutely. That’s something to trust on. So are there any mitigating circumstances with this vulnerability?
The most obvious one seems to have been quite widely neglected. There has been an option to disable this so-called lookup functionality in this tool. This option has been there since 2017, and based on what we know from the weekend, this was rarely utilized. So if that lookup functionality would’ve been disabled, we would not have this kind of trash fire currently.
The log creation and manipulation components that are typically in the backend systems, if they would’ve been properly isolated so that they would not be in a position to initiate these callback functionalities, outbound connections, that would’ve actually frustrated the number of attackers as well.
So it would’ve been extremely easy to become immune from this type of vulnerability, if basic IT hygiene guidance would’ve been followed. But the internet has not exactly been built by way of hygiene.
Absolutely. So that’s something people can already do before a patch is provided by their service providers. Is there anything else you can do while you’re waiting for that patch to drop?
Since this component and this functionality is parsing user inputted data, the input validation and input cleanup, of course, would be extremely useful before passing information to a functionality like this. So patching, disabling of unnecessary components, network configuration by the way of segmentation and input validation, would actually help bring immunity to not only this vulnerability, but to a number of other similar issues.
Yeah. I was just going to say that input validation sounds like a good idea at the best of times.
And none of these are in the hands of end users. So this is entirely on the service provider’s side. So this is frustrating as an end user that we just need to then sit and wait for the service providers and software manufacturers to do their thing.
Yeah, exactly. Maybe I should clarify that the Log4j framework itself has already been updated. There’s a patch for that. But now we’re waiting for all the various projects that use that framework to issue patches for their own services and products. So that’s where we are right now. So those teams are scrambling right now.
Yes. On Friday morning when the wider internet community learned of this issue, there was not a supported or functioning patch available. So that only became available in the AM hours, European time. Before the availability of that patch, this vulnerability had widely been used in the Minecraft community – I would imagine, for giggles. So, us serious security professionals were completely clueless of this vulnerability or the need to provide protections against this. And yet our teenage kids were happily lulling around exploiting the world and basically Microsoft Minecraft servers, using this vulnerability.
Oh, wow.
This should create the level of humbleness that this industry deserves.
That’s super fascinating. So these kids were basically just using this to pull pranks on each other without even realizing what sort of a bomb they were actually sitting on.
Indeed. They probably assumed that this would be isolated to Minecraft servers.
To illustrate, I was informing our leadership team of this “internet is on fire” type of issue. And one of them happened to mention it to his teenage kid, and the kid was immediately telling, “Yeah. Yeah. We know about this. We’ve been exploiting this. What took you so long?”
That’s awesome. Well, okay, so other than Minecraft pranks, do we have any indications or evidence of this vulnerability being exploited by attackers out there in the wild?
Yeah. After, Friday, widespread exploitation has been taking place. The earliest public sightings have been recorded by Cloudflare, if I recall correctly, already on the first of December. That was, they referred to as, probing type of testing of this existence of this vulnerability.
But since Friday, things have ballooned. There is of course, lots of testing and giggling going around. But initially, this was used to basically drop cryptominers onto affected computers. Which is, I would again argue, that this is security by the courtesy of an attacker, so luckily this didn’t have any more devastating effects.
At the same time though, it had been seen that remotely controlled Trojans or RATs were being installed on the affected devices. And this obviously means that this is an attempt to secure a beachhead, so that while the initial vulnerability gets fixed, the adversary would still have a foothold in the target organization. And of course, that can be used then to launch ransomware attacks or provide an ability to sell access to curated organizations against the price.
Sure. So given that, sort of, the goal of the attackers is to gain that persistence, what’s your thinking now? Is it the case that every organization that has been vulnerable for this vulnerability should now be rolling incident response and starting threat hunting? Is that the case?
Indeed. Yeah. It is safe to assume that your logs will be full of attempts to at least enumerate this vulnerability. The tricky thing is that when the attack is successful, the log entries look much more boring. So that is a good place for professional threat hunters to go look into the logs and to try to understand what might have happened and where to lead the incident response and forensic activities to next.
Yeah, okay. So if you’re just looking at your log files and they seem to be in pretty good shape, that might actually be a red flag in itself that they’re in a little bit suspiciously good shape.
Yeah. We’ve had these discussions that, “Yeah, it only shows errors.” Ooh, this is a big warning sign. That error should not have been triggered.
Right, right. Okay. So are there any other tips or tricks you would, at this point, like to share with people who are fighting this vulnerability?
The weekend in the security community gave us the consensus that, of course this is bad, and we’ve been extremely frustrated now trying to describe this to a layperson, to mainstream media or to the end users, that the situation is bad, although no visible effects have been seen. I still have electricity at home, and I still can use iCloud, although I don’t dare to because it’s affected. And we’ve fixed F-Secure branded products. And it seems that nothing broke. This is me knocking on wood here. (knocking sound)
But I’m assuming that over the course of this week, visible effects will be seen. So the first advice this Monday morning hour is brace yourself. Something is bound to be unearthed, and this should be treated with the level of seriousness it deserves.
Now trying to raise from the tactical level to the more strategic, this should be a wakeup call to a number of security practitioners and developers to understand that these types of seemingly isolated risk decisions that you make when you are developing your application, they actually would deserve an additional pair of eyes to check whether what you found, a convenient set of features, a nice-to-have, future-looking set of functionality, actually exposes your organization and your customers to internet-wide exploitation.
So this feature in this log handling component was introduced in 2013. For some stupid reason, it was enabled by default. Even the Log4j maintainers dislike this functionality. They would not have wanted that feature to exist or to be default in the first place. And here it is. The implementers who used this component could have used, since 2017, the disable functionality, to get rid of this lookup functionality. And yet they chose not to. And as we see even Apple iCloud backend computers, for some reason, they have unrestricted ability to initiate outbound internet connections to wherever.
That type of happy-go-lucky type of mentality in web application development should now be under serious scrutiny going forward.
But that requires a fairly detailed understanding of what’s happening, because there’s been a lot of talk during 2021 about software supply chains and sort of various software bills of lading, things like that. So that you would have that sort of, these are the components that make up my software product. If logging functionality from the Log4j framework was on that list, I’m not sure that I would assume that that means a potential RCE.
Yeah. That’s correct. And as a representative of the infosec community, I also need to be extremely careful not to give impression that I would’ve had that foresight. No, I didn’t. Before Friday, I was not even aware of such a component.
Exactly.
And I was not aware of the lookup functionality. I was not aware that it actually had been widely covered in a Black Hat talk in 2016. So I was happily clueless about what’s behind the internet facade that I see. And yeah, it would be unfair to claim that if we would just have a bill of material of each and every component that makes up an application, we would immediately be able to see what vulnerabilities or potential for misuse there would be. That would be impossible.
And that’s why some of the other kinds of IT hygiene, good practice actions are so much in need. Proper traffic filtering and network segmentation would’ve actually made it quite difficult to exploit this feature, and would’ve given organizations some breather time before they take actions to mitigate that root cause. Now that we understand where it is.
Not to mention that input validation.
The input validation, yeah. And I would imagine that if there would’ve been rational discussion about developers that, “Hey, we’re using this component to create logs based on the event feed that we see, and that event feed actually might contain user input – input that the user can arbitrarily create.” That sounds like a risky place. Should we normalize it before passing it to this component?
But most likely, since as said, this is a backend component, and these systems can be changed so that it is like the 10th server or 10th component in the data flow that actually then has this component parsing the data. You could imagine that this not accessible from the internet and we’d only accept input from this preceding set of servers.
So that would require a way more holistic understanding of the architecture than most developers either have the luxury time-wise, or even have an ability to see documentation-wise.
Thank you for talking us through this developing situation, and good luck for what I assume will be a busy week for you.
Yeah. Didn’t the week already start on Friday? So you’re late wishing me a good week.
There we go. That was the show for today. I hope you enjoyed it. Please get in touch with us through Twitter @CyberSauna, with your feedback, comments and ideas. Thanks for listening. Be sure to subscribe.
Categories