Attackers could use this coding bug to turn BIG-IP load balancers against organizations

Load balancing is an important web management process that keeps many internet services ticking. Without it, banks, governments, and other organizations providing online services to large numbers of people would struggle to keep their websites running. During a routine security assessment, F-Secure Senior Security Consultant Christoffer Jerkeby discovered that an obscure coding bug could allow attackers to exploit F5 Networks’ popular BIG-IP load balancer. Further research found that, following a successful exploit, an adversary could turn the compromised device back against the organization or even individuals using the affected services.

You can download Christoffer’s Command injection in F5 iRules whitepaper and slides for all the technical details, or you can read on for a brief summary.

No patch for coding bugs

The flaw isn’t a security vulnerability that can be fixed with a simple software update. The security issue is something organizations create when configuring (or misconfiguring) BIG-IP’s iRules. iRules are the routines written to direct incoming web traffic toward the correct web server. These iRules are created using the Tool Command Language (Tcl). Certain coding practices that may seem perfectly functional and practical to an organization can allow an attacker to inject arbitrary Tcl commands, which could be executed in the security context of the target Tcl script.

More specifically, the design of the language used for defining iRules (Tcl 8.4 to be exact) allows for substitutions in statements and commands, which attackers can use to inject code. It’s similar to the kind of injection attacks seen in SQL or shell scripting languages, where arbitrary user input is interpreted as code and executed. Some iRules parse data from incoming web requests, and incorrectly interpret that data as commands to execute.

<example>

Payload: [HTTP::respond 666 {vuln}]

URL Encoded Payload: %5BHTTP%3A%3Arespond%20666%20%7Bvuln%7D%5D

$ curl -I --cookie cookie=%5BHTTP%3A%3Arespond%20666%20%7Bvuln%7D%5D 
https://www.host.com/index.aspx | grep vuln

$ curl -I -H RequestHeader=%5BHTTP%3A%3Arespond%20666%20%7Bvuln%7D%5D 
https://www.host.com/index.aspx | grep vuln

</example>

The coding flaw and class of vulnerability is not new and has been known along with other command injection vulnerabilities (such as SQL injection attacks) in other popular languages for some time. But the consequence of these attacks can be severe. And BIG-IPs popularity amongst large organizations instigated Christoffer, who’s begun presenting his research at security conferences, to try and raise awareness of the issue.

Attacks can be as simple as filling out an online form

In some cases, exploiting a vulnerable system can be as simple as submitting a command or piece of code as part of a web request. The process basically consists of three steps:

1. Identify a field where the iRules substitute a command by: a) inputting Tcl strings in fields and header names, and b) look for indications that the code was executed
2. Test injection location using the “info” command
3. Pivot to external resources to establish persistence

Once this occurs, the exploit has effectively compromised the device hosting the BIG-IP software, allowing the attacker to use it as a beachhead to proceed with more attacks. Possibilities for follow-up attacks include stealing data from the organization, intercepting and manipulating web traffic to expose sensitive information (such as passwords), or in some cases, attacking individuals attempting to use services provided by the compromised BIG-IP device.

To make matters worse, there are cases where the compromised device will not record the adversaries’ actions, meaning there would be no evidence that an attack took place. In other cases, an attacker could delete logs that contain evidence of their post-exploit activities – severely hindering any incident investigations.

“This configuration issue is really quite severe because it’s stealthy enough for an attacker to get in, achieve a wide variety of objectives, and then cover their tracks. Plus, many organizations aren’t prepared to find or fix issues that are buried deep in software supply chains, which adds up to a potentially big security problem,” says Christoffer. “Unless you know what to look for, it’s tough to foresee this problem occurring, and even harder to deal with in an actual attack.”

Potential scope and mitigations

F-Secure is not aware of any reports involving the use of this issue in actual attacks against organizations. Christoffer is unsure how many organizations across the globe are affected by the problem. But his team found over 300,000 active BIG-IP implementations on the web by using a basic web scanning technique they developed. Considering the methodological limitations, the actual figure could be significantly higher. Not all BIG-IP implementations will be affected by this issue, however.

This security issue is a coding flaw within the iRules (Tcl scripts) themselves. It’s something companies create when they setup their BIG-IP load balancers. So, because there’s nothing wrong with BIG-IP itself, it’s not an issue that F5 or anyone can fix with a simple software patch. It’s up to organizations to investigate and fix the problem for themselves.

“The upside of this kind of security problem is that not everyone using the product will be affected. But the downside is that the problem can’t be fixed with a patch or software update from the vendor, so it’s up to organizations to do the work to check to see if they have this issue, and fix it if they find it,” explains Christoffer. “That’s why it’s important for anyone using BIG-IP to be proactive about this.”

Detecting attacks using these techniques is theoretically possible, but challenging. It’s quicker and more effective to perform a code review and address any underlying Tcl code issues than implement detection.

F-Secure and F5 are working together to support organizations’ efforts to tackle the problem. F-Secure notified F5 of Christoffer’s research several months ago. F5 has already posted a public advisory detailing the affected products, Tcl statements, and more.

Christoffer has contributed to the development of two publicly available open source tools that can analyze Tcl scripts. The first, TestTcl, is a library for unit testing BIG-IP iRules. The second, Tclscan, is a tool that (lexically) scans Tcl code specifically for command injection flaws.

Furthermore, Christoffer recommends consulting the following web pages to learn how to modify any Tcl scripts found to be vulnerable:

https://wiki.tcl-lang.org/page/double+substitution

https://wiki.tcl-lang.org/page/Brace+your+expr-essions

https://wiki.tcl-lang.org/page/Static+syntax+analysis

Because it is possible to mass scan the internet to identify and exploit vulnerable instances of the technology and, in some cases, automate this process, the issue is likely to attract attention from bug bounty hunters and attackers. Furthermore, free trial versions of the technology can be obtained from the vendor, and cloud instances can be accessed from the AWS store for a minimal cost. For these reasons, as well as the potentially severe impact of attacks using this flaw, F-Secure is advising organizations to proactively investigate whether they’re affected, and to contact us if they need help  verifying whether their code is secure.

“Unless an organization has done an in-depth investigation of this technology, there’s a strong chance they’ve got this problem,” says Christoffer. “Even someone incredibly knowledgeable about security that works at a well-resourced company can make this mistake. So, spreading awareness about the issue is really important if we want to help organizations better protect themselves from a potential breach scenario.”