What’s The Deal With Behavioral Engines?

I recently wrote a post on how scanning engines evolved from their primitive, signature-based roots in the 1980s to the present day. In that article, I touched upon how file scanning itself is just a small piece of the puzzle when it comes to protecting endpoints from threats such as malware and exploits. Today, I focus on behavioral engines, AKA host-based intrusion prevention systems (HIPS).

Put simply, behavioral engines work by monitoring the execution of processes on a system for potentially malicious actions. If a malicious action, or series of actions are spotted, the process is killed. This ensures that a malicious payload never completes its objective.

In addition to blocking malware executables, behavioral monitoring can also be used to stop web-based exploit attempts and to block malware infection vectors from non-PE sources, such as Office macros and PDFs. This is achieved by monitoring common applications such as web browsers and document viewers/editors.

For example, if a user visits a malicious site, behavioral monitoring will notice the signs of an exploit attempt in the browser process itself and kill that process before the exploit runs. As it turns out, this is a great way to stop zero-day exploits.

So, why are behavioral engines so useful? The fact is, a majority of malicious payloads use the same small bag of tricks to infect systems. If an Excel file writes executable or shell code to disk, or attempts to launch executable code on the system, we can be pretty sure it is malicious. That sort of behavior is completely unheard of in legitimate documents.

Blocking on behavior is great because you don’t need to care what the sample “looks” like. New malware is published on a  frequent basis in order to avoid signature-based scanning approaches. But all those different versions will still perform the same set of actions. If you detected the first one based on behavior you’re going to detect the rest of them too.

The only way malware authors can circumvent behavioral rules is to come up with a new set of actions. And new infection vectors don’t present themselves very often. This is why, in a majority of cases, when new malware or exploits surface, we already have them covered by our own behavioral engines. The tricks malware authors have been using for some time will, most likely, continue to be used in the future.

Behavioral monitoring on the client-side is not without its caveats, though. Any monitored process will suffer a slight performance hit. For this reason, limiting what is being monitored is important. This can be achieved in a few ways.

Whitelisting is a simple and quick way to skip all scanning, and therefore it is not uncommon for endpoint protection software to run whitelisting checks prior to any other analysis step. Files can be whitelisted based on simple metadata, such as the sample’s cryptographic hash, or on more complex metadata such as signing certificates.

Scanning engines are typically faster than behavioral engines. Hence, if a scanning engine can be used to determine if a file is malicious before it is executed (for instance, when that file is written to disk, or as it is arriving over the network), a small performance improvement can be gained.

Let’s take a look at a human analogy to illustrate how this all works.

A team of security guards at a company are tasked with monitoring the building’s physical security. They spend most of their time monitoring an array of CCTV feeds. Some of the security guards take turns patrolling the building.

People arrive at, and leave the building all day long. Most employees wear visible ID badges. Some employees are even known to the security personnel. In our analogy, these ID-wearing employees are whitelisted samples. The security guards don’t need to pay a lot of attention to them, since they’re “known good”.

Visitors also come and go during the day. Newly arriving guests are unknowns to our security guards. Visitors are instructed to wait at the reception for an escort, and obtain a temporary ID card before they are allowed to continue into the building. While on the premises, they must be escorted by an employee at all times. The process of meeting an employee, signing in and obtaining a temporary ID is roughly analogous to a sample being run through a scanning engine. The scanning engine will not determine them as “known good”, but it might possibly determine that they are not malicious.

In our hypothetical situation, at some point during the afternoon a person arrives who doesn’t follow procedure. Since not all employees remember to wear their ID cards, it is possible this new arrival is an employee. None of the guards are, however, familiar with his face. Shortly after arriving, the unknown proceeds directly past reception and tailgates an employee into the main building. The security personnel immediately notice this behavior, and keep a close eye on him via the CCTV feeds. They also contact one of the patrolling guards, asking him to proceed to the unknown’s location. This step is analogous to initiating behavioral monitoring.

As the security personnel continue to closely monitor the visitor’s behavior, they observe him navigate some corridors in the building until he finally stops, produces a crowbar from his backpack, and attempts to open one of the company’s server rooms. At this point, the security guard on the scene and prevents him from accessing the room.

A single action is not always enough to determine whether a threat is truly malicious. A series of actions, in this case, tailgating into the main building unescorted, and attempting to break into a locked room, were the series of indicators that led to finally taking action.

While behavioral monitoring on the client-side is one of the most effective ways of protecting systems against common malware and exploits, back end behavioral analysis provides us with another powerful tool.

Using cloud computing infrastructure, it is possible to run thousands of instrumented sandboxes simultaneously. Files and URLs are fed into these sandboxes for execution. Each individual run generates metadata based on what happened during the execution. This metadata can include both changes to the system where the sample was run, and an execution trace of the sample itself. This resulting metadata is then analyzed by a series of rule engines for suspicious behavior. Samples are flagged for further analysis, and in many cases, automation generates detections on the fly.

By utilizing back end sandboxes it is possible to analyze hundreds of thousands of files and URLs per day – something that would be almost impossible to do manually. This process, paired with an automated static analysis process, allows samples to be categorized and grouped easily, allows new malicious samples to be identified quickly, and allows simple cloud-based detections to be generated in almost real-time..

Behavioral monitoring is a powerful tool for protecting systems against malicious threats whether used on the client itself, or in the back end. The circumstances in which behavioral monitoring technologies come into play on the endpoint are dependent on how the system encountered the threat. As we’ve explained, behavioral blocking kicks in when visiting a malicious site, opening a malicious document or running a malicious executable (action!). For this reason, you won’t see the results of this sort of technology in VirusTotal scan reports. If you’d like to check how our products fare in real world situations, where behavioral blocking is a factor, check out the testing from AV-Comparatives and AV-Test.

If you’re interested in a more technical explanation on how our own HIPS technology works, check out this white paper.