A Journey from MSSP SOC Analyst to Threat Hunter

First impressions

After switching roles it became immediately obvious that the fundamental mindset and approach as a threat hunter is very different from that of a SOC analyst. The traditional model adopted in a SOC tends to be technology-centric, whereas threat hunting is very much people-centric – focused on the knowledge and capability of the threat hunters, and not just the tools.

“With traditional detection, you start with technology and then use people to get the most out of that technology. With threat hunting, you start with people, and then use technology to get the most out of those people.”

In traditional detection, analysts utilize a large number of tools that will automatically generate alerts for triage and investigation. This model relies on matching traditional indicators such as known malicious IPs/domains. This type of approach has proven to have high false-positive rates and be ineffective for detection, as attackers are constantly changing their infrastructure and iterating their tools. Comparatively, the threat hunting model uses large data sets with behavioral highlighting on key techniques used by attackers, along with a research-focus to enable hunts for both known and unknown threats. These differences are important at a corporate level for competency and service offering, but these also make a massive difference in job satisfaction.

The SOC/MSSP experience

The SOC analyst role is focused on dealing with the alerts generated from the various tools (SIEM, IDS, etc) that are monitored on a day-to-day basis. This relies on signatures or ‘correlation’ rules (a misnomer as these are often just aggregation rules) for detection to notify an analyst something is potentially bad, and that they need to investigate. This approach has a number of weaknesses, both from an individual and service point of view:

• Detection is signature and tool reliant
• Prone to high false-positive rates
• Causes analyst alert fatigue
• Pushes analysts (and organizations) into a reactive, rather than a proactive mind-set
• Is less agile and able to deal with new and emerging threats

To elaborate, this approach is focused primarily on finding ‘known’ threats. Analysts react to alerts triggering off of known malicious indicators, like the hash of a file, a key string related to the attack, or domain/IPs of the attacker’s command and control channel. Whilst by definition this approach should be full of high confidence rules, it is often the case that rules are poorly written or are too broad to catch all suspicious activity, resulting in lots of false-positive alerts. It is not uncommon for an analyst to have to wade through thousands of alerts a day, which leads to alert fatigue and the missing of genuine incidents amongst the excessive ‘noise’ generated by these tools.

One other major shortcoming of this approach is that it has no capability to detect new (previously unseen) threats. This is a serious weakness considering the high percentage of unique or new attacks that occur, and how easy it is for attackers to generate new variants of malware that bypass traditional detection mechanisms.

This entire approach pushes analysts and organizations into a ‘reactive’ mind-set where they are waiting for alerts to fire, and for changes to be pushed to their tooling by vendors, instead of proactively searching for intrusions and finding new threats. The traditional model is far less engaging, less technical and less interesting for security professionals, leading to fatigue, burnout and a less effective detection team with talented staff quickly moving on to other roles.

Threat Hunting: a new approach

By contrast, threat hunting is an approach that focuses less on the ability to utilize detection technology (such as a SIEM or IDS) and more on the threat hunter’s knowledge of key attacker behaviors and techniques. For example, when hunting for malicious uses of PowerShell a threat hunter is able to look at the command line arguments, parent relationships and use their experience to differentiate it from legitimate admin activity. This focus on detecting attacker behaviors rather than volatile indicators (such as hashes for known malicious files) is then coupled with a proactive mentality, with a large part of a threat hunter’s role dedicated to researching into new attack methods that can be ‘hunted’ for. As can be seen in figure 1, hunting for specific techniques can allow you to catch multiple quickly evolving variations of malware that use the same common technique.

(Figure 1 – Hunting for techniques, not indicators)

The focus on attacker techniques rather than volatile indicators has the advantage that there are far fewer potential techniques to cover, e.g. there are only so many ways to execute a file on an endpoint. Therefore focus in this area, along with continuous research into new attacker techniques does bring around the theoretical possibility of effectively covering all possible attacks, zero-day or not. By comparison, traditional indicators have an infinite number of possible values, meaning blue teams are endlessly playing catch up.

A good example of an attack technique that could be actively hunted for ahead of time, is with the latest ShadowBrokers release revealing a zero-day SMB exploit that had been in use for at least the last four years undetected. It was chained with an implant using a new kernel DLL injection technique that was already detectable for any threat hunters looking for reflective loading of DLLs. Searching for this implant using a signature-based detection model would require knowledge of this specific attack framework ahead of time, which can only be acquired after the framework is already in the public domain.

As the threat hunting approach does not rely on rules or signatures to trigger alerts for threat hunters to wade through, there is far less mental fatigue and burnout than is experienced by SOC analysts. The emphasis is instead on the threat hunter to proactively hunt, based off of the tagging of interesting techniques, anomalies within data sets, and conducting hunting sprints through generating hypotheses for new attack techniques. This proactive approach requires a platform and organization that empowers threat hunters to be agile and interactive with the environment they are hunting on. For example, the capability to pull back that suspicious executable running on a single host, the ability to run a full memory dump, or access full PCAPs to aid investigation. This will drastically reduce the effort and time spent investigating suspicious activity, allowing threat hunters to continue to hunt for further activity and give quicker response times.

One of the key strengths of threat hunting is the ability to look for low confidence indicators without drowning yourself in thousands of alerts. This is important because state-sponsored and advanced criminal groups have the resources and capability to ensure their tools do not generate high confidence indicators. This is a problem that SOCs so far have not been able to find a solution for, causing them to fall behind their offensive counterparts. The threat hunting approach means that the entire team’s ability to detect attacks is continuously improving as everyone is empowered and encouraged to make innovations to the detection methods used, and therefore able to keep up with the latest threats.

The data sets threat hunters use come in filtered and raw varieties, with machine learning and intelligence applied to them to filter out some of the less relevant data. This is a careful balance to tread, between ensuring that you don’t fall into the pitfalls purely algorithm reliant approaches do of filtering out too much useful data, but are also able to operate effectively at scale by not dealing with purely unfiltered raw data sets. The hybrid approach of having both sets available ensures the flexibility of threat hunters to operate both effectively and at scale, without missing any key data.  The mentality behind this choice of having both data sets is an important point as it highlights how threat hunting is not a one-track process of detection and is instead an approach to detection from many different angles. Hunting through data in a variety of ways ensures that different data is highlighted and that one malicious indicator that will usually blend in, will, in fact, stand out and be detected. This non-uniformity is a strength for threat hunting in the long run as it ensures malicious actors will struggle to customize exploits to bypass such a varied approach to detection.

The entire threat hunting approach puts the emphasis on the threat hunter being the intelligence behind the detection capability. This results in the large amount of money that has traditionally been invested in expensive off-the-shelf commercial software, instead of being invested in the teams to provide training, development, and improvements, as well as research and development for bespoke detection techniques. This creates a team that is highly skilled and motivated with a wealth of expertise that can be leveraged to be more agile and keep ahead of the latest security threats and trends.

One of the biggest pulls for changing from a SOC analyst to a threat hunter is the opportunity to conduct research, with a large portion of a threat hunter’s time being set aside for researching into new attack techniques and generating hypotheses. This adds another area of variety and reward for a threat hunter to counteract much of the burnout and fatigue suffered by traditional SOC analysts. The research time for threat hunters also has a massive positive multiplier for the effectiveness of the service. The approach to provide free research time for employees has been taken by many of the top technology companies worldwide and has shown to be an effective way at improving innovation in the business, which as shown above is an essential component of successful threat hunting.

(Figure 2 – The Paris Model, reference: http://threathunter.guru/blog/the-paris-model/)

The strength of having the MWR group

As the Paris Model above shows, there are many streams that must feed into threat hunting from various security disciplines. The MWR Group covers a range of different disciplines, from Cyber Defence right through to Targeted Attack Simulations (TAS), which allow the threat hunting team to draw expertise from some of the best minds across the industry and gain key insights to drive our hunt sprints. These insights could be as simple as the latest IR report providing details of a new attack technique that is being used in the wild, which we can generate a hunting use case for, to hunt for adversaries across our client base.

This approach emphasizes how threat hunting is driven by a proactive mindset that draws from many different perspectives (red and blue team), in order to aid detection and avoid getting stuck in the one track thinking of traditional SOCs that has left large holes in detection capability. It is important for any organization undertaking the development of a threat hunting capability to draw from as many different security disciplines as possible, as well as the hunt team conducting their own research into potential attacker techniques that could emerge in the future.

To conclude

Threat hunting is not just another marketing buzzword, it is a substantial shift in approach from the old traditional SOC analyst, technology-centric approach, to a more people-centric, innovative and research-focused detection mantra. Threat hunting is currently a niche area in the industry but with the many advantages it brings, it is one that is gaining significant traction and attention from many organizations who are looking to integrate it alongside their existing SOCs. At this early stage, there is no industry standard for threat hunting or even a standardized view of how it is conducted. However, a consensus is rapidly forming around some key principles and with clear value already being demonstrated for clients, this is not just another security fad that will fade away.

On a personal level, the transition from a SOC analyst to a threat hunter is not the simplest and requires the development of a new analysis skillset as well as a change in mental approach to detection, but it is one that will pay off significantly in the long run in terms of job satisfaction and development. This is important for the defense industry as there has been a historic drain of talent to other areas, largely offensive disciplines, resulting in a skill shortage and capability gulf. The creation of threat hunting roles will result in greater retention of talented individuals and help contribute to the rise in capability of the defense industry as a whole. The development of a hunt team for organizations is not as simple as going out and buying new technology but requires a substantial shift in mindset and capability. However, the need to counter advanced threat actors and the increase in capability threat hunting offers does ensure that it is a worthwhile payoff for any organization who is serious about security and is likely to be the subject of a targeted attack.

With this year having the first security conference entirely devoted to threat hunting and Gartner publishing reports describing its key advantages, one thing can be said for sure: threat hunting is here to stay.