Tracing Just-In-Time (JIT) Compilation
When a given .NET method in a loaded assembly has not yet been executed, the Common Intermediate Language (CIL) code exists in memory but the native code to actually execute that CIL does not yet exist. A C# compile will produce CIL code but it is up to the JIT engine to turn this into native code at runtime as required. This is the very nature of JIT compilation and the native code will only be generated if and when the method is actually called.
You can see some of the background on how this works in the following article (http://mattwarren.org/2017/12/15/How-does-.NET-JIT-a-method-and-Tiered-Compilation/), which shows how on first execution the native code is generated by the JIT compiler and then on subsequent calls there is no need to interpret the CIL anymore and the cached native code is used directly.
This has a really interesting side effect for introspection of .NET execution because the CLR ETW providers we covered previously have events related to the JIT compilation of methods, meaning we should be able to get an event every time a .NET method is first utilised. This gives us much more data about what is actually occurring inside a process invoking the CLR. If we enable the correct keywords to trace MethodJittingStarted events and then dynamically invoke our DemoAssembly from the previous post our script (https://gist.github.com/countercept/7765ba05ad00255bcf6a4a26d7647f6e) will give output like the following:
Now, not only can we see the dynamic loads of the assembly via byte array but we can see that both the constructor (.ctor) and the noNames() method were compiled by the JIT engine, demonstrating they were both executed at least once. In this case, we are looking at a very simple example, but you can see how this would provide powerful visibility into what an assembly might actually be doing under the hood.
An Overtly Malicious Example – Meterpreter Powershell Stager via SharpPick
Let us now consider an actual malicious example that contains more functionality than a single method call. For this example, we will use the psh-net output type from msfvenom and we will execute it via SharpPick to execute it dynamically without the actual use of powershell.exe itself.
Here we can see there are several method calls that might be considered higher risk and less common for a legitimate .NET application to be using:
- CompileAssemblyFromSource() – Dynamic compilation of C# code at runtime to generate an assembly in memory
- FromBase64String() – Plenty of legitimate use cases but also very commonly used by attack frameworks and malware, particularly for encoding of shellcode
- Native API Calls – VirtualAlloc() and CreateThread() in particularly are commonly used maliciously but .NET applications would not usually be using these functions for explicit native memory or thread management
However, if we look for MethodJittingStarted messages using the same technique as before we will be disappointed in that we will not see JIT compilation events for any of these interesting methods. We will see some interesting events related to the use of SharpPick itself though:
So in this instance we see the compilation of the Main() and RunPS() methods within the SharpPick tool itself, still useful, but none of the more interesting method calls within the supplied script. To understand why this is, we need to cast our minds back to the discussion in the first part of this blog post around the difference between standard .NET assemblies and native assemblies that have been pre-compiled using the Native Image Generator (NGEN).
Most of the common .NET assemblies provided by Microsoft have NGEN generated native assemblies present on disk and so the native code for these assemblies already exists and does not need to be JIT compiled. This explains the lack of JIT compilation methods for the activity within the meterpreter stager. However, all is not lost for there are other event types that can provide introspection in these areas in other ways.
JIT Inlining and Interop
Much like with native compilation, JIT compilation has a number of optimizations that can be performed to make code more efficient. One of these – inlining – is the process of copying the code for frequently called methods directly into the code of the calling method to avoid continuous method call overhead. A number of metrics determine when it is advantageous to do this, but generally speaking very frequently called methods that are very small in size are best suited for inlining.
This introduces an interesting possibility for introspection because the CLR ETW providers provide visibility of inlining of methods both in cases of success and failure. The consequence of this is that, even for NGEN compiled assemblies, if standard .NET assemblies make calls to methods within these assemblies then we can gain visibility of that through the generation of either the success or failure of the inlining process via the MethodJitInliningSucceeded and MethodJitInliningFailed event types:
Here we can see a failed inlining attempt of the call to FromBase64String() when it was called by the dynamic class that was generated as a consequence of SharpPick dynamically loading the supplied powershell script. This gives us a further level of visibility into the behaviour of the CLR in this instance.
Another interesting set of events we can trace are Interop events. These are generated when calls are made to the native Win32 API. This is very useful as it is common for malicious use of powershell or .NET to make use of P/Invoke to stage the execution of native code and there are often a common set of high-risk API calls associated with this, such as those used for native memory allocation, cross-process access for code injection or thread management for execution of native code. If we put all these event types together and focus in on high-risk events then the output of our meterpreter stager run via SharpPick generates is as follows:
In this case, we have focused in on a number of key events that are greater security concern:
- Use of RunPS() method indicating use of SharpPick (very specific but easily defeated indicator)
- Loading of System.Management.Automation assembly in a non-powershell process
- Loading of dynamic assemblies loaded via byte array (in this case with a randomised name)
- Use of FromBase64String() method
- Interop calls to both VirtualAlloc() and CreateThread()
Another Malcious Example – GhostPack’s SafetyKatz
Metasploit’s meterpreter serves as a great example but is also an older example. As a more modern example of malicious use of .NET, we will analyze SafetyKatz, which is part of the excellent GhostPack suite of tools (https://www.harmj0y.net/blog/redteaming/ghostpack/).
GhostPack is currently a collection of C# tools that replace many offensive tools that were previously powershell only, and so they avoid the detection and logging improvements that have been made to newer versions of powershell. One example is SafetyKatz, which uses C# to take a minidump of the lsass process and then uses a dynamic PE loading mechanism in C# to dynamically load mimikatz and extract the credentials from the minidump. Consequently, this avoids powershell logging and avoids mimikatz ever existing on disk. If we are investigating some of the output from this we can see a range of potentially dangerous interop calls.
If we look at the associated namespace, these come from within Microsoft assemblies. This is related to the internal implementation of some .NET methods used by SafetyKatz, such as querying if the process has high integrity etc. Whilst this is definitely interesting, it is likely to be more false positive prone. However, if we continue focusing in on a specific list of dangerous functionality as well as paying attention to namespaces, we can see a really useful overview of the primary malicious behaviour:
Now we can see a few aspects of behaviour that are clearly of concern:
- Interop calls from the main space of the application
- MiniDumpWriteDump() for acquiring the actual process dump
- VirtualAlloc(), LoadLibrary(), CreateThread() etc which for the PE loading of mimikatz dynamically
- JIT events for MemoryStream() and FromBase64String() related to the unpacking of the mimikatz binary in-memory
A Note on Performance
In order to trace some of the method-level event types, we need to enable verbose logging in the ETW provider. Some of the JIT events can be extremely noisy, particularly if you have a lot of non-native .NET assemblies loading on your system. The PyWintrace library is only intended as a research and exploration tool for ETW and is extremely inefficient at processing ETW events compared with using a C# or C/C++ implementation. However, it can still be useful for introspecting malicious .NET execution in a lab environment.
Whilst any attempts to perform CLR event tracing in a production setting would no doubt make use of an efficient ETW processing pipeline, it is possible that JIT-level tracing may still produce a high load. However, assembly load and interop tracing combined are still very powerful and produce a much lower data rate than full JIT tracing.
In this blog post we built on the previous work and looked at how JIT and Interop tracing can be used to gain a much deeper insight into the behaviour of a process invoking the .NET CLR by gaining method-level visibility, rather than simply assembly loading information. This data is extremely useful from both a detection and dynamic malware analysis perspective.
As a couple of examples, we looked at how this can easily show highly suspicious indicators when using SharpPick to dynamically execute a meterpreter powershell stager via .NET as well as using the recently released SafetyKatz tool, for executing mimikatz via C#, that is part of the GhostPack framework.