Endpoint Detection of Remote Service Creation and PsExec

Lateral movement is when attackers move from a compromised host to other hosts to expand their access and reach their goal. If threat hunters can detect malicious activity on an endpoint they may see similar indicators appearing on new machines when lateral movement has occurred. But if they can detect the lateral movement as it is happening it can be much quicker to see how the attacker is moving around, decreasing response times and possibly providing opportunities for quick containment actions.

In this article, we’ll explore remote service creation as a lateral movement technique, and illustrate how we might spot it on an endpoint. We will use this as an opportunity to explore Event Tracing for Windows (ETW), as well as how RPC calls work in Windows.

After a primer on ETW, we’ll look first at two built-in Windows utilities for creating a service, sc.exe and WMI, and then look at the Sysinternals tool PsExec, which uses remote service creation as a way of executing commands on a remote host.

ETW

Event Tracing for Windows (ETW) is a kernel-level tracing facility built into Windows that allows a wide range of system activity to be traced in real time. This makes it a great telemetry source for attack detection.

Since Windows Vista, the Windows Event Log has been built on top of ETW and both log events and ETW events have similar metadata associated with them. As we will see later, this means we can sometimes directly correlate logs with ETW events.

There are many ways to interact with ETW, including several different Microsoft utilities, as well as custom written code. For this post we will be using the built-in logman  tool to capture data, and will make use of Microsoft Message Analyzer  as a convenient way of searching through the results. You can also use Message Analyzer to capture data, but I found it simpler to script up logman (which is command line) on my lab machines to grab the data to analyze later.

ETW events are obtained through providers, each being identified by a GUID and, in many cases, a human-readable name. You can get a list of the providers registered with the OS with the following command:

logman query providers

This will be a long list, but it still isn’t all the providers available. Applications can define their own providers, and you can get a list for a given process by adding the “-pid <processid>” argument to the above command. You may not get human-readable names for all of the applications providers though, so mileage may vary.

A provider will split events up by level (verbose, informational, warning, etc.) and keyword, and when subscribing to a provider you can pass in a combination of levels and keywords that you want to receive. To get details of what a given provider has available you can query it by name or GUID:

logman query providers "Microsoft-Windows-RPC"

This will list the keywords and levels the provider supports, as well as the processes on the system that use the provider.

There are lots of options for how to capture data from ETW, but to do a basic capture to a file starting immediately, you can just do the following:

logman -ets start some_test_log -p SomeProvider "keyword1,keyword2" win:Informational

When you are done, stop with:

logman -ets stop some_test_log

This will log everything to some_test_log.etl.

If you leave out the keywords and levels, it will default to all keywords and all levels. This can be useful when testing to see what appears, but some providers will produce a torrent of data when you do this.

You can also enable multiple providers by creating a file listing, a provider name or GUID – one per line – and passing the filename to logman using “-pf” rather than “-p”. You can add keywords and levels after each provider name if you need to, as well.

In the lab, I use “-pf” to enable a large number of providers, which I create by grepping through the full list for any that sounds relevant to what I’m looking at. This is a great way to discover new providers of interest.

To analyze the data, we can use Microsoft Message Analyzer to help filter down to the interesting events, as well as make use of its parsers to get as much readable information out of the events as we can.

If I have a capture with more than one provider, I like to use Message Analyzer’s “Group” functionality to group events by provider. This puts events under their respective provider and lets you collapse them down to consider each provider separately, but still allowing you to search and correlate between providers. Right click the Module column header and select “Group” to do this.

For production use, there are several APIs that can be used to consume ETW events in real time. The specifics of this are beyond the scope of this article, but you can find more information from the TechNet blog post, Hidden Treasure: Intrusion Detection with ETW (Part 2). Another option might be using Python with pywintrace though this may not be as performant as .NET or native options.

Remote Service Creation

It is possible to create and start a Windows service on a remote machine. This can be used to move laterally by using existing credentials or tokens to create and start a service running your code on another machine. This is actually how tools like PsExec work under the hood as well, so looking at remote service creation in general can be very useful.

To create a service, you need to specify a service executable on the target. This usually means you need to get your service executable onto the machine first, and a common way to do this is via a hidden share such as C$ or ADMIN$, which are standard administrative shares. Remember, this is lateral movement, so it is assumed that the attacker has an account with administrative privileges on some hosts already.

After this, the service can be created and started using standard Windows utilities such as the sc.exe command line tool, or using a facility like WMI. For this post, we’ll be looking at both of these to discover how they operate under the hood, find ways to generically detect those mechanisms on an endpoint, and finally have a look at PsExec to see which method it uses to remotely create its service.

So, let’s see what remote service creation looks like first for sc and then WMI, broken up into three stages for each:

1. Copying the service executable to the target;
2. Creating the service entry;
3. Starting the service.

For this example, everything was performed on Windows 7 x64.

Sc.exe

File Copy

Copying the file to the target is just a standard SMB file transfer to a hidden administrative file share. This is certainly not specific to service creation, and while not the focus of this post I will mention that you could log such share access to the event log with some combination of the Audit File Share, Audit Detailed File Share, and possibly even Audit File System settings; these could generate a high volume of logs depending on the environment.

Service Create

Service creation is achieved with a command like the following:

sc \\targethost create evilservice binPath= C:\evilservice.exe

Like many Windows features, this command uses RPC to communicate with the remote host. RPC on Windows can use a variety of underlying network transport protocols, including basic TCP, HTTP, or SMB named pipes. This means RPC can look very different on the wire, but it all appears in the RPC ETW provider in a similar way, making it a great place to catch this activity.

RPC is also used extensively as an IPC mechanism between processes on the same machine, and this usually uses a special Lightweight Remote Procedure Call (LRPC) transport. This is a useful thing to know if you want to quickly filter through RPC events and find those that originated remotely.

To capture these events, enable the “Microsoft-Windows-RPC” provider. You may want to disable the verbose level as this generates a huge amount of data that has not proven so useful for what we want to do.

Even at informational level you will see a lot of events from this provider as there is always plenty of RPC happening. For RPC events, in the Summary field, a common pattern will be seen where an RPC call goes through these stages:

Client RPC call started.
Server RPC call started.
Server RPC call completed.
Client RPC call completed.

Whether you see the entries for client or server depends – of course – on whether you are on the host making the call or receiving it, and for local RPC you see both. The “completed” entries just give the call status (usually 0 on success) and in this case we’re less concerned with that, so we can reduce clutter by filtering out these rows. At the top of the Message Analyzer window you can enter a filter string to do this (remember to click ‘Apply’):

Service endpoints get a lot of incidental traffic on any host; this is because they provide basic SMB functions to get information about hosts to facilitate SMB connections in general. So our final filter is:

!*Summary contains "LRPC" and !*Summary contains "completed." and !*Summary contains "Extended Error Information:" and !*Summary contains "srvsvc" and !*Summary contains "wkssvc"

Using this filter we can more easily pick out some interesting events:

Server RPC call started. InterfaceUuid: e1af8308-5d1f-11c9-91a4-08002b14a0fa OpNum: 0x00000003 Protocol: TCP Endpoint 135

Server RPC call started. InterfaceUuid: 367abb81-9844-35f1-ad32-98f038001003 OpNum: 0x0000000F Protocol: Named Pipes Endpoint \pipe\ntsvcs

Server RPC call started. InterfaceUuid: 367abb81-9844-35f1-ad32-98f038001003 OpNum: 0x0000000C Protocol: Named Pipes Endpoint \pipe\ntsvcs

Server RPC call started. InterfaceUuid: 367abb81-9844-35f1-ad32-98f038001003 OpNum: 0x00000000 Protocol: Named Pipes Endpoint \pipe\ntsvcs

Server RPC call started. InterfaceUuid: 367abb81-9844-35f1-ad32-98f038001003 OpNum: 0x00000000 Protocol: Named Pipes Endpoint \pipe\ntsvcs

The first of these is just a call to the endpoint mapper service; we can tell this from the TCP port number, and also the interface ID if we care to look it up. After this we see calls via the \pipe\ntsvcs named pipe, which certainly sounds like a services-related endpoint.

We can further resolve the interface ID and OpNums to find the actual RPC functions being called here, assuming the interface is documented publically as many built in interfaces are. We find the interface ID is defined in [MS-SCMR]: Service Control Manager Remote Protocol  and section 3.1.4 lists OpNums and the functions they relate to, with details of what each function does. We can then translate this into the following series of calls:

1. 0x0000000F (15) – ROpenSCManagerW
2. 0x0000000C (12) – RCreateServiceW
3. 0x00000000 (0) – RCloseServiceHandle
4. 0x00000000 (0) – RCloseServiceHandle

A fairly intuitive series of calls, with RCreateServiceW being the most meaningful (in general terms you may also want to look out for 24 RCreateServiceA, 44 RCreateServiceWOW64A, and 45 RCreateServiceWOW64W).

These events do not, however, tell us any details of the service that was created, nor do they tell us what remote host made the calls. If we have telemetry from the attacking host as well, we can see corresponding calls to the same interface and OpNums, but we also see the destination host. Interestingly, the endpoint is a different named pipe:

NetworkAddress \\targethost  Endpoint \pipe\svcctl

[MS-SCMR] actually lists this pipe name as being associated with the interface ID. Alternative names can be used for some named pipe RPC operations, which is one reason knowing the RPC interface ID can be very useful.

Correlating events from the attacker and target could allow a more complete picture of what accounts and hosts were involved in the lateral movement, and the event log could provide details of the service being created.

It is also worth mentioning that with default settings we see an entry in the event log that is very relevant here. From the Service Control Manager in the System log, event ID 7045, we see the following:

A service was installed in the system.

Service Name:  evilservice

Service File Name:  C:\evilservice.exe
Service Type:  user mode service
Service Start Type:  demand start
Service Account:  LocalSystem

This is a pretty clear indication that a service was created. It is hard to tell if this may be part of lateral movement however, as there is no information on whether the service was created remotely. We do get to see the details of the service itself though, such as its name and executable.

Service Start

The newly created service can be started with a command such as:

sc \\targethost start evilservice

We get another Service Control Manager log entry, in the System log, event ID 7036:

The evilservice service entered the running state.

In ETW, we will be able to see some events from the Microsoft-Windows-Services provider. This did not produce anything when a service was created, but it does now show the service changing state:

Event_105{ExecutionPhase=0,CurrentState=2,StartType=3,PID=0,ServiceName=evilservice,ImageName=,… }

Event_105{ExecutionPhase=0,CurrentState=2,StartType=3,PID=3100,ServiceName=evilservice,ImageName=C:\evilservice.exe, … }

Event_105{ExecutionPhase=0,CurrentState=2,StartType=3,PID=3100,ServiceName=evilservice,ImageName=C:\evilservice.exe, … }

Event_105{ExecutionPhase=0,CurrentState=4,StartType=3,PID=3100,ServiceName=evilservice,ImageName=C:\evilservice.exe, … }

Ignoring the bit of repetition of state 2, this shows the service moving from state 2 to state 4. We can see what these values mean in various places in Microsoft documentation, including https://docs.microsoft.com/en-us/windows/desktop/services/service-status-transitions, which tells us that 2 is SERVICE_START_PENDING and 4 is SERVICE_RUNNING. We also get the PID of the service process here, which may be useful to know.

Turning to RPC, if we use our previous filter for interesting non-LRPC calls, we will see a series of remotely initiated calls via \pipe\ntsvcs, which translate to:

1. 0x0000000F (15) – ROpenSCManagerW
2. 0x00000010 (16) – ROpenServiceW
3. 0x00000013 (19) – RStartServiceW
4. 0x00000028 (40) – RQueryServiceStatusEx
5. 0x00000000 (0) – RCloseServiceHandle
6. 0x00000000 (0) – RCloseServiceHandle

So a clear remote call to RStartServiceW (in general terms you may also want to look out for RStartServiceA, OpNum 31).

WMI

Windows Management Instrumentation (WMI) is a whole framework for managing Windows systems. It also has the concept of a “provider”, but these are very different from ETW providers. Rather they provide code that implements WMI classes, which can perform various management activities. For service management, WMI has a Win32_Service class that has methods to both create and start services.

The wmic command line tool can be used to call these methods on the Win32_Service class. It has a special alias mode that simplifies accessing common classes, and as we shall see Win32_Service can be used with the alias “service”.

We will not look at what service creation through WMI looks like on endpoints. The file copy stage is the same as before, so we do not look at that here.

Service Create

A service can be remotely created with WMI using a command like the following:

wmic /NODE:"targethost" service call Create PathName=C:\evilservice.exe DisplayName=evilservice DesktopInteract=false StartName="NT AUTHORITY\SYSTEM" StartPassword="" ErrorControl=3 LoadOrderGroup="" LoadOrderGroupDependencies="" Name= evilservice ServiceDependencies="" ServiceType=16 StartMode=Manual

Although many arguments are blank here, all are required to be present to make a valid call.

This will generate an event log entry just like creating a service with sc.exe, event ID 7045. It is not possible to tell from the event log alone what method was used to create the service.

The ETW provider Microsoft-Windows-WMI-Activity shows this very clearly on the target:

GroupOperationId = 283917; OperationId = 283917; Operation = IWbemServices::Connect; ClientMachine = TARGETHOST; User = TESTDOMAIN\Administrator; ClientProcessId = 3124; NamespaceName = \\targethost\ROOT\CIMV2

GroupOperationId = 283918; OperationId = 283919; Operation = Start IWbemServices::ExecMethod – Win32_Service::Create; ClientMachine = TARGETHOST; User = TESTDOMAIN\Administrator; ClientProcessId = 3124; NamespaceName = \\.\ROOT\CIMV2

ProviderInfo for GroupOperationId = 283918; Operation = Provider::ExecMethod – Win32_BaseService::Create; ProviderName = CIMWin32; ProviderGuid = {d63a5850-8f16-11cf-9f47-00aa00bf345c}; Path = %systemroot%\system32\wbem\cimwin32.dll

The first of these events shows a connect event, but has a different GroupOperationId to the rest, so is less useful to pinpoint specific activity. However, the second event shows exactly what action is being performed, a call to Win32_Servie::Create, and has the user context as well. Unfortunately, it does not tell us the name of the service or the service executable. The third event shows a lower level call to Win32_BaseService::Create, but this adds no more useful information for us.

If we filter out remote RPC events, we see several interfaces being used, which are all part of either the [MS-WMI] or [MS-DCOM] specifications. WMI uses DCOM for remote calls, and DCOM uses RPC as a transport protocol, which is why we see this activity. However, it does not provide enough detail to really see what is happening beyond that an unknown method is being called on a WMI object remotely.

What we do get from the RPC events, however, is the TCP port (specified in “Endpoint”), as seen in the example below:

Server RPC call started. InterfaceUuid: 00000143-0000-0000-c000-000000000046  OpNum: 0x00000003  Protocol: TCP  Endpoint 49154

The TCP port may be helpful to correlate with any network data you may also be working with.

Service Start

The service can be started with WMI using the following command:

wmic /NODE:"targethost" service evilservice call StartService

This produces an identical event log entry as starting the service with sc.exe, event ID 7036.

We also get ETW events from Microsoft-Windows-Services, similar to those when starting the service with sc.exe.

From Microsoft-Windows-WMI-Activity we get a number of events, most of which can be grouped together by the GroupOperationId. These show the service being looked up by name in the WMI database before we see the one event that provides all the information we want:

GroupOperationId = 62668; OperationId = 62677; Operation = Start IWbemServices::ExecMethod - \\TARGETHOST\ROOT\CIMV2:Win32_Service.Name="evilservice"::StartService; ClientMachine = ATTACKINGHOST; User = TESTDOMAIN\Administrator; ClientProcessId = 2092; NamespaceName = \\.\ROOT\CIMV2

This shows a call to StartService on a Win32_Service instance for our named service, with both the attacking and target host named, the user account being used, and even the PID of the resulting service process.

We also get RPC events, which – much like for service create – do not add much useful information beyond the TCP port in use; but for the curious, below is a table of what methods we can see being called remotely as part of the WMI/DCOM/RPC that is in action.

PsExec

Having seen what remote service creation looks like with two different built-in system utilities – sc.exe, which uses the RPC based Service Control Manager Remote Protocol, and WMI, which uses its own protocol over DCOM (itself RPC based) – let’s have a look at what PsExec uses to create its service.

PsExec is part of the Sysinternals Suite of tools, and allows commands to be executed remotely with the command line input/output redirected to give the feel of execution locally. To achieve this, PsExec remotely creates a service on the target host that handles I/O redirection and execution of the command on the target. By default, PsExec creates a service named PSEXESVC, although this can be changed.

For this example, the following command was used, which simply executed ipconfig on the target:

psexec64 \\targethost ipconfig

PsExec creates its service, performs its task, then cleans up the service, stopping and deleting itself. And we do indeed see the familiar event log entries from the SCM, along with an entry for the service being stopped.

Event ID 7045:

A service was installed in the system.

Service Name:  PSEXESVC
Service File Name:  %SystemRoot%\PSEXESVC.exe
Service Type:  user mode service
Service Start Type:  demand start
Service Account:  LocalSystem

Event ID 7036:

The PSEXESVC service entered the running state.

Event ID 7036:

The PSEXESVC service entered the stopped state.

Of course, the name of the service reveals PsExec is in use, but there is no reason an attacker would necessarily keep the default. Otherwise, these are standard service lifecycle events, and we have no idea if the service is being manipulated remotely or by what method.

From the Microsoft-Windows-Services ETW provider we can see the service changing states, along with its PID:

Event_105{ExecutionPhase=0,CurrentState=2,StartType=3,PID=0,ServiceName=PSEXESVC,ImageName=, …}

Event_105{ExecutionPhase=0,CurrentState=2,StartType=3,PID=2024,ServiceName=PSEXESVC,ImageName=C:\Windows\PSEXESVC.exe, …}

Event_105{ExecutionPhase=0,CurrentState=2,StartType=3,PID=2024,ServiceName=PSEXESVC,ImageName=C:\Windows\PSEXESVC.exe, …}

Event_105{ExecutionPhase=0,CurrentState=4,StartType=3,PID=2024,ServiceName=PSEXESVC,ImageName=C:\Windows\PSEXESVC.exe, …}

Event_105{ExecutionPhase=0,CurrentState=3,StartType=3,PID=2024,ServiceName=PSEXESVC,ImageName=C:\Windows\PSEXESVC.exe, …}

Event_105{ExecutionPhase=0,CurrentState=1,StartType=3,PID=0,ServiceName=PSEXESVC,ImageName=, …}

This amounts to the service changing through the following states:

1. (2) SERVICE_START_PENDING
2. (4) SERVICE_RUNNING
3. (3) SERVICE_STOP_PENDING
4. (1) SERVICE_STOPPED

We also see remote RPC calls to the SCM interface {367abb81-9844-35f1-ad32-98f038001003}, the same one sc.exe used. The calls made are as follows:

The calls highlighted in bold show the major high level actions; RControlService is a little less intuitive, but its documentation shows that one of the control codes it accepts is SERVICE_CONTROL_STOP, which would make sense here.

Seeing a service created remotely, which is started, stopped, and deleted quite rapidly, is a fairly suspicious event. Of course, PsExec is used legitimately in many organizations, so in such environments it cannot be taken immediately as malicious, but the genuine tool used legitimately will most likely maintain the standard executable name for both the tool itself and the service.

Another interesting point is that while PsExec uses the same RPC interface as sc.exe did, it makes the RPC calls directly over TCP rather than going via SMB named pipes. This makes the traffic look very different on the wire, with SMB traffic using port 445 and the TCP transport using a random unprivileged port. However, the ETW RPC provider captures all this in the same place and in the same format.

Despite these differences on the wire, an honourable mention must go to Wireshark, whose dissectors are able to carve out the SCM protocol (as “SVCCTL”) from both SMB and TCP in a very similar looking way:

Conclusion

We have seen some of the traces remote service creation leaves behind on an endpoint in the form of event logs and other ETW providers that can be utilized. One advantage of using ETW is that by its nature it is easier to process in real time, allowing rapid detection of service events as they happen. But we have also seen that the event logs are not specific to what method was used to manipulate the service, and give no indication if a service was created or started remotely, and if so, from where. By supplementing this with ETW events, we can determine if RPC or WMI calls were initiated remotely, from where, and under what user context.

We have also seen how the RPC provider gives information on calls in one place in a common format, regardless of the underlying transport protocol, which can look very different on the wire. As so many things are driven by RPC in Windows, this could prove useful in other detection scenarios as well.

A summary of indicators relevant to service creation as a lateral movement technique is given below:

• Event log IDs:
  • – 7045 – Service create
  • – 7036 – Service state change (running, stopped, etc.)
• ETW Microsoft-Windows-Services provider:
  • – Identify service with ServiceName / ImageName, track state with CurrentState
• ETW Microsoft-Windows-RPC provider:
  • Interface {367abb81-9844-35f1-ad32-98f038001003}, OpNums:
  • – 12, 24, 44, 45 – Create service
  • – 19, 31 – Start service
  • – 2 – Delete service
• ETW Microsoft-Windows-WMI-Activity provider:
  • Operation:
  • – Start IWbemServices::ExecMethod – Win32_Service::Create
  • – Start IWbemServices::ExecMethod – \\<TARGET_HOST>\ROOT\CIMV2:Win32_Service.Name=”<service_name>”::StartService
• – Rapid sequences of service create, start, delete could indicate PsExec-like behaviour.