Microsoft's Accidental Enterprise DFIR Tool

SCCM can be a goldmine when hunting for evil, all you need to do is enable some inventory collections, send them to Splunk and get creative. While the data is snapshot in time (usually the last 24-hours) it can be a great first start when dealing with incidents, plus most enterprises already have SCCM.

In this post all breeze over the setup & configuration of and focus more on some of the searches you can do to find evil. As a bonus, you can you use this to monitor the health and coverage of your security agents with some pretty dashboards in Splunk.

Some enterprise wide hunting-for-evil examples I’ll cover:

Finding service outliers

Find least frequency of occurrence for persistent mechanisms via autoruns.

Find least frequency of occurrence of installed services.

Least frequency of occurrence of executables.

Services running from abnormal paths.

Triage enterprise using IoC

Ingredients

1 Microsoft SCCM server + access to the SCCM SQL server.

1 Splunk server

1 Splunk DB Connect App

* Adult beverage is optional but strongly recommended.  

Preparing SCCM

In SCCM enable the collection of Autoruns, Browser Helper Objects (BHO) and inventory collection of all executables. This collected data will be pulled into Splunk or directly querying with SQL statements.

Preparing DB Connect App in Splunk

The DB app executes the SQL queries on a scheduled basis and dumps them into a defined index.

Under "Identities" add in MS SQL server credentials. In "Connections" add in SCCM SQL server. Click validate to make sure everything is configured correctly.

Now to get useful data into Splunk you’ll need to make SQL queries. Your SQL server table names will probably be different so verify directly on SQL manager to make sure your queries work and pull back data in a way that is expected. As an example we’ll use autoruns, however this process will be the same for all queries.

Autorun lookup

Autorun has a ton of good info so you’ll want to have this data imported after the collection process runs, default (I think) is every 24-hours. Here is the query I used to import all the autorun data:

SELECT [Name] AS Hostname ,

[Description0] ,

[FileName0] ,

[FilePropertiesHash0] ,

[FilePropertiesHashEx0] ,

[FileVersion0] ,

[Location0] ,

[Product0] ,

[ProductVersion0] ,

[Publisher0] ,

[StartupType0] ,

[StartupValue0]

FROM [SCCM].[dbo].[v_GS_AUTOSTART_SOFTWARE]

LEFT JOIN [SCCM].[dbo].[v_ActiveClients] Name ON ResourceID = MachineResourceID

The LEFT JOIN statement will give you the hostname based upon the ResourceID along with all of the autorun details. Modify this to your needs and paste in “Input SQL” field.

Select batch and past in your SQL statement:

Click more settings and set the lookup period to every 24-hours, set sourcetype and index:

Save the input, take a swig and you’re done.

Hunting

If you’re not able to import the entire inventory of executables due to $plunk license, consider outputting the aggregated data from these examples or investigate Elasticsearch.

Since we now have all autorun data going to Splunk, we can use the following query to show autorun persistence that appear on less than 10 systems in the enterprise (by filename).

If you see a filename like ‘Update.exe’ which only exists on one system, probably going to want to investigate it. Get creative, search by product name, description or a combination of both. Since everything is timestamped you might consider adding a report on servers or high-valued targets like executives and their administrative staff.

Find malware that attempts to hide by using common Windows system filenames like ‘svchost.exe’ that are in the wrong directory:

If you see ‘svchost.exe’ in an appdata or temp folder, that would be red flag to investigate further. To take this example further you could include other common windows filenames with:

WHERE FileName IN ('wininit.exe', 'svchost.exe', ‘services.exe’) group by [FilePath]

It will be obvious where to investigate further by looking at the outliers. If you investigated enough malware infections you will no doubt have a list of clever names used my malware authors such as 'system.exe', 'lsass.exe', 'services.exe'. Also keep in mind the data collected by SCCM the next time you read about a new threat and how it can be used to quickly search for indictors of compromise.

Executables in $Recycle.Bin

Service outliers, least frequency of occurrence by path on disk:

Search for a service where the executable is located in an odd place like %temp% or %appdata% by appending: 

WHERE [Name0] LIKE ‘%temp%

Using the same methodology, search for least frequency of occurrence based on filename. Since you have the data in Splunk you might even add an alert based upon new service running on a domain controller or publicly accessible server.

If you’re feeling really adventurous, try checking out any executables hanging out in temp folder.

Fast Triage All the Files in < 1 Minute.

If you have an incident occur or just want to triage some intelligence across the enterprise using static indicators like:

Filename

Filepath

Internal File Description

Internal File Version

Files Modified Timestamp

Autorun persistence: Registry key/value or file/lnk in startup folders.

You can use those indicators to quickly triage every binary file (or any file metadata you’re collecting) in your enterprise that was available at the time the client inventory process was ran. If you don't have the files internal file description in the provided intelligence, consider looking up the MD5 on Virustotal to get it.

In previous investigations I came across malware with the internal metadata company name of “Microsoft Corp.” which is unique since Microsoft doesn’t abbreviate Corporation. Using this we could triage every executable in the enterprise to see any others executables have this company name:

The same could be done with filename, path or using a combination of the collected metadata e.g. any executable between 55K and 57K modified in the last week. This is a good way to quickly search the enterprise while your other hunting tools are still running, but keep in mind this is searching a snapshot of time, you should know at what frequency your SCCM server collects this information.

Security Agent Health Monitoring & Coverage

As a bonus, throw in some queries to check if your security agents are installed and if their service is started. As an example, here is query to check for hosts that do not have GRR installed. In this example you can replace GRR with any application, keep in mind this query checks if the service is not present. Its also possible to query add/remove programs but you get more bang for the buck by querying services.

Add the following DB input query:

SELECT DISTINCT [ResourceID],

[Name] AS Missing_GRR

FROM [SCCM].[dbo].[v_GS_SERVICE]

LEFT JOIN [SCCM].[dbo].[v_ActiveClients] [Name]

ON [ResourceID] = [MachineResourceID]

WHERE [ResourceID] NOT IN

(select [ResourceID] FROM [SCCM].[dbo].[v_GS_SERVICE]

WHERE [DisplayName0] = 'GRR Service') AND [Name] IS NOT NULL

Set the input to run every 24-hours and now you can use a simple query in Splunk to see what hosts don’t have your security agents installed:

Add in another DB input to query the status of services for your security agents that are installed:

In Splunk you can search for disabled agents with:

index=sccm  StartMode=Disabled |stats count  by Service

Now you have pretty picture to show you how many of your security agents are missing from endpoints.

As you can see there is a lot of good data to find evil and also to search for indicators of compromise from known threats or when you receive new intelligence. This is only scratching the surface but hopefully will give you some ideas that are useful in your enterprise. While SCCM doesn’t collect everything, sometime it collects enough to get you what you need for hunting or during an incident. If you found anything useful for malware hunting or DFIR in SCCM database, please share it with me +Keith Tyler 

Cheers and happy hunting!

Automating Forensic Artifact Collection with Splunk and GRR

Recently I had the need for GRR to collect forensic artifacts when a Splunk alert was triggered. The point of this is to collect the forensics data when a incident ticket is generated to save IR staff time and eliminate redundant redundant tasks.

Example Scenario
When a pre-defined malicious event is seen, Splunk will send an email with event details to the ticketing system and IR folks will investigate. One of the first step in the example below is to acquire the files in question with GRR. To save time we will want to automate the collection of evidence.

AV does a horrible job of detecting malicious scripts like JS.Proslikefan.B (and anything malicious in general). However, with the help of WLS this is simple to detect and alert on. Splunk search:

`wlslogs` (EventID=4688 OR EventID=592) InternalName=wscript.exe BaseFileName!=wscript.exe

To briefly explain this alert, JS.Proslikefan maintains persistent by executing a 'random filename.lnk' file in the startup folder. The LNK file executes a randomly named copy of 'wscript.exe' in the appdata folder along with the malicious script. When a person logs on to an infected machine it would generate a WLS event like this (snippet):

BaseFileName="udpbat.exe" InternalName="wscript.exe" CommandLine="C:\Users\tupac\AppData\Roaming\avseda\udpbat.exe  C:\Users\tupac\AppData\Roaming\avseda\vnyqxluw.js"

Process Overview 
1. Splunk alert finds execution of 'wscript.exe' when BaseFileName is not 'wscript.exe'.
2. Splunk alert launches 'wrapper.py' which then launches 'grrRemoteGetFile.py'. 
3. 'grrRemoteGetFile.py' sends GRR an API request to acquire files in question.
4. Profit.


Splunk uses its own python version which doesn't have modules like 'requests'. Rather than installing modules into Splunk's python, we can just use a wrapper which will use the system default version of python. 

Creating the Splunk Alert
Run the search and when you're satisfied your search has minimal false positives, save it as an Alert.

When going through the alert wizard check the 'enable' box under Run a script and enter wrapper.py

wrapper.py (Mashed together from a few examples on the Splunk forums):

 #!/usr/bin/python   
 import gzip, os, sys, csv   
 from subprocess import call   
   
 python_executable = "/usr/bin/python"   
 real_script = "/opt/splunk/bin/scripts/grrRemoteGetFile.py"   
   
 for envvar in ("PYTHONPATH", "LD_LIBRARY_PATH"):   
  if envvar in os.environ:   
   del os.environ[envvar]   
   
 def openany(p):   
  if p.endswith(".gz"):   
   return gzip.open(p)   
  else:   
   return open(p)   
   
 results_file = sys.argv[8]  
   
 for row in csv.DictReader(openany(results_file)):   
  my_command = [ python_executable, real_script, row["host"], ]   
  call(my_command)   

The wrapper script does the following: 

• Remove environment path and LD_LIBRARY_PATH
• Opens the splunk search results (unzip's and reads csv for host value) .
• Execute 'grrRemoteGetFile.py' with the systems default python along with the hostname that triggered the alert.

Splunk will pass 9 variables to the script when it executes. Variable 8 contains the path to the gzip'd search results in csv format. The other variables are documented here.


grrRemoteGetFile.py 

 #!/usr/bin/python    
 import sys, json, urllib2, base64, requests   
 from requests.auth import HTTPBasicAuth   
   
 hostname = sys.argv[2]  
   
 grrserver = 'https://grrserver:8000'   
 username = 'Tupac'   
 password = 'isAlive'   
   
 base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')   
 authheader = "Basic %s" % base64string   
   
 index_response = requests.get(grrserver, auth=HTTPBasicAuth(username, password))   
 csrf_token = index_response.cookies.get("csrftoken")   
   
 headers = {   
  "Authorization": authheader,   
  "x-csrftoken": csrf_token,   
  "x-requested-with": "XMLHttpRequest"   
  }   
   
 data = {   
  "hostname": hostname,   
  "paths": ["%%users.appdata%%\Roaming\*\*.{js,exe}",   
            "%%users.appdata%%\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\*.lnk"],    
  "pathtype": "OS"    
  }   
   
 response = requests.post(grrserver + "/api/clients/" + hostname + "/flows/remotegetfile",   
        headers=headers, data=json.dumps(data),   
        cookies=cookies, auth=HTTPBasicAuth(username, password))   

'grrRemoteGetFile.py' will start a FileFinder flow on the 'hostname' variable passed to it by the 'wrapper.py' script. When IR staff is able to review the ticket, the files will be available in GRR to download and review.

In Summary...
This is just a basic example to demonstrate how all the pieces fit together. There are some really cool things you can do with these tools to automate stuff. Some things I've been playing with:

Automatically launch Incident Response Collector ($MFT, Registry, Browser History, etc.) and full memory image when a known bad MD5 or static indicator is seen in Splunk.

Utilize WLS's hash tracking to automatically submit new binaries to internal malware analysis tools. Splunk alert would be:

`wlslogs` (EventID=4688 OR EventID=592) NewHash=True

In the wrapper.py, you would add row["NewProcessName"] and pass it to grrRemoteGetFile.py to download (instead of the static path in the example above).

Get any executable downloaded from the internet and send it to internal malware analysis tools.

`wlslogs` (EventID=4688 OR EventID=592) Zone=3

Get any compressed file attachment opened from Outlook email and send to internal malware analysis tools.

`wlslogs` (EventID=4688 OR EventID=592) CreatorProcessName=OUTLOOK BaseFileName=winzip*

If you have any examples/suggestions on automation with GRR and WLS/Splunk, share them on the GRR user group, I'm really interested to hear what other folks have done.

DFIR with Windows Logging Service (WLS)

WLS is logging service built with forensics and incident response in mind. The best way to explain what WLS is to show an example:

Here is what you get from a process creation event from Windows:

2014 Nov 21 21:39:28 10.10.10.10 WINTETST.domain.com MSWinEventLog 0 Security 2099 nov 21 16:39:28 2014 4688 Microsoft-Windows-Security-Auditing domain\WINTETST$ N/A Success Audit WINTETST.domain.com Process Creation A new process has been created.    Subject:   Security ID:  S-1-5-18   Account Name:  WINTETST$   Account Domain:  DOMAIN   Logon ID:  0x3e7    Process Information:   New Process ID:  0xf0c   New Process Name: C:\Windows\System32\dllhost.exe   Token Elevation Type: TokenElevationTypeLimited (3)   Creator Process ID: 0x2d4    Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.    Type 1 is a full token with no privileges removed or groups disabled.  A full token is only used if User Account Control is disabled or if the user is the built-in Administrator account or a service account.    Type 2 is an elevated token with no privileges removed or groups disabled.  An elevated token is used when User Account Control is enabled and the user chooses to start the program using Run as administrator.  An elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the Administrators group.    Type 3 is a limited token with administrative privileges removed and administrative groups disabled.  The limited token is used when User Account Control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using Run as administrator.

Here is what WLS logs:

Aug 23 12:57:57 win-1ujak6s06vk. Security: LogType="WLS", BaseFileName="notepad.exe", Channel="Security", CommandLine="'C:\Windows\system32\notepad.exe'", CompanyName="Microsoft Corporation", Computer="WIN-1UJAK6S06VK", CreatorProcessName="explorer", Entropy="6.95893575618574", EventID="4688", EventRecordID="72208", ExecutionProcessID="4", ExecutionThreadID="56", FileDescription="Notepad", FileVersion="6.1.7600.16385 (win7_rtm.090713-1255)", InternalName="Notepad", Keywords="0x8020000000000000", Language="English (United States)", Length="193536", Level="0", MD5="F2C7BB8ACC97F92E987A2D4087D021B1", NewProcessId="0x8a8", NewProcessName="C:\Windows\System32\notepad.exe", Opcode="0",ProcessId="0x8ec",ProductVersion="6.1.7600.16385",ProviderGuid="{54849625-5478-4994-A5BA-3E3B0328C30D}", ProviderName="Microsoft-Windows-Security-Auditing", SessionId="2",SESSIONNAME="Console",SHA1="7EB0139D2175739B3CCB0D1110067820BE6ABD29",Signed="Catalog",SSDeep="3072:QOrerAgXWMI6vKoTN6p0frxJLgf7nDVF6PUp1Yo3ICgx:QWDcRgNpex5gfzDVlVXg",SubjectDomainName="WIN-1UJAK6S06VK",SubjectLogonId="0x1fc0a2",SubjectUserName="testuser",SubjectUserSid="S-1-5-21-874994001-2474262622-3605836291-1008", Task="13312", TokenElevationType="TokenElevationTypeDefault (1)",ValidSignatureDate="False", Version="1", WindowStation="Winsta0\Default", Zone="0"

All the useless information is replaced with useful information. More details on WLS can be found here http://digirati82.com/. I’m writing this blogpost to share some methods for detecting malicious behavior and malware using WLS and Splunk.  

Sticky Keys Authentication Bypass

Using sticky keys to bypass authentication is a favorite since it doesn’t involve malicious binaries. This method relies on replacing sethc.exe, utilman.exe, osk.exe, magnify.exe or narrator.exe with cmd.exe in the windows\system32\ directory. Since we are capturing the file metadata on execution, detecting this malicious behavior is simple. Create a schedule Splunk job to email you when event count is >0 for this search:

index=main (EventID=4688 OR EventID=592) FileDescription="Windows Command Processor" BaseFileName!=cmd.exe

A legitimate execution of the command prompt will contain the file description of “Windows Command Processor" with the BaseFileName of “cmd.exe”

Webshell / Compromised Web Server Detection

In some web server compromises attackers will add a webshell allowing them command line access for lateral movement among other things. Since we know that the IIS service account shouldn’t normally be running ‘cmd.exe’ or commands like whoami, netstat, “net localgroup administrators pwnu /add” or even “net user guest /active:yes” we can create a splunk alert to notify us when this behavior does occur.

Here is an example WLS event snippet of a compromised IIS webserver executing the command prompt through a webshell:

BaseFileName="cmd.exe", Channel="Security", CompanyName="Microsoft Corporation", Computer="WIN-1UJAK6S06VK", CreatorProcessName="w3wp", EventID="4688", SubjectUserSid="S-1-5-82-3006700770-424185619-1745488364-794895919-40

04696415", NewProcessName="C:\Windows\System32\cmd.exe", SubjectUserName="Defaul

tAppPool"

An example Splunk search to alert on this activity:

index=main (EventID=4688 OR EventID=592) CreatorProcessName=”w3wp”

NewProcessName="C:\Windows\System32\cmd.exe"

Maybe it’s a good idea to see ALL command line activity by the IIS user:

index=main WLS_CommandMonitor User=”IIS APPPOOL\\DefaultAppPool"

In any case we know what an IIS server shouldn’t do, creating alerts on known bad behavior is a simple way to increase your odds of finding a compromised server.

Svchost.exe Injection

Some malware will start a legitimate copy of svchost.exe and inject code into it. This method used to hide on a system is easy to alert on. Since services.exe should execute svchost we can create a Splunk alert anytime svchost.exe is executed when the Creator Process Name is not services.exe. Here is a snippet of what a malicious execution would look like:

BaseFileName="svchost.exe", Channel="Security", CommandLine="'c:\Windows\System32\svchost.exe", CompanyName="Microsoft Corporation", Computer="WIN-1UJAK6S06VK", CreatorProcessName="mspswls.tmp",  EventID="4688",

Our Splunk search to alert on this behavior would look like this:

index=main (EventID=4688 OR EventID=592) BaseFileName=svchost.exe CreatorProcessName!=”services.exe”

Alerting on Attacker Behavior

Some attackers follow a pattern after gaining access; gather system information, escalate privileges, lateral movement, persistence, collect & exfil data.  Their tools are constantly changing to evade detection but their behaviors and the tools behaviors don’t evolve so quickly. A great example is the staging of data for exfil. Attackers will sometime use a custom version of rar to password protect a file, in several version found on Virustotal they all use the standard command flags of ‘a -ph’ to create an archive and to password protect the data.

SomeExecutable.exe a -hpSomePassword RandomFlename RandomFilePath

Based on this we can add a scheduled search in Splunk for the following:

index=main WLS_CommandMonitor Command=”*a -hp*”

Example WLS CommandMonitor Log

Aug 23 13:39:09 win-1ujak6s06vk. WLS_CommandMonitor: LogType="WLS",

Command="a.exe a -hpqwerty123 data1.rar supersecret.doc", ProcessId="0x11c", ProcessName="cmd", Type="Added", User="WIN-1UJAK6S06VK\testuser", WLSKey="122"

Some attackers use the “makecab.exe” for compressing data before exfil. It’s on all windows systems and not many people use it so alerting on its execution might be a good idea.

Other APT groups are also using powershell as a method for persistence as indicated by this FireEye article. Using the same method above we create a scheduled Splunk search on the “-enc” parameter passed to powershell.

Example attacker activity from https://www2.fireeye.com/rs/fireye/images/rpt-m-trends-2015.pdf  

"C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe –NonInteractive –enc SQBuAHYAbwBrAGUALQBDAG8AbQBtAGEAbgBkACAALQBDAG8AbQBwAHUAdABl..."

Example Splunk alert

index=main WLS_CommandMonitor Command=”powershell –NonInteractive –enc *”

The other popular method of persistence is by scheduling “at” jobs. Since this data is logged and not too many people use this, results should be low:

index=main WLS_CommandMonitor: LogType="WLS", Command="at *”

Or you could alert on the execution of “at.exe”:

index=main (EventID=4688 OR EventID=592) BaseFileName=”at.exe”

Malicious Microsoft Office Documents

Word documents embedded with malicious macros to download & execute malware is still common. WLS logs the creator process name so it’s pretty simple to search for any event where Microsoft Word creates a new process:

index=main (EventID=4688 OR EventID=592) CreatorProcessName=WINWORD

For Excel, Powerpoint or Access you would just substitute the appropriate “CreatorProcessName”.

Entropy

In many cases attackers use packed binaries which result in high entropy. While this is not a candidate for an alert it can certainly be used for malware hunting. An example from WLS event would report it as Entropy="4.61108706992542". In splunk you could search for unsigned binaries executed with entropy greater than 7:

index=main (EventID=4688 OR EventID=592) Signed=False Entropy>=7

I will be updating this post with additional searches when time allows. Hopefully this information is helpful, if you have any questions or comments hit me up on twitter.