Intro to Threat Hunting - Part 1

If you're wanting some amazing introductory threat hunting training for free, look no further than Chris Brenton's Cyber Threat Hunting Level 1 course. The training is available here on YouTube along with the accompanying VM listed in the description to complete the labs. I use VMware Workstation 17 Pro, so I went with the VMware option, but if you use VirtualBox you'll find a VM option for you, as well. What follows is my own understanding of the material, and my attempt to explain it well enough to myself and the reader so that we can all increase our knowledge and understanding of cybersecurity topics; and, maybe, even get inspired to dig deeper into threat hunting! Enjoy the hunt ;)

Introduction

The goals for this class include:

• Define “cyber threat hunting”

• Identify how to perform a threat hunt

• Define and identify connection persistency

• Learn how to investigate endpoints

• Hands on lab time running down real C2 channels used in the wild

What is threat hunting? According to Brenton, it is “actively searching your environment for compromised systems.” Threat hunting is something new, and anyone who says they have been doing threat hunting for decades is not understanding the concept of threat hunting. Advances in storage and processing technology are what have allowed for threat hunting to be a reality today, and to be able to analyze and scrutinize all of the connections that leave our environment. Threat hunting is an active process, meaning that if you are reacting to alerts in your environment, or to an event that has already occurred, you are not threat hunting. Actively going through a number of different processes and validating the integrity state of all systems in your environment is threat hunting. Timing is usually what decides when threat hunting will take place, such as performing a threat hunt once per week/month/quarter, and is not decided based on getting an alert on your SIEM. Again, we do not know if we have been compromised before starting a threat hunt. We are not reacting to alerts and calling that threat hunting.

A proper threat hunt includes everything and not just the systems that are submitting logs to your SIEM, so if you are confining your threat hunting processes only to using SIEM data then you are missing the systems on your network that are not feeding data to the SIEM (e.g. IP cameras, network hardware other than desktops and servers, printers, etc.). Threat hunting has to start at the network, since the network is what sees everything. Relying on data from endpoint agents set up to alert on malicious activity means, by definition, that you are going to miss things in your environment since endpoint agents can’t be installed on every system in your environment (e.g. temperature sensors, IP cameras, a Cisco router, etc.).

There is some debate about whether we as practitioners are getting better at early threat detection. The cost to recover from an incident and the damage done to the company’s environment has not changed over time, leading some to believe that we really are not getting better at early threat detection. Although dwell time for threats in an environment has dropped considerably it is not correlated with reduced impact of breaches on compromised environments. We need a new approach, and not one that has been in place for years and is considered the bare minimum for complying with PCI-DSS, ISO 27001, SOC 2, FedRAMP and the like. We need to make sure we have a centralized way to collect information in our environment and go through it regularly (e.g. checking our SIEMS, reviewing host logs, and searching for attack patterns we recognize). It’s time… for threat hunting.

The purpose of threat hunting is to decrease the time between infiltration and detection. That’s it. Plain and simple. Since threat hunting is a relatively new discipline, there are no industry standards or best practices established yet. If you watch 4 videos on threat hunting you’ll get 4 different ideas on how to approach it. The creators of this course have their way of conducting threat hunting, and it is based on thousands of threat hunting engagements, and they adapt their processes in accordance with new evidence and what is getting the best results. Their approach is not that of a vendor attempting to capitalize on the sexiness of threat hunting by developing a product and claiming that it is the final grandmaster solution that you’ll ever need to do effective threat hunting.

So, why start with the network?

Well, because you see everything. And, you can’t hide packets. Savvy attackers can hide their processes and malicious code on endpoints very successfully against forensic tools, but the packets are always going to be seen going by in a network.

How does the threat hunting process work?

The first thing to do is identify connection persistency. If you identify an endpoint that is reaching out to an IP address in China once per minute and it’s been going on constantly for several days then that is interesting. You then want to ask yourself if there is a business need for that connection, such as patching or checking time. If you can’t identify a business need for that connection then we have a problem. From here, start a deeper analysis on the protocol. Is there anything odd about it? [The following situation was provided by Chris as an example of odd behavior] – “[the internal endpoint] is talking to a web server and it’s accessing a JPEG file but the server is identifying that JPEG file as actually being a text file, and when I look at the contents it looks like it’s actually taking a zipped file that’s been converted to HEX and it’s directly loading it into RAM instead of loading it to disk.”

You may then decide to do some investigation on the internal system. If I see that my system is talking to China once per minute every day then check to see which process is doing that. If the system is communicating with a Slack server, and we use Slack internally in our environment, then the likelihood of that traffic being malicious has just decreased. Now, if there is a process we don’t expect to see, or a process that doesn’t usually behave the way it’s behaving, then that could be a problem. Chris’s example is to consider notepad reaching out to the internet. That is not a process we would normally see, and if we did it should be reaching out to Microsoft and not some random place on the internet (in which case, this might not be the notepad we think it is!) Attackers can disguise their work to try to make it appear that a benign process is running on your machine, so being familiar with how processes normally behave is important for being effective at threat hunting.

Moving along in our process, we want to create a disposition for every internal IP address on our network. If we identify traffic patterns on a system that are potentially malicious and then conclude they aren't, we’ll want to move that traffic pattern to a safe list. For example, if we see traffic going out to an Amazon EC2 server to some marketing website and we have no idea why, and we go to our purchasing department and they confirm that they are paying that company for a service then that tells us this is a legitimate business need, and we can add this traffic to a safe list. If there is evidence that a system is compromised then we trigger our incident response plan, which is beyond the scope of this threat hunting introduction.

Why do we not want to isolate a system immediately after we find out it’s been compromised?

Intuitively, we may think this is exactly what we should do. “If we isolate the system then the attacker can’t do anymore damage!” You may say. This is when we need to gather some more information. Check to see if there are any other systems in your environment that are reaching out to the malicious IP address. If there are, then that means that the attacker has taken over multiple machines in your environment, and if you jump the gun and isolate a system that can tip off the attacker that you’re on to them, and they may start going “scorched earth” on your servers and data, deleting information and all sorts of bad things, because they realize they only have a limited amount of time now to accomplish whatever goal they have in mind. You may spare your environment massive damage by not immediately taking action to isolate systems as soon as you discover they’ve been compromised. Said another way, you do not want to cross the passive/active line when it comes to threat hunting. The attacker should not be able to detect your data gathering efforts as a threat hunter. We’re going to be pulling data from our SIEM and from PCAP files and such. We are not going to the terminal and typing any commands into these compromised systems!

Once we have gathered the data we need we can begin incident response. This should involve individuals across multiple disciplines, including penetration testers. Having a room of people with a diverse range of skills makes it far more likely that you won’t make a bad incident response decision (like immediately isolating a system that was just found to be compromised!)

C2 Detection Techniques

Where do we start with C2 detection? You guessed it. We start on the network because you see everything, and you can't hide packets. What tools should we use to detect data on the network? Historically, we as security professionals would use packet captures. The great thing about packet captures is you have full fidelity – you see everything that comes across the network. Its strength is also its weakness, in that packet captures collect everything, so you may only have 5 days worth of pcap files stored because of the sheer volume of information being collected and saved. An alternative option is to use a tool like Zeek. Zeek still functions as a packet capturing tool but only keeps the parts of the packet that we as security practitioners care about, dramatically increasing the amount of data we can save. If we are trying to get information about when an attacker first infiltrated our network, and we only have 5 days worth of data saved then it’s likely we won’t have visibility into when the attacker first started popping machines. With a tool like Zeek, that same amount of storage could give us about 100 days of Zeek logs, making it more likely that we’ll be able to find out when an attacker first got into our network.

Typical Packet Capture Deployment

AWS deployments will be different but on your internal physical networks you’ll want to set up a span port connection for packet sniffing. Basically, you’re going to configure it so that traffic coming into the network on a certain port will be copied to the packet capture that’s running on another port. Packets sent from the compromised system will also be copied to the packet capture as they’re sent out of the network over the C2 channel. From here, we can use a tool like AC-Hunter to analyze the packets we are getting from the packet capture. Head on over to part 2 where we'll spin up AC Hunter and look at some of its core features and functionality.