A Practical Example of Non-technical Indicators and Incident Response

Once upon a time there was a network security analyst slash NSM engineer who, like any sane person, ran full packet capture, IDS/IPS, session capture, and passive fingerprinting inline at the ingress/egress of his home network. His setup was most similar to diagram two in IDS/IPS Placement on Home Network.

This security analyst was casually going about his business one day when he opened the basement door of his house and found a tennis ball wedged between the door frame and the storm door. “That’s odd!” he thought. “Who would do that?”

After removing the tennis ball, he thought, “Well, this storm door is really loud when it closes those last few inches. Maybe someone did it to quietly enter or exit the house.” It just so happens that the daughter of said analyst was in high school and her bedroom was down the hall from the basement door. He promptly entered her room and took a quick look around. Lo and behold, the screen from her window was under her bed and the window itself was unlocked. Since this room was on the ground floor, the analyst immediately had some good ideas about what was happening with the window and the basement door. Someone was sneaking in or out of the house!

The analyst confronted his teenage daughter when she got home from school and received denial after denial about any possible wrongdoing. The denials did not sound sincere.

Enter the network security monitoring. He stated, “I told you I would respect your privacy with your email and other electronic communications unless you gave me a reason not to. I consider you in violation of these Terms of Service and I’m going to see what you’ve been up to lately.”

At this point it was late in the evening and the analyst had to get up early for work. This was some years back when AIM was quite common, so he briefly used Sguil to look at recent sessions of AOL Instant Messenger traffic. He decided to get some sleep for work the next day and put off additional investigation. In the meantime, his daughter's privileges were highly restricted.

A day or two later, after trying to manually sift through some of the ASCII transcripts of the packet captures the analyst quickly decided there was a better way. He whipped up a short shell script to loop through all the packet captures, run Dug Song’s msgsnarf, and pipe the output into an HTML file for later examination. This required a little tweaking to make the HTML easily readable, but it was fairly quick to write and test the script.
The next morning there were many hundreds of lines of AIM conversations to examine. He started working from the most recent and reading backwards. After a few minutes he quickly confirmed that his daughter had been sneaking out of the house to go to parties and get into other mischief.

Another conversation with his daughter finally led to her confession, a long discussion, and suitable punishment. Despite the severity of her actions, the HTML file containing the chat transcripts also contained a few endearing nuggets.

Daughter: OMG they know everything!
Accomplice: what do u mean everything
Daughter: my dad can read all my chats
Daughter: he does computer security for [company redacted]
Daughter: he’s a computer genius
Daughter: DAD I’M INNOCENT!

Upon telling this story to a current colleague, he mentioned that the last few lines are the best father’s day gift the analyst would ever receive.

I think there are a few obvious lessons here that can translate to network monitoring.

First, the initial indicator of the problem was in the physical world. Network security monitoring or any other type of technical monitoring and prevention will fail. I have experienced many times when phone calls from users have been one of the earliest indicators of malicious activity. Particularly in the case of insider threats, it's important to note that many initial indicators of malicious activity are non-technical, like a person's behavior, personnel action, or in this case a physical indicator of a security problem.

Second, sometimes you need to be flexible to solve a problem quickly and with minimal effort. The analyst could have manually looked at the AIM traffic, but because he judged that the threat of another incident was already mitigated by talking to the daughter, digging up the traffic wasn’t urgent. Instead, The analyst decided to write the script that would pull all the traffic and convert it to a readable format. The analyst also had the luxury of knowing that all the packet captures would still be there since his home bandwidth at the time meant well more than 30 days of pcap storage.

Third, network monitoring is a means to an end. In this case, there was a security problem that could be addressed with the help of technical means. In many obvious cases you are trying to protect data. In other cases, you can be trying to protect people or things in the physical world that could be harmed if the wrong information is revealed. It is important to stay focused on what really matters and not get caught worrying about the wrong things because your instrumentation or technologies push you towards priorities that don’t make sense.

Last, attackers are not static. The daughter definitely learned the value of encryption and even using out-of-band communication in the form of SMS over the phone network if she did not want the network sensor recording her conversations in plain text. Technology advancement also makes attackers evolve, for instance with the move to Facebook chat or SMS from older forms of IM.

Tweet