Snort 2.8.1 changes and upgrading

Snort 2.8.1 has been out since April, so this post is a little late. I wanted to upgrade some Red Hat/CentOS systems from Snort 2.8.0.2 to 2.8.1. When I write RHEL or Red Hat, it will include CentOS since what is applies to one should apply to the other.

I quickly found that the upgrade on RHEL4 was not exactly straightforward because the version of pcre used on RHEL4 is pcre 4.5, which is years old. Snort 2.8.1 requires at least pcre 6.0. For reference, RHEL5 is using pcre 6.6.

PCRE Changes

I had assumed an upgrade from 2.8.0.2 to 2.8.1 was minor, but the pcre change indicated there were definitely significant changes between versions. A few of the pcre changes made it into the Snort ChangeLog, and there is more in the Snort manual. I also had a brief discussion about the pcre changes with a few SourceFire developers that work on Snort and some other highly technical Snort users.

The most significant changes seems to be adding limits to pcre matching to help prevent performance problems and denial of service through pcre overload. At regular expression compile time, there will be a maximum number of state changes that can be passed and on evaluation libpcre will only change states that many times. There is a global config option that sets the maximum number of state changes and the limit can also be disabled per regular expression if needed.

The related configuration options for snort.conf are pcre_match_limit:

Restricts the amount of backtracking a given PCRE option. For example, it will limit the number of nested repeats within a pattern. A value of -1 allows for unlimited PCRE, up to the PCRE library compiled limit (around 10 million). A value of 0 results in no PCRE evaluation. The snort default value is 1500.

and pcre_match_limit_recursion:

Restricts the amount of stack used by a given PCRE option. A value of -1 allows for unlimited PCRE, up to the PCRE library compiled limit (around 10 million). A value of 0 results in no PCRE evaluation. The snort default value is 1500. This option is only useful if the value is less than the pcre_match_limit

The main discussion between the SourceFire folks and everyone else was whether it was wise to have the limits turned on by default and where the default of 1500 came from. I think leaving the pcre limits on by default makes sense because those that don't fiddle much with the Snort configuration probably need the protection of the limits more than someone that would notice when Snort performance was suffering.

The argument against having the limits on by default is that it could make certain rules ineffective. By cutting short the pcre, the rule may not trigger on traffic that should cause a Snort alert. I suspect that not many rules will hit the default pcre limit.

Other Changes Included in Snort 2.8.1

From the release notes:

[*] New Additions
* Target-Based support to allow rules to use an attribute table
  describing services running on various hosts on the network.
  Eliminates reliance on port-based rules.

* Support for GRE encapsulation for both IPv4 & IPv6.

* Support for IP over IP tunneling for both IPv4 & IPv6.

* SSL preprocessor to allow ability to not inspect encrypted traffic.

* Ability to read mulitple PCAPs from the command line.

The SSL/TLS preprocessor helps performance by allowing Snort to ignore encrypted traffic rather than inspecting it. I haven't looked at the target-based support yet, but it definitely sounds interesting.

I also noticed something from 2007-12-10 in the ChangeLog:

 * configure.in:
       Add check for Phil Woods pcap so that pcap stats are computed
       correctly.  Thanks to John Hally for bringing this to our
       attention.

That should be good for those running Phil Wood's pcap who may not have been seeing accurate statistics about packet loss.

A last note of interest about Snort 2.8.1 is that it fixes a vulnerability related to improper reassembly of fragmented IP packets.

Upgrading

Upgrading on RHEL5 was pretty simple, but upgrading on RHEL4 required downloading the source RPM for the pcre included with RHEL5 and building a RPM for RHEL4.

First, I installed compilers and the rpm-build package and the RPM source package for pcre-6.6. (Note that I'm using CentOS4 in the example, but the procedures for Red Hat should be nearly identical except for URLs). Next I built the pcre-6.6 RPMs and installed both pcre and pcre-devel.

# up2date -i cpp gcc gcc-c++ rpm-build
# rpm -Uvh http://isoredirect.centos.org/centos/5/updates/SRPMS/pcre-6.6-2.el5_1.7.src.rpm
$ rpmbuild -bb /usr/src/redhat/SPECS/pcre.spec
# rpm -U /usr/src/redhat/RPMS/i386/pcre-*rpm
# rpm -q pcre pcre-devel
pcre-6.6-2.7
pcre-devel-6.6-2.7

Now I can configure, make and make install Snort as usual. The snort.conf file will also need to be updated to reflect new options like the SSL/TLS preprocessor and non-default pcre checks. Note that the README.ssl also recommends changes to the stream5 preprocessor based on the SSL configuration.

$ ./configure --enable-dynamicplugin --enable-stream4udp --enable-perfprofiling
$ make
# make install

If you are building on a production box instead of a development box, some would recommend removing the compilers afterwards.

Here is an example of a SSL/TLS and stream5 configuration using default ports. By adding the default SSL/TLS ports to stream5_tcp settings, you also have to enumerate the default stream5_tcp ports or they will not be included. I bolded the default SSL/TLS ports, and the rest are default stream5_tcp ports:

preprocessor stream5_global: track_tcp yes, track_udp yes, track_icmp yes
preprocessor stream5_tcp: policy first, ports both 21 23 25 42 53 80 \
     110 111 135 136 137 139 143 443 445 465 513 563 636 \
     989 992 993 994 995 1433 1521 3306
preprocessor stream5_udp
preprocessor stream5_icmp
preprocessor ssl: \
     noinspect_encrypted

I have tested to make sure Snort runs with this configuration for these preprocessors, but don't just copy it without checking my work. It is simply an example after reading the related README documentation. Note that the Stream 5 TCP Policy Reassembly Ports output to your message log when starting Snort will be truncated after the first 20 ports but all the ports listed will be included by Stream 5.

Tweet

Query sguildb for alert name then do name lookup

I wrote a Perl script to query for a specific alert name and then get the NetBIOS or DNS name of the systems that triggered the alert. This script is useful to me mainly as an automated reporting tool. An example would be finding a list of systems associated with a specific spyware alert that is auto-categorized within Sguil. The alert will never show up in the RealTime alerts but I use the script to get a daily email on the systems triggering the alert.

Some alerts aren't important enough to require a real-time response but may need remediation or at least to be included in statistics. I don't know if this would be useful for others as it is written, but parts of it might be.

As always with Perl, I welcome suggestions for improvement since I'm by no means an expert. (By the way, does anyone else have problems with Blogger formatting when using <pre> tags?)

#!/usr/bin/perl
#
# by nr
# 2008-05-11
#   Script to query db for a specific alert
#   and do name lookup based on source IP address in results

use strict;

# Requires MySQL, netbios and DNS modules
use Mysql;
use Net::NBName;
use Net::DNS;

my $host = "localhost";
my $database = "sguildb";
my $tablename = "event";
my $user = "sguil";
my $pw = "PASSWORD";
my $alert = 'ALERT NAME Here';

# Set the query
my $sql_query = "SELECT INET_NTOA(src_ip) as src_ip,count(signature) as count FROM $tablename \
    WHERE $tablename.timestamp > DATE_SUB(UTC_DATE(),INTERVAL 24 HOUR) AND $tablename.signature \
    = '$alert' GROUP BY src_ip";

# perl mysql connect()
my $sql_connect = Mysql->connect($host, $database, $user, $pw);

print "\"$alert\" alerts in the past 24 hours: \n\n";
print "Count    IP Address  Hostname\n\n";

my $execute = $sql_connect->query($sql_query); # Run query

# Fetch query results and loop
while (my @result = $execute->fetchrow) {
    my @hostname; # 1st element of this array is used later for name queries
    my $ipresult = $result[0]; # Set IP using query result
    my $count = $result[1]; # Set alert count using query result
    my $nb_query = Net::NBName->new; # Setup new netbios query
    my $nb = $nb_query->node_status($result[0]);
    # If there is an answer to netbios query
    if ($nb) {
        my $nbresults = $nb->as_string; # Get query result
        # Split at < will make $hostname[0] the netbios name
        # Is there a better way to do this using a substitution?
        @hostname = split /</, $nbresults;
    } else {
        # Do a reverse DNS lookup if no netbios response
        # May want to add checks to make sure external IPs are ignored
        my $res = Net::DNS::Resolver->new;
        my $namequery = $res->query("$result[0]","PTR");
        if ($namequery) {
            my $dnsname = ($namequery->answer)[0];
            $hostname[0] = $dnsname->rdatastr;
        } else {
            $hostname[0] = "UNKNOWN"; # If no reverse DNS result
        }
    }
format STDOUT =
@>>>    @<<<<<<<<<<<<<< @*
$count, $ipresult,      $hostname[0]
.
    write;
}

Tweet

NoVASec: Memory Forensics

Richard Bejtlich arranged a NoVASec meeting on memory forensics for Thursday, April 24. Aaron Walters of Volatile Systems was the scheduled speaker. George Garner of GMG Systems, Inc., also showed up, so we were lucky enough to get two speakers for the price of one. (If you aren't aware, NoVASec is actually free). Aaron primarily talked about performing forensics and analysis on memory dumps, and afterwards Richard asked George to come up from the audience and talk about the challenges of actually acquiring the memory dumps.

Both Aaron and George were very knowledgeable and had a lot of interesting things to discuss. In fact, most of us didn't leave until after 22:00 so there was a good two and a half hours of technical discussion. It wouldn't do them justice for me to try and recap their talks, but I will mention a couple brief thoughts I jotted down while listening. If I'm getting anything wrong here, someone please pipe up and let me know.

First is that I saw some parallels between points mentioned by Aaron and Network Security Monitoring. Aaron stated that a live response on a system requires some trust of the system's operating system, is obtrusive, and is unverifiable. Dumping the RAM and performing an analysis using a trusted system helps mitigate these problems though I don't think he meant it solves them completely. Similarly, in NSM we use information that is gathered by our most trustworthy systems, network sensors that allow limited access, rather than trusting what we find on the host. In forensics and NSM, steps are taken to increase the trustworthiness and verifiability of information that is gathered.

Second, Aaron and George both seemed to agree that acquiring memory contents is not easy. Not only can it be difficult, but even a successful acquisition has issues. George pointed out that if you don't isolate the system, an attacker could be altering the system or memory as you acquire it. He also pointed out that dumping the memory is actually sampling, not an image, because the RAM contents are always changing even on a system that has been isolated from the network. One memory dump is just one sample of what resided in memory at a given time. More evidence and more sampling will increase the reliability of the evidence attained. Also, gathering evidence from multiple sources, for instance hard drive forensics, memory forensics and NSM, increases the probability evidence will be accurate and verifiable.

There was also some discussion of PCI and video devices as they relate to both exploiting systems and memory forensics. Acquiring memory can be an issue on systems using PAE since reading from the space used by PCI devices can crash the system. On the exploit side, the GPU and RAM on video cards can be used to help facilitate attacks, as can certain PCI devices. There is a lot of interesting work going on in this field, and George even mentioned that he has been working on tools for acquiring the contents of memory from video cards.

It was an excellent meeting.

Tweet

Defcon 16 Race to Zero

There have been articles about Defcon's Race to Zero since it was announced. I first read about it on the Daily Dave mailing list when the announcement was posted a couple days ago on 27 April. Apparently, some vendors and media are unhappy and criticizing the competition. While this is not surprising, it strikes me as pointless to complain about a competition that is just demonstrating what can be and already is done in the wild.

From the Race to Zero site:

The event involves contestants being given a sample set of viruses and malcode to modify and upload through the contest portal. The portal passes the modified samples through a number of antivirus engines and determines if the sample is a known threat. The first team or individual to pass their sample past all antivirus engines undetected wins that round. Each round increases in complexity as the contest progresses.

Anyone that has submitted real malware samples to a service like VirusTotal already knows how pitiful and inconsistent anti-virus software is at detecting malware, particularly if it is new or newly modified. There is a reason we see so many variants of the same malware, and it's not because anti-virus is so effective that malware authors have to completely rewrite their code.

1. Reverse engineering and code analysis is fun.
2. Not all antivirus is equal, some products are far easier to circumvent than others. Poorly performing antivirus vendors should be called out.
3. The majority of the signature-based antivirus products can be easily circumvented with a minimal amount of effort.
4. The time taken to modify a piece of known malware to circumvent a good proportion of scanners is disproportionate to the costs of antivirus protection and the losses resulting from the trust placed in it.
5. Signature-based antivirus is dead, people need to look to heuristic, statistical and behaviour based techniques to identify emerging threats
6. Antivirus is just part of the larger picture, you need to look at controlling your endpoint devcies [sic] with patching, firewalling and sound security policies to remain virus free.

Although I have very limited and basic experience reverse engineering malware, it does seem fun and interesting. I also totally agree that vendors need to be called out.

Heuristic, statistical and behavior-based techniques may indeed help, but point number six seems equally important. I don't really know what the best solution is, but hopefully some vendors will eventually realize that their methods and models need to change to become more proactive instead of reactive.

Tweet

PADS signatures, NSMWiki, OpenPacket

I added a few PADS signatures to the NSMWiki. Anyone else that has some should definitely contribute since the standard signature set is fairly small and has a huge potential for improvement. I'm sure that any other useful contributions to NSMWiki are also appreciated.

Richard Bejtlich posted about OpenPacket being online. I think the idea is great and there is a strong community of people that have signed on to help him with various aspects of the site.

OpenPacket.org is a Web site whose mission is to provide a centralized repository of network traffic traces for researchers, analysts, and other members of the digital security community.

For anyone just starting out in digital security or looking to get into the field, I strongly encourage you to participate in the security community as a whole. The number of ways to participate are too numerous for me to list, but there is definitely a lot to be learned from others who are more experienced, less experienced, or just have different types of experience. Just reading blogs, news, mailing lists and other sites can be enlightening, and once you get your feet wet you may find yourself contributing in short order.

Tweet

Upgrading from Sguil 0.7.0 CVS to RELEASE

Sguil 0.7.0 was released this week. The upgrade from 0.6.1 takes a little more effort because of the change to multiple agents, but since I was already running 0.7.0 from CVS, upgrading was fairly easy.

The Sguil overview page at NSMWiki notes some of the differences between 0.6.1 and 0.7.0. It also has some nifty diagrams someone (***cough***me***cough***) contributed that may help people visualize the data flow in Sguil 0.7.0.

Here is how I upgraded from CVS to the release version.

First, I pre-staged the systems by copying the appropriate Sguil components to each system. Then I shut down the agents on all sensors and stop sguild on the server.

Looking in my sguild directory, there is not that much that will actually need to be replaced.

$ ls /etc/sguild/
CVS/                  certs/    sguild*        sguild.email    sguild.users
archive_sguildb.tcl*  contrib/  sguild.access  sguild.queries  sql_scripts/
autocat.conf          lib/      sguild.conf    sguild.reports  xscriptd*

I start by making a backup of this whole directory. The files or directories that I don't want to lose are:

sguild.conf: sguild configuration file
sguild.users: sguild's user and password hash file
sguild.reports: I have some custom reports, including some based on the reporting and data mining page of NSMWiki.
autocat.conf: used to automatically categorize alerts based on specific criteria, and most people that have done any tuning will hopefully have taken advantage of autocat.conf
certs/: sguild cert directory

Some people may also have added standard global queries in sguild.queries, or access controls in sguild.access. These are all basically configuration files, so if you have changed them you may want to keep them or include the changes in the new files.

After deciding what I need to keep, I upgrade the server.

$ mv -v /etc/sguild/server ~/sguild-old
$ cp -R ~/src/sguil-0.7.0/server /etc/sguild/
$ cp -R ~/sguild-old/certs /etc/sguild/server/
$ cp ~/sguild-old/sguild.users /etc/sguild/server/
$ cp ~/sguild-old/sguild.conf /etc/sguild/server/
$ cp ~/sguild-old/sguild.reports /etc/sguild/server/
$ cp ~/sguild-old/autocat.conf /etc/sguild/server/

Then I edit my sguild init script to remove "-o" and "-s" since encryption is now required instead of optional. The new version of sguild and the agents will give errors if you start them without removing the switches.

I start sguild and see that it is working, so next is the sensor. On the sensor, I backed up the conf files first.

$ cp -v /etc/sguil-0.7.0/sensor/*.conf ~/
$ rm -Rf /etc/sguil-0.7.0/sensor/
$ cp ~/src/sguil-0.7.0/sensor /etc/sguil-0.7.0/
$ cp ~/*.conf /etc/sguil-0.7.0/sensor/

Then, I edited all the agent init scripts to remove the "-o" switch. The agents are pads, pcap, sancp and snort. Now I can reconnect the agents to the server and the only thing left to do is upgrade my client. For the client upgrade, I replace everything except the sguil.conf file. If I made any modifications to my client, I would also need to incorporate those into the new client.

Tweet

Passive Tools

I love passive tools, what I like to think of as the "M" in NSM.

I recently posted about PADS. Sguil also uses p0f for operating system fingerprinting, and sancp for session-logging.

Even the IDS and packet-logging components of Sguil are passive. There are plenty of other good passive tools available.

You can learn a lot just by listening.

You can also run Snort inline and active, which goes a little beyond monitoring, for better or worse.

Tweet