Inline devices and fail open

There was a post to the sguil-users mailing list asking for recommended fail-open network cards in the hopes that it would be less expensive than alternatives. Richard Bejtlich pointed out that the long-term costs and lower downtime make bypass switches worth the initial expense. This make a lot of sense.

One thing people need to note when talking about inline devices and fail-open hardware is that many devices will only fail open in a powered off state. For example, if you're running Snort inline with a fail-open NIC and the Snort process dies, then the box will no longer pass traffic. Your link is down unless you fix the problem by restarting Snort or you shut off the system completely, which will then cause the fail-open NIC to cross-connect and pass traffic.

If a system only fails open when the power is off, you still need to be aware that an operating system or application failure can take down your link if the fail-open hardware remains powered on. NetOptics has some bypass switches that have a heartbeat feature to address this problem.

An exclusive Heartbeat feature monitors link status between the Bypass and monitoring tools for enhanced reliability. A configurable Heartbeat packet is injected into the monitor port link to help determine availability of attached monitoring tools. For instance, the Bypass Switch can automatically switch network traffic around an unresponsive IPS appliance – even if the IPS is still powered on. Once the IPS re-establishes a connection, traffic is re-routed to the monitor port for continued operation.

This is a better solution than a NIC that will only fail open when the power is off, but not all bypass switches have the feature. If anyone knows of other vendors or hardware that have a similar feature, please let me know.

It's also important to know that the use of the terms "fail open" and "fail closed" is not always consistent.

Tweet

Upgrading to Snort 2.8.0

I finally upgraded my test sensor from Snort 2.6.1.5 to Snort 2.8.0. David Bianco mentioned some of the new features in his blog a couple of months ago, so I won't get into the differences. I am mainly documenting the things I had to do for the upgrade so I have a reference if needed at a future date.

The first thing I did was look through the documentation in the "docs" directory, reading some of the README files for the preprocessors. The README.variables file was of particular interest since Snort 2.8 allows port lists. I also looked at the snort.conf file in the "etc" directory of the source tree to see how it differed from my current configuration file.

Next, I made a copy of my snort.conf from 2.6.1.5 and edited it for the changes in 2.8.0. I changed the HTTP_PORTS variable to list a few other ports besides 80, including 8080 and 8000. The portvar variable was used in the examples of multiple HTTP_PORTS.

portvar HTTP_PORTS [80,8000,8080,8888]

Although I only run HTTP on port 80, the Snort web-client ruleset and a number of Bleeding rules use the $HTTP_PORTS variable to detect attacks against web clients like Internet Explorer, Mozilla Firefox, media players, and more. After that simple change, I configured and installed Snort.

./configure --enable-dynamicplugin --enable-inline --enable-perfprofiling
make
make install

After installing, I tried to start Snort. The first problem I encountered was with my stream5 configuration. I had previously been using the stream4 and the flow preprocessors, but when changing the configuration to use stream5 I had not removed the flow preprocessor configuration. Stream5 handles everything that used to be handled by the combination of flow and stream4, so I removed the flow configuration. I also had to add the stream5_udp and stream5_icmp options.

A check of the configure help will also show that --enable-dynamicplugin is the default with 2.8.0, so it should not actually be needed in the configuration command.

After fixing stream5, I tried again and had one more problem. I was getting the following errors:

Oct 23 19:18:52 sensor snort[3117]: /etc/snort/snort.conf(206) unknown dynamic preprocessor "ftp_telnet"
Oct 23 19:18:52 sensor snort[3117]: /etc/snort/snort.conf(210) unknown dynamic preprocessor "ftp_telnet_protocol"
Oct 23 19:18:52 sensor snort[3117]: /etc/snort/snort.conf(221) unknown dynamic preprocessor "ftp_telnet_protocol"
Oct 23 19:18:52 sensor snort[3117]: /etc/snort/snort.conf(226) unknown dynamic preprocessor "ftp_telnet_protocol"
Oct 23 19:18:52 sensor snort[3117]: /etc/snort/snort.conf(238) unknown dynamic preprocessor "smtp"
Oct 23 19:18:52 sensor snort[3117]: /etc/snort/snort.conf(307) unknown dynamic preprocessor "dcerpc"
Oct 23 19:18:52 sensor snort[3117]: /etc/snort/snort.conf(313) unknown dynamic preprocessor "dns"
Oct 23 19:18:52 sensor snort[3117]: FATAL ERROR: Misconfigured dynamic preprocessor(s)

This was pretty easy to fix. I just needed the proper path to the dynamic preprocessors.

dynamicpreprocessor directory /usr/local/lib/snort_dynamicpreprocessor/

After I fixed the snort.conf, the next try to start Snort 2.8.0 was successful. Now that I have it installed and running with very similar settings to 2.6.1.5, it's time to dig deeper into the differences and possibly test other configuration changes.

Tweet

Running IDS Inline

When do you run IDS inline?

Although the idea of running an IDS inline to prevent attacks rather than just detect them is appealing, reality is harsh. There are a lot of problems running an IDS inline.

The first and most significant problem with running IDS inline is that most people don't adequately tune IDS. This is a problem whether you run inline or passive IDS. I get the feeling that the general management view towards IDS, perpetuated by marketing, is that you should be able to drop an IDS on the network and it is ready to go. This is far from the truth. Network monitoring requires tuning all involved systems and humans to analyze the information provided by the systems.

When I set up my first Snort deployment, I spent countless hours tuning and tweaking the configuration, rules, performance and more. Along the way, I added Sguil, which was useful both for more thorough analysis and more effective tuning based on those analyses. Even after all that, tuning is a daily requirement and never should stop.

Every time you update rules, you should tune. Every time the network changes, you may need to tune. Every time your IDS itself is updated, you may need to tune. Every time a new attack is discovered, you may need to tune. Tuning does not and should not end unless you want your network security monitoring to become less effective over time. I would be willing to bet that a majority of inline IDS don't come close to taking full advantage of blocking features because of a lack of tuning. Passive IDS suffers from the same problem, but the results can be even worse if an inline IDS starts blocking legitimate traffic.

Another problem, somewhat related to tuning, with running IDS inline is that rules are never perfect. Do you trust your enabled rules to be 100% perfect? If not, what percentage of alerts from a given rule must be accurate to make the rule acceptable in blocking mode? Even good rules will sometimes or often trigger on benign traffic.

For Snort, one of the most reliable rule sets are policy violations. Both VRT rules and Bleeding rules are extremely reliable when detecting policy violations with a minimal number of alerts triggered on unrelated network traffic. Spyware rules are similarly accurate.

Rules designed to detect more serious malicious activity are much less consistent. Most are simply not reliable enough to use in a mode that blocks the traffic the rule is designed to detect. That does not mean the rules are necessarily bad! Good rules still aren't perfect. This is one of many reasons why there will always be work for security analysts. IDS or other NSM systems are no substitute for the analysis that an experienced human can provide. They simply are tools to to be used by the analyst.

Last, don't forget that running anything inline can seriously impact the network. If my Snort process at home dies for some reason, I can survive without Internet access until I get it restarted. This isn't always the case for businesses, so consider whether your traffic should be going through a system that fails open if there are problems. Even at home I don't queue port 22 to snort_inline because I want to be able to SSH into the box from the Internet if there are problems.

The real question that has to be answered is whether the benefits of dropping traffic iare worth the risks of running inline.

Tweet

Transparent Bridging, MMAP pcap, and Snort Inline

I use Snort and the Sguil Analyst Console for NSM, but there is always room to experiment and/or improve. Up to this point, I have used Snort either on mirrored (SPAN) ports or with a network TAP, both common configurations. After finally getting a second-hand CPU and motherboard to replace a dead CPU, I had a spare system to set up Snort inline at home.

The upgrade from Snort 2.4.x to 2.6.x.x was quite taxing on performance, so I decided it was also time to play with Phil Wood's MMAPped libpcap. The modified libpcap will make drastically fewer system calls when compared to the standard libpcap sniffing on a busy network. Although my home network certainly isn't high bandwidth, I wanted the experience of setting up Snort with the Phil Wood's modified version of libpcap. Since I actually did all this many months ago and am just now posting about it, I can say that I have seen a huge performance improvement when going from the standard libpcap to Phil Wood's libpcap in high bandwidth environments.

How would I implement Snort Inline on a home network? The two choices were to replace one of my routers with a BSD or Linux system configured as a router, or to set up Linux as a transparent bridge. For those that prefer FreeBSD, you would have to configure it as a router since the bridge code in FreeBSD does not support the ipfw divert socket. I am not familiar enough with other BSD versions to say whether their bridge code is the same or not.

I much preferred a bridge rather than a router since it would avoid the time-consuming process of reconfiguring my network topology. This meant that I had to use Linux, and my distribution in this case is Slackware-current. The process should not be much different for any distribution.

Installing Software

Because plugging in an untested bridge between my LAN and the Internet could interrupt my connection, I decided it would be easiest to get and install all the software prior to configuring the bridge and putting it inline.

My first step was to install the modified libpcap, which needs either flex and bison, or yacc. This was essentially a freshly built Slackware system and I didn't have them installed, so I used swaret to install the packages.

swaret --install flex
swaret --install bison

I was now ready to install libpcap.

cd /usr/src/
wget http://public.lanl.gov/cpw/libpcap-current.tar.gz
tar xvzf libpcap-current.tar.gz
ln -s libpcap-0.9.20060417 libpcap
cd libpcap
./configure
make
make install

Snort will need the header files from libpcap and the install did not copy them anywhere, so I manually copied the files to /usr/include/. Another option would be to create a link to the files in the include directory.

cp /usr/src/libpcap/pcap.h /usr/include/
cp /usr/src/libpcap/pcap-bpf.h /usr/include/
cp /usr/src/libpcap/pcap-namedb.h /usr/include/

Because this is a modified libpcap, all the software that depends on libpcap must also be compiled against the version I just installed. I will definitely be using tcpdump when I test the bridge.

wget http://www.tcpdump.org/daily/tcpdump-current.tar.gz
tar xvzf tcpdump-current.tar.gz
ln -s tcpdump-2007.01.07 tcpdump
cd tcpdump
./configure
make
make install

Now I could test that tcpdump worked with the PCAP_FRAMES option available because of the modified libpcap. For some reason, perhaps because of my kernel version, PCAP_FRAMES=max did not work but I was able to use it by manually setting the value. I was able to bump the value of PCAP_FRAMES quite high, above 300000, before it resulted in errors. I have yet to determine what that really means for performance. Here are two commands I used to show that the newly installed tcpdump worked with the modified libpcap.

PCAP_FRAMES=65535 PCAP_VERBOSE=1 PCAP_TO_MS=0 PCAP_PERIOD=10000 /usr/local/sbin/tcpdump \
-i eth0 -s 1514 -w /dev/null -c 100

PCAP_FRAMES=65535 /usr/local/sbin/tcpdump -v -i eth0

Snort needs libnet-1.0.2a when configured with --enable-inline, so I had to install libnet first.

tar xvzf libnet-1.0.2a
cd libnet-1.0.2a
./configure
make
make install

Finally, install Snort 2.6.x.x. (Note: Since this document was written quite a while ago, there are quite a few newer versions of Snort available). Another alternative to using the --enable-inline option with mainline Snort is to download snort_inline, which is maintained by William Metcalf and Victor Julien. There are a number of added features and conveniences when using snort_inline, as highlighted on Victor's blog. However, I used mainline Snort in the following example.

tar xvzf snort-2.6.1.2.tar.gz
cd snort-2.6.1.2
./configure --enable-dynamicplugin --enable-inline
make
make install

I tested snort with -V to check that it would start and was compiled to work inline. It shows that Snort was configured with the inline option.

snort -V

 ,,_     -*> Snort! <*-
o"  )~   Version 2.6.1.2 (Build 34) inline
 ''''    By Martin Roesch & The Snort Team: http://www.snort.org/team.html
         (C) Copyright 1998-2006 Sourcefire Inc., et al. 

Configuring Linux for Transparent Bridging

Configuring the bridging is fairly simple. The only module I had to manually insert was ip_queue. Other modules that may be needed are ip_tables, iptable_filter and bridge.

In this case, eth0 is my separate management interface, I named the bridge interface bridge0, and the physical interfaces joining bridge0 were eth1 and eth2. The bridge device can be configured with an IP address if you do not want to use a separate NIC for management. In either case, make sure to secure the management NIC on your Snort box, for example limiting connections to the management IP so only source IP addresses in your private IP space are allowed to connect. Here, I created the bridge interface, added eth1 and eth2 to it, and brought them up:

/sbin/brctl addbr bridge0
/sbin/brctl addif bridge0 eth1
/sbin/brctl addif bridge0 eth2
/sbin/ifconfig eth1 up
/sbin/ifconfig eth2 up
/sbin/ifconfig bridge0 up

After some iptables configuration, bridge0 should work. Assuming the system is dedicated to being a bridge running Snort Inline, the only addition necessary to make bridging work is the following:

iptables -I FORWARD -o bridge0 -j ACCEPT

Now bridge0 should be ready to pass packets.

Once the bridge is connected, iptables can be used to show packet statistics and confirm that the bridge is forwarding. You can also use tcpdump -v -i bridge0 to confirm that traffic is passing if you skip ahead to installing libpcap and tcpdump prior to plugging in the bridge.

iptables -vL
--snip--
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source    destination
527  168K ACCEPT     all  --  any    bridge0 anywhere  anywhere
--snip--

The bridge is seeing packets! tcpdump showed that they were more than just broadcast packets as I accessed the Internet through the bridge.

PCAP_FRAMES=65535 /usr/local/sbin/tcpdump -v -i bridge0
tcpdump: WARNING: bridge0: no IPv4 address assigned
tcpdump: listening on bridge0, link-type EN10MB (Ethernet), capture
--snip--
57 packets captured
57 packets received by filter
0 packets dropped by kernel 

Testing Snort Inline

Finally, I'm ready to configure and test Snort Inline. I am not going to cover Snort configuration, but I did write one test rule to put in local.rules and disabled all the other rule sets in snort.conf. The following rule should drop any outbound connection on the HTTP_PORT set in snort.conf.

drop tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"Test rule outbound HTTP"; \
classtype:misc-activity; sid: 3000000;)

Now I run Snort. The -v will print packets to the console so I can confirm that snort is seeing the traffic and the -Q tells snort to accept input from the iptables queue.

/usr/local/bin/snort -Qvc /etc/snort/snort.conf -l /data

Note that I forgot to add the PCAP_FRAMES value. In production on a busy network, I would add it permanently to my environmental variables and/or to my init scripts for Snort so it would always take advantage of Phil Wood's libpcap.

I add the necessary rule to iptables. I don't want to risk losing all connections by queueing everything, so I just queue port 80. This is what the line in my iptables-save file looks like:

-A FORWARD -i bridge0 -p tcp -m tcp --dport 80 -m state --state NEW -j QUEUE

I try to browse to a web site and get the following alert in /data/alert:

[**] [1:300000:0] Test rule outbound HTTP [**]
[Classification: Misc activity] [Priority: 3]
01/11-22:22:24.458394 xx.xx.xx.xx:60813 -> 66.35.250.209:80
TCP TTL:63 TOS:0x0 ID:52831 IpLen:20 DgmLen:60 DF
******S* Seq: 0xE79DA525  Ack: 0x0  Win: 0x16D0  TcpLen: 40
TCP Options (5) => MSS: 1460 SackOK TS: 604992578 0 NOP WS: 2

Everything is working. Now I have to get everything ready for production by fully configuring and tuning Snort. Once that is done, I will queue all traffic except port 22. Queuing port 22 could result in not being able to connect using SSH if the Snort process were to die or had to be restarted. If snort-inline is not running, all queued traffic is effectively dropped since Snort is required to pass the traffic from the queue back to iptables.

Tweet