Extracting emails from archived Sguil transcripts

Here is a Perl script I wrote to extract emails and attachments from archived Sguil transcripts. It's useful for grabbing suspicious attachments for analysis.

In Sguil, whenever you view a transcript it will archive the packet capture on the Sguil server. You can then easily use that packet capture to pull out data with tools like tcpxtract or tcpflow along with Perl's MIME::Parser in this case. The MIME::Parser code is modified from David Bianco's blog.

As always with Perl or other scripts, I welcome constructive feedback. The first regular expression is fairly long and may scroll off the page, so make sure you get it all if you copy it.

#!/usr/bin/perl

# by nr
#   2009-05-04
# A perl script to read tcpflow output files of SMTP traffic.
# Written to run against a pcap archived by Sguil after viewing the transcript.
#   2009-05-07
# Updated to use David Bianco's code with MIME::Parser.
# http://blog.vorant.com/2006/06/extracting-email-attachements-from.html

use strict;
use MIME::Parser;

my $fileName; # var for tcpflow output file that we need to read
my $outputDir = "/var/tmp"; # directory for email+attachments output

if (@ARGV != 1) {
    print "\nOnly one argument allowed. Usage:\n";
    die "./emailDecode.pl /path/archive/192.168.1.13\:62313_192.168.1.8\:25-6.raw\n\n";
}

$ARGV[0] =~ m
    /.+\/(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}):(\d{1,5})_(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}):(25)-\d{1,3}\.raw/
    or die "\nIncorrect file name format or dst port is not equal to 25. Try again.\n\n";

system("tcpflow -r $ARGV[0]"); # run tcpflow w/argument for path to sguil pcap

my $srcPort = sprintf("%05d", $2); # pad srcPort with zeros
my $dstPort = sprintf("%05d", $4); # pad dstPort with zeros

# Put the octest and ports into array to manipulate into tcpflow fileName
my @octet = split(/\./, "$1\." . "$srcPort\." . "$3\." . "$dstPort");

foreach my $octet(@octet) {
    my $octetLength = length($octet); # get string length
    if ($octetLength < 5) { # if not a port number
        $octet = sprintf("%03d", $octet); # pad with zeros
    }
    $fileName = $fileName . "$octet\."; # concatenate into fileName
}

$fileName =~  s/(.+\d{5})\.(.+\d{5})\./$1-$2/; # replace middle dot with hyphen
my $unusedFile = "$2-$1"; # this is the other tcpflow output file

# open the file and put it in array
open INFILE, "<$fileName" or die "Unable to open $fileName $!\n";
my @email = <INFILE>;
close INFILE;

my $count = 0;
# skip extra data at beginning
foreach my $email(@email) {
    if ($email =~ m/^Received:/i) {
        last;
    }
    else {
        delete @email[$count];
        $count ++;
    }
}

my $parser = new MIME::Parser;
$parser->output_under("$outputDir");
my $entity = $parser->parse_data(\@email); # parse the tcpflow data
$entity->dump_skeleton; # be verbose when dumping

unlink($fileName, $unusedFile); # delete tcpflow output files

Tweet

Do we need anti-virus software?

My friend Richard has a good post about Verizon's 2009 Data Breach Report. One of his last comments really struck me since it is something I have seen firsthand again and again.

Most companies are probably relying on their anti-virus software to save them. This is too bad, because the explosion in customized malware means it probably won't.

Anti-virus software just does not work against most recent malware. The table from the Verizon report shows a drastic upswing in customized malware and my experience tells me that doesn't tell half the story. Even only small changes will often evade anti-virus software.

I'm not saying anything new here. Anyone that does penetration tests, reverse engineers malware, writes exploits, or is involved with information security in a number of ways already knows that anti-virus software is terrible at detecting new malware. I have even written about it before and pointed out that more subtle methods of exploitation aren't always necessary because of the effectiveness of commodity malware.

My question is, do we really need anti-virus software?

When you take into account the amount of resources spent running anti-virus in the enterprise, is it a good investment in risk reduction? We pay for hours worked to setup the anti-virus infrastructure, update, and troubleshoot. If you are in an enterprise, you're paying for the software, not using a free alternative. You're probably paying for support and also paying for hardware.

What does it get you? I find malware on a weekly basis, sometimes daily, that is not detected by the major vendors. I submit the malware to some of these vendors and places like VirusTotal, but the responses from anti-virus vendors are inconsistent at best. Even after definitions are updated, I'll then run across malware that is obviously just an altered version of the previous but is once again not detected.

I don't pretend to have the answers, but I do wonder if all the resources spent on anti-virus by a business, particularly large or enterprise businesses, might be better spent somewhere else. Is it really worth tens or hundreds of thousands of dollars in software, hours, and hardware to make sure old malware is detected? If not, how much is it worth? Does the occasional quick response to emerging malware make it more worthwhile? If you have enough influence on the vendor, does being able to contact them directly to help protect against a specific attack make it more valuable?

Anti-virus software is too ingrained in corporate culture to think it is realistic that companies will stop using it altogether, but we need to keep asking these types of questions.

Tweet