A lot of people noticed the recent Congressional ethics probe that was disclosed because a junior staff member put a sensitive document on her computer at home. Not surprisingly, her computer also had file-sharing software installed and she inadvertently was sharing the document on a peer-to-peer network. Some are calling for a review of congressional cybersecurity policies after the breach. One thing to remember is that this sort of thing is not unique, new or surprising.
David Bianco wrote about a similar topic in 2006 and covers the important points, though I would add that the problem also extends to personal systems, not just mobile devices. Whether the vulnerability is a mobile device that is easily lost or stolen (laptop, smart-phone, music player, etc), or a personal system running software that would never be allowed in a work environment, don't put sensitive information on systems that are difficult to control.
19 November, 2009
SNAFU: Peer-to-peer and Sensitive Information
17 November, 2009
SANS WhatWorks in Incident Detection Summit 2009
I am scheduled to be a part of several discussion panels at the SANS WhatWorks in Incident Detection Summit 2009 on 9-10 December. There are a lot of good speakers participating and the agenda will cover many topics related to incident detection. I believe there is still space available for anyone that is interested in attending.
From SANS:
Following the success of the 2008 and 2009 editions of the SANS WhatWorks in Forensics and Incident Response Summits, SANS is teaming with Richard Bejtlich to create a practioner-focused event dedicated to incident detection operations. The SANS Incident Detection Summit will share tools, tactics, and techniques practiced by more than 40 of the world's greatest incident detectors in two full days of content consisting of keynotes, expert briefings, and dynamic panels.
http://www.sans.org/incident-detection-summit-2009/
19 October, 2009
Hackintosh Dell Mini 9
If you want a small laptop running Mac OSX, this is a pretty cool. The Dell outlet sometimes still has these laptops, but remember to factor in that they are old and need a larger SSD. Also take into account that installing even a retail copy of OSX on non-Apple hardware may violate their EULA.
12 October, 2009
Adding GeoIP to the Sguil Client
This is a post I meant to publish months ago, but for some reason slipped off my radar. I was reading the Sguil wishlist on NSMWiki and saw something that looked simple to implement. Here are a couple diffs I created after adding a menu item for GeoIP in sguil.tk and a proc for it in lib/extdata.tcl. All I did was copy the existing DShield proc and menu items, then edit as needed to change the URL and menu listings.
I think it should work and I downloaded a pristine copy of the files before running diff since I've hacked Sguil files previously, but no warranty is assumed or implied, et cetera.
Ideally, I would love to help out and tackle some of the other items on the wishlist. My time constraints make it hard, but at least I now have a Tcl book.
sguil.tk
2865a2866 > .ipQueryMenu add cascade -label "GeoIP Lookup" -menu $ipQueryMenu.geoIPMenu 2873a2875,2876 > menu $ipQueryMenu.geoIPMenu -tearoff 0 -background $SELECTBACKGROUND -foreground $SELECTFOREGROUND \ > -activeforeground $SELECTBACKGROUND -activebackground $SELECTFOREGROUND 2917a2921,2922 > $ipQueryMenu.geoIPMenu add command -label "SrcIP" -command "GetGeoIP srcip" > $ipQueryMenu.geoIPMenu add command -label "DstIP" -command "GetGeoIP dstip"lib/extdata.tcl
211a212,243
> proc GetGeoIP { arg } {
>
> global DEBUG BROWSER_PATH CUR_SEL_PANE ACTIVE_EVENT MULTI_SELECT
>
> if { $ACTIVE_EVENT && !$MULTI_SELECT} {
>
> set selectedIndex [$CUR_SEL_PANE(name) curselection]
>
> if { $arg == "srcip" } {
> set ipAddr [$CUR_SEL_PANE(name) getcells $selectedIndex,srcip]
> } else {
> set ipAddr [$CUR_SEL_PANE(name) getcells $selectedIndex,dstip]
> }
>
> if {[file exists $BROWSER_PATH] && [file executable $BROWSER_PATH]} {
>
> # Launch browser
> exec $BROWSER_PATH http://www.geoiptool.com/?IP=$ipAddr &
>
> } else {
>
> tk_messageBox -type ok -icon warning -message\
> "$BROWSER_PATH does not exist or is not executable. Please update the BROWSER_PATH variable\
> to point your favorite browser."
> puts "Error: $BROWSER_PATH does not exist or is not executable."
>
> }
>
> }
>
> }
>
15 September, 2009
MySQL replication on RHEL
I recently configured MySQL for replication after first enabling SSL connections between the two systems that would be involved with replication. I have to say that MySQL documentation is excellent and all these notes are simply based on what is available on the MySQL site. I have included links to as many of the relevant sections of the documentation as possible.
For reference, here is the MySQL manual on enabling SSL: 5.5.7.2. Using SSL Connections
Before beginning, it is a good idea to create a directory for the SSL output files and make sure all the files end up there.
MySQL’s RHEL5 packages from mysql.com support SSL by default, but to check you can run:
$ mysqld --ssl --help mysqld Ver 5.0.67-community-log for redhat-linux-gnu on i686 (MySQL Community Edition (GPL))Copyright (C) 2000 MySQL AB, by Monty and others This software comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to modify and redistribute it under the GPL license Starts the MySQL database server Usage: mysqld [OPTIONS] For more help options (several pages), use mysqld --verbose --help
The command will create an error if there is no SSL support.
Next, check that the MySQL server has SSL enabled. The below output means that the server supports SSL but it is not enabled. Enabling it can be done at the command line or in the configuration file, which will be detailed later.
$ mysql -u username -p -e"show variables like 'have_ssl'" Enter password: +---------------+----------+ | Variable_name | Value | +---------------+----------+ | have_ssl | DISABLED | +---------------+----------+
Documentation on setting up certificates:
5.5.7.4. Setting Up SSL Certificates for MySQL
First, generate the CA key and CA certificate:
$ openssl genrsa 2048 > mysql-ca-key.pem Generating RSA private key, 2048 bit long modulus ............................................+++ ............+++ $ openssl req -new -x509 -nodes -days 356 -key mysql-ca-key.pem > mysql-ca-cert.pem You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [GB]:US State or Province Name (full name) [Berkshire]:California Locality Name (eg, city) [Newbury]:Burbank Organization Name (eg, company) [My Company Ltd]:Acme Road Runner Traps Organizational Unit Name (eg, section) []:Acme IRT Common Name (eg, your name or your server's hostname) []:mysql.acme.com Email Address []:acme-irt@acme.com
Create the server certificate:
$ openssl req -newkey rsa:2048 -days 365 -nodes -keyout mysql-server-key.pem > mysql-server-req.pem Generating a 2048 bit RSA private key .............................+++ .............................................................+++ writing new private key to 'mysql-server-key.pem' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [GB]:US State or Province Name (full name) [Berkshire]:California Locality Name (eg, city) [Newbury]:Burbank Organization Name (eg, company) [My Company Ltd]:Acme Road Runner Traps Organizational Unit Name (eg, section) []:Acme IRT Common Name (eg, your name or your server's hostname) []:mysql.acme.com Email Address []:acme-irt@acme.com Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []: $ openssl x509 -req -in mysql-server-req.pem -days 356 -CA mysql-ca-cert.pem -CAkey mysql-ca-key.pem -set_serial 01 > mysql-server-cert.pem Signature ok subject=/C=US/ST=California/L=Burbank/O=Acme Road Runner Traps/OU=Acme IRT/CN= mysql.acme.com/emailAddress=acme-irt@acme.com Getting CA Private Key
Finally, create the client certificate:
$ openssl req -newkey rsa:2048 -days 356 -nodes -keyout mysql-client-key.pem > mysql-client-req.pem Generating a 2048 bit RSA private key ................+++ .................+++ writing new private key to 'mysql-client-key.pem' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [GB]:US State or Province Name (full name) [Berkshire]:California Locality Name (eg, city) [Newbury]:Burbank Organization Name (eg, company) [My Company Ltd]:Acme Road Runner Traps Organizational Unit Name (eg, section) []:Acme IRT Common Name (eg, your name or your server's hostname) []:mysql.acme.com Email Address []:acme-irt@acme.com Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []: $ openssl x509 -req -in mysql-client-req.pem -days 356 -CA mysql-ca-cert.pem -CAkey mysql-ca-key.pem -set_serial 01 > mysql-client-cert.pem Signature ok subject=/C=US/ST=California/L=Burbank/O=Acme Road Runner Traps/OU=Acme IRT/CN=mysql.acme.com/emailAddress=acme-irt@acme.com Getting CA Private Key [nr@mysqld mysqlcerts]$ ls mysql-ca-cert.pem mysql-client-key.pem mysql-server-key.pem mysql-ca-key.pem mysql-client-req.pem mysql-server-req.pem mysql-client-cert.pem mysql-server-cert.pem
To enable SSL when starting mysqld, the following should be in /etc/my.cnf under the [mysqld] section. For this example, I put the files in /etc/mysql/openssl:
ssl-ca="/etc/mysql/openssl/mysql-ca-cert.pem" ssl-cert="/etc/mysql/openssl/mysql-server-cert.pem" ssl-key="/etc/mysql/openssl/mysql-server-key.pem"
To use any client, for instance mysql from the command line or the GUI MySQL Administrator, copy the client cert and key to a dedicated folder on the local box along with ca-cert. You will have to configure the client to use the client certificate, client key, and CA certificate.
To connect with the mysql client using SSL, copy the client certificates to a folder, for instance /etc/mysql, then under the [client] section in /etc/my.cnf:
ssl-ca="/etc/mysql/openssl/mysql-ca-cert.pem" ssl-cert="/etc/mysql/openssl/mysql-client-cert.pem" ssl-key="/etc/mysql/openssl/mysql-client-key.pem"
In MySQL Administrator, the following is an example you would put into the Advanced Parameters section if you want to connect using SSL.
SSL_CA U:/keys/mysql-ca-cert.pem SSL_CERT U:/keys/mysql-client-cert.pem SSL_KEY U:/keys/mysql-client-key.pem USE_SSL Yes
Replication
Before configuring replication, I made sure to review the MySQL replication documentation.
16.1.1.1. Creating a User for Replication
Because MySQL stores the replication user’s name and password using plain text in the master.info file, it’s recommended to create a dedicated user that only has the REPLICATION SLAVE privilege. The replication user needs to be created on the master so the slaves can connect with that user.
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'192.168.1.50' IDENTIFIED BY ‘password’;
16.1.1.2. Setting the Replication Master Configuration
Edit my.cnf to uncomment the “log-bin” line. Also uncomment “server-id = 1”. The server-id can be anything between 1 and 2^32 but must be unique.
Also add “expire_logs_days” to my.cnf. If you don’t, the binary logs could fill up the disk partition because they are not deleted by default!
expire_log_days = 4
16.1.1.3. Setting the Replication Slave Configuration
Set server-id to something different from the master in my.cnf. Although not required, enabling binary logging on the slave is also recommended for backups, crash recovery, and in case the slave will also be a master to other systems.
16.1.1.4. Obtaining the Master Replication Information
I flush the tables to disk and lock them to temporarily prevent changes.
# mysql -u root -p -A dbname mysql> FLUSH TABLES WITH READ LOCK; Query OK, 0 rows affected (0.00 sec)
If the slave already has data from master, then you may want to copy over data manually to simplify things, 16.1.1.6. Creating a Data Snapshot Using Raw Data Files. However, you can also use mysqldump, as shown in Section 16.1.1.5, “Creating a Data Snapshot Using mysqldump”.
Once the data is copied over to the slave, I get the current log position.
Mysql> SHOW MASTER STATUS; +------------------+----------+--------------+------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | +------------------+----------+--------------+------------------+ | mysql-bin.000002 | 16524487 | | | +------------------+----------+--------------+------------------+ 1 row in set (0.00 sec) mysql> UNLOCK TABLES;
16.1.1.10. Setting the Master Configuration on the Slave
Finally, configure the slave. The log file and log position tell the slave where to begin replication. All changes after that log position will be replicated to the slave.
mysql> CHANGE MASTER TO -> MASTER_HOST=’192.168.1.50’, -> MASTER_USER=’repl’, -> MASTER_PASSWORD='replication_password', -> MASTER_LOG_FILE=’ mysql-bin.000002’, -> MASTER_LOG_POS=16524487, -> MASTER_SSL=1, -> MASTER_SSL_CA = '/etc/mysql/openssl/mysql-ca-cert.pem', -> MASTER_SSL_CAPATH='/etc/mysql/openssl/', -> MASTER_SSL_CERT = '/etc/mysql/openssl/mysql-server-cert.pem', -> MASTER_SSL_KEY = '/etc/mysql/openssl/mysql-server-key.pem'; Query OK, 0 rows affected (0.74 sec) mysql> START SLAVE;
Replication can start!
The slave status can be checked via the following command:
mysql> show slave status;
06 September, 2009
Two years
It has been two years since I started this blog. Here is a quick recap of notable posts that consistently get a substantial number of page views.
IR/NSM:
- Building an IR Team: People
- Building an IR Team: Organization
- Transparent Bridging, MMAP pcap, and Snort Inline
- Snort Performance and Memory Map Pcap on RHEL
- Upgrading to Snort 2.8.0
- Snort 2.8.1 changes and upgrading
- Snort shared object rules with Sguil
- JavaScript decoding and more
- Querying Session Data Based on Snort Rule IPs
- Setting up OpenLDAP for centralized accounts
- OpenLDAP continued
- OpenLDAP Security
- Using parted and LVM2 for large partitions
15 July, 2009
Building an IR Team: Documentation
My third post on building an Incident Response (IR) team covers documentation. The first post was Building an IR Team: People, followed by Building an IR Team: Organization.
Good documentation promotes good communication and effective analysts. Documentation is not sexy, and can even be downright annoying to create and maintain, but it is absolutely crucial. Making it as painless and useful as possible will be a huge benefit to the IR team.
Since documentation and communication are so intertwined, I had planned on making one post to cover both topics. However, the amount of material I have for documentation made me decide to do a future post, Building an IR Team: Communication, and concentrate on keeping this post to a more digestible size.
There are quite a few different areas where a Computer Incident Response Team (CIRT) will need good documentation.
Incident Tracking
Since I am writing about computer IR teams, it is obvious that the teams will be dealing with digital security incidents. For an enterprise, you will almost certainly need a database back-end for your incidents. Even smaller environments may find it best to use a database to track incidents.You will need some sort of incident tracking system for many reasons, including but not necessarily limited to the following.
- Tracking of incident status and primary responder(s)
- Incident details
- Response details and summary
- Trending, statistics and other analysis
However, off-the-shelf software may not have great support for the incident details. A great example is IP addresses and ports. Logging IP addresses, names of systems, ports if applicable, and what type of vulnerability was exploited can be extremely useful for trending, statistics, and historical analysis. A field for IP addresses can probably be more easily be queried than a full text field that contains IP addresses. If I see that a particular IP address successfully attacked two systems in the previous shift, or a particular type of exploit was used successfully on two systems, I want to be able to quickly check and see how many times it happened in the past week. I also want to be able to pull that data out and use it to query my NSM data to see if there was similar activity that garnered no response from analysts.
Reponse details can be thought of as a log that is updated throughout the incident, from discovery to resolution. Having the details to look back on is extremely useful. You can use the details for a technical write-up, an executive summary, to recreate incidents in a lab environment, for training, lessons learned, and more. My general thought process is that the longer it takes to document an incident, the more likely the documentation is to be useful.
Trending and statistical analysis can be used to help guide future response and look back at previous activity for anything that was missed, as I already mentioned. It is also extremely useful for reports to management that can also be used to help gain political capital within the organization. What do I mean by political capital?
Say you have noticed anecdotally that you are getting owned by web servers over HTTP, but the malicious sites are usually known to be malicious, for instance when searching Google or using a anti-malware toolbar. Your company has no web proxy and you recommend one with the understanding that most of the malicious sites would be blocked by the web proxy. The problem is that the networking group does not want to re-engineer or reconfigure, and upper management does not think it is worth the money. With a thorough report and analysis using the information from incident tracking, and by using that data to show the advantages of the proxy solution, you could provide the CIRT or SOC management the political capital they need to get things moving when faced with other parts of the company that resist.
Standard Operating Procedures (SOP)
Although analysts performing IR need to be able to adapt and the tasks can be fluid, a SOP is still important for a CIRT. A SOP can cover a lot of material, including IR procedures, notification and contact information, escalation procedures, job functions, hours of operation, and more. A good SOP might even include the CIRT mission statement and other background to help everyone understand the underlying purpose and mission of the group.
The main goal of a SOP should be to document and detail all the standard or repetitive procedures, and it can even provide guidance on what to do if presented with a situation that is not covered in the SOP. As an example, a few bullet points of sections that might be needed in a SOP are:
- Managing routine malware incidents
- Analyzing short term trends
- Researching new exploits and malicious activity
- Overview of security functions and tools, e.g. NSM
- More detailed explanation and basic usage information for important tools, e.g. how to connect to Sguil and who are the administrators of the system
I also like analysts to think about the most efficient way to analyze an incident. Some may gather information and investigate using slightly different methodology, but each analyst should understand that something simple should be checked before something that takes a lot of time, particularly when the value of the information returned will be roughly equal. The analysis should use what my boss likes to call the "Does it make sense?" test. Gathering some of the simplest and most straightforward information first will usually point you in the right direction, and a SOP can help show how to do this.
Knowledge Base
A knowledge base can take many different forms and contains different types of information than SOP, though there also may be overlap. There are specific knowledge base applications, wikis, simple log applications, and even ticketing or tasking systems that provide some functionality for an integrated knowledge base. A knowledge base will often contain technical information, technical references, HOWTOs, white papers, troubleshooting tips, and various other types of notes and information.
One of my favorite options for a knowledge base is a wiki. You can see various open knowledge bases that are using wikis, for instance NSMWiki and Emerging Threats Documentation Wiki, but if you want organization- and job-specific knowledge bases then you will also need something to hold the information for your CIRT.
The reason I pick those two wikis as examples is because they contain some of the exact type of information that is useful in a knowledge base for your CIRT. The main difference is that your knowledge will be specific to your organization. One good example are wiki entries for specific IDS rules as they pertain to your network, in other words an internal version of the Emerging Threats rule wiki. There may be shortcuts to take with regard to investigating specific rules or other network activity to quickly determine the nature of the traffic, and a wiki is a good place to keep that information.
Similarly, documentation on setting up a NSM device, tuning, or maintenance can be very effectively stored and edited on a wiki. The ease of collaboration with a wiki helps keep the documentation useful and up to date. If properly organized, someone could easily find information needed to keep the team running smoothly. Some example of documentation I have found useful when put it in a wiki:
- How to troubleshoot common problems on a NSM sensor
- How to build and configure a NSM sensor
- How to update and tune IDS rules
- List and overview of scripts available to assist incident response
- Overviews of each available IR tool
- More detailed descriptions and usage examples of IR tools
- Example IR walk-throughs using previously resolved incidents
- Links to external resources, e.g. blogs, wikis, manuals, and vendor sites
Shift Logs
In an environment with multiple shifts, it is important to keep shift logs of notable activity, incidents, and any other information that needs to be passed to other shifts. Although I will also discuss this in Building an IR Team: Communication, the usefulness of connecting the shifts with a dedicated log is apparent. Given the amount of email and incident tickets that are generated in an environment that requires 24x7 monitoring, having a shift log to quickly summarize important and ongoing event helps separate the wheat from the chaff.
Since my feeling is that shift logs should be terse and quick to parse, what to use for logging may not be crucial. The first examples that come to my mind are software designed for shift logs, forum software, or blogging software. The main features needed are individual accounts to show who is posting, timestamps, and an effective search feature. Anything else is a bonus, though it may depend on exactly what you want analysts logging and what is being used to handle incident tracking.
One thing that is quite useful with the shift log is a summary post at the end of each shift, and then the analysts should verbally go over the summary at the shift change. This can help make sure the most significant entries are not missed and it gives the chance for the oncoming shift to ask questions before the outgoing shift leaves for the day.
As usual, I can't cover everything on the topic, but my goal is to provide a reference and get the gears turning. The need for good documentation exists and documentation is important to use to the IR team's advantage.
Subscribe to:
Posts (Atom)
