With the recent news about the latest Apache backdoor on systems using cPanel, I thought it would be pertinent to show the process of adding an OSSEC agent that connects to a Security Onion server. Why is this relevant? Because OSSEC and other file integrity checkers can detect changes to binaries like Apache's httpd.
"OSSEC is an Open Source Host-based Intrusion Detection System that performs log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response."Many systems include integrity checking programs in their default installs these days, for instance Red Hat with AIDE. AIDE is also available in repositories for a number of other Linux distributions plus FreeBSD.
This case in particular would require using something other than the default options for integrity checking because cPanel installs Apache httpd in /usr/local/apache/bin, a non-standard directory that may not be automatically included when computing file hashes and doing subsequent integrity checks.
The reason I'm demonstrating OSSEC here is that it easily integrates with the Sguil console, and in Security Onion the sensors and server already have OSSEC configured to send alerts to Sguild. OSSEC also has additional functionality compared to AIDE. In this case, I'm installing the agent on a Slackware server.
$ wget http://www.ossec.net/files/ossec-hids-2.7.tar.gz
---snipped---
$ openssl sha1 ossec-hids-2.7.tar.gz
SHA1(ossec-hids-2.7.tar.gz)= 721aa7649d5c1e37007b95a89e685af41a39da43
$ tar xvzf ossec-hids-2.7.tar.gz
---snipped---
$ sudo ./install.sh
OSSEC HIDS v2.7 Installation Script - http://www.ossec.net
You are about to start the installation process of the OSSEC HIDS.
You must have a C compiler pre-installed in your system.
If you have any questions or comments, please send an e-mail
to dcid@ossec.net (or daniel.cid@gmail.com).
- System: Linux webserver 3.8.4
- User: root
- Host: webserver
-- Press ENTER to continue or Ctrl-C to abort. --
1- What kind of installation do you want (server, agent, local, hybrid or help)? agent
- Agent(client) installation chosen.
2- Setting up the installation environment.
- Choose where to install the OSSEC HIDS [/var/ossec]:
3- Configuring the OSSEC HIDS.
3.1- What's the IP Address or hostname of the OSSEC HIDS server?: 192.168.1.20
- Adding Server IP 192.168.1.20
3.2- Do you want to run the integrity check daemon? (y/n) [y]:
- Running syscheck (integrity check daemon).
3.3- Do you want to run the rootkit detection engine? (y/n) [y]:
- Running rootcheck (rootkit detection).
3.4 - Do you want to enable active response? (y/n) [y]:
3.5- Setting the configuration to analyze the following logs:
-- /var/log/messages
-- /var/log/auth.log
-- /var/log/syslog
-- /var/adm/syslog
-- /var/adm/auth.log
-- /var/adm/messages
-- /var/log/xferlog
-- /var/log/proftpd.log
-- /var/log/apache/error_log (apache log)
-- /var/log/apache/access_log (apache log)
-- /var/log/httpd/error_log (apache log)
-- /var/log/httpd/access_log (apache log)
- If you want to monitor any other file, just change
the ossec.conf and add a new localfile entry.
Any questions about the configuration can be answered
by visiting us online at http://www.ossec.net .
--- Press ENTER to continue ---
---snip---
- Init script modified to start OSSEC HIDS during boot.
- Configuration finished properly.
- To start OSSEC HIDS:
/var/ossec/bin/ossec-control start
- To stop OSSEC HIDS:
/var/ossec/bin/ossec-control stop
- The configuration can be viewed or modified at /var/ossec/etc/ossec.conf
Thanks for using the OSSEC HIDS.
If you have any question, suggestion or if you find any bug,
contact us at contact@ossec.net or using our public maillist at
ossec-list@ossec.net
( http://www.ossec.net/main/support/ ).
More information can be found at http://www.ossec.net
--- Press ENTER to finish (maybe more information below). ---
- You first need to add this agent to the server so they
can communicate with each other. When you have done so,
you can run the 'manage_agents' tool to import the
authentication key from the server.
/var/ossec/bin/manage_agents
More information at:
http://www.ossec.net/en/manual.html#ma
Next, I add the agent to my Security Onion server.
$ sudo /var/ossec/bin/manage_agents **************************************** * OSSEC HIDS v2.6 Agent manager. * * The following options are available: * **************************************** (A)dd an agent (A). (E)xtract key for an agent (E). (L)ist already added agents (L). (R)emove an agent (R). (Q)uit. Choose your action: A,E,L,R or Q: A - Adding a new agent (use '\q' to return to the main menu). Please provide the following: * A name for the new agent: webserver * The IP Address of the new agent: 192.168.1.5 * An ID for the new agent[001]: Agent information: ID:001 Name:webserver IP Address:192.168.1.5 Confirm adding it?(y/n): y **************************************** * OSSEC HIDS v2.6 Agent manager. * * The following options are available: * **************************************** (A)dd an agent (A). (E)xtract key for an agent (E). (L)ist already added agents (L). (R)emove an agent (R). (Q)uit. Choose your action: A,E,L,R or Q: e Available agents: ID: 001, Name: webserver, IP: 192.168.1.5 Provide the ID of the agent to extract the key (or '\q' to quit): 001 Agent key information for '001' is: ---snip--- ** Press ENTER to return to the main menu.
Now copy the key, go back to the web server, paste and import the key.
$ sudo /var/ossec/bin/manage_agents **************************************** * OSSEC HIDS v2.7 Agent manager. * * The following options are available: * **************************************** (I)mport key from the server (I). (Q)uit. Choose your action: I or Q: i * Provide the Key generated by the server. * The best approach is to cut and paste it. *** OBS: Do not include spaces or new lines. Paste it here (or '\q' to quit): ---snip--- Agent information: ID:001 Name:webserver IP Address:192.168.1.5 Confirm adding it?(y/n): y
If I was running a system with cPanel that was vulnerable to Cdorked.A then I would want to make sure OSSEC is monitoring the directories with the Apache httpd files. The OSSEC default configuration from my recent install is /var/ossec/etc/ossec.conf and the relevant lines are below:
<syscheck>
<!-- Frequency that syscheck is executed - default to every 22 hours -->
<frequency>79200</frequency>
<!-- Directories to check (perform all possible verifications) -->
<directories check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
<directories check_all="yes">/bin,/sbin</directories>
So by default OSSEC would apparently not be checking the integrity of cPanel's Apache installation and I would need to add /usr/local/apache to the directory checks. After making any changes for my particular system, I check the status of OSSEC and it is not yet running.
$ sudo /etc/rc.d/rc.ossec status ossec-logcollector not running... ossec-syscheckd not running... ossec-agentd not running... ossec-execd not running... $ sudo /etc/rc.d/rc.ossec start Starting OSSEC HIDS v2.7 (by Trend Micro Inc.)... Started ossec-execd... Started ossec-agentd... Started ossec-logcollector... Started ossec-syscheckd... Completed.
Note after adding the OSSEC agent on the remote system then adding it on the OSSEC server, you must restart ossec-hids-server in order for the ossec-remoted process to start listening on 1514/udp for remote agents.
$ sudo /etc/init.d/ossec-hids-server status ossec-monitord is running... ossec-logcollector is running... ossec-remoted not running... ossec-syscheckd is running... ossec-analysisd is running... ossec-maild not running... ossec-execd is running... $ sudo /etc/init.d/ossec-hids-server restart Killing ossec-monitord .. Killing ossec-logcollector .. ossec-remoted not running .. Killing ossec-syscheckd .. Killing ossec-analysisd .. ossec-maild not running .. Killing ossec-execd .. OSSEC HIDS v2.6 Stopped Starting OSSEC HIDS v2.6 (by Trend Micro Inc.)... OSSEC analysisd: Testing rules failed. Configuration error. Exiting. 2013/04/30 23:13:59 ossec-maild: INFO: E-Mail notification disabled. Clean Exit. Started ossec-maild... Started ossec-execd... Started ossec-analysisd... Started ossec-logcollector... Started ossec-remoted... Started ossec-syscheckd... Started ossec-monitord... Completed. $ sudo /etc/init.d/ossec-hids-server status ossec-monitord is running... ossec-logcollector is running... ossec-remoted is running... ossec-syscheckd is running... ossec-analysisd is running... ossec-maild not running... ossec-execd is running... $ netstat -l | grep 1514 udp 0 0 *:1514 *:*
Note the error corresponding to the FAQ entry about getting an error when starting OSSEC. However, since I'm running OSSEC 2.7 this did not seem to apply. Poking around, I realized the ossec-logtest executable had not been copied to /var/ossec/bin when I ran the install script. After I manually copied it to the directory, restarting OSSEC no longer caused the "Testing rules failed" error.
Once you have installed OSSEC on the system to be monitored, added the agent on the server, imported the key on the system to be monitored, restarted the server process, and started the client process, you will start getting alerts from the newly added system in Sguil. For example, the content of Sguil alerts will look like this after updating gcc:
Integrity checksum changed for: '/usr/bin/gcc' Old md5sum was: '764a405824275d806ab5c441516b2d79' New md5sum is : '6ab74628cd8a0cdf84bb3329333d936e' Old sha1sum was: '230a4c09010f9527f2b3d6e25968d5c7c735eb4e' New sha1sum is : 'b931ceb76570a9ac26f86c12168b109becee038b'
In the Sguil console, if I wanted to view all the recent OSSEC alerts I could perform a query as pictured below. Note you need to escape the brackets or remove them in favor of the MySQL wildcards '%%'.
Finally, to show an example of the various types of alerting that OSSEC can do in addition to checksum changes, here is a query and output directly from the MySQL console.
mysql> SELECT count(signature),signature FROM event WHERE signature LIKE '%%OSSEC%%' GROUP BY signature ORDER BY count(signature) DESC; +------------------+---------------------------------------------------------------------------------------+ | count(signature) | signature | +------------------+---------------------------------------------------------------------------------------+ | 388 | [OSSEC] Integrity checksum changed. | | 149 | [OSSEC] Host-based anomaly detection event (rootcheck). | | 46 | [OSSEC] Integrity checksum changed again (2nd time). | | 39 | [OSSEC] IP Address black-listed by anti-spam (blocked). | | 12 | [OSSEC] Integrity checksum changed again (3rd time). | | 4 | [OSSEC] Web server 400 error code. | | 3 | [OSSEC] Receipent address must contain FQDN (504: Command parameter not implemented). | +------------------+---------------------------------------------------------------------------------------+ 7 rows in set (0.00 sec)The highest count alert, plus the alerts indicating "2nd time" and "3rd time", are the basic functionality needed to detect changes to a file, my original use case. The "rootcheck" is alerting on files owned by root but writable by everyone. The balance of the alerts are from reading the system logs and detecting the system rejecting emails (anti-spam, 504) or web server error codes.
Back to the original problem of Cdorked.A, the blog posts on the subject also indicate that NSM could detect unusually long HTTP sessions, and there are no doubt other network behaviors that could be used to create signatures or network analytics resulting in detection. File integrity checks are just one possible way to detect a compromised server. Remember you need to have known good checksums for this to work! You ideally install something like OSSEC prior to the system being live on the network or at the least prior to it running any listening services that could be compromised before computing the checksums.
