This post is really just my thoughts and notes on managing multiple Sguil sensors. A lot of the thoughts could be applied to managing multiple UNIX-like systems in general, whether or not they are NSM systems. I would love any feedback on what other people recommend for managing Unix deployments.
In the past, most Linux or FreeBSD systems I've managed have been single systems, not part of a larger group of systems. Currently, I'm actually well past the point where I should have come up with a more elegant and automated solution to managing multiple systems of similar configurations.
In my current environment, I have multiple sensors with almost identical configurations. The configuration differences are minor and mainly because of different sensor locations.
Most of my management amounts to patching operating systems, patching software, and updating Snort signatures. To update Snort signatures, I generally run Oinkmaster on a designated system, review rule changes, disable or enable new rules as needed, then run a script to push out the updated rules and restart Snort with the new rules on each sensor.
When I build new sensors, I manually install the operating system. With RHEL systems, I should really be using Kickstart to help automate the installation and configuration of new systems.
After I install the operating system, I have a set of scripts I use to automate configuration and installation of software. These scripts are similar to David Bianco's InstantNSM except mine are much more specifically tailored for my environment and include configuration of the operating system, not just installing NSM software.
User account management is another facet of system management. For NSM sensors, since access to the systems is limited to an extremely small number of people, account management is not a huge issue and I won't be getting into that aspect of system management.
One site I've seen mentioned as a reference for managing numerous systems is Infrastructures.Org. The first thing I noticed on the Infrastructures site was the recommendation to use version control as a way of managing configuration files. This is probably obvious to a lot of people, but I never really thought of version control for that purpose. Subversion, CVS, or your version control of choice may be useful for managing systems, not just software. Normally, when I thought version control I thought of it in reference to software projects, not system management. Another option would be something like rdist, which is a program to maintain identical copies of files over multiple hosts.
One other thing some people do is local software repositories. On RHEL systems this might mean a local RPM repository for software that has been customized or is not available in the official Red Hat repositories. This could also mean considering something like Red Hat Satellite Server, depending on exactly what you want out of it. In my case, I think Satellite Server may be overkill but it certainly has some intersting things to offer.
There are several things I see myself needing to do to make management of my systems better. These are just places to start, and hopefully once I explore the options I will be posting my experiences.
- Kickstart for OS installation, which would also probably replace some of my scripts used to configure new systems.
- Modify my scripts for NSM setup to work with Kickstart if necessary.
- Version control and automating the process of systems synchronizing files.
- Local software repositories for customized software, including making RPMs of the modified software instead of compiling the software and then pushing it out.