CERT is hiring

The company I work for is hiring. For those that don't know, CERT is part of the Software Engineering Institute at Carnegie Mellon University. CERT was created in 1988 as part of the response to the Morris worm. You can find out more on CERT's "About Us" page.

If you are interested, please read more about our hiring process and browse some of the available positions. The positions are primarily in Pittsburgh with a few openings in Arlington, VA. The open positions cover network security analysis, security architecture, malware analysis, software development, vulnerability analysis, and more.

I consider our hiring process fairly grueling but also stimulating. It gives the prospective employee and prospective coworkers a good chance to really learn if the relationship will work. It is an opportunity not just for the candidate to get interviewed, but also for the candidate to interview those that already work at CERT.

One of the reasons we have so many vacant positions is because we try to maintain high standards when considering candidates. Most of our positions require a fair amount of experience and expertise. My colleagues are smart, diligent, and largely enthusiastic about their chosen professions. Don't get me wrong -- we still have bad days when we are less enthusiastic or unhappy about the state of information security, but this is a pretty cool place to work. We do a wide variety of both research and more operationally focused work, tackling a lot of big problems. We also get a fair amount of freedom to find interesting and challenging areas of work.

If I know you or know of your work, please contact me about using my name as a referral. A referral from a current CERT employee can be helpful when applying. The best way to contact me regarding a referral or to ask other questions is via email or LinkedIn. You can also post questions more publicly here on my blog if it seems appropriate. In the interest of disclosure, I have an interest in recruiting people that I will want to work with but also could potentially get a referral bonus if you list me when you apply.

Tweet

Building an IR Team: Growth

This is a long overdue continuation of my posts regarding Building an Incident Response Team. I had a very rough outline of this post going all the way back to 2009! The good response I got on some of my previous posts on building IR teams made me come back and work on finishing the posts I had planned when I first started the series.

Previous posts:

• Building an IR Team: Documentation 
• Building an IR Team: Organization 
• Building an IR Team: People
  

I believe one of the hardest things to deal with when building a successful IR team is growth. If you build an IR team that is successful and gets management buy-in as a result, there is a good chance that responsibilities, the amount of work, the number of incidents detected, and the size of the team will all grow. This will invariably cause growing pains, setbacks, and reevaluation of procedures.

I honestly could go on and on about dealing with the growth of an IR team. There are so many things to consider that it is daunting to plan for growth ahead of time instead of just dealing with the hurdles as they come. However, if you have a team that is growing it really helps to take a step back and plan for both immediate and long-term growth. It is so important that a fair amount of this post will reiterate what I have explicitly or implicitly said in some of my previous "Building an IR Team" posts.

There are a number of questions to keep in mind when an IR team grows. What are the additional duties causing the addition of positions? Are the additional positions adequate to cover the additional duties and responsibilities? If not, how can expectations be managed so superiors understand what is actually feasible? Are the duties just a higher volume of what the team already is responsible for, or are there new areas that will require different types of team members and different types of training? What works well now but may be problematic with a larger team? Do we need to restructure? How do we maintain the success that led to the IR team growth? The last question is one of the most fundamental.

Relationships

At one point I worked on a team that, over the course of a few years, increased the number of personnel fourfold. This completely changed the dynamics of the team, from the lead all the way down to the most junior analyst. The more people you add, the more complex the relationships become. This applies not only to relationships within the team, but also relationships with other parts of your organization and management.

With such growth, it became a lot more important to clearly define roles and responsibilities, the command structure, and get management support of decisions.

• Command structure: As the team grows, other groups in the company are less likely to know each person on the team. This means in a lot of cases it is helpful to have a few key people known to those other groups. These key people don't have to always be the ones to communicate with a specific group, but can be used as a fallback if the other group's first instinct is to be more adversarial with those people they don't know.
• Intra-team relationships: The more people you have, the more you have to keep an eye on the working relationship between members. When you have a team that numbers single digits, it is almost natural to know all the ins and outs of the working relationships, for example who complements each other and who can be a good mentor to more junior analysts. It takes more conscious effort to track as you increase the number of people. Not only that, it requires more actively setting expectations about what you expect of them.
• Management support and inter-team relationships: As a team gets bigger, its profile is raised throughout the company. This can make dealing with other groups easier, more difficult, or most likely a little bit of both. As we all know, IR teams sometimes need to make decisions or do things that are not popular and people outside the team view as irritating to say the least. It is very important to have management support when you invariably have conflicts with those outside the IR team. It's also important to have a manager that knows when to tell you that you're being unreasonable and the outside groups have a reasonable concern or complaint.

This is by no means a complete list of things to consider. The bottom line is that a larger team makes both intra- and inter-team relationships more complex.

Other Growing Pains

The simplest example I have from the past regarding growing pains was when I was on a team that was not gaining new areas of responsibility but was switching to coverage 24 hours a day seven days a week. As I covered in another blog post, it is important come up with the proper organization and make sure every shift was productive. Increasing the number of hours of coverage also obviously means hiring new analysts, plus the possibility of shifting current analysts to drastically different schedules.

Restructuring can often cause conflicts beyond those involving work schedules. On a small team, most people gravitate to a niche and can often be allowed to work in it as long as they also can handle the more generalized response duties. In a larger team, it's much harder to let members naturally gravitate towards certain areas while maintaining the ability to get all the work done. It certainly is nice to keep everyone happy and specializing in the areas they are most interested in, but it's not always realistic. One way to help with this is to make sure you follow the advice for redundancy in the "Organization" post, plus allow members to rotate through different areas of specialty. This means they won't be stuck in one particularly area in addition to providing redundancy of skills.

Another issue is making sure you formalize reporting to some degree. In a team of a few people, it's readily apparent what each person is doing. When you have a score of people, you need to get both formal and informal reporting from shift leads, team leads, mentors, and even individual analysts to properly understand who is doing what, workloads, what is working well, and what is not working. Regardless, the structure of a larger IR team probably needs to be more formal when it is larger. Notice the "probably." I think it is safe to say there may be exceptions to all these points! The key is to find the proper balance that enables useful reporting while avoiding unneeded bureaucracy.

Hiring can also create growing pains. I must stress that you should do everything possible to maintain standards when hiring. That said, a bigger team can mean more room and opportunity for less experienced analysts. One weak link among five people is a much bigger deal than one weak link among 30, so a larger team can allow you to take a chance or two when hiring. I've always been an advocate of getting smart people that can learn and are legitimately interested in the field over those who have experience but less potential for growth, and a larger team can sometimes make this easier to justify.

Evaluation of Procedures and Operations

This advice really applies to all IR teams, but becomes more important with growth. Incident response procedures that work well in a small team may not work as well with a larger group. Even if your team has not grown, you may want to regularly reevaluate IR workflow, reporting, or just about any existing procedures and standards of operations. Sometimes it may mean more clearly codifying what were once informal standards, while other times it may mean completely rethinking how you operate because you have several tiers of analysts. Having good metrics so you can try to make reevaluation more objective and less subjective also helps. Unfortunately, metrics is a huge topic that I can't address in this post, but there are many sites, papers, books, and more to help anyone interested in the topic.

Standards for working with the field may also need to change. If you are in an enterprise where the IR team often is reaching out to "boots on the ground" like local system administrators or IT staff, there may need to be changes in areas of responsibility when the IR team is larger. I partially covered this when mentioning inter-team relationships. Even if your IR team is comfortable contacting those in the field directly, those managing the people in the field may want a more formal command structure so they can track requests and other communications from the IR team. Contacts in the field may also want their roles and responsibilities more formally or clearly defined. This is easier to work through when the IR team only has a few people, but once there are dozens it can cause problems if those in the field don't know upfront what the IR team expects and what qualifies as an unusual request from the IR team.

Training

A larger IR team means the company is spending a lot more money on the team and security in general. It also means you may have enough team members to form a class-sized group. Whether you use in-house training, outsource, or a combination, a larger team means you will need to think about more formal training where a large group is in a classroom environment. This doesn't mean one-on-one or one-on-few mentoring and training should go away, but you will need to adapt to training larger groups. You also should consider setting aside money specifically for training if that was not done previously.

Be Flexible

Note that this is all based on my experiences in the past 10 or more years, but it is just the tip of the iceberg. Different teams may have different issues to consider when growing. Depending on the specific IR team, none of what I wrote may apply directly. I think there are two overriding concerns when an IR team grows. One is to be flexible as the team grows so your organization can really see what works and what does not. Two is to plan for the growth instead of just letting it happen haphazardly. Some teams do quite well with very little change after they've grown, while some may need drastic changes just because of adding a few people or analyst turnover.

Other Resources

There are some resources available to help deal with creating IR teams, and much of what applies at creation of a team can apply to the growth of a team. When a team goes from a few people to 20-30 people, you essentially are destroying the old team and creating a new one. Most of the questions considered when creating an IR team can be asked once again and reevaluated as the team grows.

• Creating a Computer Security Incident Response Team: A Process to Getting Started from CERT.
• Handbook for Computer Security Incident Response Teams (PDF) from CERT, which is not recent but still has a lot of useful information.
• Building a Successful Security Operation Center from HP.
• RFC 2350: Expectations for Computer Security Incident Response from 1998.

Richard Bejtlich has posted on his blog about many aspects of building and maintaining SOCs, and also mentioned that he will have a chapter in his new book titled "Network Security Monitoring Operations," focused on sharing "the author’s experience building and leading a global Computer Incident Response Team (CIRT), such that readers can apply those lessons to their own operations." I presume anyone regularly reading my blog is already reading Taosecurity, and also anticipate that his new book will be quite useful.

I hope to have at least one more post in my "Building an IR Team" series. I may also have additional material, or collate and improve all my existing posts if I feel it is worthwhile.

Tweet

Reflections on Over Five Years of Blogging

My first post to this blog was in September, 2007. Professionally speaking, I have gone through major changes since then. I've changed employer, though amazingly enough in this line of work that happened only once during that time. I have also learned a lot and my duties have changed quite a bit.

Though I try to stay plugged in to incident response, NSM, and all those other operational bits I love, I am definitely a step back from directly responding to incidents compared to a lot of my previous experience. Another big change for me is that I no longer run a bunch of NSM sensors though I still do that type of administration on my home network. On the other hand, one of the wonderful things about my current employer is that they allow us a lot of freedom to identify problems or challenges then take them on without trying to pigeonhole us. I look forward to 2013 as a year in which I will continue being challenged by taking on some new projects of interest to me.

I've gotten a number of links and traffic bursts on some of my past blog posts, which is flattering. I don't particularly feel like a unique snowflake that should get a ton of web traffic and don't usually get a ton of traffic, but occasionally I will really hit the nail on the head with a technical post and get a lot of traffic and links from other bloggers. Unsurprisingly, many of my top posts are in the system administration category since the more security-focused posts probably have a narrower target audience.

I attended FloCon 2013 in January, which made me reflect on a couple things. First, I am going to try and blog a little more often this year. It was very flattering to talk to people at the conference and have them say they have read my blog or to find they were using content I had contributed to NSMWiki. When I started this blog, my two main goals were to provide references for myself and to make those references available to others in case they also found them useful. It is good to know that my blog and other public contributions have been useful to others. I would not be where I am without similar help from others and I think that sharing of information, advice, experience, and debate is a great thing about much of the security community.

The second thing it drove home is that I need to end the semi-anonymous nature of this blog. At FloCon I found that I had coworkers following me on Twitter without even realizing it was me that they were following!

My previous employer knew about my blog and did not give me any grief whatsoever, but at the same time they were somewhat nervous about it. My current employer embraces public engagement to a much larger degree. Plenty of people already knew my name prior to this and Richard Bejtlich even linked to my blog using my name at least once, but generally I did not promote myself as the author. It is time to change that.

Tweet