The Changing Nature of Incident Response: Part 2
October 26th, 2011
Perhaps the biggest single step in the life cycle of an incident response team is going operational. The major problem with getting the Department of Energy’s incident response team operational was that there was nothing–no policies, no standards, and virtually no procedures concerning incident response at that time. The CERT/CC team claimed that it was operational at that time, but if that were true, there would have been some kind of indication that operations were taking place, and there was no indication whatsoever. One of the best ways that my management ever helped at that time was to inform me of other emergency response teams and to try to get me in touch with people who managed such efforts. One such team was the Nuclear Energy Search Team (NEST) at Lawrence Livermore National Laboratory. Discussions with the manager and some of the more senior staff members of this team helped me better understand the kind of procedures that would have to be performed, the kinds of communication that would have to occur, and how action priorities would have to be determined. Still, the nuclear arena is not all that closely aligned with the information security arena, and after I was finished meeting with NEST members I developed a kind of sinking feeling that there was much more to do than I had ever imagined. And at the time the team I managed consisted only of myself.
I was allowed to do some hiring, and my first inclination was to turn to a friend, Unix guru and former grad student at UC Davis, Ana Maria de Alvare. Additionally, I turned to another friend, technical guru, and then current grad student at UC Davis, Tom Longstaff. With these two excellent people aboard the team, we were able to better define and prioritize team goals and to start thinking about what we would do once we had to go operational. Back in those days (in early 1989), there was no SecurityFocus or bugtraq–in fact, the Worldwide Web did not even exist then. So we decided that one of the best things we could do was to offer a vulnerability notification service to the DOE sites, all 78 of them at the time. We figured that if people at least patched their systems and applications, they would be far less likely to have incidents, and preventing incidents altogether seemed better than experiencing incidents and then having to devote time and effort in responding to them. So we did our best to learn about vulnerabilities and then distribute them to the security managers (called computer protection program managers or CPPMs) at the various DOE sites.
Our first few bulletins were pretty pathetic. We neglected to determine what the format and content of bulletins would be, and our having failed to do so showed in the quality of the bulletins. But after receiving some flak from disaffected recipients, we started to get the hang of it. CERT/CC has started to issue bulletins about the same time, and it did not take long to realize that the CERT/CC bulletins were not meeting the needs of system administrators in that they were too high level. So we decided to devote the first part of our bulletins to a kind of high-level summary of each vulnerability and then the rest to what often amounted to detailed procedures for installing patches and/or workarounds. The result was very gratifying–we started to win the hearts not only of CPPMs at DOE sites, but also of rank and file system and network administrators at these sites, as well as others to which our bulletins had been forwarded.
As our constituency warmed up to us, we found them increasingly willing to share vulnerabilities they had discovered with us, and we soon were in the position of having to play middleman between technical people in the field and vendors. This was an awkward position, as vendors in that day were with a few exceptions incredibly uninterested in hearing about, let alone fixing vulnerabilities in their products. The worst was Sun Microsystems, which eventually let us know that their people were committed to dealing only with CERT/CC. Apparently, some kind of exclusive relationship between the two had been set up. This was unfortunate given that it became very apparent very early in the life of the DOE CIAC team that in wanting to be the only power player in the response arena, CERT/CC was not cooperating much with other teams. In some cases, CERT/CC team members were even playing dirty tricks against other teams, CIAC very much included. But that is another story.
Back in the late 1980s DOE sites were under no coercion from DOE headquarters to install patches to fix vulnerabilities described in DOE bulletins. Years later, things changed, and sites were required to install patches. But then not long afterwards, individual department and agency teams started to be phased out. After producing almost nothing for well over a decade, CIAC was phased out a little over three years ago in favor of a Las Vegas-based team that cost far less and produced far more. NASCIRC, NASA’s incident response team, fell by the wayside in favor of the U.S. CERT. And CERT/CC no longer issues vulnerability bulletins, nor is it involved in incident response operations any more. Additionally, vendors are doing a much better job getting information about vulnerabilities in their products to their customers. Furthermore, powerful and efficient patch management tools are now widely available. With a few exceptions, incident response teams no longer spread the word about vulnerabilities. So we are in many ways much better off than we were in the late 1980s, even if many of us are not very diligent in patching vulnerabilities in our systems and applications. Meanwhile, incident response teams are now freer to engage in other tasks more centrally related to incident response support.
–Gene Schultz, Ph.D., CISSP, CISM, GSLC
– – – – – – – – – – – – – – – – –
Dr. Eugene Schultz is the CTO at Emagined Security, an information security consulting practice based in San Carlos, California. He is the author/co-author of five books, and has also written over 120 published papers. Gene has been the editor-in-chief of two journals and is currently on the editorial board of three journals. He is also a SANS instructor, member of the SANS NewsBites editorial board, co-author of the 2005 and 2006 CISM preparation materials, and is on the technical advisory board of three companies. Gene has previously managed an information security practice as well as a national incident response team. He has also been professor of computer science at several universities and is retired from the University of California. He has received the NASA Technical Excellence Award, the Department of Energy Excellence Award, the ISACA John Kuyers Best Speaker/Best Conference Contributor Award, the Vanguard Conference Top Gun Award (for best presenter) twice, the Vanguard Chairman’s Award, and the National Information Systems Security Conference Best Paper Award. A Distinguished Fellow of the Information Systems Security Association (ISSA), Gene has also been named to the ISSA Hall of Fame and has received ISSA’s Professional Achievement and Honor Roll Awards. He is currently a member of the accreditation board of the Institute of Information Security Professionals (IISP). Dr. Schultz has provided expert testimony before committees within the U.S. Senate and House of Representatives on various security-related issues, and has served as an expert witness in legal cases.