The Changing Nature of Incident Response: Part 1
October 22nd, 2011
I’ve been affiliated with incident response in one way or another since 1988. I am not saying this boastfully, as I’ve made many mistakes both in responding to incidents technically and in managing incident response efforts. At the same time, however, when I first entered the incident response arena, there were no policies, standards, and procedures, and not really any requirements, either, to guide incident response efforts. Everyone who played in this arena originally had to use a combination of intuition and learning from mistakes just to get by.
Contrary to what the Software Engineering Institute’s (SEI’s) Computer Emergency Response Team Coordination Center (CERT/CC) says, the beginning of incident response as we know it today was at the University of California’s Berkeley Lab. Dr. Cliff Stoll, an astrophysicist who ran out of research funding took on system administration work there just to keep his time card filled. He noticed a slight discrepancy in computing time charges between a Unix host’s built-in accounting function and a home-grown accounting program that ran on that host. His scientific curiosity aroused, Cliff looked into the matter and found not only that someone had tampered with the host’s accounting function, but also that there were unexplained connections from unlikely places. You’ll have to read The Cuckoo’s Egg, Cliff’s best-selling book, to learn the whole story. Suffice it to say, the most significant impact of Cliff’s story from an incident response point of view is that the break-ins, which ostensibly were financed by the Soviet Union’s KGB, showed that some kind of organized and systematic response effort was needed to deal with what was then becoming an increasingly serious problem of remote intrusions into systems.
Because Berkeley Lab was and still is a Department of Energy (DOE) laboratory, officials within the DOE decided that this department needed a DOE-wide response team. Long before anyone even thought about starting CERT/CC, they solicited proposals from DOE labs; Los Alamos National Laboratory (LANL), Oak Ridge National Laboratory (ORNL) and Lawrence Livermore National Laboratory (LLNL) responded. LLNL’s proposal was accepted, and because I had written the final version of this proposal, somehow I became the project manager in an area about which I knew virtually nothing.
Meanwhile, an LLNL employee leaked the news that the DOE was forming an incident response team to a friend who worked at the SEI and suggested that an Internet-wide incident response team be established. Several SEI staff members acted on this suggestion and requested and obtained funding from the Defense Advanced Research Projects Agency (DARPA) before the LLNL incident response project received its funding from the DOE. What still amazes me is that CERT/CC claims that it was created in response to the Morris Worm of 1988, when in reality people who became members of this team had applied for funding and had defined their proposed team months before.
Anyway, when the Morris Worm made its debut on the evening of November 2, 1988, both CERT/CC and the precursor to the DOE’s incident response team had virtually no positive impact whatsoever. While members of both teams flailed, largely due to a lack of procedures, academics at the Massachusetts Institute of Technology (MIT) and Purdue University led the way in reverse engineering the worm code and providing help and advice to the then ARPAnet (shortly afterwards, the NSFnet, and then later the Internet) community. But in the long run, the Morris Worm was an extremely beneficial event for the development of incident response in that it showed the fledgling incident response community just how big the task at hand was. It also helped make government bureaucrats and others who had money and power aware that relying only on preventative and detective controls was grossly insufficient.
Oh by the way, I mentioned that when I first entered the incident response arena, I knew absolutely nothing about incident response. Just a few weeks after the DOE response team became operational, I was regularly contacted by writers for Government Computer News, the Federal Computer Week, and other media outlets. After writers had interviewed me, they would run stories with captions such as “Government security expert says that…” Funny–I knew a lot about MIL-STD 5200 then, but to call me an expert was really stretching it! It all goes to show that you cannot believe everything you read…
–Gene Schultz, Ph.D., CISSP, CISM, GSLC
– – – – – – – – – – – – – – – – –
Dr. Eugene Schultz is the CTO at Emagined Security, an information security consulting practice based in San Carlos, California. He is the author/co-author of five books, and has also written over 120 published papers. Gene has been the editor-in-chief of two journals and is currently on the editorial board of three journals. He is also a SANS instructor, member of the SANS NewsBites editorial board, co-author of the 2005 and 2006 CISM preparation materials, and is on the technical advisory board of three companies. Gene has previously managed an information security practice as well as a national incident response team. He has also been professor of computer science at several universities and is retired from the University of California. He has received the NASA Technical Excellence Award, the Department of Energy Excellence Award, the ISACA John Kuyers Best Speaker/Best Conference Contributor Award, the Vanguard Conference Top Gun Award (for best presenter) twice, the Vanguard Chairman’s Award, and the National Information Systems Security Conference Best Paper Award. A Distinguished Fellow of the Information Systems Security Association (ISSA), Gene has also been named to the ISSA Hall of Fame and has received ISSA’s Professional Achievement and Honor Roll Awards. He is currently a member of the accreditation board of the Institute of Information Security Professionals (IISP). Dr. Schultz has provided expert testimony before committees within the U.S. Senate and House of Representatives on various security-related issues, and has served as an expert witness in legal cases.