The ultimate test of the value of an incident response team is how that team handles crises. Crises are generally not everyday occurrences. In fact, most issues with which an incident response team must deal are not of a bona fide emergency nature. That is why from the very onset of computer incident response teams I have objected to any incident response team name that includes “emergency” or “crisis” in it, because these terms represent little more than massive embellishment of the true nature of most of their activities.
At the same time, however, an incident response team must be prepared to deal with genuine emergencies. The first such emergency that the CIAC team that I then managed encountered was less than a year after the team’s having gone operational. The WANK (Worms against Nuclear Killers) worm infected one VMS host after another on what was at the time the DECnet. Given that the Department of Energy (DOE) and NASA computing communities depended heavily on VMS and the DECnet, the worm comprised a genuine crisis.
The CIAC team had some technical expertise in VMS, but not as much as was needed to deal with this nasty worm. Fortunately, several VMS experts, most notably Kevin Oberman of Lawrence Livermore National Laboratory and Ron Tencati of NASA’s NASCIRC team, did far more than their fair share of reverse engineering and making recommendations for containing infections. CIAC team members were constantly on the phone with Kevin, Ron and other people from DOE and other sites. At the same time, they valiantly struggled to write worm-related updates and bulletins that were quickly disseminated to the computer protection program managers (CPPMs) and other key personnel at these sites.
Murphy’s Law has stood the test of time better than virtually any other human-derived axiom, and once more the truth of Murphy’s Law became evident when a massive earthquake hit the San Francisco Bay Area right in the middle of the rapid spread of the WANK (later to become the WANK/OILZ) worm. Telephone communications in the Bay Area were out, severing the major link between CIAC and its constituency. Back in late 1989 when these events occurred, the Internet (which at that time was actually the ARPAnet) was comparatively small and insignificant compared to what it currently is. Many CPPMs and other significant security players within the DOE community did not use the ARPAnet. Consensus among them favored FAX-based communications, but because such communications depend on telephone lines, no FAXes could be sent. ARPAnet-based communications were also temporarily disrupted. Reaching CIAC’s constituency for a period of nearly 48 hours was thus impossible–a worst case scenario for an incident response team.
After three weeks, the WANK (and later the WANK/OIL) worm finally slowed to a crawl, giving CIAC and the rest of the incident response community time to catch their breaths. Firsthand accounts of disruption and stories related to this worm’s temporary “reign of terror” started coming in, the most amusing of which was from a Japanese researcher who expressed gratitude to the worm for being so kind as to honor his country with its “visit!”
Several extremely valuable “lessons learned” emerged from the events that transpired:
- Cooperation is everything. No incident response team was or is an island, and the cooperation between the teams that did the most in dealing with the WANK – WANK/OILZ worms proved invaluable. At the same time, resources from outside of the response team community proved to be at least as if not more helpful, showing that no matter how technically proficient it is, the response team community needs to fall back on other resources and expertise sometimes.
- Out-of-band communications needs must be anticipated, designed, implemented and extensively tested long before situations in which they will be needed occur.
- Training for incident response team members must include how to deal with crisis situations as well as other situations. Emphasizing the need and how to escalate critical information during emergencies needs to be one of the most fundamental parts of such training.
Again, however, incident response today is different from how incident response used to be. The Internet community’s response to the various versions of the Conficker worm lacked the kind of extensive involvement by incident response teams that the response to the WANK – WANK/OILZ worms did. Technical staff from Microsoft, Symantec and other companies that had a vested interest in countering worm outbreaks as well as researchers from organizations such as SRI International instead led the way in responding to the Conficker worm. Today’s incident response teams are still important, but given the greater availability of knowledge and capabilities within the Internet community, each such team is more likely to serve a specific purpose or set of purposes for the specific organization for which it has been created
–Gene Schultz, Ph.D., CISSP, CISM, GSLC
– – – – – – – – – – – – – – – – –
Dr. Eugene Schultz is the CTO at Emagined Security, an information security consulting practice based in San Carlos, California. He is the author/co-author of five books, and has also written over 120 published papers. Gene has been the editor-in-chief of two journals and is currently on the editorial board of three journals. He is also a SANS instructor, member of the SANS NewsBites editorial board, co-author of the 2005 and 2006 CISM preparation materials, and is on the technical advisory board of three companies. Gene has previously managed an information security practice as well as a national incident response team. He has also been professor of computer science at several universities and is retired from the University of California. He has received the NASA Technical Excellence Award, the Department of Energy Excellence Award, the ISACA John Kuyers Best Speaker/Best Conference Contributor Award, the Vanguard Conference Top Gun Award (for best presenter) twice, the Vanguard Chairman’s Award, and the National Information Systems Security Conference Best Paper Award. A Distinguished Fellow of the Information Systems Security Association (ISSA), Gene has also been named to the ISSA Hall of Fame and has received ISSA’s Professional Achievement and Honor Roll Awards. He is currently a member of the accreditation board of the Institute of Information Security Professionals (IISP). Dr. Schultz has provided expert testimony before committees within the U.S. Senate and House of Representatives on various security-related issues, and has served as an expert witness in legal cases.