The TDL3 rootkit is viewed by many malware experts as the most sophisticated rootkit to surface in real-world settings. It has fared very well in its efforts to avoid detection and eradication by anti-virus software. Yet previous versions of this rootkit have been limited by several protection mechanisms that Microsoft has built into recent versions of Windows operating systems (e.g., the 64-bit versions of Vista and Windows 7:

1. A very systematic digital signature check keeps rogue drivers out of kernel memory. A driver must be digitally signed if it is to be loaded into kernel memory. Because malicious code is almost never digitally signed, rogue drivers are prevented from getting where they need to be to hook kernel memory processes.

2. Windows Kernel Patch Protection (“PatchGuard”) prevents kernel mode drivers from changing anything within the Windows kernel, including the System Service Descriptor Table (SSDT), Interrupt Descriptor Table (IDT), and the kernel code itself.

3. In Windows systems one must run as the Administrator to be able to install code. Both Vista and Windows 7 do not by default allow users to run as the Administrator.

What is different about the version of TDL3 that targets 64-bit Windows systems is that it gains control very early during the boot sequence–by altering the Master Boot Record (MBR) as the target system is booting so that it can intercept and take ownership of routines that run during boot time and then load its own malicious drivers. This enables the latest version of TDL3 to circumvent the first two of the Windows protection mechanisms described above. But there is still another barrier that this rootkit faces–having the Administrative privileges needed to infect the MBR. Unless the user on the targeted machine throws all caution to the wind and logs on as the Administrator, the 64-bit targeting version of TDL3 cannot do its dire deeds. But it gets around this barrier by causing the targeted host to restart, thereby getting the maliciously altered MBR to be read and loaded before other processes can intervene.

The presence of a rootkit that targets 64-bit Windows OSs is a very significant and frightening development. Other similar rootkits are bound to follow–quickly. Not only will many more Windows systems become infected, but once again the anti-virus vendors will be left standing flatfooted, wondering what to do. Unfortunately, the bad guys keep showing just how far they are ahead of the white hat community, and there appears to be no easy and cheap way to remedy this sad situation.

Not unexpectedly, a great deal of the content covered in this workshop had to do with cloud computing. Whether we like it or not, many people equate virtualization and cloud computing. The difference in the SANS Virtualization Summit was that the cloud computing talks actually had some substance to them, unlike most of the other talks on this subject that I have attended at other meetings and conferences. My favorite one was by Matt Linton, who described an eloquent network architecture in which security processes were well-integrated into cloud services at NASA-Ames Research Center. In another session Alexander Meisel, CEO of Art of the Defense, pointed out an often overlooked advantage of cloud computing, namely that it can offer such extensive computing power that some complex business processes that may not run well in conventional IT environments may be able to run well in the cloud.

My main interest is not cloud computing, however, but rather virtualization, especially virtualization security. A few sessions focused on the difficulty of knowing just how many virtual machines (VMs) are running in virtualized environments. One IT director talked about an audit that was conducted. The auditors found a total of 250 VMs, but the director knew of only 50 of them. Imagine the security (let alone auditing) issues that this “VM sprawl” created! Other sessions focused on the differences between conventional physical networking and virtual networking. Traffic between VMs may travel routes that network and system administrators have never envisioned. Conventional security barriers such as firewalls and intrusion prevention systems might thus not be able to inspect and in some cases block some of the traffic in virtual networks. Fortunately, technology such as virtual firewalls that mitigate this problem has become available in the past few years.

One of the most interesting talks was by the conference chair, Tom Liston. He described his research efforts in trying to discover and exploit a vulnerability in VMware that would allow someone who had access to a guest VM to obtain access to the host VM without authorization. Many virtualization specialists claim that “bare metal” virtualization prevents this kind of thing from happening. He refuted this claim, saying that it is impossible for a virtual machine monitor (VMM) to run directly on hardware–there must be at least some operating system instructions available if the VMM layer is to function. This gives an attacker the ability to identify and exploit vulnerabilities in the operating system, even if the operating system is razor thin. Tom found that operating system instructions were written in assembly language, and that some of the commands were not part of the conventional instruction set, but were rather custom instructions. He discovered some coding flaws that could be exploited; one allowed him to crash one of the guest VMs to gain unauthorized access to the host VM on the same physical machine.

A good part of the second day of the conference dealt with compliance issues. Speakers and panelists described numerous complications that cloud computing and virtualization have presented in achieving compliance with various regulations. They then advocated solutions ranging from the use of certain technology products to forming partnerships with providers and/or writing and enforcing SLA provisions that help ensure that compliance requirements are being met.

Ed Ray and I co-presented a talk in which we asserted that the most fundamental problem with security in virtualized environments is vulnerabilities in virtualization software. We described vulnerabilities that have surfaced in virtualization products such as various flavors of VMware, Denali, and Windows Server 2008 Hyper-V over the years, and said that if you do other things right for security in virtualized environments, but do not create and implement a vulnerability patching process, virtualized environments are wide open to attackers. If you would like a copy of this talk, just send email to seminar@emagined.com.
I entered the virtualization security arena over four years ago, and have found it to be fascinating. I have done my best to learn as much as I can, and I have learned quite a bit over the years, but the SANS Virtualization Summit did more to accelerate my knowledge and understanding than anything prior to it. Good job, SANS, good job, Tom Liston.

Not unexpectedly, a great deal of the content covered in this workshop had to do with cloud computing. Whether we like it or not, many people equate virtualization and cloud computing. The difference in the SANS Virtualization Summit was that the cloud computing talks actually had some substance to them, unlike most of the other talks on this subject that I have attended at other meetings and conferences. My favorite one was by Matt Linton, who described an eloquent network architecture in which security processes were well-integrated into cloud services at NASA-Ames Research Center. In another session Alexander Meisel, CEO of Art of the Defense, pointed out an often overlooked advantage of cloud computing, namely that it can offer such extensive computing power that some complex business processes that may not run well in conventional IT environments may be able to run well in the cloud.

My main interest is not cloud computing, however, but rather virtualization, especially virtualization security. A few sessions focused on the difficulty of knowing just how many virtual machines (VMs) are running in virtualized environments. One IT director talked about an audit that was conducted. The auditors found a total of 250 VMs, but the director knew of only 50 of them. Imagine the security (let alone auditing) issues that this “VM sprawl” created! Other sessions focused on the differences between conventional physical networking and virtual networking. Traffic between VMs may travel routes that network and system administrators have never envisioned. Conventional security barriers such as firewalls and intrusion prevention systems might thus not be able to inspect and in some cases block some of the traffic in virtual networks. Fortunately, technology such as virtual firewalls that mitigate this problem has become available in the past few years.

One of the most interesting talks was by the conference chair, Tom Liston. He described his research efforts in trying to discover and exploit a vulnerability in VMware that would allow someone who had access to a guest VM to obtain access to the host VM without authorization. Many virtualization specialists claim that “bare metal” virtualization prevents this kind of thing from happening. He refuted this claim, saying that it is impossible for a virtual machine monitor (VMM) to run directly on hardware–there must be at least some operating system instructions available if the VMM layer is to function. This gives an attacker the ability to identify and exploit vulnerabilities in the operating system, even if the operating system is razor thin. Tom found that operating system instructions were written in assembly language, and that some of the commands were not part of the conventional instruction set, but were rather custom instructions. He discovered some coding flaws that could be exploited; one allowed him to crash one of the guest VMs to gain unauthorized access to the host VM on the same physical machine.

A good part of the second day of the conference dealt with compliance issues. Speakers and panelists described numerous complications that cloud computing and virtualization have presented in achieving compliance with various regulations. They then advocated solutions ranging from the use of certain technology products to forming partnerships with providers and/or writing and enforcing SLA provisions that help ensure that compliance requirements are being met.

Ed Ray and I co-presented a talk in which we asserted that the most fundamental problem with security in virtualized environments is vulnerabilities in virtualization software. We described vulnerabilities that have surfaced in virtualization products such as various flavors of VMware, Denali, and Windows Server 2008 Hyper-V over the years, and said that if you do other things right for security in virtualized environments, but do not create and implement a vulnerability patching process, virtualized environments are wide open to attackers. If you would like a copy of this talk, just send email to seminar@emagined.com.
I entered the virtualization security arena over four years ago, and have found it to be fascinating. I have done my best to learn as much as I can, and I have learned quite a bit over the years, but the SANS Virtualization Summit did more to accelerate my knowledge and understanding than anything prior to it. Good job, SANS, good job, Tom Liston.

Several days ago some startling news started circulating on the Internet. On August 20, 2008 Spanair flight JK 5022 crashed immediately after it took off from the Marajas Airport in Madrid, Spain, killing 172 people on board. The plane had some mechanical problems, the most significant of which was having its flaps and slats retracted during takeoff. An earlier takeoff attempt the same day was aborted due to mechanical problems, and mechanical problems in the plane had also been reported a few days earlier. A computing system at the airport ran an application designed to report problems such as the ones JK 5022 experienced. Had it worked properly, the pilot would almost certainly have aborted the takeoff, but it did not. The El Pais, a daily newspaper in Spain, claims that the central computing system was so infected with Trojan horse programs that it did not function properly.

This news item needs to be interpreted with caution, as there is no conclusive evidence that the central computing system in question was infested with malware. An investigation into the incident is underway, and findings are due to be released by the end of this calendar year. If malware infestation turns out to be the cause of this unfortunately incident, however, the incident would be a landmark in information security in that this would be the first time that a security-related cause would be at the root of a massive catastrophe that caused many deaths.

Security letdowns have resulted in deaths before, however. In the 1980s a PC in a medical center that controlled the amount of radiation delivered to cancer patients undergoing radiation treatments malfunctioned because of a virus infection, causing it to deliver far too much radiation. Fortunately, only one fatality resulted from this malfunction, but even one is too much. Additionally, a control system at a textile plant that controlled a carpeting rolling machine malfunctioned, causing two employees to be suffocated because they were caught up into and trapped in a carpet roll. Analysis of the control system later revealed that a malicious insider had tampered with it.

If malware turns out to be the cause of the Spanair crash, the incident will serve as a compelling wake-up call that people will not be able to ignore. When security problems turn into human fatalities, even the most security-hostile managers will be forced to ask questions such as “could something like this happen in our organization if we do not adequately control security risk?”

Incidentally, the “blame game” in this crash has taken an interesting twist. A plane mechanic who checked the plane before its takeoff attempt and a maintenance chief are being investigated. They are likely to face manslaughter charges in the future. But how would individuals who perform the roles they do detect malware on the central computing system in question? Did anyone even think of running anti-malware tools on this system? Was the Spanair information security staff aware of the risk of malware running on this system and did the staff make recommendations for mitigating this risk? We do not presently know the answers to these questions, but I suspect that in time we will. Meanwhile, the Spanair incident shows just how important it is to anticipate events of this magnitude and to ensure that processes that address the associated risks keep risk down to an acceptable level.

I have previously argued that IDSs (especially signature-based IDSs) in and of themselves are of little value anymore. They “see” only what they are configured to “see,” a fraction of the multitude of events that occur within networks and hosts. But they can at least provide one source of input for attack identification. Event correlation based on input from multiple sources, IDSs very much included, is necessary if organizations are going to be more proficient in identifying attacks. Security Information and Event Management (SIEM) technology is thus in theory ideally suited for this purpose, but as if have said so many times before, not all SIEM tools are equal. Only a few have excellent event correlation capability–the rest are essentially nothing more than log data aggregators and managers that provide little if any help whatsoever in incident detection.

So now I have a new concern regarding IDSs–using them can delude people into believing that they are really informing them about attacks that have occurred, when they in reality are not (unless, of course, they are being used as data reporters in event correlation). Just look at how long it took TJX, Heartland Payment Systems, the University of California-Berkeley, and UCLA to detect their recent massive data security breaches. I am confident that too often information security and IT staff members deploy IDSs without attempting to benchmark them for their proficiency (or lack thereof) and to estimate the amount of residual risk associated with them. These people somehow need to be educated concerning the limitations of today’s IDSs as well of the need to benchmark them and perform residual risk analyses on them.

At the same time, we need to wake up to the fact that even if an organization has good event correlation capability, organizations still need to share data about attacks with each other to improve the ability to recognize attacks. The US Government set up numerous Information Sharing and Analysis Centers (ISACs) in an attempt to achieve this and other goals. The Financial ISAC, for example, promotes sharing of security-related information within the financial sector, whereas the Energy ISAC facilitates information sharing within the energy section. Although ISACs have a way to go, they have at least established the value of such centers in cooperative information sharing that leads (among other things) to incident recognition.

Intrusion detection methods need to evolve as attacks and detection evasion methods get better. Right now event correlation and cooperative information sharing currently provide the best results, but other, more proficient methods will certainly surface in the future.

Now comes Donn Parker, one of the preeminent researchers and thinkers in information security, who may have taken the idea of the death of risk a little bit too far. Parker writes in a recent journal article,

“Think of security as a necessary overhead cost of doing business just as are facilities management, legal, audit, human resources, payroll, and accounting, and like the other overheads, it does not produce a return on investment (The return of savings from expenditures for security is unknown since the incidents that would have caused the savings did not occur.) I suggest that you gradually limit the extent of your risk assessments and reporting to meet only the minimum requirements of the law and regulations and remove the word “risk” from your writings, job titles, and job descriptions.”

[emphasis mine] Aaaaaaacckk! (that was me running screaming from the room!).
The article appeared in last month’s issue of The ISSA Journal, volume 8 – issue 7, page 12, entitled “Our Excessively Simplistic Information Security Model and How to Fix it.” The main point of the article is Parker’s proposal to expand our traditional “CIA” model (confidentiality, integrity and availability) to the new “Parkerian Hexad” in which he introduces three additional attributes or objectives of security, utility, authenticity and possession, explains why these three new attributes cover things that weren’t covered before and how they complete the model of information security. Parker also adds many new types of controls (yes, Virginia, prevention, detection and correction are not enough anymore). And he also proposes new objectives for information security, including: avoidance of negligence; an orderly & protected society; compliance with laws, regulations, and audits; ethical conduct; and successful commerce; and competition. The problem is, these new objectives replace risk reduction. Unfortunately, “removing risk” from the definitional model of information security is impossible and absurd and to advocate it throws our whole profession on its ear.

There are many things in Parker’s article that I agree with. The three new attributes he proposes can be shown to relate to “CIA” in a very complementary way. As the velocity of information through our systems and networks continues to accelerate, we will very likely see more instances in which a breach in possession occurs when confidentiality is unaffected; availability will be fine but utility will be in the impaired, and integrity will be intact but authenticity will be revealed to be bad. Parker’s point here is we have oversimplified the things we are striving for, and in so doing have missed important elements. However, Parker’s arguments against risk are themselves simplistic and rhetorically throw the baby out with the bathwater. He mostly rails against the calculation of aggregate risk for a company or organization and notes, quite correctly, that the actual risks from disparate and ostensibly independent threats may be in many cases highly correlated yet we have no idea exactly how they interrelate.
Parker correctly exposes the computation of aggregate risk for a company – or, for that matter the combination of precise risk results from uncharacteristic threats (say, the sum of the ALEs from airplane disasters and earthquakes) – as at best intellectual adventures or, worse, dangerous illusions. But “remove risk from your writings”? No way does that make sense.

In the late 1970s Alfred Kahn, an economist from Cornell University, once intemperately referred to a potential outcome of unwise policy as likely to lead to a recession, or a deep depression. White House advisors to President Carter objected and thereafter Kahn promised never to refer to a “recession” by name. He later jokingly said to reporters that while he could not comment on the likelihood of an economic downturn, he could talk about a “banana.” Later, he changed the fruit to a kumquat after a large banana company complained (I’m not making this up…). What are we to do without “risk” to talk about? The ideas of active acceptance of risk, residual risk, and inappropriate risk that we have worked so diligently to nurture, are good and we benefit from that common language. Risk is important…no, it is crucial. Without it we are simply mumbling about the abstruse. If I start talking about an orderly society and ethics, as Donn Parker argues in his article, I’ll use up all of my boardroom time slot explaining terms, then my controls proposal will get voted down because no one will have a clue what I’m talking about. In the movie “The Blues Brothers” Egon describes the elevated amount of psychokentic energy in New York as a, “Twinkie 35 feet long weighing approximately 600 pounds.” When your latest pen test report discloses multiple severity one exposures and shocking unpatched vulnerabilities, instead of referring to this as inappropriate risk, you can quote Winston Zeddemore from the movie,

“That’s a big Twinkie.”

In part 2 of this blog, I’ll talk about why I think Risk has an undeservedly bad name, what problems have emerged by a careless use of risk terms and how to deal with all of this in a way that helps your program. If you are busy “removing risk” from your writings, at least turn “Track Changes” on so you can get it back later…

The recent Gulf of Mexico catastrophe is only one of a number of highly-related incidents involving BP’s failure to mitigate operational and safety risks. You may, for instance, have read about the explosion at a BP refinery in Texas in 2005 that resulted in the deaths of 15 workers and injuries to another 170. In 2005 the Occupational Safety and Health Administration (OSHA) ordered BP to pay a fine of $21 million for numerous safety infractions at the refinery. This fine was increased to $50.6 million last year because of BP’s failure to implement safety measures mutually agreed upon by BP and OSHA, and at the same time BP was ordered to set aside at least $500 million more to implement safety and other measures that the (OSHA) has determined are still missing. Labor Secretary Hilda Solis went so far as to say that the amount of these fines parallels “BP’s disregard for workplace safety and shows that we will enforce the law so workers can return home safe at the end of their day.” OSHA has in fact over the years cited BP for many hundreds of safety infractions in Texas and in other locations; recently BP had to pay another $5.9 million dollars for the most recent round of these infractions. Remember, too, the BP oil spill in Prudhoe Bay, Alaska in 2006 involving over 210,000 US gallons of crude oil. In late 2007, BP Exploration-Alaska pled guilty to negligent discharge of oil in violation of the US Clean Water Act and was ordered to pay a fine of $20 million.

With hate talk about BP being very popular nowadays, you might think that I am just following suit, but I really am not. My focus is instead on enterprise governance within BP (or the apparent lack thereof). BP’s executive management must have an incredibly high level of risk tolerance. BP’s goal ostensibly is to bring in the money, lots of it, without adequately dealing with a variety of serious operational and safety-related risks. If catastrophic events occur, BP then uses legal maneuvers, blames its contractors and affiliates to reduce the PR damage, and deploys other tricks in an attempt to reduce legal and regulatory liability and other negative consequences. For all practical purposes BP was caught completely off guard when the Gulf well blew up; apparently BP’s executive management felt that the probability of this or any other well blowing up in the fashion that it did was so small that it did not justify the cost involved in meeting safety standards and creating and testing a disaster recovery plan. When forced to appear before a Congressional committee investigating the oil spill, the then president of BP attempted to put the blame on the contractor who supplied the drilling rig equipment. Hmmm–the BP risk management modus operandi should be becoming very apparent.

Is BP’s executive management stupid? Although there may be ethical issues within this team, on the surface it appears to be anything but stupid. So far BP has had to pay a total of less than $600 million in fines over the years, but in the first quarter of this year this company was making $93 million each day. Risk mitigation costs should never exceed risks. Face the facts–BP’s executive management has ostensibly weighed the costs versus the risks and has decided that risk mitigation is too costly. So BP operations continue without adequate concern for safety of its workers and for the welfare of those who are unfortunate enough to live close enough to BP oil rigs and refineries to be affected by each catastrophic incident.

BUT–BP’s executive management may have failed to consider a very pertinent and serious risk–stock price devaluation. The price of BP stock has plummeted since the Gulf oil spill. Perhaps it is thus time for BP’s executive management team to rethink their risk management strategy.

So what does all this have to do with information security? As I have said before, if an organization has little or no enterprise governance, it is nearly impossible for an information security program to achieve high levels of governance. If BP’s executive management tolerates the amount of risk that it does and if safety and the welfare of the public is of little or no consideration within executive ranks, whoever the CISO of BP is must not be getting far in managing information security-related risk. Support of executive management is critical in obtaining the authority and resources needed to manage security-related risk to a truly acceptable level. I wish the BP CISO lots of luck, but it might be a good time for this person to do a resume’ update.

I’ve mentioned in previous blog postings that responsibility for fraud-related losses is different in the U.S. from the way it is in Europe. In the U.S., banks and merchants absorb the loss when customers are defrauded, whereas in Europe, customers are generally not reimbursed. Faced with the possibility of losing hundreds if not thousands of Euros, Europeans tend to be more cautious concerning banking and other transactions. Additionally, they generally do not rail when additional controls that cause extra work on their part are required. Smart card-based credit cards are commonplace in Europe, as is two-factor authentication in financial transactions in numerous countries. In the U.S., however, relatively few of those who engage in Internet-based transactions have smart chip-embedded credit cards, nor do many of them use two-factor authentication.

The originators of electronic banking and similar transactions did not envision (nor should they have been expected to envision) the pervasiveness of security threats on the Internet today. Had they had any inkling that organized crime in countries such as Russia, the Ukraine, Belarus and Brazil would be so successful in perpetrating Internet-based fraud and also that powerful  tools such as the financial Trojan Zeus would be so widespread and easy to obtain, they might have discontinued their efforts and moved on to something else.

This having been said, what can we do (if anything) to improve the security of electronic banking and electronic transactions in the U.S. and elsewhere? It is not likely that U.S. banks and merchants will require users to use increasingly stringent controls for reasons mentioned earlier in this blog entry. I am confident that instead U.S legislation that requires more security in banking and other financial transactions will be passed and go into effect. I do not believe that the federal government will just sit by idly as fraud-related loss figures grow to the point that they threaten to cripple the economy. Banks and electronic merchants will greatly benefit from the legislation–they will be able to tell their customers that any extra inconvenience in electronic transactions is not their fault, but rather the government’s fault.

Sometimes my predictions are right, and sometimes they are dreadfully wrong, so stay tuned. All I know is that when it comes to electronic transactions, we cannot keep going the way we are currently going. Something has to give. And, unfortunately, if and when users have to use more stringent security controls in banking and other transactions, the world of organized computer crime will invent new attacks that will keep them in the driver’s seat.

First, SANS offers the advantage of allowing certification candidates to schedule the date and location of the exams they must take. This is extremely advantageous to examinees, as it allows them to take an exam when they are ready to do so–but with a few constraints, too. For example, those who sign up for the GSLC exam when they enroll in the GSLC course have four months from the time they took the course to take the exam. I really do not like mass exams that must be taken only on certain dates and times and at limited locations–especially as I grow older. So I very much appreciated being allowed to schedule the exam on the date of my choice.
At the same time, I found the testing conditions to be a bit strange. I showed up at a testing center in Fremont, California, showed two pieces of identification, answered some questions, surrendered my cell phone, and was seated in a room that looked like a bullpen. I had four hours to take 150 questions. Several other people were already there. Presumably, they were taking some kind of exam, e.g., perhaps even a SANS certification exam, and, fortunately, talking was forbidden. Mounted in the ceiling were many video cameras that taped everything everyone did. No eating and drinking were allowed, and the fact that I had to take the test over lunch (10 a.m. – 2 p.m.) resulting in my being really hungry and thirst as the testing progressed. But I could understand why these measures were in place–SANS is doing everything it could reasonably do to keep people from cheating, and for that I take my hat off to SANS.

The testing was computer-controlled. Once I entered an answer, there was no going back. I was allowed to skip questions about which I was not immediately sure of the answer, but then I ran into a wall–I was allowed to skip only five questions. Frankly, I would rather have taken a paper and pencil test, one that allowed me to skip any number of questions and then go back to them if I so chose. But there was also an unfamiliarity factor that influenced my perception of the exam. The only other tests I have ever taken have been paper and pencil tests. Still, unfamiliarity aside, the process of taking the GSLC was nevertheless quite manageable and reasonable.

Almost without exception every test item was written clearly, something that I very much appreciated. Honestly, I do not think that I have ever taken any kind of test that had such a high proportion of well-written items. I knew many answers right away. The fact that I have taught the GSLC course so many times was highly advantageous, yet some questions forced me to think really hard.

Something else that made an impression on me is that I found out my score the minute I completed the test. There is something about instant feedback that I very much like. Normally, a person who takes a certification test does not learn about the results until two or more months afterwards–something that I consider a major flaw in the testing process.

The bottom line here is that they say you cannot teach old dogs new tricks. I am an old dog–a grisly, battle-scarred veteran of the information security arena. I am even one of those older generation individuals who eschew Facebook and Twitter. Yet when all was said and done, I must admit that what was initially a rather strange experience turned out to make sense and to not be so strange after all. Perhaps old dogs can learn new tricks after all. And, oh by the way, the fact that I got a good score (remember–I very frequently teach the GSLC course, so if I hadn’t, sometime would have been seriously wrong with me!) may also have had something to do with my final perceptions.

The most important single consideration is the reputation and trustworthiness of the service provider. When an organization commissions a pen test of its networks, hosts and applications, it is in many ways opening its IT environment to whoever is chosen to perform the test. This is why I strongly advise not hiring any so-called “hacker” or a (hopefully) “recently reformed” member of the black hat community. Someone who has shown bad judgment by making a living based on violating laws and crossing far past professional ethical standards is not going to suddenly become a person of good judgment, even if that person sincerely means to change. I am aware of dozens of cases in which an organization hired a “hacker” to perform pen testing, only to greatly rue its decision. In one case, an organization that hired a “hacker” cut the pen test short. The “hacker” retaliated by launching a massive denial of service attack against the organization’s network, causing a prolonged and costly outage. In another case, an apparently much better intentioned “hacker” made a series of honest but naïve mistakes that repeatedly crashed routers and switches within the customer’s network.

Another consideration is proven excellence in pen testing. This is why organizations that have obtained outstanding pen testing services from a provider would do well to stay with that provider. If the quality of a potential pen testing service provider is unknown, the provider should be able and willing to provide references that the potential customer can contact to determine how well the service provider has performed for them in the past. The major limitation of using references is that service providers tend to provide only the names of highly satisfied customers. Still, when the quality of pen testing services is in question, having references to contact is far better than nothing. But the best method of determining how good a candidate service provider’s services are is to set up a virtual, non-production IT environment with a few exploitable vulnerabilities, define the rules of engagement, and then turn the potential service provider loose for an agreed period of time. The candidate pen tester’s skill level will rapidly become obvious. An increasing number of organizations is using this strategy, often to competitively pit service providers against each other to determine which one is the best. Having candidate service providers prepare and submit write-ups is the proverbial topping on the cake–there are so, so many technical geniuses who can perform amazing pen testing, but are seriously deficient when it comes to ability to write. The chosen service provider should not only perform outstanding pen tests, but should also be able to write a clear and technically-detailed report that contains not only a description of the methodology used and the results, but also prioritized recommendations for vulnerability remediation.

Oh, and by the way, I must close by saying that I know that I am biased (please forgive me), but I am extremely proud in that I work for a company that prides itself in its excellence in performing pen testing.

What troubles me is information security staff members within an organization using the results of pen testing as the basis for drawing conclusions concerning the security posture of the organization. So, for example, if an organization with 10,000 hosts discovers that only two moderately severe vulnerabilities can be exploited in only ten hosts, the CISO may be tempted to come to the conclusion that the organization has a very good security posture. I disagree. I would instead say that the organization does very well in vulnerability identification and remediation, but no more–going any farther is not justified. This organization might be grossly deficient in areas such as policy and procedures, may have completely dysfunctional incident response, business continuity and disaster recovery efforts, and may be seriously negligent in identifying, classifying and assigning ownership to critical information. In this hypothetical example, vulnerability eradication might be the only bright spot in an otherwise dismal information security practice.

I would not expect most information security professionals to fall into this erroneous way of thinking about pen test results. Most are aware that although vulnerability eradication is a particularly important part of an information security program, other areas are also critical, as ISO/IEC 27001/2 tells us. The temptation to equate pen testing results with the effectiveness of a security practice is instead strong among professionals whose main focus is technical. Just three weeks ago I was part of a very interesting panel at SANS-Rocky Mountain that consisted of fellow SANS instructors. Two of the panelists, both gurus, maintained that good pen testing results indicate a strong security posture. I argued for the view I have taken in this blog posting. I was encouraged when these brilliant individuals softened their view about the relationship between good pen test results and security posture as time went by. They started to realize that many other factors should be considered when one is deciding what the security posture of an organization is.

I’ll go farther by saying that many times highly technical people fall into the trap of thinking that the only vulnerabilities that exist are in operating systems, networks, applications and databases. Many other types of vulnerabilities exist, however. Unlocked server rooms or server rooms that have push button locks with never-changed default combinations are also examples of critical vulnerabilities. The same applies to users’ susceptibility to social engineering attacks and user error in following security procedures. Come to think of it, there are actually far more security-related vulnerabilities outside of the world of computing than within it. Almost anything that concerns human beings is a potential vulnerability, and we know that humans comprise the greatest threat vector.

So just to set the record straight, pen test results are just that and no more. They are extremely valuable because patching vulnerabilities is one of the most critical things an information security staff can do. But they in and of themselves do not provide a very complete picture of the goodness of an information security practice.

The most frequently used type of pen testing is external pen testing in which the pen testers launch attacks against hosts, devices and applications within an organization’s network from the outside. The pen testers typically encounter security barriers such as perimeter firewalls and network address translation (NAT). Success in penetrating systems is thus often limited. The same is not usually true for Web applications, however, because they by necessity must generally be accessible by anyone. Many of these applications have at least several externally exploitable vulnerabilities. Externally-launched pen tests are valuable in that they show how vulnerable an organization is to Internet attacks. At the same time, however, because of the previously mentioned barriers, they generally uncover only some of the vulnerabilities that exist in hosts, devices and applications. There is more work to do if an organization wants a more comprehensive and genuine view of all the vulnerabilities that exist.

Another kind of pen test, an internal pen test, starts from within an organization’s network. The testing team is normally given access to several parts of the internal network. Because there are no firewall, NAT complications, and other barriers in this kind of testing, more vulnerabilities in more hosts can be tested, yielding a more complete picture of the range and number of vulnerabilities within the organization’s IT environment.

Sometimes pen testing is conducted in connection with a mandated external audit, e.g., one in connection with determining whether an organization is compliant with a regulation such as FERC/NERC. In this case pen testing often uncovers a plethora of vulnerabilities, prompting the information security manager to complain that the test was unfair and that it should have been limited to testing from the outside. I strongly disagree. With security perimeters becoming less effective over time, making the assumption that attackers are able to attack only from outside a network is specious. Conducting testing by starting within the organization’s network is fair and realistic. But conducting both an external and an internal pen test is the better way to go if a more complete view of vulnerabilities is to be achieved.

But simply conducting an external and internal pen test still does not provide a complete picture of vulnerabilities that exist. Many of today’s attacks target Web browsers, which have become one of the major causes of security breaches. An even more complete pen test will thus also include testing of susceptibility of browsers (and possibly also users who use them) to downloading malicious content. Additionally, the possibility of an insider or sometimes even an outsider walking up to a machine and gaining unauthorized access or installing a physical sniffer or rogue code either clandestinely or through social engineering if the user is present is ever present. A really thorough penetration test would thus also include having pen testers attempt to gain physical access to systems.

How much pen testing is enough? As in most cases in information security, the answer is that it depends. In some cases, an external test is sufficient for the purposes for which it is conducted. In others, only a very complete test will do. The more complete the testing, the higher the cost will be, but once again, you get what you pay for.