AV Software Testing: Caveat Emptor
August 8th, 2011
One of the platitudes with which people attending information security awareness programs quickly become familiar is “run AV software and keep it updated.” Informing users accordingly is important. Following up to verify that they engage in actions corresponding to these platitudes is every bit as important. But are the same platitudes preached in the 1990s and early 2000s concerning AV software still true?
AV vendors are ostensibly on a roll. They sell millions of copies of their products. At the same time, they too often find a friendly “pay-for-play” organization that will certify their product. In the case of AV products, certifying organizations are likely to certify products as being highly efficient (allegedly 98 – 100 percent) in detecting and eradicating viruses, worms and Trojan horse programs. Unfortunately, the testing process conducted by certifiers too often leaves much to be desired. Testing is too often based on a selected (but not random) subset of well-known but not validated malware. In some cases, the testing organization pre-notifies the to-be-tested vendor of the malware to be used in the so-called “tests.” This gives the vendors ample time to update their software to be able to detect malware that would have not been detected had the vendor not been pre-notified of the malware used in the test.
NSS Labs, which is rapidly becoming my favorite when it comes to independent testing of information security products, recently conducted comprehensive tests of AV products. The results do not at all correspond to the typical “pay-for-play” certification results. NSS Labs tested many products, including F-Secure, Kaspersky, AVG, Eset, F-Secure, McAfee, Norman, Sophos, Symantec, Panda, and Trend Micro. Based on a creative set of methodologies in which new, previously unknown malware was used, NSS Labs found that malware detection and eradication proficiency varied between 67 and 96 percent. Some of the best selling tools did not fare very well. Trend Micro scored the best overall.
Having worked for a software vendor for a number of years in the past, I must admit that I have lost a lot of confidence in information security software vendors. A mediocre (or even worse) product can get rave reviews if the vendor is willing to spend enough money. If an organization such as Gartner jumps aboard a vendor’s proverbial bandwagon (often after the Gartner staff member in charge of the vendor’s upcoming Magic Quadrant rating has performed consulting services for the vendor in question–a very obvious conflict of interest) without performing systematic and detailed testing, heaven help us all.
So I have two main points here. Most importantly, I urge you to use AV software as part of a defense-in-depth control scheme. Even though AV software is not nearly as proficient in detecting and eradicating malware as most vendors claim, it at least does some good. The second is that you should use not just any AV product, but rather a product that is as proficient as possible in detecting and eradicating malware. Achieving defense-in-depth with mediocre products does not work very well. Low barriers, no matter how many, are easy to defeat. Check NNSS Labs’ results for help in selecting a suitable AV product.
A great deal of deception concerning information security products exists. I urge you to not fall prey to this deception. Product certifications may or may not be valid, but in my experience more are invalid than valid. Organizations are all too quick to take in money and then quickly pronounce a product to be certified. But proclaiming a product as certified may or may not correspond to reality. So whether or not the issue is AV products, caveat emptor.