Home > Uncategorized > Interpreting Information Security Research Results

Interpreting Information Security Research Results

An abundance of information security research is performed every year. Surveys that measure the foci and activities of information security practices, funding allocated to IT security, types of security controls that are being used, attitudes concerning compliance, number and types of incidents that have occurred, and amount of incident-related financial loss are just a few of the many that are taken. Large organizations such as the Computer Security Institute and ISACA and corporations such as the Big Four accounting firms are particularly likely to conduct these surveys. No matter what the year is, results generally indicate that funding and staffing are never sufficient, that senior management is prone to overlook information security-related risk, that the cost of security breaches is growing, and that certain types of security-related technology is used more widely than others.

The fact that so much information security-related research is conducted is a good thing, but too often the way the research is conducted and the results of such research are interpreted greatly troubles me. For example, suppose that results of a study indicate that the amount of funding for information security and the amount of security breach-related losses are inversely proportional to each other, that is, the more spending, the lower the losses, and vice versa. Too often the conclusion drawn is that spending more money on security produces results in less financial loss due to security breaches. This conclusion may make sense to individuals who do not know about scientific research, but it is completely specious to those who do. Controlled experiments were not conducted, and as such, causative conclusions cannot be drawn, no matter how high the absolute value (positive or negative) the correlation coefficient turned out to be.

Similarly, many studies compare two or more groups of individuals, organizations, security practices, or other entities. Results typically show that one group of individuals, organizations, or security practices scored higher on one or more measures than did others. For example, I am familiar with several studies, the results of which show that those who engage in black hat activities are more anti-social and introverted than others. The problem with so many of these studies is that the subjects of these studies have been chosen by the researcher or may have preselected themselves by volunteering to participate in a survey that was posted on a public Web site rather than being randomly selected. As such, being able to derive valid generalizations from the results is impossible, because the results may have been due solely to selection factors.

Another gripe I have concerning much of the research conducted in the information security arena is using far too small sample sizes. Incredibly, I have seen grandiose claims based on research in which only 50 or 60 individuals were involved in a study. Again the problem is being able to derive valid generalizations from the results.

The information security arena has some excellent certifications, but curiously none of them tap being able to adequately interpret information security-related research results, something that competent information security professionals need to be able to do. This oversight needs to be corrected—the sooner, the better.

Categories: Uncategorized Tags:
  1. No comments yet.
  1. No trackbacks yet.
You must be logged in to post a comment.