There is a vision for the future of assessing cybersecurity: The goal is a system of cyber metrics that are transparent, auditable, practical, scalable and widely agreed upon. To that end, it is useful—indeed, imperative—to evaluate various approaches to cyber risk quantification with the aim of informing the development of a public standard for measuring cybersecurity.
As I’ve noted before on Lawfare:
[W]hen governments, commercial actors and private citizens think about new deployments of cybersecurity measures, they either explicitly or implicitly balance the costs to be incurred (whether monetary or nonmonetary, this includes disruptions caused by changes to the enterprise and the resulting, temporary, reductions in efficiency) against the benefits to be derived from the new steps under consideration. And yet there are no generally accepted metrics by which to measure and describe cybersecurity improvements. …
One group of expert practitioners says that the quest for good cybersecurity metrics is a phantasm. Another, equally confidently, asserts that the problem has already been, to a large degree, solved. Meanwhile, most people in the middle don’t know the answer to the question and have only begun to think about how the answer might be defined. Which, of course, is why the question needs to be asked.
This post is the first in a series that will look at the problem of cyber metrics from different perspectives. The goal of this series is to lay out in understandable terms various ways in which metrics might be developed and to assess their respective strengths and weaknesses.
A Thesis: Outside-In Measurement
Most ideas about cyber metrics conceive of the assessment as one that works from the inside out. The thought is to conduct an internal analysis of an enterprise based on some combination of assessments of its governance, its processes, and the ways in which it implements security solutions. To a large degree this is an attractive and familiar construct. It is modeled on how we normally think about assessing other areas of concern such as the environmental compliance, health or safety of an enterprise. In regulatory America, any enterprise would be experienced with this audit/assessment/compliance construct.
The inside-out construct, however, suffers from a number of limitations. First, it requires internal access to the enterprise in order to be assessed—access that is often difficult to achieve for legal, policy, social or political reasons. Second, the method often requires some form of external standard-setting by the government. And, finally, the audit/compliance system is cumbersome, adversarial and frequently difficult to scale. For example, the IRS currently audits less than 0.6 percent of all returns—hardly an effective measure of tax compliance.
So, the question arises—can a reasonable model to assess cybersecurity from external information be developed? Put another way, are there market indicators that can advance the public’s understanding of the cybersecurity of an enterprise?
The answer to that question leads to a set of second-order inquiries: What steps are necessary to enable those market indicators to function appropriately? Should the government have a role in creating greater transparency on how well or poorly companies are managing cyber risk? Would a public standard and mandated disclosure be valuable?
The Markets and Cyber Risk
The Securities and Exchange Commission (SEC) noted—in a 2018 statement—the increasing risk of cybersecurity incidents for investors. As the SEC put it: “[G]iven the frequency, magnitude and cost of cybersecurity incidents, the Commission believes that it is critical that public companies take all required actions to inform investors about material cybersecurity risks and incidents in a timely fashion, including those companies that are subject to material cybersecurity risks but may not yet have been the target of a cyber-attack.” As two observers, including a former SEC enforcement officer, wrote in the Harvard Business Review:
[M]ake no mistake, the SEC is now viewing cybersecurity risks as it does all other economic and business risks—particularly as it relates to internal controls, financial reporting, and requisite related public disclosures. The SEC certainly is calling for a meaningful change in the approach corporate boards have been taking with respect to cybersecurity oversight and the discharge of their governance responsibility over this core area of enterprise risk.
As this suggests, there is at least a possibility that cyber risk can be treated as any other business risk—quantified by the market and made the basis of analysis and investment. It also suggests, however, that existing structures for market transparency are inadequate. Pick almost any SEC Form 10-K—the annual financial report required by the SEC for every company—at random and it will have some form of cyber risk disclosure along the following (representative) lines: “Significant data breach or other interference with our technology infrastructure could disrupt our operations and result in the loss of critical confidential information, adversely impacting our reputation, business, or results of operations.” This is pablum so anodyne as to offer nothing of any value to any discerning investor.
We do know, from recent experience, that—at least with respect to postbreach analysis—the markets are generally pricing cyber incidents more severely than enterprises are in their public disclosures. When Equifax suffered a breach, it put its estimated balance sheet loss at $1.4 billion. But its actual loss in shareholder value relative to its peer organizations at the six-month postbreach mark was closer to $4 billion. Likewise, Boeing’s estimated balance sheet loss from a recent breach was $8 billion, but its actual market loss was closer to $20 billion. Whatever the mechanism, it is clear that the financial markets are assessing cyber losses in a way that is very different from how enterprises themselves make their assessments.
Hedging Risk
If markets are assessing losses postbreach, it is plausible that they can likewise assess risk of loss—a proxy for cyber(in)security—before the losses occur. What is necessary, and perhaps possible, going forward is a more sophisticated method of discerning cybersecurity status from other metrics. The objective is to be able to hedge against the risk of cyber-insecurity in the same way that the market hedges against any other catastrophic risk. In the past 40 years, the financial markets have gone through two phases of growth—one from the globalization of the economy and one from increased securitization. Both times the markets evolved mechanisms for quantifying and hedging against the changed risk.
Today, we are in the throes of a third broad transition to the digitization of much of the economy. The market now has significant value derived from digital assets that are at risk—a trend that is only accelerating during the coronavirus pandemic. Yet this relatively new form of value is subject to much less consideration than traditional assets.
In the aviation sector, one form of asset (airframes) is subject to stringent regulation via airworthiness standards, and that asset is insured against loss, while the capital markets hedge against failures in the enterprise or downturns in the industry. This is obviously limited by capability and experience, however, as there was no hedge against the coronavirus. But other forms of valuable assets—information about their customers, their products and their operations—are not similarly quantified, insured and hedged.
If, as suggested, this path has been trod before, that realization also suggests that hedging cyber risk can be done using traditional methods. Managing technology risks in financial terms will require updating outdated accounting standards that are not directly applicable. But that should be feasible.
In short, theoretically it should be possible to create a predictive index for an enterprise that assesses potential shareholder value change as a result of cyber incidents. This measure should be effective in assessing security from the outside even though, according to Mandiant’s 2020 report, the majority of actual breaches remain undetected. That same metric reasonably would relate to operational performance in the enterprise and to financial risks including extreme possibilities like credit downgrades or defaults.
Stemming the growing scale of cyber-related financial losses and better managing this systemic risk, however, requires more than new accounting standards. What is needed is a mix of market incentives and, probably, some form of standard-setting or regulation. Historically, when the markets looked to manage the risks associated with globalization, market incentives such as foreign exchange derivatives became indispensable. Cyber risk is no different. Consider how much companies would improve management of digital assets if they not only had to disclose how well or poorly they were managing them but also were rewarded for managing them well. As with any macro issue of the past, the challenge is finding the right mix of incentives and regulation that will prompt the change in behaviors required to strike the balance between growth and security.
Imputing Cybersecurity Risk
So what might that mix look like? What might be the data that would form the base of our metric of security? What fields of information exist that might be available to external observers to provide a good basis for assessing the cyber risk of any particular enterprise? For example, if Standard & Poor’s were to start issuing cybersecurity ratings as they currently do financial security for bonds, what factors ought they to consider?
Though much of this remains speculative, we are already beginning to see the possibility that certain externally measurable factors correlate with good—or poor—cybersecurity practices. Here are some example of candidate metrics that, when appropriately weighted, are relevant to the security of an enterprise:
● To what degree, if any, does the enterprise exchange network communications using Tor connections—or another anonymized internet routing system?
● Is a large volume of spam email propagated to a large number of enterprise recipients? Conversely, has an enterprise taken measures to prevent such activity?
● What is the size of the enterprise’s external network—based on IP addresses that are visible and detectable to the public?
● Are private internal subnet IP address records accessible to the public?
● What proportion of vulnerabilities or misconfigurations detected in the past quarter has been remediated?
● To what extent, if any, is darknet traffic entering or originating from the enterprise?
The list could go on. In a less mechanistic way, one could also focus on spend rates and publicly reported purchases of cybersecurity tools—provided those are appropriately adjusted for the size of the enterprise, the nature of its information technology platform, and so on. Whatever the metrics, the idea would be that a rating company or agency could differentiate—with a high degree of confidence—between enterprises that were evaluated at “5 Star” and ones that were evaluated at “1 Star.”
To see one example of this phenomenon, consider the indices created by Cyberhedge, which appear to successfully differentiate market performance just on this basis, and even seem to have a predictive value in anticipating the possibility of future breaches from poor cyber governance ratings. [Full disclosure: Cyberhedge is a donor to the R Street Institute, where I serve as a senior fellow.] If such a model proves to have long-term efficacy, it would be worth exploring as a component of any cyber metrics project.
In any event, the purpose here is not to suggest an exhaustive list; rather, it is to suggest the possibility of many competing evaluative measures. Whatever the structure, in a well-developed market more than one rating company will build its own secret sauce for assessing “cyber governance”—just as Moody’s and Standard and Poor’s offer competing ratings for the bond market.
Signs of Movement—The Cyberspace Solarium Commission
Earlier this year, the Cyberspace Solarium Commission issued its final report to Congress. The report was intended to capture recommendations for improving America’s cybersecurity across the entire spectrum of national effort. Two aspects of the report bear directly on the questions I’ve examined in this post.
First, the commission recommended that Congress establish and fund a National Cybersecurity Certification and Labeling Authority. As the commission accurately described the problem, the lack of differentiation among enterprises as to cybersecurity implementation leads to the lack of demand for more secure products and to the inability of investors to have adequate visibility into the operational risks posed by cyber management in those same enterprises. The suggestion here was for the government to designate a nonprofit organization as a labeling authority that would be charged with creating a cybersecurity certification program. Implemented properly, the labeling would be grounded both in process evaluations and in product testing and could, of course, ultimately serve as an external indicator of cyber governance.
Second, the commission recommended that Congress establish a Bureau of Cyber Statistics. This organization would be charged with collecting, processing, analyzing, and disseminating essential statistical data on cybersecurity, cyber incidents, and other aspects of the cyber ecosystem. It would also be in a position to develop and publish new cyber metrics that might measure risk reduction. The goal would be the collection of data that would help to clarify the effectiveness of varying cybersecurity measures in reducing risk to technologies or enterprises, thereby enabling the allocation of resources and the assessment of otherwise concealed risk preferences.
Both of these recommendations, though particularly the latter, could lead to precisely the sort of external metric that could be of value in providing an enhanced ability to assess the security of enterprises. Properly developed, they would provide information that is practical, auditable and scalable to a national system of evaluation. Combined with some of the other metrics I’ve identified, they would certainly be of utility in creating a generalized metric of cyber risk.
Limitations
We should not be overly sanguine about this idea. It is quite promising, but it goes only so far. External measurement of the cybersecurity of an enterprise is likely to be a component, but only a component, of any comprehensive measurement system.
First, and most importantly, this type of measurement applies to only one side of the equation. It may—or may not—prove to be an excellent predictor of the vulnerability of an enterprise and also capture some of the consequences of poor cybersecurity management. And that, by itself, would be a significant improvement over the current state of affairs. But a full risk metric would also incorporate measurements of threat—from nation-state or other malicious actors—to an enterprise. Those factors are outside the scope of the external examination suggested here, which would leave much of that measurement to be done by other means.
Second, a critical component of any external methodology is transparency. Since outside assessments rely—perforce—on information available to those who are outside an enterprise, the entire process can be frustrated by a lack of adequate information. Naturally, most enterprises are likely to seek to maintain the confidentiality of their internal evaluations—and that makes it possible, and perhaps even likely, that critical components of external metrics may be obscured from view.
Any long-term success for this idea of cybersecurity measurement will need to create adequate transparency. Indeed, lack of transparency may, itself, be a measure of insecurity. As a result, greater visibility into enterprise security may occur organically, as the market calls for more information. But it seems likely that some form of governmental compulsory transparency—as suggested by the Cyberspace Solarium Commission—may, ultimately, be required—with all of the political challenges that such a requirement would entail.
Third, and rather obviously, this type of risk hedging works only in a market economy of the sort operated in the United States and it works only to the extent it is designed to assess enterprises that are part of that market. This concept may be effective in assessing risk for enterprises listed on the New York Stock Exchange or in some other market, but the vast majority of enterprises—and quite possibly the locus for the majority of cyber risk—are smaller enterprises that are not exchanged in open markets or subject to the same transparency rules.
Finally, it may be impossible to create an external metric system without some type of affirmative government encouragement or intervention. Perhaps this is not a “limitation” per se but, rather, merely a statement of conditionality. But any system that depends, ultimately, on governmental activity is inherently subject to rent-seeking behavior that may prove problematic.
Conclusion
When all is said and done, this exercise leaves us with several questions that need to be answered for an effective system of external cyber metrics to be developed:
● First, do we need a standard metric, akin to generally accepted accounting principles, or are divergent standards appropriate?
● If there is a standard, who sets it?
● Is there another macro-risk standard that serves as a good example or reference point?
● And, finally, what would an assurance/audit process look like?
These are by no means trivial questions. Nor can one confidently say that they are capable of being answered definitively at this time. But I suspect that they are capable of resolution and, thus, that the search for cybersecurity metrics can be advanced by looking to external evaluation as one component of a larger suite of tools.
Featured Publications
