On metrics and prioritization of investments in hardware security

Abstract

The security risks posed by electronics are numerous. There are typically a variety of risk-reducing countermeasures for a given system or across an enterprise. Each countermeasure is associated with both a level of risk reduction and its lifecycle costs. Given budgetary constraints, risk managers and systems engineers must determine what combinations of countermeasures cost-effectively maximize risk reduction, and what metrics best guide the investment process. In this paper, we seek to answer these questions through exploration of risk reduction metrics from the field of security economics, including the benefit/cost ratio, return on security investment (ROSI), expected benefit of information security (EBIS), and expected net benefit of information security (ENBIS). The results suggest that ratio-based metrics are not strongly correlated with risk reduction, while EBIS is equivalent to risk reduction and ENBIS is equal to risk reduction minus cost.

1 INTRODUCTION

The modern economy, telecommunications, infrastructure and transportation systems, healthcare, and almost every aspect of our daily lives are enabled and enhanced by digital connectivity.¹ This connectivity is made possible by microelectronics. Due to the critical nature of these components and their applications, assurance of the security and trust of these components, and of the global supply chain, is a national priority. A number of sources of risk can arise throughout an electronic component's life cycle. For example, electronic components in the supply chain may be counterfeit.^2-4 There are seven basic counterfeit types: recycled, remarked, overproduced, out of specification/defective, cloned, forged documentation/part substitution, and tampered.⁵ Moreover, malicious Trojans can be inserted into microelectronics that can exfiltrate sensitive data or grant unauthorized access to a system.

There are a number of negative impacts associated with these lifecycle risks. Firms may lose revenue and may incur additional costs through the activities to identify and repair systems that may have been compromised by counterfeits. Legal liabilities, intellectual property (IP) theft, and reputational impacts must also be considered.⁶ Consumers may experience negative consequences as well, with potentially degraded performance of products and safety impacts. For government consumers of microelectronics, counterfeits and the lack of a trusted supplier base can impact the ability to meet national security objectives.⁷ Concerns over the security and trust of components in 5G telecommunications systems have prompted a reevaluation of the nation's global supply chain sourcing strategies.⁸

A number of trends and factors related to security and trust across a system's lifecycle are important for systems engineers to consider. One of the drivers is the extremely high capital costs associated with building and maintaining a foundry (on the order of tens of billions of dollars) which has resulted in horizontal integration and contract manufacturing.⁹ This has given rise to so-called “fabless” firms that focus only on the design portion of the lifecycle, and outsource the manufacturing to a foundry. These foundries are increasingly international. The U.S. share of global fabrication capacity has fallen from 37% to 12%, while Asia has grown to an 80% share.¹⁰ While ideally the foundry is trustworthy and secure, there may be cases where IP is stolen, reverse engineered, or overproduced and resold on the market for a discount, eroding the profits of the IP holder.¹¹

End of life factors are also important for systems engineering, including the consideration of obsolescence.¹² Obsolescence in general can occur for many reasons, including changes in consumer preferences and demand, sunsetting of support, or market entry of a competitor.¹³ For microelectronics, especially in defense platforms, the main driver of obsolescence is the fact that the end product or system tends to have very long lifetimes (on the order of several decades), and the rate of technological innovation of the electronic components outpaces the life of the product or system they enable.¹⁴ This is referred to as diminishing manufacturing sources and material shortages (DMSMS), that is, the “loss or impending loss of original manufacturers of items or suppliers of items or raw materials.”¹⁵ Since there is still demand for the electronics but lack of authentic supply, opportunities for counterfeiting emerge.

Sources of risk can occur during any of the phases of the system's lifecycle, and the task of identifying what specific countermeasures are used to reduce risk and increase security and trust is complicated by the many potential alternatives available. Methods for security and trust of hardware and the supply chain include the use of RFID tags, blockchain-based approaches, logic locking, IC camouflaging, split manufacturing, and watermarking.⁵ While one can take a “defense in depth” approach and implement many safeguards, the question remains as to which ones in particular accrue the most risk reduction benefits at the lowest cost across the “Systems Engineering V”.

Seeking an effective “mix” of countermeasures naturally lends itself to a portfolio modeling approach. A portfolio is simply a set of objects that are considered together.¹⁶ In particular, each security countermeasure is associated with a certain level of effectiveness in mitigating risks in exchange for an installation and/or upkeep cost. Risk management decisions must balance the benefits and costs and other concerns from upper management and organizational departments such as supply chain management, finance, and IT.

However, a number of risk reduction metrics can be used in prioritization of security countermeasures. In this paper, we compare a number of risk reduction metrics, representing various ways to rank order the potential “returns” or “benefits” of investment in hardware security. We explore how such metrics might guide the prioritization of countermeasures for risk reduction.

2 BACKGROUND

2.1 The economics of security

One of the fundamental questions surrounding security is how much should be spent to protect the organization. Spending too little may leave the organization vulnerable, while spending too much is a suboptimal use of resources. After all, “you don't want to secure yourself out of business.”¹⁷ Since budgets are finite, each dollar spent on security has an opportunity cost, and there are competing priorities within an organization that make justifying security investment necessary. Security investments must therefore be framed in terms of enterprise-wide objectives, including the impacts of security (or lack thereof) on productivity and profitability.¹⁸ Additionally, underinvestment in security exposes a firm to potential legal liability and accusations of negligence.^19-21 Ultimately, “economics – not technology – determines what security technologies get used”.²²

The need to justify security expenditures has prompted the burgeoning field of security economics, which posits that traditional economic insights, such as externalities, adverse selection, and moral hazard can be brought to bear on information security problems.²³ A growing body of literature on cyber insurance has emerged,^24-26 as well as innovative market-based solutions such as vulnerability-trading markets and exploit derivatives.^{27, 28}

Notably, the vast majority of research in this area is applied to more “traditional” aspects of cyber security, such as software and network security, rather than hardware. Hardware security, however, suffers from a number of market failures, including information asymmetry, prisoner's dilemmas, misaligned incentives, and free riding.²⁹ Research has been conducted using game theory to detect and protect against hardware Trojans.³⁰ However, in general, the research on the economics of security investment for hardware security and trust is an undeveloped area.

The following sub-sections describe common metrics used in security economics.

2.2 Return on security investment

An area of research within the field of security economics is the quantification of the return on security investment (ROSI). One of the main difficulties in justifying security investment remains the fact that the “return” is quantified as avoided costs rather than additional profits. Any ROSI measure must be framed as being financed through borrowing against the costs of future losses.³¹ ROSI seeks to define a return on investment (ROI) measure based on the reduction of an annual loss expectancy (ALE). The ALE was proposed as a risk metric in the “Federal Information Processing Standard (FIPS) 65, Guidelines for Automatic Data Processing Risk Analysis”,³² and is equal to:

$$$$\begin{equation}{\rm{ALE\ }} = \mathop \sum \limits_{i = 1}^n I\left( {{O}_i} \right){\rm{*}}{F}_i\ \end{equation}$$$$(1)

where I(O_i) is the impact of harmful outcome O_i in dollars and F_i is the frequency of O_i. ROSI has been applied to case studies such as software engineering^{33, 34} and cloud migration.³⁵ Rosenquist documented cost savings at Intel, through the use of ROSI metrics in their business decision making, of $18 million annually.³⁶

As discussed by European Network and Information Security Agency (ENISA), “security is not usually an investment that provides profit but loss prevention”.³⁷ The return from a cyber countermeasure should be calculated based on the amount of loss avoided, or how much is expected to be saved. The ROSI metric is used to quantify the benefit, or return, from investments in security countermeasures (European Network and Information Security Agency, 37).

To calculate ROSI, first the single loss expectancy (SLE) must be estimated, which is “the expected amount of money that will be lost when a risk occurs”.³⁷ In this approach, SLE can be considered as the total cost of an incident assuming its single occurrence. The SLE can be described as the product of the Asset Value (AV) and the Exposure Factor (EF):

$$$$\begin{equation}SLE\ = \ AV*EF\end{equation}$$$$(2)

where AV is the overall cost of the damaged assets from the cyber-attack, and EF is the percentage of the AV that would be lost.³⁷

To determine the annualized return of each cyber mitigation, the ALE is the product of the SLE and annual rate of occurrence (ARO), a measure of the amount of times the asset is attacked each year. The ARO is defined by a frequency of occurrence and not a probability of occurrence.³⁷ In addition, it is worth noting that these metrics are typically defined by annual statistics. A significant amount of variance often exists in regards to the true number of annual incidents.

$$$$\begin{equation}ALE\ = \ SLE*ARO\end{equation}$$$$(3)

Finally, the ROSI is calculated as a simple ROI by dividing the net gains (initial investment subtracted from loss prevented) by the cost of the initial investment. This is reflected as follows:

$$$$\begin{equation}ROSI\ = \frac{{Annual\ Savings - Annual\ Cost}}{{Annual\ Cost}}\ = \frac{{ALE - mALE - Annual\ Cost}}{{Annual\ Cost}}\ \end{equation}$$$$(4)

where ALE is the annual loss expectancy without the cyber defense countermeasure, and mALE is the new annual loss expectancy with the countermeasure implemented.³⁷ To find the mALE, one can define the Mitigation Ratio (MR) as the percentage of attacks successfully thwarted with the mitigation applied. Noting that

$$$$\begin{equation}mALE\ = \ ALE*\left( {1 - MR} \right)\end{equation}$$$$(5)

one can rewrite our ROSI formula as:

$$$$\begin{equation}{\rm{ROSI\ }} = \frac{{ALE*MR - {\rm{Annual\ Cost}}}}{{{\rm{Annual\ Cost}}}}\ \end{equation}$$$$(6)

2.3 Expected net benefit of information security

A leading model within the field of security economics is the Gordon–Loeb model.³⁸ Rather than taking a cost minimization approach, common in risk management, in which the sum of the expected losses and the security investment are minimized (Figure 1A), the Gordon–Loeb model takes a benefit maximization approach, where the benefits are expressed in terms of risk reduction. The authors define an expected loss function as a product of the monetary loss given a security breach, the probability of the threat, and the vulnerability of the asset, which is the probability that the threat will be successful, given that it is realized. By investing resources into information security, the value of the vulnerability probability can be decreased. Assuming that as security investment increases, security increases at a diminishing rate, Gordon and Loeb define the expected benefits of investment in information security (EBIS) as the difference between the vulnerability of the asset before and after security investment, multiplied by the expected loss (i.e., the threat probability multiplied by the monetary loss). The expected net benefits from the investment in information security (ENBIS) is the EBIS less the amount of money invested. The approach of the Gordon–Loeb model is shown in Figure 1(B).

FIGURE 1

Economic risk management approaches. (A) (left): Cost minimization strategy. (B) (right): Benefit maximization strategy based on the Gordon–Loeb model.

Specifically, the ENBIS is calculated as follows:

$$$$\begin{equation}ENBIS\ \left( z \right) = \left[ {v - S\left( {z,v} \right)} \right]\ t\lambda - z\end{equation}$$$$(7)

where z represents the amount of money invested, v is the vulnerability of the asset without any countermeasures implemented (measured as a probability), t is the threat of a breach (measured as a probability), λ is the monetary loss associated with a breach, and S(z,v) is the new probability of a security breach for the asset with initial vulnerability v and a security investment of z. Therefore the term v-S(z,v) measures the reduction in probability of a breach resulting from security investments.³⁸

3 METHODOLOGY

Given several alternatives that exist, a risk manager or systems engineer must consider which risk reduction metric to use. Cox compared three such metrics using a set of generic risks simulated as uniform random variables.³⁹ The first metric was risk itself, corresponding to the strategy of “address the largest risks first”. The second was risk reduction, corresponding to the strategy of “address the largest risk reductions first,” while the third was a risk reduction/cost ratio, corresponding to “address the largest risk reductions per unit cost first”.

Noting that risk reduction corresponds to EBIS from the Gordon–Loeb model, and the risk reduction/cost ratio is similar to ROSI, we propose the following metrics to investigate:

• Risk reduction (EBIS): This figure of merit sets priorities based on the amount of risk reduced by the implementation of a countermeasure. It is equivalent to ALE-mALE.
• Risk reduction minus cost (ENBIS): This figure of merit is similar to EBIS but subtracts cost to give the net benefits. It is equivalent to ALE-mALE-Cost.
• Benefit/cost ratio: This figure of merit is a benefit cost ratio, where the benefit is the risk reduction, calculated as (ALE-mALE)/cost.
• Return on security investment: ROSI is a ROI metric, and is calculated as (ALE-mALE-cost)/cost.

Based on the illustration described in the next section, we rank-ordered the countermeasures from highest to lowest, corresponding to the various strategies implied by each of the four alternative metrics. We selected each countermeasure in order until a budget constraint was reached. The budget constraint was varied to allow us to compare portfolios of countermeasures across different costs.

4 ILLUSTRATION: IoT-ENABLED SMART WAREHOUSING FOR VACCINE STORAGE

4.1 Motivation

With the advent of the Fourth Industrial Revolution, or Industry 4.0., characterized by ubiquitous connectivity and the internet of things (IoT), traditional warehouses are being transformed to “smart” warehouses. The key characteristics of a smart warehouse are automated, unmanned, and paperless activities for pickup, delivery, and bookkeeping functions, facilitated by connected cyber physical systems (CPS).⁴⁰ Warehousing is an important logistics function in many supply chains, serving as a nexus for inbound, storage, and outbound activities, such as receiving, storing, tracking, planning, picking, and shipping.^{41, 42}

Smart warehouses are examples of socio-technical systems comprising potentially thousands of CPS devices, depending on hardware, software, and human interactions. For example, in smart warehouses, each component of inventory may be affixed with tags or devices (such as RFID tags), along with Wi-Fi access points and Bluetooth beacons, cameras, and other sensors, as well as connected devices such as tablets and laptops. Smart warehouses may also utilize robots or other automated systems to carry out various tasks, and human workers monitoring and controlling the operations.⁴⁰ Other assets such as pallets, forklifts, and machines may be IoT enabled.⁴² Coordination and tracking of items and activities is achieved through a warehouse management system (WMS), that facilitates receiving, location management, order picking, transport, and inventory management.^{42, 43} 5G networks enable the transmission of larger quantities of data with less interference and interruption compared to 4G and Wi-Fi.⁴⁴ An IoT-enabled smart warehouse architecture includes sensors connected to microcontrollers, that transmit to an IoT gateway, in turn communicating with servers, handheld devices, and cloud services.⁴⁵ With potentially thousands of sensors and IoT-enabled devices within a smart warehouse, the need for secure and trusted hardware is a critical requirement.

For example, the warehouse environment must be maintained within certain temperature ranges while avoiding fluctuations that may cause spoilage or expiration for some temperature-sensitive goods such as certain foods, pharmaceuticals, and specialty paints and coatings.⁴⁵ For example, for the COVID-19 vaccine to remain effective, depending on the manufacturer, it must be stored at temperatures ranging from −20 to −70°C, colder than the winter in Antarctica.⁴⁶ “Freezer farm” warehouse facilities for the COVID-19 vaccine were constructed, and freezer boxes have been designed with GPS-enabled temperature sensors for shipping.⁴⁷ The necessary cooling systems and temperature monitoring infrastructure require trusted, assured, and secure electronic hardware.

4.2 Expert elicitation

First, a realistic illustrative application for the model was created in order to frame the data elicitation exercise. The illustration considered a vaccine storage facility and its associated systems. Specifically, we introduced the problem of a smart warehouse used to store the extremely temperature sensitive COVID-19 vaccine. Building upon the research surrounding the structure of vaccine “cold chains”,^{48, 49} and their associated costs,^{50, 51} the illustration was developed. A generic architecture was presented (Figure 2) for a smart warehouse used for cold vaccine storage. The following question was posed: Suppose an adversary wanted to purposefully alter the storage temperatures (e.g., by changing the tolerances on the sensors). How would you protect the system from such malicious tampering?

FIGURE 2

Smart warehouse architecture for cold vaccine storage.

Two experts in hardware security and trust with backgrounds in electrical engineering were recruited for the elicitation exercise. The interview was held via video conferencing software, wherein the experts were presented with an introduction to smart warehouse systems, information regarding the COVID-19 vaccine and its associated risks/costs, and the hypothetical warehouse in question. The AV at risk was estimated to be $2,250,000 using available market data.^{52, 53} This estimate accounts for a large warehouse holding approximately 150,000 doses.

The experts were asked to identify potential countermeasures, and for each mitigation, to estimate parameters such as the mitigation cost, its years of use, and the effectiveness (in terms of percentage of attacks successfully blocked). The inputs were entered into the model in real time.

It was assumed that any successful attack on the warehouse rendered the entire stock of vaccines unusable. Thus, the EF was assumed to be 100%, setting the SLE equal to the entire AV at risk. ARO values were set to 10 for the purposes of the illustration. The mitigation rate, or percentage of attacks blocked, was elicited from the experts as a point estimate. The cost and years of use of the security measure was assigned based on elicited data and supplemented by additional research.

4.3 Results

Ten potential countermeasures were identified by the experts. Table 1 summarizes the elicited security countermeasures and associated economic values.

TABLE 1. Security countermeasures.

Security measure	1 – Firewall Protection	2 –Wired sensors – platinum systems (peripherals of warehouse)	3 – Two part/factor authentication for firmware updates (IoT)	4 – Sensor redundancy measures (independent networks)	5 – Sensor polling processes (tests/monitors)	6 – Premium (trusted) sensors (i.e., domestically manufactured)	7 – Buddy system (handling protocol improvements)	8 – Security protocols (improved staff hygiene)	9 – Intermittent attack protection/monitoring	10 – Power grid protection/monitoring
Asset value	$2,250,000	$2,250,000	$2,250,000	$2,250,000	$2,250,000	$2,250,000	$2,250,000	$2,250,000	$2,250,000	$2,250,000
EF (%)	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
SLE	$2,250,000	$2,250,000	$2,250,000	$2,250,000	$2,250,000	$2,250,000	$2,250,000	$2,250,000	$2,250,000	$2,250,000
ARO	10	10	10	10	10	10	10	10	10	10
ALE	$22,500,000	$22,500,000	$22,500,000	$22,500,000	$22,500,000	$22,500,000	$22,500,000	$22,500,000	$22,500,000	$22,500,000
MR	60%	15%	50%	99%	23%	70%	20%	60%	25%	52%
mALE = ALE(1−MR)	$9,000,000	$19,125,000	$11,250,000	$225,000	$17,325,000	$6,750,000	$18,000,000	$9,000,000	$16,875,000	$10,800,000
Total cost	$20,000	$300,000	$250,000	$600,000	$1,000	$360,000	$80,000	$24,000	$13,594	$7,933
Years of use	5	20	7	15	2	20	1	1	5	5
Annual cost	$4,000	$15,000	$35,714	$40,000	$500	$18,000	$80,000	$24,000	$2,718	$1,586
Risk reduction (EBIS)	$13,500,000	$3,375,000	$11,250,000	$22,275,000	$5,175,000	$15,750,000	$4,500,000	$13,500,000	$5,625,000	$11,700,000
Annual net return (ENBIS)	$13,496,000	$3,360,000	$11,214,285	$22,235,000	$5,174,500	$15,732,000	$4,420,000	$13,476,000	$5,622,281	$11,698,413
Benefit/cost ratio	337500%	22500%	31500%	55688%	1035000%	87500%	5625%	56250%	206893%	737426%
ROSI	337400%	22400%	31400%	55588%	1034900%	87400%	5525%	56150%	206793%	737326%

• Abbreviations: ALE, annual loss expectancy; ARO, annual rate of occurrence; EBIS, expected benefit of information security; EF, exposure factor; ENBIS, expected net benefit of information security; MR, mitigation ratio; SLE, single loss expectancy; ROSI, return on security investment.

Table 2 describes the prioritization of countermeasures based on the various risk reduction metrics. We can see that the rankings for EBIS and ENBIS are mostly the same. The one exception is that countermeasure 1 and 8 have equal risk reduction values (i.e., EBIS), so they are tied at a rank of 3, whereas for the ENBIS value, countermeasure 1 is ranked third whereas countermeasure 8 is ranked fourth. For the purposes of portfolio selection, we assigned countermeasure 8 a rank of 4 since it is more expensive than countermeasure 1. Similarly, the rankings for the benefit/cost ratio and ROSI were equal. However, comparing across the two sets (EBIS and ENBIS vs. benefit/cost ratio and ROSI), we see differences in rankings. The ratio-based rankings diverge from the non-ratio rankings. For example, countermeasure 4 ranked first in terms of EBIS and ENBIS, but only seventh in terms of B/C ratio and ROSI, while countermeasure 5 was ranked eighth in terms of EBIS and ENBIS, but ranked first in terms of B/C ratio and ROSI. We find that countermeasures 5, 10, and 1 are the first, second, and fourth most inexpensive countermeasures. The ratio-based metrics, which calculate a risk reduction per unit cost, naturally favor small costs in the denominator of the expression. Therefore, while benefit/cost ratio and ROSI will identify countermeasures that have a large risk reduction per unit cost, a collection of such countermeasures may not necessarily deliver the largest total risk reduction, especially if the countermeasures that deliver large risk reductions happen to also be expensive.

TABLE 2. Comparison of figures of merit.

Counter-measure	EBIS	ENBIS	B/C ratio	ROSI	EBIS rank	ENBIS rank	B/C ratio rank	ROSI rank
1	$13,500,000	$13,496,000	337500%	337400%	3	3	3	3
2	$3,375,000	$3,360,000	22500%	22400%	10	10	9	9
3	$11,250,000	$11,214,285	31500%	31400%	6	6	8	8
4	$22,275,000	$22,235,000	55688%	55588%	1	1	7	7
5	$5,175,000	$5,174,500	1035000%	1034900%	8	8	1	1
6	$15,750,000	$15,732,000	87500%	87400%	2	2	5	5
7	$4,500,000	$4,420,000	5625%	5525%	9	9	10	10
8	$13,500,000	$13,476,000	56250%	56150%	4^a	4	6	6
9	$5,625,000	$5,622,281	206893%	206793%	7	7	4	4
10	$11,700,000	$11,698,413	737426%	737326%	5	5	2	2

• Abbreviations: EBIS, expected benefit of information security; ENBIS, expected net benefit of information security; ROSI, return on security investment.
• ^aCountermeasures 1 and 8 are tied with respect to EBIS, and cost was used as a tie-breaker.

In terms of portfolio selection, using each set of metrics yielded different portfolio compositions (see Tables 3 and 4). Using EBIS and ENBIS, no projects were recommended until a budget of $40,000, when countermeasure 4 was recommended. On the other hand, using the benefit/cost ratio and ROSI, initially both countermeasures 5 and 10 were recommended with a $5000 budget, as they are both inexpensive, and in the next increment, countermeasures 1 and 9 were recommended. Figure 3 shows the number of countermeasures recommended as the budget is increased. The ratio-based metrics generally select more countermeasures as compared to EBIS and ENBIS. Figure 4 depicts the cost and surplus for each budget level, where surplus is defined as the portfolio budget minus the portfolio cost.

TABLE 3. Portfolios recommended using EBIS or ENBIS as a figure of merit.

Budget	Countermeasures selected	Cumulative cost	Surplus
$5,000		$-	$5,000
$10,000		$-	$10,000
$15,000		$-	$15,000
$20,000		$-	$20,000
$25,000		$-	$25,000
$30,000		$-	$30,000
$35,000		$-	$35,000
$40,000	4	$40,000	$-
$45,000	4	$40,000	$5,000
$50,000	4	$40,000	$10,000
$55,000	4	$40,000	$15,000
$60,000	4,6	$58,000	$2,000
$65,000	4,6,1	$62,000	$3,000
$70,000	4,6,1	$62,000	$8,000
$75,000	4,6,1	$62,000	$13,000
$80,000	4,6,1	$62,000	$18,000
$85,000	4,6,1	$62,000	$23,000
$90,000	4,6,1,8	$86,000	$4,000
$95,000	4,6,1,8,10	$87,587	$7,413
$100,000	4,6,1,8,10	$87,587	$12,413
$105,000	4,6,1,8,10	$87,587	$17,413
$110,000	4,6,1,8,10	$87,587	$22,413
$115,000	4,6,1,8,10	$87,587	$27,413
$120,000	4,6,1,8,10	$87,587	$32,413
$125,000	4,6,1,8,10,3	$123,301	$1,699
$130,000	4,6,1,8,10,3,9,5	$126,520	$3,480
$135,000	4,6,1,8,10,3,9,5	$126,520	$8,480
$140,000	4,6,1,8,10,3,9,5	$126,520	$13,480
$145,000	4,6,1,8,10,3,9,5	$126,520	$18,480
$150,000	4,6,1,8,10,3,9,5	$126,520	$23,480
$155,000	4,6,1,8,10,3,9,5	$126,520	$28,480
$160,000	4,6,1,8,10,3,9,5	$126,520	$33,480
$165,000	4,6,1,8,10,3,9,5	$126,520	$38,480
$170,000	4,6,1,8,10,3,9,5	$126,520	$43,480
$175,000	4,6,1,8,10,3,9,5	$126,520	$48,480
$180,000	4,6,1,8,10,3,9,5	$126,520	$53,480
$185,000	4,6,1,8,10,3,9,5	$126,520	$58,480
$190,000	4,6,1,8,10,3,9,5	$126,520	$63,480
$195,000	4,6,1,8,10,3,9,5	$126,520	$68,480
$200,000	4,6,1,8,10,3,9,5	$126,520	$73,480
$205,000	4,6,1,8,10,3,9,5	$126,520	$78,480
$210,000	4,6,1,8,10,3,9,5,7	$206,520	$3,480
$215,000	4,6,1,8,10,3,9,5,7	$206,520	$8,480
$220,000	4,6,1,8,10,3,9,5,7	$206,520	$13,480
$225,000	4,6,1,8,10,3,9,5,7,2	$221,520	$3,480

• Abbreviations: EBIS, expected benefit of information security; ENBIS, expected net benefit of information security.

TABLE 4. Portfolios recommended using benefit/cost ratio and return on security investment (ROSI) as a figure of merit.

Budget	Countermeasures selected	Cumulative cost	Surplus
$5,000	5,10	$2,087	$2,913
$10,000	5,10,1,9	$8,805	$1,195
$15,000	5,10,1,9	$8,805	$6,195
$20,000	5,10,1,9	$8,805	$11,195
$25,000	5,10,1,9	$8,805	$16,195
$30,000	5,10,1,9,6	$26,805	$3,195
$35,000	5,10,1,9,6	$26,805	$8,195
$40,000	5,10,1,9,6	$26,805	$13,195
$45,000	5,10,1,9,6	$26,805	$18,195
$50,000	5,10,1,9,6	$26,805	$23,195
$55,000	5,10,1,9,6,8	$50,805	$4,195
$60,000	5,10,1,9,6,8	$50,805	$9,195
$65,000	5,10,1,9,6,8	$50,805	$14,195
$70,000	5,10,1,9,6,8	$50,805	$19,195
$75,000	5,10,1,9,6,8	$50,805	$24,195
$80,000	5,10,1,9,6,8	$50,805	$29,195
$85,000	5,10,1,9,6,8	$50,805	$34,195
$90,000	5,10,1,9,6,8	$50,805	$39,195
$95,000	5,10,1,9,6,8,4	$90,805	$4,195
$100,000	5,10,1,9,6,8,4	$90,805	$9,195
$105,000	5,10,1,9,6,8,4	$90,805	$14,195
$110,000	5,10,1,9,6,8,4	$90,805	$19,195
$115,000	5,10,1,9,6,8,4	$90,805	$24,195
$120,000	5,10,1,9,6,8,4	$90,805	$29,195
$125,000	5,10,1,9,6,8,4	$90,805	$34,195
$130,000	5,10,1,9,6,8,4,3	$126,520	$3,480
$135,000	5,10,1,9,6,8,4,3	$126,520	$8,480
$140,000	5,10,1,9,6,8,4,3	$126,520	$13,480
$145,000	5,10,1,9,6,8,4,3,2	$141,520	$3,480
$150,000	5,10,1,9,6,8,4,3,2	$141,520	$8,480
$155,000	5,10,1,9,6,8,4,3,2	$141,520	$13,480
$160,000	5,10,1,9,6,8,4,3,2	$141,520	$18,480
$165,000	5,10,1,9,6,8,4,3,2	$141,520	$23,480
$170,000	5,10,1,9,6,8,4,3,2	$141,520	$28,480
$175,000	5,10,1,9,6,8,4,3,2	$141,520	$33,480
$180,000	5,10,1,9,6,8,4,3,2	$141,520	$38,480
$185,000	5,10,1,9,6,8,4,3,2	$141,520	$43,480
$190,000	5,10,1,9,6,8,4,3,2	$141,520	$48,480
$195,000	5,10,1,9,6,8,4,3,2	$141,520	$53,480
$200,000	5,10,1,9,6,8,4,3,2	$141,520	$58,480
$205,000	5,10,1,9,6,8,4,3,2	$141,520	$63,480
$210,000	5,10,1,9,6,8,4,3,2	$141,520	$68,480
$215,000	5,10,1,9,6,8,4,3,2	$141,520	$73,480
$220,000	5,10,1,9,6,8,4,3,2	$141,520	$78,480
$225,000	5,10,1,9,6,8,4,3,2,7	$221,520	$3,480

FIGURE 3

Number of countermeasures recommended for various budgets.

FIGURE 4

Portfolio cost and surplus for various budgets.

4.4 Limitations

The results suggest a limitation of using ratio-based metrics for prioritization of security countermeasures: the propensity of the cost/benefit ratio and ROSI to favor inexpensive countermeasures that may not be strongly correlated with risk reduction. One must think carefully about the risk reduction metrics to guide the prioritization of countermeasures. A portfolio optimization approach, such as an integer programming method, that maximizes risk reduction and uses cost as a constraint may be a better approach than to use cost/benefit ratio-based metrics for countermeasure selection.⁵⁴

Additional general limitations are as follows. First, a systems engineer will need to have access to security experts who can identify particular countermeasures. When identifying countermeasures, one can use an aid such as the Cyber Defense Matrix (CDM).⁵⁵ The CDM is an organizational construct for teams to identify their security needs as they head into the security vendor marketplace. The CDM identifies five operational functions (identify, protect, detect, respond, recover) and five asset classes (devices, applications, networks, data, users) to create a 5 × 5 grid. Therefore, each cell represents a category for a particular mitigation to perform an operational function relative to an asset class (e.g., a way to identify threats to devices).

Furthermore, the maturity of security economic calculations is still in the early stages. More research is needed to better parameterize the elements that go into the estimates of these risk reduction metrics. For example, estimating probability distributions that model the frequency of cyber-attacks is a difficult and ongoing area of research.^{56, 57} One approach is to utilize the triangular distribution⁵⁸ which has the convenient property that it is simple to elicit – generally an expert is asked to provide a maximum, minimum, and most-likely value for some variable of interest.⁵⁹

Empirical data are typically not available for certain parameters such as ARO and MR, and therefore experts must be consulted to estimate them. As the field of security economics evolves, methods to estimate and forecast such uncertain values should be investigated. The use of expert elicitation of uncertain parameters is well known to potentially introduce error based on a variety of human factors and cognitive biases, and best practices have been proposed about the calibration of experts and the weighting of their responses.^{60, 61} While such elicitation approaches have been applied to information security,⁶² parameterizing cybersecurity risk models remains difficult, in part due to the unique and evolving nature of cyber threats.⁶³ Theories and methodologies to better understand the dynamic and quickly changing nature of cyber threats are needed, as well as tools to capture the current state of the cyber environment. While we can never be entirely sure of their costs, savings, frequency of incidents, etc., in the future (i.e., post investment), it has been argued that “something is better than nothing”.⁶⁴

5 CONCLUSIONS

In this paper, we examined a number of risk reduction metrics for prioritizing hardware security countermeasures. We found that the benefit/cost ratio and ROSI are not strongly correlated to risk reduction. Ranking projects by EBIS and ENBIS closely mirrors the benefits of risk reduction, as EBIS is equivalent to risk reduction, and ENBIS is risk reduction less costs. As ratio-based metrics, the cost/benefit ratio and ROSI tend to favor inexpensive investments, leading to potentially misleading rankings of countermeasures.

The illustration has explored the above approach with a prioritization of hardware security countermeasures for integrated hardware and software (IoT) systems. While the illustration was based on a fictional system, which serves to concretize the methodology, future work will involve in-depth validation on real systems.

In terms of managerial implications, there will always be a need to justify security investment within an organization. Among the ways to justify security spending includes recognizing that security is an urgent problem, promoting an “action state of mind,” showing that investments will be effective in terms of performance, and that the investments will be cost effective.⁶⁵ By framing security returns as avoided risks, one can recommend data-informed, economically sound investments in security countermeasures for the enterprise.

ACKNOWLEDGMENTS

This effort was supported by the National Science Foundation under Grant 1916760 “Phase I IUCRC University of Virginia: Center for Hardware and Embedded Systems Security and Trust (CHEST),” and the Commonwealth Center for Advanced Logistics Systems (CCALS).

SIGNIFICANCE AND PRACTITIONER POINTS

For a given system, there may be a number of threats, and a large pool of potential security countermeasures. Systems engineers must be able to answer the question: “for a given budget, what countermeasures should I prioritize alone and in combination?” Based on insights from the emerging field of security economics, this paper explores several metrics that could be helpful to prioritize selection of countermeasures for risk reduction.

CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest.

Biographies

• 

  Zachary A. Collier is Assistant Professor in the Department of Management at Radford University. He earned his PhD in Systems Engineering from University of Virginia (Charlottesville, VA), a Master of Engineering Management from Duke University (Durham, NC), and a Bachelor of Science in Mechanical Engineering from Florida State University (Tallahassee, FL). He is President of Collier Research Systems, a consultancy providing decision making and analytics services for clients across multiple industries. His prior work experience includes the U.S. Army Engineer Research and Development Center, where he was a member of the Risk and Decision Science Team and served as PI and Co-PI on a number of interdisciplinary research projects. Dr. Collier is a member of the Society for Risk Analysis, where he has served as President of the Decision Analysis and Risk Specialty Group and President of the Resilience Analysis Specialty Group. He currently serves as Co-Chair of the NDIA Electronics Division's Trust and Assurance Committee and is a member of the INFORMS Advocacy Governance Committee. Dr. Collier is a Fellow of the Center for Risk Management of Engineering Systems at University of Virginia, a Visiting Scholar at the Center for Hardware and Embedded Systems Security and Trust (CHEST), and contributes as a subject matter expert to the development of industry standards through SAE International. He is Managing Editor of the Springer journal "Environment Systems & Decisions", and is a member of the Editorial Board of "Risk Analysis".

• 

  Brett Briglia is a senior analyst with PGIM and is currently working in commercial real estate finance. He focuses on multifamily assets and has worked alongside his team to finance over $2 Billion dollars in volume. He has a background in real estate development and enjoys the application of economic and financial principles to the field. He is a 2020 graduate from the University of Virginia with degrees in Economics and Spanish.

• 

  Tom Finkelston is a Technology Consultant at Ernst & Young, and is currently based out of their New York City office. He functions predominantly within Banking, Capital Markets, and ESG, and has developed an affinity for Data Analytics and Technology Strategy focused projects. While at the University of Virginia, Tom received his bachelor degrees in Systems Engineering and Economics. He continues to enjoy the marriage of these two fields, and the application of analytics to financial services and environmental sustainability.

• 

  Mark Manasco is president and executive director for the Commonwealth Center for Advanced Logistics Systems (CCALS), a collaboration between industry and Virginia universities that works to improve logistics operations in the key areas of data analysis, cost control, quality assurance, security, and demand forecasting. Formerly director of workforce development for the Greater Richmond Chamber of Commerce and of the University of Richmond's Center for Systems Assurance, Manasco brings more than two decades of public policy, higher education, information technology, and human resource management experience to the leadership post at CCALS. He earned a BA in Economics from University of Richmond, and a MA in Economics from Virginia Commonwealth University.

• 

  Professor Slutzky is currently a Research Associate Professor in the Science, Technology and Society Program of the Department of Engineering and Society at the University of Virginia. David Slutzky has spent the last 35 years as an entrepreneur, and as a public policy expert. He founded ERC, which become the largest provider of Environmental Site Assessments in the US. Slutzky co-founded the nation's first environmental data company, ERIIS. David Slutzky later co-founded Skeo Solutions, a Virginia based environmental policy consulting firm, which employs 65 professionals and where he currently serves as Board Chairman. In 2011, Slutzky founded Fermata Energy, a tech start-up that is using Vehicle-to-Grid technologies to accelerate the adoption of electric vehicles while providing the energy storage needed to enable the transition of our electric power grid from coal and nuclear to renewables. Professor Slutzky is widely recognized as a thought leader in the V2G industry. Mr. Slutzky served as a Senior Policy Advisor at the U.S. EPA, and later at the White House, where he coordinated the International Task Force of the President's Council on Sustainable Development. Mr. Slutzky was elected to serve on the Albemarle County Virginia Board of Supervisors from 2005 to 2009, and was chosen to Chair the Charlottesville-Albemarle Metropolitan Planning Organization. Slutzky earned his BA and pursued graduate studies in Political Science at the University of Chicago, and received his law degree from the Program on Energy and the Environment at Chicago-Kent College of Law.

• 

  James H. Lambert is a Professor of Engineering Systems and Environment (Program in Systems Engineering, Program in Civil Engineering), Director of the Center for Risk Management of Engineering Systems, and Member of the Technical Advisory Council of the Commonwealth Center for Advanced Logistics Systems, each at the University of Virginia. He is a Site Director of the NSF funded Center for Hardware and Embedded Systems Security and Trust (CHEST). Professor Lambert's research interests are engineering systems and risk analysis. He is a Fellow of the AAAS (F.AAAS), Fellow of the IEEE (F.IEEE), Fellow of the ASCE (F.ASCE), Fellow of the SRA (F.SRA), Diplomate (D.WRE) of the American Academy of Water Resources Engineers, member of the American Association for the Advancement of Science, member of the International Council on Systems Engineering, and licensed Professional Engineer (P.E.). He is a Past President (2015–2016) of the Society for Risk Analysis (SRA). He is Editor-in-Chief of the Springer journal Environment Systems & Decisions. He is an Area Editor of the Wiley journal Risk Analysis. He is an Associate Editor of the ASCE/ASME Journal of Risk & Uncertainty in Engineering Systems. He represents the University of Virginia to the Council of Engineering Systems Universities (CESUN). He received a PhD and MS in Civil Engineering at the University of Virginia, and a BSE in Mechanical Engineering with a Certificate in Engineering Physics at Princeton University.

Open Research

DATA AVAILABILITY STATEMENT

No new data were produced in this paper.