4

New Data and New Metrics for Evaluating Impact

Turning from metrics used in the past to assess SBIR programs, the third panel considered new data and new metrics that could be used in program evaluation.

DIRECT ECONOMIC IMPACTS

In 2021, the number of people in the United States who worked in the research and development (R&D) sector was 2.4 million out of a population of 130 million full-time workers. “Most people experience innovation policy indirectly,” observed Alexander Whalley, associate professor of economics and business at the University of Calgary. “They don’t experience patenting, publications, or things like that. They experience things like job creation, earnings, and things like that. How can we measure these broad-based economic effects?”

Whalley described two broad approaches. One, the indirect method, tries to quantify the effects of innovation policy on innovation outcomes such as patents or publications. Adding up the economic impact of each of these outcomes yields an overall impact for the economy.

The other approach, the direct method, evaluates the impacts of innovation policy on economic activity. In this way, it directly estimates the overall economic impacts of a policy.

Focusing on the second approach, Whalley first considered agricultural research. When the land grant universities were founded in the 19th century, they were not doing much agricultural research. That changed in 1890 with the launch of experiment stations under the Hatch Act, which sought to boost agricultural innovation. The success of this approach can be seen in the relationship between agricultural productivity and distance from a land grant institution. As research accrues, the effects of this research on nearby farms grow while distant farms benefit less. This effect appears gradually and fades over a few decades, but it demonstrates the power of innovation.

Whalley has also studied the space race, which represents one of the biggest R&D investments of all time—at its peak, the space program was consuming about 1 percent of the nation’s gross domestic product, and about 400,000 people were making things for the program. Many different technologies emerged from the space race, including computers, communication devices, and sensors. To measure the impacts of these products, Whalley and his colleagues looked at the differences in value added between firms that were space capable and firms that were not space capable.¹ Before the space race, these firms were similar in value added, but they diverged appreciably in the 1960s and 1970s, after which the difference between the two categories of firms subsided.

One advantage of the direct method is that it seeks to determine the economic effects of a policy, which is important given that government programs can spend money on different things. “We’re making decisions about whether we spend more money on R&D programs or more money on early childhood education or health programs—you can compare across different areas.” It also can gauge the responses to a policy by workers and firms; this can provide context-specific results and yield policy complementarities.

Among the disadvantages of the direct method are its significant data requirements and lag in estimates, Whalley observed. “If you’re thinking about a program and you want to know what’s happening right now, it might be 5 years, 7 years, or later until you see an impact.” The linkage between innovation policy and economic outcomes can also be unclear, and it can be hard to capture all the economic impacts of an innovation policy. New methods, such as textual data analysis, may counter some of these difficulties, Whalley said. For example, in the space race project, analysis of declassified Central Intelligence Agency documents was used to define space technology, after which those technologies could be tracked.

Textual analysis points toward other possible new metrics for evaluating SBIR projects, Whalley observed. For example, analysis of the text in job advertisements can indicate what skills firms are using and seeking. As a specific

___________________

¹ Shawn Kantor and Alexander T. Whalley. 2024. Moonshot: Public R&D and growth. National Bureau of Economic Research. Working Paper 31471. https://www.nber.org/papers/w31471

case in point, job ads for vacancies requiring skills in artificial intelligence document the growth in this technology.

Another new metric could be company revenues. This number is often hard to get, but a real-time measure of revenues could be website traffic, Whalley pointed out, adding that “there should be some correspondence between website traffic and revenue.” As an example, website traffic to OpenAI and competing firms exploded in 2023, and measures of their traffic could be used to estimate revenues.

Human capital could be another valuable metric. Individual principal investigators can become involved with companies and drive their success. Wage data indicate that contributions of individuals matter significantly to the success of firms.² “When we start to do that decomposition, what we see is the person matters a lot.”

These new data and methods open up new opportunities to apply the direct approach to capturing the economic impacts of innovation policy, Whalley concluded. At the same time, better understanding of where innovation policy impacts occur can improve evaluation. For example, are the primary impacts on firm revenue or on worker human capital?

FINDING “INVISIBLE” INNOVATION

Daniel Gross, assistant professor of business administration at Duke University’s Fuqua School of Business and a faculty research fellow at the National Bureau of Economic Research, discussed innovations that tend to be “invisible,” including innovations that receive support from the SBIR program. Such innovations have always existed but were especially common in World War II, when nuclear fission, radar, electronic communication, synthetic rubber, cryptography, and other technologies were classified as secret while other technologies, such as penicillin, were not patented (in the case of penicillin because it occurs naturally).³ Today, some technologies, such as hypersonics or directed energy weapons, remain classified, while others, such as software or artificial intelligence, are harder than traditional technologies to quantify and describe.

Beginning with software, Gross pointed out that the Apollo Guidance Computer in 1969 had about 145,000 lines of code; the original Space Shuttle in 1981 had about 400,000; the Lockheed F-35 Joint Strike Fighter in 2006 had more than 8 million (with another 24 million lines of code in support programs); and the Army Future Combat Systems, before it was cancelled in 2009, had more than 63 million. “Software has gotten much more complex over the past 50 years, and this has a number of implications.” First, as Marc Andreessen observed in 2011,

___________________

² John M. Abowd, Francis Kramarz, and David N. Margolis. 1999. High wage workers and high wage firms. Econometrica 67(2):251–333.

³ Daniel P. Gross and Bhaven N. Sampat. 2023. America, jump-started: World War II R&D and the takeoff of the U.S. innovation system. American Economic Review 113(12):3323–3356.

“More and more major businesses and industries are being run on software and delivered as online services—from movies to agriculture to national defense.” DOD hardware used to be and still is hardware intensive, but it is growing more software intensive too. Software requirements can make or break major projects; intelligence now includes the harvesting of digital information; and war arenas now include cyberspace, where physical systems are no longer very relevant.

Gross quoted a blog post from the defense company Anduril: “American technological supremacy will require a pivot from the large, exquisite, hardware-defined systems that won us the conflicts of last century to larger numbers of lower-cost, attributable, smaller, software-defined systems.” This will require a fundamental shift from hardware-first to software-first acquisition, he said. For example, The New York Times reported that the Air Force is planning to split up aircraft and software as separate purchases, with algorithms essentially functioning as “robot wingmen” that will fly alongside human pilots.⁴ Similarly, the Defense Innovation Board has, in a study of software acquisition practices, proposed to create a new appropriations category for software capability.⁵

Gross compared the share of DOD SBIR awards with text in the abstracts from 1980 through 2022 and found that the occurrence of “software,” “algorithm,” “simulation,” and “programming” grew for the first two decades and has remained roughly constant, at about 30 percent, during the 21st century. Meanwhile, the occurrence of “computers” and “computation” has steadily fallen to less than 10 percent. Similarly, “autonomy” and “automation” have been rising very quickly, and “artificial intelligence” and “machine learning” have exploded in the past few years. “Is software eating SBIR?” he asked. “It already has.”

The value of defense software innovation is very difficult to measure, he pointed out. Conventional metrics such as patents and publications are not very helpful. Software is generally not patented and is more often protected by copyright. Publications may be useful for measuring some algorithm development, and textual analysis can provide additional information, “but that won’t get you all the way there.” An important question, he noted, is how to measure software intensity in physical systems.

Secrecy remains an issue in innovation metrics, he observed. For example, the U.S. Patent and Trademark Office (USPTO) has the authority to issue compulsory secrecy orders for patent applications and the inventions in them if they present a threat to national security. A patent application that is a candidate for secrecy will be forwarded to evaluators who are seconded from the DOD, after which the application can go under seal. A secrecy order can prohibit publication or disclosure of invention and prohibit filing in a foreign country. Such an order, which is subject to annual (and possibly indefinite) renewal, permits the invention to be disclosed to relevant agencies, such as the DOD, and the inventor can claim

___________________

⁴ Eric Lipton. 2023. A.I. brings the robot wingman to aerial combat. The New York Times, August 27.

⁵ Defense Innovation Board. 2019. Software is never done: Refactoring the acquisition code for competitive advantage. https://media.defense.gov/2019/Mar/26/2002105909/-1/-1/0/SWAP.REPORT_MAIN.BODY.3.21.19.PDF

royalties for government use. Penalties for violating the order include losing the patent, fines, and jail terms.

In World War II, around 5 percent of USPTO-filed inventions were ordered into secrecy—over 11,000 altogether. This compulsive secrecy had substantial impacts, Gross noted.⁶ These inventions were less likely to be built upon. They also could not be commercialized while the secrecy order was in effect. Although secrecy orders are less frequent today, they are still being issued. About 0.02 percent of USPTO-filed inventions, or about 100–150 per year, are ordered into secrecy.⁷ These inventions are probably defense specific, but that is not known for sure. Some organizations, such as the Federation of American Scientists, have been tracking patents that emerge from secrecy orders, but patents still under secrecy orders are not publicly accessible. “This is another one of the ways in which innovation might become invisible.”

Finally, Gross discussed the trend of selective participation in the SBIR program. Arcane and burdensome acquisition regulations, slow contract awards, a low probability of procurement or recurring revenue, and an inability to conduct or incorporate classified work all can stop firms from engaging with DOD SBIR programs. This raises a different kind of measurement challenge: who is not engaging with SBIR programs, and why? Growing software requirements of defense R&D programs (among other things) point toward an increased need to engage the commercial sector and widen the application funnel, he said.

In response to a question about assessing the impacts of secrecy in any given technological area, Gross reiterated the possibility of looking up public data about technologies that have come out of secrecy, especially if they have been patented. “Those should be observable and either already available or FOIA-able.”⁸ In that way, it may be possible to see if there are subclasses of technologies or companies where secrecy is commonly applied.

PANEL DISCUSSION

Asked what this work means for measuring the impact of innovations, Whalley pointed to the importance of assessing the impacts of SBIR funding beyond direct impacts on firms that receive the funds. For example, individual firms are connected to other firms and other local actors in local economies, and “there are metrics you can use to think about how connected firms are to each other.” One is the sharing of workers when someone leaves one firm and is hired by another, which transfers human capital from one place to another. Another is related to the capacity of firms within local economies, such as the ability of firms to take advantage of work going on in nearby universities. Firms with more

___________________

⁶ Daniel P. Gross. 2023. The hidden costs of securing innovation: The manifold impacts of compulsory invention secrecy. Management Science 69(4):2318–2338.

⁷ Steven Aftergood. 2023. “FAS project on government secrecy.” https://sgp.fas.org/othergov/invention/index.html

⁸ FOIA = Freedom of Information Act.

“absorptive capacity can absorb ideas and then make something out of them.” Finally, connections within innovation ecosystems can indicate innovative activity, such as when a patent citation or research paper lists the sources upon which it drew. “You could think about the intellectual property [SBIR firms] generate, which other firms are reading until their patents are connected to that ecosystem.”

Regarding the role of human capital, Gross speculated about the possibility of studying firms that are located near military establishments. Relationships between institutions could promote SBIR applications, and it may be possible to measure these relationships and their effects when a military establishment closes or moves. “The nexus of small businesses and the military is something I would want to understand better, especially in understanding how much SBIR contributes to local economic development or local high-tech development around specific technologies that are particularly related to mission needs of whatever branch operates in that locale.”

A workshop attendee called attention to the federal acquisition regulation known as authorization and consent that grants government organizations the right to use any U.S. patent regardless of whether the government supported the work that led to the patent. This creates a large disincentive for small businesses to file patents and leads them to rely on trade secrecy. Gross pointed out that the same thing can happen with contracts that assign data rights, which includes not just datasets but software as well and can represent foregone revenue opportunities for SBIR awardees.

In response to a question about unconventional research methods, Whalley again cited textual analysis. Such analyses provide lists of characteristics or elements of technology that then provide search terms. Gross mentioned the value of firsthand accounts of how programs are run. For example, an “amazing set of oral histories with DARPA⁹ directors over the past 50 years” is available that provides valuable insights into DARPA’s history and accomplishments.

In response to a question about assessing the numbers of companies that go out of business after being supported by SBIR awards, Whalley noted that the end of a company is a tricky thing to measure. “There’s lots of entry and exit, especially for small startups. Sometimes they change their names, sometimes they change control.” As a result, this measure is a hard one to attain. One suggestion, from Noah TK, is to look for web domain names that have disappeared, which may be an indication that the company has gone out of business. As the moderator of the panel, Maryann Feldman observed, “Companies don’t have obituaries. They just sort of disappear.”

Asked whether success could be differentiated as technology programs move between various phases of research, development, and commercialization, Whalley pointed to the work of Jeffrey Liebman and Neale Mahoney on the

___________________

⁹ DARPA = Defense Advanced Research Projects Agency.

effectiveness of government spending toward the end of each fiscal year.¹⁰ He also has explored this phenomenon with basic research funding in general, asking whether contracts issued right before or right after the end of the fiscal year have different levels of effectiveness.

Finally, in response to a question about studying the impact of innovation ecosystems on the success of SBIR/STTR programs, Whalley called attention to policy complementarity. “What’s the effect of being in a thriving ecosystem? You can think about two different firms getting SBIR awards in two different ecosystems—one that’s really connected with a high level of human capital versus one that’s not.” Differences in impact between such firms could point to the importance of policies separate from SBIR policies. Another possibility would be close analysis of the many pages of descriptive information that accompany each year’s budget request, particularly regarding plans for transitioning technologies. As Feldman pointed out, such analyses could be used for training purposes to do text mining.

___________________

¹⁰ Jeffrey B. Liebman and Neale Mahoney. 2017. Do expiring budgets lead to wasteful year-end spending? Evidence from federal procurement. American Economic Review 107(11):3510–3549.

This page intentionally left blank.