Introduction

In response to a congressional mandate, a committee of the National Academies of Sciences, Engineering, and Medicine is conducting a review of the Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs at the Department of Defense (DOD). These programs, created in 1982 and 1992, respectively, and operated within the DOD and a number of other agencies within the federal government, encourage the participation by small business concerns in the federal research and development and procurement processes through competitively based awards.

The National Academies was mandated by Congress to conduct periodic reviews of the programs at the five largest research and development agencies—the departments of Defense and Energy, the National Institutes of Health, the National Aeronautics and Space Administration (NASA), and the National Science Foundation (15 USC § 638). Recently completed National Academies SBIR/STTR program reviews have covered the Department of Energy, the National Institutes of Health, and the National Science Foundation,¹ and separate National Academies committees are currently conducting a review of the programs at NASA² and a follow-on review of the programs at the Department of Energy.³

As part of its review of the DOD’s SBIR/STTR programs, and in response to a request by the DOD, the present study committee held a workshop on December 7–8, 2023, entitled “Data and Metrics for the DOD SBIR and STTR Programs.” The impetus for the meeting stems from Congress mandating that the Secretary of Defense “conduct a study on the metrics necessary to assess the effectiveness of the SBIR and STTR programs of the Department of Defense in

___________________

¹ Previous reports in the series are available at https://nap.nationalacademies.org/catalog/25674/review-of-the-sbir-and-sttr-programs-at-the-department-of-energy, https://nap.nationalacademies.org/catalog/26376/assessment-of-the-sbir-and-sttr-programs-at-the-national-institutes-of-health, and https://nap.nationalacademies.org/catalog/26884/review-of-the-sbir-and-sttr-programs-at-the-national-science-foundation

² See https://www.nationalacademies.org/our-work/review-of-the-small-business-innovation-research-and-small-business-technology-transfer-programs-at-nasa

³ See https://www.nationalacademies.org/our-work/assessment-of-the-small-business-innovation-research-and-small-business-technology-transfer-programs-at-the-department-of-energy

meeting the mission needs of the Department, including by developing metrics and collecting and assessing longitudinal data necessary for evaluation of those metrics.”⁴ The National Academies workshop was convened to facilitate the DOD’s development of recurring, quantifiable metrics for measuring the ability of the SBIR and STTR programs to deliver products and services that meet the DOD’s mission needs, while also informing the National Academies committee’s overall study of the programs at the DOD.⁵ The statement of task for the committee’s overall study and this workshop are found in Box I-1.

This proceedings of a workshop was prepared by a workshop rapporteur as a factual summary of what was presented and discussed at the workshop. The planning committee’s role was limited to organizing the workshop. The statements made are those of the rapporteur and do not necessarily represent positions of the workshop participants as a whole; the planning committee; or the National Academies of Sciences, Engineering, and Medicine.

As committee cochair Maryann Feldman, Watts Professor in the Department of Public Policy at Arizona State University, said in her opening remarks at the workshop, the DOD is “critically important” in the series of SBIR/STTR reviews being undertaken by the National Academies. With a budget that accounts for more than half of total federal SBIR/STTR spending, it is further notable because the DOD SBIR/STTR programs specifically support national defense, giving it a broader purpose beyond supporting small companies and the commercialization of their products.⁶

Relatedly, the committee’s other cochair, Scott Stern, David Sarnoff Professor of Management at the Massachusetts Institute of Technology Sloan School of Management, said “The mission and needs of the Department of Defense are very different than many other federal agencies.” He said that “There’s a particularly timely and important need for trying to understand the measurable ways in which we can capture impact.” Are the data currently available to support the assessment of metrics, including the identification of existing data gaps?, he asked. What are the means and methods available to the department for collecting data in an automated fashion? Can an analysis and assessment methodology framework be developed to make trade-offs between metrics and between existing DOD commercialization benchmarks?

THE SBIR/STTR PROGRAMS AT THE DOD

As the DOD states on its SBIR/STTR webpage, the SBIR and STTR programs are “Congressionally authorized small business set-asides established

___________________

⁴ The James M. Inhofe National Defense Authorization Act for Fiscal Year 2023 (H.R.7776). U.S. Congress, Public Law No. 117-263 (December 23, 2022).

⁵ Materials presented at the workshop are available at https://www.nationalacademies.org/event/41337_12-2023_data-and-metrics-for-the-dod-sbir-and-sttr-programs-a-workshop

⁶ At the workshop and in this summary of the event, the term “SBIR programs” should be interpreted to encompass STTR programs as well as SBIR programs.

to support scientific excellence and technological innovation through investment of Federal research funds in critical American priorities to build a strong national economy.”⁷ SBIR involves a competitive three-phase process for soliciting proposals for research/R&D (R/R&D), production, services, or any combination of these activities to meet the agency’s needs or missions. STTR is a parallel program to SBIR with the added requirement that small businesses partner with colleges, universities, federally funded R&D centers, or qualified nonprofit research institutions on cooperative R/R&D. The goal of these programs is to “stimulate technological innovation for the DOD to maintain technological superiority and military readiness to deter military operations from U.S. adversaries” by:

• increasing private-sector commercialization of federal R&D to increase competition, productivity, and economic growth;
• stimulating a partnership of ideas and technologies between innovative small businesses and research institutions;
• enabling small businesses to explore their technological potential and providing incentives to profit from the commercialization of technologies; and
• stimulating innovation by including qualified small businesses in the nation’s R&D arena.

“The SBIR/STTR program is vital to our national security,” said Jagadeesh Pamulapati, director of Science and Technology Foundations in the Office of the Under Secretary of Defense for Research and Engineering, in introductory remarks to the workshop. “Without these programs, we risk the advantage that we’ve historically held over other nations.” Funding for SBIR/STTR programs at the DOD is about $2 billion, well over half of

___________________

⁷ Information about the program and how to participate in it is available at https://www.defensesbirsttr.mil/SBIR-STTR/Program. Information in this paragraph is taken from this webpage.

SBIR/STTR spending across all federal agencies. “We are the elephant in the room,” he said, adding that inevitably, questions about measuring the effectiveness of the program arise, including identifying the right things to measure, how they should be measured, and how long data should be collected to ensure good evaluations of commercialization or transition.

The Under Secretary of Defense for Research and Engineering, Heidi Shyu, has been very active in advocating for the program, Pamulapati reported. She hosts bimonthly roundtables with small businesses to get a sense of their perspectives and challenges. The DOD continues to upgrade the Defense SBIR/STTR Innovation Portal, which accepts proposals across all the solicitations from the department over the course of a year.⁸ It also recently launched an innovation pathways website that allows one-stop shopping for anyone looking to deliver innovations to the department.⁹ The site, he said, serves as a gateway for industry, small businesses, universities, and even students to understand the department’s priorities and to connect them to the relevant parts of the department.

Legislators have directed the DOD to look at how it could increase the number of new entrants into the SBIR/STTR program, said Pamulapati, with a particular focus on socially and economically disadvantaged small businesses and those in underserved states. In response, the department has been increasing its outreach and social media presence. “We’re looking to broaden, expand, and diversify the number of small businesses that are participating.” The department has also been making increased use of open topics solicitations, as is required by statute.

Pamulapati described how the DOD has a number of programs designed to support commercialization efforts. An example is the Office of the Secretary of Defense Transitions SBIR/STTR Technologies program, which was begun in 2020 to further mature technologies and help companies avoid the “valley of death” between R/R&D and commercialization.¹⁰ The DOD’s Accelerate the Procurement and Fielding of Innovative Technologies (APFIT) program is designed to transition technologies from pilot programs, prototype projects, and research projects into production, with priority given to those developed by small businesses and/or nontraditional defense contractors.

In a recent study of the SBIR/STTR programs by TechLink and the Business Research Division of the Leeds School of Business at the University of Colorado Boulder, “it was astounding how much economic benefit we found from our investment in this program,” said Pamulapati.¹¹ To the extent that data and metrics are available to advocate for the program, “we can put it on a stronger foundation,” he concluded. “This is a really important effort that you are undertaking.”

___________________

⁸ The portal is available at https://www.dodsbirsttr.mil/submissions/login

⁹ The innovation pathways website is available at https://www.ctoinnovation.mil

¹⁰ Information about the program is available at https://www.defensesbirsttr.mil/OTST

¹¹ The report is available at https://media.defense.gov/2022/Mar/10/2002954100/-1/-1/0/DODSBIR-STTRECONOMICIMPACTSTUDY.PDF

In response to a question about the motivations and rewards for program managers or program executive officers to transition small businesses so that they become a more fundamental part of the defense industrial base, Pamulapati pointed out that program managers do not receive monetary awards for successful commercialization but have many other incentives to promote transitions. For example, SBIR/STTR programs can function as part of an acquisition strategy without having to undertake a subsequent competition.

In response to another question about the main challenges in identifying metrics that could be used to assess the effectiveness of SBIR/STTR programs, Pamulapati observed that the commercialization of programs is self-reported onto a Small Business Administration website, which requires placing trust in the companies doing that reporting and raises questions about the accuracy of the database. Also, a small business could develop a technology that is not transitioned into a solution for some time. “If the programs are going through a fairly tight time sequence, we can measure the transition as it goes from Phase II or Phase III.” But the transition can be overlooked in other circumstances. “There’s no sticker that says ‘SBIR/STTR’ inside,” Pamulapati said.

STRUCTURE OF THE SUMMARY

This workshop proceedings follows the structure of the workshop agenda (see Appendix). Chapter 1 summarizes the keynote address on innovation and national security given by Josh Lerner, Jacob H. Schiff Professor at Harvard Business School and codirector of the school’s Private Capital Project. Chapter 2 situates SBIR/STTR programs within the DOD’s broader missions and goals. Chapter 3 looks at metrics that have been used in the past to evaluate the government’s innovation and commercialization programs and that are now being extended in new directions. Chapter 4 considers new kinds of data and metrics that could be used in SBIR evaluations. Chapter 5 samples the experiences of representatives of the small business community with SBIR programs. Chapter 6 offers the reflections of committee members on the presentations and discussions that occurred during the workshop.