Data and Metrics for the DOD
SBIR and STTR Programs

_____

Steve Olson, Rapporteur

Board on Science, Technology, and
Economic Policy

Policy and Global Affairs

Proceedings of a Workshop

NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001

This material is based upon work supported by the Army Research Office under Contract No. W911NF-23-D-0002. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the Army Research Office.

International Standard Book Number-13: 978-0-309-72724-2
International Standard Book Number-10: 0-309-72724-3
Digital Object Identifier: https://doi.org/10.17226/27984

This publication is available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242; http://www.nap.edu.

Copyright 2024 by the National Academy of Sciences. National Academies of Sciences, Engineering, and Medicine and National Academies Press and the graphical logos for each are all trademarks of the National Academy of Sciences. All rights reserved.

Printed in the United States of America.

Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: National Academies Press. https://doi.org/10.17226/27984.

The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.

The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. John L. Anderson is president.

The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.

The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.

Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.

Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task.

Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies.

Rapid Expert Consultations published by the National Academies of Sciences, Engineering, and Medicine are authored by subject-matter experts on narrowly focused topics that can be supported by a body of evidence. The discussions contained in rapid expert consultations are considered those of the authors and do not contain policy recommendations. Rapid expert consultations are reviewed by the institution before release.

For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo.

COMMITTEE ON THE REVIEW OF THE SMALL BUSINESS INNOVATION RESEARCH AND SMALL BUSINESS TECHNOLOGY TRANSFER PROGRAMS AT THE DEPARTMENT OF DEFENSE

MARYANN P. FELDMAN (co-chair), Watts Professor, Department of Public Policy, Arizona State University

SCOTT STERN (co-chair), David Sarnoff Professor of Management, Sloan School of Management, Massachusetts Institute of Technology

MICHAEL J. ANDREWS, Associate Professor, Department of Economics, University of Maryland Baltimore County

ANDREA BELZ, Vice Dean of Transformative Initiatives, Viterbi School of Engineering, University of Southern California

JANET BERCOVITZ, Deming Professor of Entrepreneurship, Leeds School of Business, University of Colorado Boulder

M. DIANE BURTON, Joseph R. Rich ‘80 Professor, School of Industrial and Labor Relations, Cornell University

RAMALINGAM CHELLAPPA (NAE), Bloomberg Distinguished Professor, Johns Hopkins University

DONNA GINTHER, Roy A. Roberts and Regents Distinguished Professor of Economics, University of Kansas

JORGE GUZMAN, Gantcher Associate Professor of Business, Columbia University

WARREN KATZ, Chairman, The Alliance for Commercial Technology in Government (until January 1, 2024)

LAUREN LANAHAN, Associate Professor of Management, Inman Research Scholar, Lundquist College of Business, University of Oregon

ELLEN M. LORD, Former Under Secretary of Defense, Acquisition and Sustainment

VICTOR R. MCCRARY, Vice President for Research, University of the District of Columbia

J. MICHAEL MCQUADE, Special Advisor to the President, Carnegie Mellon University

KYLE MYERS, Assistant Professor, Technology and Operations Management Unit, Harvard Business School

ARUN SERAPHIN, Executive Director, Emerging Technologies Institute, National Defense Industrial Association

STEPHANIE S. SHIPP, Research Professor, Department of Statistics, Iowa State University

ROSEMARIE ZIEDONIS, Professor of Strategy and Innovation, Questrom School of Business, Boston University

Staff

GAIL E. COHEN, Senior Director

DAVID DIERKSHEIDE, Program Officer

ERIN ROONEY, Senior Program Assistant

EMILY SCHMITZ, Associate Program Officer (as of July 2024)

ATHENA SHRESTHA PRAJAPATI, Senior Finance Business Partner

Consultant

STEVE OLSON

Reviewers

This Proceedings of a Workshop was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making each published proceedings as sound as possible and to ensure that it meets the institutional standards for quality, objectivity, evidence, and responsiveness to the charge. The review comments and draft manuscript remain confidential to protect the integrity of the process.

We thank the following individuals for their review of this proceedings:

ANDRÉ GUDGER, Eccalon

JOHN MCKEON, Boston University

HALIMAH NAJIEB-LOCKE, Pallas Advisors

BHAVEN SAMPAT, Arizona State University

Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the content of the proceedings nor did they see the final draft before its release. The review of this proceedings was overseen by EVAN JOHNSON, University of North Carolina at Chapel Hill. He was responsible for making certain that an independent examination of this proceedings was carried out in accordance with standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the rapporteur and the National Academies.

This page intentionally left blank.

Contents

Introduction

1 National Security Considerations and Entrepreneurship

IMPLICATIONS FOR SBIR/STTR PROGRAMS

FURTHERING COMMERCIALIZATION

2 Using SBIR and STTR to Achieve DOD Missions and Goals

DIVERSE OBJECTIVES

METRICS FOR MICROELECTRONICS

GAUGING THE INTEREST OF COMPANIES IN DOD WORK

THE ROLES OF METRICS

PANEL DISCUSSION

3 Perspectives on Conventional Innovation and Commercialization Metrics

USES OF EXISTING DATABASES

NATIONAL ECONOMIC IMPACTS

METRICS USED IN THE EVALUATION OF SBIR/STTR PROGRAMS AT THE NIH

PANEL DISCUSSION

4 New Data and New Metrics for Evaluating Impact

DIRECT ECONOMIC IMPACTS

FINDING “INVISIBLE” INNOVATION

PANEL DISCUSSION

5 Perspectives from the Small Business Community

A LACK OF DATA ON COMMERCIALIZATION

ACCOUNTING FOR DEMAND-SIDE SUCCESSES

FLAWS IN THE SBIR DATABASE

FIRST PANEL DISCUSSION

MEETING NATIONAL DEFENSE NEEDS

PROVIDING VALUE TO END USERS

CONTINUING SUPPORT

SECOND PANEL DISCUSSION

6 Reflections on the Workshop

DAY 1 REFLECTIONS

FINAL THOUGHTS

Appendix Public Meeting Agenda

Box and Figures

BOX

I-1 Statement of Task

FIGURES

3-1 The total economic value of SBIR awards yields a high return on investment (green and yellow lines)

3-2 Based on FY 2000–2018 NIH SBIR/STTR application data and PubMed data, firms whose applications were awarded did better than firms whose applications were discussed but not awarded or not discussed

Acronyms and Abbreviations