7

Findings and Recommendations

The findings and recommendations in this chapter reflect the congressional objectives for the SBIR program, as reiterated in the recent program reauthorization and in the subsequent SBA policy Directive that guides program implementation at all agencies.¹ Section 1c of the Directive states program objectives as follows:

The findings below review the extent to which each of these program objectives is being addressed at NSF, as well as examine some specific aspects of program management.

FINDINGS

The Small Business Innovation Research (SBIR) program at the National Science Foundation is having a very positive overall impact. It is meeting three

___________________

¹See Box 1-2 and the discussion in Chapter 1 of the committee’s task.

²SBA SBIR Policy Directive, October 18, 2012, p. 3.

of the four legislative objectives of the program; we find that more needs to be done to “foster and encourage participation by socially and economically disadvantaged small businesses (SDBs), and by women-owned small businesses (WOSBs), in technological innovation.” Even so, the program is supporting the birth and growth of small innovative companies, which indirectly impacts all of the congressional objectives.

I. Commercialization

Each agency has its own priorities for the program. At NSF, the overwhelming emphasis has been on commercialization, which here means that projects are commercially successful in private sector markets.

1. SBIR projects at NSF commercialize at a substantial rate.³
   • Substantial Sales Reported: About 70 percent of Phase II respondents reported sales, based on responses to the 2011 Survey.⁴ This represents a slight increase from rates reported in the previous Academies report.
   • Sales Anticipated: An additional 19 percent of Phase II respondents reported that they anticipate future sales.⁵
   • Scale of Commercialization: Despite the high rates of commercialization, the scale of commercialization was limited: no projects surveyed reported aggregate sales of $50 million or more, and only 6 projects (out of 114) reported project-related sales of $10-50 million.⁶
2. Academies survey data show that NSF SBIR projects are primarily commercializing in the domestic private sector.
   • Just under 60 percent of respondents with sales reported revenues from domestic private sector customers.⁷
   • About 15 percent reported export customers.⁸
   • About 16 percent of responses identified customers in the public sector (primarily defense and defense contractors.)⁹
3. Further investment in SBIR is additional evidence of commercial activity. Subsequent investment provides further evidence that SBIR projects are expected to generate significant commercial value even if they have not yet reached the market. Academies survey data show that

___________________

³NSF declined to share the results of its own internal commercialization tracking system on confidentiality grounds.

⁴See Table 4-1.

⁵See Table 4-1.

⁶See Table 4-2.

⁷See Table 4-3.

⁸See Table 4-3.

⁹See Table 4-3.

4. • More than 60 percent of Phase II survey respondents reported additional investment funding.¹⁰
   • The most likely source of additional funding for Phase II recipients (other than their own company and personal funds) was non-SBIR federal funding, which is funding provided by from federal agencies from other budgets outside the SBIR programs (39 percent of responses), closely followed by other companies (32 percent), private equity/angels (28 percent). About 15 percent reported venture capital (VC) funding.¹¹
5. SBIR is associated with job growth. With respect to direct job growth, Academies survey data indicate that the median size of firms with NSF Phase II awards grew from 6 employees at the time of award to 10 employees at the time of survey.¹²
6. SBIR funding makes an important difference to project outcomes. SBIR funding makes the difference in determining project initiation, scope, and timing. Academies survey data show that
   • 68 percent of Phase II respondents reported that the project probably or definitely would not have proceeded without SBIR funding.¹³
   • 75 percent of those who would likely have proceeded anyway reported that the project would have been narrower in scope.¹⁴
   • About two thirds of those who would likely have proceeded anyway reported that the project would have been delayed by at least one year.¹⁵
7. Commercialization and venture funding
   • NSF has focused on an expected pattern of product development and commercialization that is modeled on that for projects funded by venture capitalists (VCs).
   • This model is not an appropriate model for NSF SBIR project commercialization for several reasons:
     • VCs seek projects that meet a number of narrow criteria, including timeline to market exit, size of opportunity, amount of funding required, capabilities of the management team, and industry sector. The vast majority of NSF SBIR projects do not and will not meet these criteria as they are typically specified by VCs.
     • NSF appears to make a point of funding very small companies at or soon after startup. The median size of Phase II company at

___________________

¹⁰See Table 4-6.

¹¹See Table 4-8.

¹²See Tables 4-4 and 4-5.

¹³See Table 4-19.

¹⁴See Table 4-20.

¹⁵See Table 4-21.

II. Fostering the Participation of Women and Other Underserved Groups in the SBIR Program

1. Current outcomes data show that this objective has not been met.
   • Levels of participation by underserved groups are low and declining.
     • Data from NSF indicate that approximately 10 percent of SBIR Phase I awards go to Minority-Owned Small Businesses (MOSBs). These figures have declined over the past decade, as has the MOSB share of Phase II awards.¹⁸ MOSBs also show lower success rates for both Phase I and Phase II applications than non-MOSB applicants, with the success for Phase I MOSB applicants being lower in every year, and success rate of Phase II MOSBs averaging 12 percentage points lower over the period.¹⁹
     • NSF data show that about 13 percent of SBIR Phase I awards were to Woman-Owned Small Businesses (WOSBs) and make up just over 10 percent of Phase II awards.²⁰ The higher share for WOSBs compared to MOSBs is based on the large numbers of awards won by 3 well established WOSBs.
     • NSF does not maintain data on woman and minority Principal Investigators (PIs). Data from the Academies survey indicates that these numbers are also low and not rising.
     • The Academies survey indicated that 13 percent of PIs were female.²¹
     • These discouraging results correspond to similar poor results across many STEM education fields in the United States.
   • NSF has no separate data on African-American-, Hispanic-, or Native American-owned small businesses.
     • The Academies survey, however, indicated that black- and Hispanic-owned small businesses are themselves a very small

___________________

¹⁶See Table 4-4.

¹⁷See Tables 4-6 and 4-8.

¹⁸See Figure 3-10 for percentages of SBIR Phase I awards going to MOSB, and Figure 3-24 for percentages of SBIR Phase II awards going to MOSB.

¹⁹See Figure 3-8 for Phase I MOSB comparative success rates for applications receiving awards, and Figure 3-22 for Phase II MOSB comparative success rates.

²⁰See Figure 3-15 for percentages of SBIR Phase I awards going to WOSB, and Figure 3-28 for percentage of SBIR Phase II awards going to WOSB.

²¹See Chapter 2, “Evidence provided by Academies Survey Data,” and 2011 Survey, Question 14A.

2. • • share of MOSBs overall. Black-owned small businesses accounted for approximately 1 percent of all respondents; Hispanic-owned firms, about 3 percent.²²

     • In the Academies survey, companies reported that 17 percent of Phase I PIs and 15 percent of Phase II PIs were minority. However, further analysis indicates that less than 1 percent overall were African American, with 1.5 percent in Phase I and none in Phase II. Less than 3 percent overall were Hispanic, with 3.9 percent in Phase I and 1.1 percent in Phase II. There were no Native Americans in Phase I or Phase II.²³
   • There are further questions raised by the data regarding the shares of Phase I and Phase II awards accounted for by WOSB.
     • The Academies survey data show that WOSBs accounted for 17.1 percent of Phase I companies but only 11.3 percent of Phase II companies responding to the Academies survey.²⁴
3. NSF’s efforts to “foster and encourage” the participation of woman-owned and minority-owned small businesses have not been effective.
   1. No substantial program has been implemented for outreach to these communities.
      • No examples or other evidence was found of persistent efforts by NSF to reach out to WOSB.
      • The NSF SBIR/STTR Advisory Committee also raised this issue in 2012.
   2. NSF does not report on or sufficiently track participation by WOSB and MOSB.
      • NSF does not provide an annual report that covers either data on participation or efforts to foster and encourage participation.
      • No evidence was found that NSF tracks MOSBs at a level sufficient to meet congressional intent.
      • No evidence was found that NSF tracks the participation of woman and minority principal investigators in the program.
4. There appears to have been insufficient effort made at NSF to understand why these rates remain low. Concerted effort would be needed to determine what practical steps can be taken to improve participation and hence both meet congressional objectives of the program and expand the pool of qualified applicants and capabilities.²⁵

___________________

²²Calculated from data reported in the text of Chapter 2, just prior to Table 2-9.

²³Calculated from data reported in Table 2-7 and accompanying text.

²⁴See Table 2-10.

²⁵A discussion of conclusion about woman and minority participation and NSF’s limited efforts to address the issue is provided in more detail in Chapter 2, “Conclusions: Woman and Minority Participation in the NSF SBIR Program.”

III. Stimulating Technological Innovation and Meeting Agency Mission Needs

NSF’s agency mission is focused on the advancement of science.²⁶ Thus for the NSF SBIR program, the twin objectives to use small business to meet federal R/R&D needs and to stimulate technological innovation are close complements and are therefore discussed together in this section.

1. Meeting federal R/R&D needs: The SBIR program at NSF supports the development and adoption of technological innovations. Selection of both topics and individual projects for funding reflects a strong focus on technological innovation.²⁷
   • Focused Topic Selection: Program Directors strongly assert that topic selection is driven by efforts to enhance and support innovation, and topics undergo a review process within the agency toward that end.
   • Solicitations are not Exclusive: NSF does not formally require that projects specifically match the topics described in the solicitation. Under the terms of the solicitation, non-matching applications can also be accepted, in an effort to ensure that potentially important innovations are not excluded by the topic structure. However, NSF does not track whether applications match the solicitation topics, there is no evidence to confirm or refute if solicitations are nonexclusive.
2. Emphasis on Technological Innovation: Scoring for individual projects is weighted toward technological innovation: NSF merit criteria emphasize the importance of technical innovation, and selection decisions are—according to Program Directors who make recommendations for those decisions—also heavily weighted toward the support of potentially transformational technologies. SBIR continues to connect companies and universities. Survey data indicate that NSF SBIR projects continue to utilize universities in a variety of ways (excluding the even closer connection through Small Business Technology Transfer [STTR]):
   • Faculty and Student Participation: Just under 60 percent of Phase II survey respondents reported a link to a university. In about 35 per-

___________________

²⁶“The National Science Foundation (NSF) is an independent federal agency created by Congress in 1950 “to promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense…” See National Science Foundation, “NSF—At A Glance,” accessed on June 2, 2014. A more extended description from the same source states that goals are “to advance the frontiers of knowledge, cultivate a world-class, broadly inclusive science and engineering workforce and expand the scientific literacy of all citizens, build the nation’s research capability through investments in advanced instrumentation and facilities, and support excellence in science and engineering research and education through a capable and responsive organization.”

²⁷See Chapter 2 for a discussion of NSF’s topic selection and funding decisions.

3. • cent of responses, university faculty worked on the project (not as PI); 30 percent employed graduate students; and in almost a quarter of Phase II projects reporting, universities were subcontractors.²⁸ These figures are broadly similar to those reported from the Academies’ 2005 survey.

   • Project Partners: 114 different universities were identified by survey respondents as project partners; 13 were mentioned by more than 3 respondents.²⁹
   • Academic Founders: More than 80 percent of NSF Phase II SBIR companies responding to the survey reported at least one academic founder, and about a quarter reported that the most recent prior employment of the founder was at a university.³⁰
4. Academies survey data show that SBIR projects generate substantial knowledge-based outputs such as patents and peer reviewed publications.
   • Patents: Patenting remains an important component of knowledge diffusion (and protection).
     • More than 70 percent of the Academies’ Phase II survey respondents reported filing at least one patent related to the surveyed project.
     • 23 percent reported filing five or more related patents.
   • Publications: Publication of peer reviewed articles remains the primary currency of scientific discourse, and despite the need to protect ideas in the commercial environment of small businesses, the Academies’ Phase II Survey shows that SBIR firms continue to participate deeply in scientific publication.
     • About 80 percent of surveyed Phase II projects reported at least one resulting peer reviewed publication.³¹
     • 32 percent reported more than three publications resulting from the surveyed project.³²
     • Many of the companies interviewed for case studies made a point of indicating that they take a great deal of pride in the number of peer reviewed publications developed by their scientists and engineers, both within and outside of the SBIR program.
5. SBIR funds projects with social benefits that may not be attractive to commercial sources of funding.³³

___________________

²⁸See Table 4-17.

²⁹See Table 4-18.

³⁰See Table 4-28 and Chapter 4 text, just before “Conclusions: Knowledge Effects.”

³¹See Table 4-16.

³²See Table 4-16.

³³See Chapter 6, “Funding Otherwise Un-fundable Projects.”

IV. Fostering Innovative Companies

1. Company formation and early funding: Many SBIR companies surveyed by the Academies report reported that SBIR funding was instrumental in the founding of the company. The formation of new innovative companies is a positive outcome for the program.
   • 45 percent of Phase II respondents to the Academies survey said that the company was founded entirely or in part because of the SBIR program.³⁵
   • For some companies included among the case studies, SBIR funding permitted the shift from an exploratory to a professional operation.³⁶
2. De-risking subsequent investment and new markets
   • Mitigating Risks: One important impact of the NSF SBIR program is to provide funding that reduces the risk for subsequent investors. Risk mitigation often leads to the leveraging of SBIR funding while retaining the power of markets to make final decisions about funding.³⁷
   • Matching Funds: The NSF Phase IIB program lowers risks by providing matching funds to companies with external investors or other approved sources of funding.³⁸
   • Early Stage: More widely, many respondents to the Academies survey and a number of case study interviewees said that NSF SBIR funding was provided at a stage when the project was sim-

___________________

³⁴See Appendix E: Touch Graphics Case Study.

³⁵See Table 4-29.

³⁶See “Validation Effects” in Chapter 6, and Box 6-7.

³⁷See section on “Risk” in Chapter 6, and survey results for risk reduction in Box 6-4.

³⁸See discussion in Chapter 5 of requirements and effects of matching funding for Phase IIB projects, and tables 5-2 and 5-3 that show a 3:1 ratio of matching investments to NSF funding over a dozen years.

3. • ply too risky for commercial sources of funding. Once the project proceeded further, risk was lower and additional funding could be acquired.³⁹

   • Support for Technology Development: NSF SBIR funding supports technology development, which can be supported through commercial funding further downstream. SBIR is particularly important for funding proof of concept for new technologies.⁴⁰
   • Validation: NSF SBIR funding has itself been important validation for companies seeking further investments, according to case study interviews and survey responses. The strength of the selection process and growing understanding of SBIR among investors appears to be strengthening this effect.⁴¹
   • NSF funding can support company efforts to enter new markets.⁴²
   • Exploit technology platforms: In some cases, companies use SBIR funding to build off existing platform technologies specifically to enter new markets. This platform-driven approach is used by a number of the interviewed companies. (See Chapter 6.)
   • Strategic corrections: Innovative companies must often make mid-course corrections. NSF funding has—according to Academies survey recipients—helped a number of surveyed companies successfully make what are often difficult changes that are hard to fund.
4. Long term positive effect on SBIR awardee companies. Commercialization in the long run requires sustainable companies, and SBIR has supported the development of an ecosystem of small innovative companies in the United States.⁴³ The Academies survey provided SBIR companies with the opportunity both to report the overall impact of SBIR on the company, and to identify specific kinds of impacts.
   • 35 percent of Phase II winning recipients indicated that the NSF SBIR program had a “transformative” effect on their company. Another 54 percent said that it had a “substantial positive long-term effect.”⁴⁴
   • Of the 179 detailed comments received in the Academies survey, none reported negative effects. Widely differing kinds of impacts were reported, as summarized in Box 7-1.⁴⁵

___________________

³⁹See “Company Formation and Very Early-stage Funding” in Chapter 6, and Box 6-2.

⁴⁰See “Funding Core Technology Development” in Chapter 6, and Box 6-5.

⁴¹See “Validation Effects” in Chapter 6, and Box 6-7.

⁴²See “Expansion into New Markets” in Chapter 6.

⁴³See “Capacity Building—Human Capital” in Chapter 6, and Box 6-8.

⁴⁴See Table 4-23.

⁴⁵See “Key Aspects of SBIR-Driven Transformation” in Chapter 4, as well as Chapter 6 and cases in Appendix E.

4. Limiting Company dependence on SBIR
   • Limits on Multiple Awards: The NSF program tightly limits the capacity of firms to garner multiple awards.⁴⁶
     • Firms are permitted to apply for only two awards per year.⁴⁷
     • The most prolific firms received fewer than 10 SBIR Phase II awards over a period of 10 years.⁴⁸
     • Interviews with Program Directors indicate a preference for spreading Phase II awards more widely.⁴⁹
   • NSF firms are not dependent on SBIR awards: Less than 20 percent of companies responding to the 2011 Survey report that SBIR accounts for more than half of current revenues;⁵⁰ however, a considerable number of surveyed firms reported in textual responses that SBIR is the most important source of funding prior to reaching the market.⁵¹
   • SBIR Innovation is nonlinear: Most projects at most companies do not proceed directly from Phase I to Phase II to commercialization.⁵²
     • About 85 percent of Phase II survey respondents reported at least one additional SBIR Phase II award related to the surveyed project.⁵³
     • About one third reported at least two additional related Phase II awards.⁵⁴
     • As noted above, more than 60 percent of Phase II respondents report additional investment funding related to the project subsequent to the SBIR award.⁵⁵

V. The Phase IIB Program

The NSF Phase IIB program supports the accelerated commercialization of SBIR-funded research through the provision of matching NSF funds provided companies can generate additional investment funding of their own. The evidence shows that Phase IIB projects tend to commercialize more than Phase II-only projects. In part, this reflects the rigorous selection process—NSF selects Phase IIB projects specifically for their commercial potential, so improved commercial

___________________

⁴⁶Previous Academies studies discussed the question of multiple SBIR awards to companies. The 2009 report concluded that “mills” are not a significant problem.

⁴⁷National Science Foundation, June 2014 SBIR program Solicitation, NSF-13-599, p. 1.

⁴⁸See Chapter 3, Annex Table 3-6.

⁴⁹See text in Chapter 3 at the end of the section, “New Companies in the SBIR Program.”

⁵⁰See Table 4-36.

⁵¹See Chapter 6 and case studies in Appendix E.

⁵²See Chapter 4’s section, “Prior Use of the SBIR Program.”

⁵³See Table 4-38.

⁵⁴See Table 4-38.

⁵⁵See Table 4-6.

outcomes should be expected. Moreover, evidence from surveyed recipients and from case studies strongly suggests that the program does have a catalyzing effect, attracting additional funding, and that it does make a positive difference to company activities and outcomes.⁵⁶

1. The Academies survey of Phase IIB projects reports increased commercialization and larger revenues.
   Phase IIB program attracts third-party investors, thereby accelerating and enhancing the commercialization of SBIR-funded technologies:
   • Phase IIB projects reach the market at a high rate—81 percent of surveyed Phase II projects reported that they had achieved some sales.⁵⁷
   • Products in use. Just under half of the respondents to the Phase IIB survey report that their products, processes, or services are in use today.⁵⁸
   • Projects with large revenues. Almost a quarter of Phase IIB projects surveyed reported project-based revenues of $3 million or more percent. Median reported sales were $650,000.⁵⁹
   • Important role of Phase IIB. Eighty percent of respondents to the Phase IIB survey indicated that Phase IIB has accelerated commercialization of their product or services.⁶⁰ This is an important point because the program by design focuses on firms with maximum commercial potential. Confirmation from firms about the role and impact of the program is therefore highly relevant.
2. The Academies survey of Phase IIB projects reports employment effects and opportunities for female PI’s.
   • Employment growth. Respondents to the Phase IIB survey reported employment growth from a median of 8 employees at the time of the award to 17 at the time of the survey.⁶¹
   • More opportunities for female principal investigators. Phase IIB respondents reported more female principal investigators than Phase II projects—18 percent versus 7 percent.⁶²
3. The Academies survey of Phase IIB projects reports that by requiring matching funds, the Phase IIB program provides incentives for firms to acquire additional investment.

___________________

⁵⁶See Chapter 5 for a description of the Phase IIB program and its effects.

⁵⁷See Table 5-8. See also Table A-5 for information on the Phase IIB survey responses. A total of 281 survey responses were received, reflecting an effective response rate of 48.3 percent.

⁵⁸See Table 5-7.

⁵⁹See Table 5-9.

⁶⁰See Table 5-27.

⁶¹See Tables 5-11 and 5-12.

⁶²See Table 5-6.

___________________

⁶³There were no statistically significant differences between survey responses from Phase IIB respondents and agency data from the NSF awards database. See Table 5-23.

⁶⁴See Table 5-23.

⁶⁵See Table 5-24.

⁶⁶See Table 5-17.

⁶⁷See Table 5-26.

⁶⁸See Table 5-26.

⁶⁹See Table 5-29.

⁷⁰See Table 5-30.

4. paperwork” or “less than 2 weeks FTE for company staff.”⁷¹ About one third of respondents believed the application process was easier than for comparable Federal R&D applications, while nineteen percent thought it more difficult.⁷²

5. Companies report positive views of Phase IIB impacts.
   Companies had remarkably positive views of the impact of Phase IIB funding on both their projects and their companies as a whole.
   • Worth Applying: Ninety-eight percent of Phase IIB respondents believed that the funding they received was worth the effort involved in applying—even though more than a fifth had spent at least two months of senior management time on the application.⁷³
   • Long-term Impact: Ninety-six percent of Phase IIB respondents indicated that Phase IIB had a positive long-term impact on the company. Nearly two-thirds stated that the program had a substantial positive impact, and nearly one in ten responded that the program had had a “transformative effect” on the company.⁷⁴
   • Broad Support: Comments from respondents indicate that Phase IIB supports a wide range of activities tied to a considerable variety of commercialization strategies and approaches.
6. Other Aspects of the Phase IIB program.
   • Strong support for companies from NSF program officers. Survey respondents and interviewees both had generally strongly positive views on the role of NSF program officers in the application process (and indeed during the course of the award—see Chapter 2 and Appendix E).
7. Improving Phase IIB. The study draws attention to potential areas of improvement for the Phase IIB program. These include the following:
   • Confusion about matching funds criteria. There was considerable confusion even among recipients about the criteria for identifying appropriate matching funds. Many respondents identified funding sources that were in fact not acceptable as matching funds; and many failed to identify equity funding and sales, both of which are permitted.⁷⁵
   • Matching funds criteria. Aside from the confusion about criteria noted above, almost half of Phase IIB awardees indicated that NSF matching fund criteria were too stringent in excluding in kind con-

___________________

⁷¹See Table 5-19.

⁷²See Table 5-20.

⁷³See Table 5-34.

⁷⁴See Table 5-28.

⁷⁵See Table 5-21.

VI. Program Management

1. The NSF Management Model
   • Key Role of Program Directors: The NSF model relies on a small cadre of highly qualified Program Directors who have strong technical and commercial backgrounds to provide guidance and support to companies that apply for SBIR grants and that seek to commercialize SBIR technologies. Evidence from the Phase II survey and from case studies suggests that in many cases the relationship between the Program Director and the company is strong and that the model works as anticipated.⁷⁷
   • There is evidence from several sources that the Program Directors face a heavy workload.⁷⁸
     • Program Directors interviewed for this study indicated that they are responsible for about 30 Phase II awards at any one time. Given this workload, it is not clear how much time they are able to devote to any one company.
     • Phase II Recipient survey responses indicate that in some cases companies believe that the workload for Program Directors is too great for them to provide appropriate support.
2. Best Practices
   • Effective Program Directors: Companies strongly appreciate the guidance offered by program officers in most cases, and in interviews suggest that they benefit considerably from the high quality of the NSF program directors.⁷⁹
   • Limits on submissions per company: These limits are an appropriate mechanism for ensuring that the number of submissions is limited while at the same time not excluding any specific project.

___________________

⁷⁶See Table 5-37.

⁷⁷See “Role and Effectiveness of Program Directors” and “Qualifications of PDs” in Chapter 2 and Box 2-6.

⁷⁸See Tables 2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, and 2-18, and related text in Chapter 2.

⁷⁹See Box 2-6 in Chapter 2.

3. • Companies decide which projects to submit, reducing the number of applications that program staff must review and ensuring that if a project is funded, it will be one that was at or near the top of the company’s priority list.⁸⁰

   • Inclusive Topic Deployment: The broad topics offered by NSF—in conjunction with the increase in the number of solicitations by topic to two per year—indicate that the agency is working hard to ensure that promising projects are not excluded unnecessarily. NSF program officers indicate that they believe any technically and commercially worthwhile project could be funded under the published solicitation guidelines.⁸¹ Some companies, however, disagree.
   • The Phase IIB Program: This program has been an important initiative at NSF, and has since been adopted in different forms at other agencies.⁸²
   • Partnerships: NSF is to be commended for exploring a range of partnerships and mechanisms through which to support the commercialization of funded projects.
   • i-Corps, an NSF program that offers entrepreneurship training (primarily for academics), seeks to ensure that some key steps in the development of a sustainable company are taken at an early stage in the product development cycle.⁸³
   • The Innovation Accelerator, a private organization partnering with NSF to facilitate the commercialization efforts of high-technology small businesses funded by grants from the NSF SBIR program, is also potentially valuable, although its reach may have to be expanded beyond the information technology sector.
4. Key Concerns
   • Data Tracking, Management, and Use: There are broad challenges in tracking commercialization, at both the company and project levels. Companies move in and out of the program, and tracking is harder once they have left. More generally, commercialization may come years after an award, and may involve multiple awards plus considerable additional funding. All this makes it difficult to assert that any specific outcome “results from” an SBIR award. But there are also specific challenges with existing tracking tools.
   • Recognition of NSF efforts to track outcomes. NSF’s existing effort to track outcomes is commendable. It predates similar efforts at other agencies by a number of years, and by seeking information

___________________

⁸⁰See, for example, Chapter 3, last paragraph before “Demographics of Phase I Winning Companies.”

⁸¹See “Topic Selection” in Chapter 2, and Box 2-9.

⁸²For a description of the Phase IIB Program at NSF, see Chapter 5.

⁸³See I Corps program highlights in Chapter 2, Box 2-3.

___________________

⁸⁴It appears that this may be in part a result of compliance with the Paperwork Reduction Act, which limits agency tracking capabilities.

RECOMMENDATIONS

While the NSF SBIR program generates substantially positive outcomes, the committee has identified a series of recommendations to improve its processes and outcomes.

I. Addressing Underserved Populations

NSF should immediately examine past and current efforts to address the clear congressional mandate to foster the participation of underserved populations in the SBIR program, examine and report on best practices and create benchmarks to relate the impact of such activities.

1. Quotas are not necessary. NSF should not develop quotas for the inclusion of selected populations into the SBIR program. Such an approach is not necessary to meet congressional intent, and is likely to reduce program effectiveness.
2. NSF should develop new benchmarks and metrics.⁸⁶
   • Improve participation metrics: The SBIR/STTR program office should work with the NSF indicators group to develop much improved metrics for benchmarking the participation of underserved populations, developing and publishing clear benchmarks based on a defensible analysis of existing data.
   • Disaggregate Benchmarks: Measures of the participation of socially disadvantaged groups must be disaggregated by race or ethnicity, and attention focused on the clear congressional intent to support “minority” participation. We do not believe a focus on the current SBA definition of “socially and economically disadvantaged” is sufficient to meet this objective.

___________________

⁸⁵But others see the NSF Phase I and Phase II success rates as not low relative to the acceptance rates of other competitive R&D funding programs. Similarly, some see failed applications as a waste of company and reviewer time, while others see knowledge benefits and capacity building in company efforts to develop proposals for the SBIR program even if they are not successful in obtaining an SBIR grant. Formulated plans may go on to generate benefits in other ways.

⁸⁶See Finding II-A and II-C.

3. • Customize Benchmarks: Points of reference should be developed separately (though perhaps drawing on a shared methodology) for women and minorities. These benchmarks should be shared with other SBIR agencies. Benchmarks should address key questions that would include the following metrics, all of which should include both absolute levels and trends over time:
     • Shares of applications from companies owned by women and minorities
     • Shares of applications with woman and minority principal investigators
     • Shares of Phase I awards to companies owned by women and minorities
     • Shares of Phase I awards with woman and minority principal investigators
     • Shares of Phase II awards to companies owned by women and minorities
     • Shares of Phase II awards with woman and minority principal investigators
     • Shares of Phase IIB awards to woman and minority owned firms
     • Shares of Phase IIB awards with woman and minority principal investigators
   • Track Related Program Operations: Metrics should also track related program operations including outreach efforts. (See below.)
4. NSF should undertake an investigation to understand better why its efforts to date to expand the participation of under-served populations have been largely unsuccessful.⁸⁷
   • Develop Outreach Strategy: Based on this investigation, NSF should develop a coherent and systematic outreach strategy that provides for cost-effective approaches to enhance recruitment of both woman- and minority-owned companies and female and minority PI’s, developed in conjunction with other stakeholders and with experts in the field, including relevant programs at the state level. Outreach efforts should aim to expand SBIR awareness among potential applicants from underserved demographics.
   • Integrate Outreach Effort: NSF should ensure that outreach to selected populations is an integral part of its overall outreach.
   • Provide Management Resources: NSF should provide significant management resources given that improving participation is likely to be both difficult and a long term effort.

___________________

⁸⁷See Finding II-B.

4. • Designate Staff: NSF should consider designating a senior staffer to work exclusively on improving women and minority participation in order to identify strategic initiatives, implement them, and provide consistent and timely reporting on their impacts.
5. NSF should review internal award and selection data and processes to identify and address disparities between Phase I and Phase II awards to selected populations. The goal is to ensure that there are no biases in the selection process that are adversely affecting the selection of women and minorities.⁸⁸
   • NSF should carefully review patterns of success rates for WOSB and MOSB to identify potential disparities.
   • As part of the review, NSF should investigate the possibility of selection-bias in the review process and take actions to make sure that it is avoided—including ensuring that review panels are reasonably diverse.

II. Commercialization

The NSF SBIR program is tightly focused on commercialization that is based on a venture capital model. However, it is worth considering some possible adjustments to the current approach.

• Broaden Perspective to include strategic partners:⁸⁹ The limited funding for seed and startup projects from U.S. venture capital in general, and the low numbers of NSF firms that report venture capital funding, suggests that NSF should not focus too tightly on commercialization models that rest on venture capital funding. Many alternatives exist—for example, a deeper focus on finding strategic partners. The VC-focused commercialization model narrows the program in several ways—by limiting the timeframe viewed as appropriate for commercialization, and also by anticipating certain levels of commercial scale needed to attract VC-type funding. We recommend that NSF review its conceptual approach to commercialization with a view to ensuring that different paths to commercial success are fully included, such as angel funding and strategic investments by other companies. In addition, NSF should explore newer alternatives—for example, open source models or use of equity crowdfunding. This is to some degree implicit in the I-Corps model, and NSF should ensure that its grantees are aware of alternative funding paths.

___________________

⁸⁸See Finding II-A, points 2 and 3.

⁸⁹See Finding I-F.

III. Phase IIB

1. NSF should continue to operate the Phase IIB program.⁹⁰
   • The program has had a positive impact on the commercialization of SBIR funded research.
   • Respondents strongly support retention of the program.
2. NSF could consider expanding the size of the Phase IIB program.⁹¹
   • The strongly positive impact suggests that there may be additional projects that could benefit from Phase IIB funding.
   • But NSF should be cautious about the impact of expansion on other aspects of the program. A majority of Phase IIB respondents would not support the addition of Phase IIB awards at the cost of fewer Phase I or Phase II awards.
3. NSF should not increase the dollars per Phase IIB maximum award if that reduces the number of Phase IIB awards.⁹²
   • The program appears to successfully meet its mission at current award levels. Two-thirds of respondents agreed that the amount of funding was about right.
   • NSF should be cautious about reducing the number of Phase IIB awards in order to increase the size of each award. The analysis identified no evidence to suggest that this would be an improvement to the program or that recipients would support such an adjustment.
   • Survey responses indicated that while some supported such an adjustment, more than twice as many did not favor a trade-off of larger awards for fewer awards.
4. NSF should clarify the criteria under which funding qualifies as an acceptable match for Phase IIB purposes.⁹³
   • Survey and interview data indicated confusion among recipients (and presumably potential applicants) about acceptable criteria, and there are inconsistencies between different statements on the NSF Phase IIB web page.
5. Matching funds requirements.⁹⁴
   • Explore use of In-Kind Contributions: NSF may wish to explore allowing the limited use of some specified in-kind contributions as part or all of the matching funds. Nearly half of survey respondents agreed. Perhaps NSF might opt to allow in-kind contributions in “exceptional circumstances,” offering a limited but potentially important degree of flexibility.

___________________

⁹⁰See Finding V-A.

⁹¹See Finding V-A.

⁹²See Finding V-C.

⁹³See Finding V-F

⁹⁴See Finding V-F.

IV. Improving Monitoring, Evaluation, and Assessment

The development of more careful monitoring and more sophisticated analysis of key variables are necessary to improve program outcomes. The NSF deserves considerable credit for its early efforts to track outcomes, but more remains to be done in this area. Newly available administrative funding can improve this area of program operations.

1. Improving current data collection approaches and methodologies.⁹⁵ Data collected through the current process are a good start but are far from sufficient to underpin a data driven program.
   • Improve Data Collection and Organization:
     • NSF should more systematically collect a range of quantitative data and standardize key questions to improve program evaluation, management, and outcomes.
     • This data collection effort should address the entire range of congressionally mandated outcomes, not just commercialization.
     • NSF should develop a dataset that can provide a basis for longitudinal analysis.
   • Track Commercialization Outcomes:
     • NSF should track commercialization outcomes in ways similar to the now widely-accepted methodology developed for the Academies studies. This approach focuses on multiple metrics in order to provide a deeper and more nuanced basis for analysis. (See methodology discussion in Chapter 1 and Appendix A).
     • Although NSF already tracks outcomes for several years after the end of an award, this period should be extended to more fully assess long term impacts as these may not become fully discernible during a more limited period of review.
   • Manage Information:
     • NSF should explore the development of an integrated information management system to improve the management of its SBIR program.
     • NSF is now developing a database of Phase IIB projects for which it is to be commended. However, program management

___________________

⁹⁵See Findings II-A and VI-C.

2. • would be on a firmer basis if managers had readily at hand information about company participation in all of the program’s activities, including Phase IIB, various supplements, I-Corps, and the Innovation Accelerator.

   • Collect Demographic Data:
     • NSF should take immediate steps to improve its collection of demographic data, using the Fastlane electronic submission program.
     • NSF should extend its collection of the demographics of company ownership to show which of the SBA’s socially and economically disadvantaged categories an applicant belongs to. In addition, applicants should be asked the same demographic questions about the principal investigator.
   • Collect qualitative outcomes data:
     • NSF should also develop and adopt a more systematic and critical approach to the use of detailed case studies and success stories. Success stories can provide inspiration lessons learned, and important information not available elsewhere about program impacts. Case studies can inform the roles played by SBIR awards, the challenges faced by small businesses, insights into needed improvements in process, lessons learned, and other important information not available elsewhere about program impacts.
3. Use available commercial tools to gather more current data on a long-term basis. While we recommend utilizing modern information management and data visualization tools, we stress that effective products are widely available on the commercial market.⁹⁶
   • Use New Technologies: NSF should explore ways to use new technology such as social media to collect more current data. SBIR companies—like “customers” in other markets—are an important source of information about program strengths and weaknesses. This knowledge is currently not systematically included in internal program evaluation by NSF’s SBIR program.
   • Develop Feedback Tools: NSF should develop pathways to provide ongoing feedback from companies about program activities and operations. These could include various electronic communication tools.
   • Improve Networking: Similarly, NSF should consider developing mechanisms—like electronic tools—through which recipients can share information about their SBIR projects, helping them both to find technical or marketing or investment partners and to navigate

___________________

⁹⁶ See Finding VI-C.

3. the often-complex regulatory and technical environment of NSF programs.

4. Improved Data Utilization. NSF has collected data on outcomes for some years, but it is not clear whether these data are systematically employed to guide program management.⁹⁷
   • Develop a Plan for Data Analysis: NSF should seek to develop a more sophisticated approach to analyzing and applying the data that are already collected. For example, there appear to be no plans in place within NSF to use the data collected over a number of years to evaluate factors that tend to encourage successful transitions between Phases, into Phase IIB, and then into full scale commercialization. Such an approach could identify key issues for program management.
   • Undertake Regular Analysis of Data: By collecting more and better data on outcomes and participation, NSF will be positioned to undertake regular analysis—either internally or with third-party help—on key program management issues, such as
     • What is the long term impact of partnership programs and other commercialization supports?
     • Is Phase IIB simply picking successful companies or is it at least, in part, causing companies to be successful?
     • How well do NSF selection processes predict eventual successful projects?
   • Recognize Impacts of Data Collection and Analysis: In some cases, simply measuring something more closely will likely provoke action. Closely tracking the participation of women and minorities could help assure a fair process and surface problem issues early, when they can be most easily corrected. For example, systematically tracking woman and minority participation would have surfaced the issue of differential Phase I and Phase II success rates identified earlier.
5. Annual Report to the National Science Board and Congress: The SBIR/STTR program should provide a comprehensive annual report to the National Science Board and Congress and the public on its operations. While the precise details should be left to the agency we would recommend that it consider including six areas of program operations:⁹⁸
   • Inputs, including aggregate current and longitudinal data on numbers of applications and awards, broken out by relevant subgroups (such as demographics, state)

___________________

⁹⁷ See Finding VI-C.

⁹⁸ See Findings II-B and VI-C.

5. • Program initiatives, which would provide NSF with an opportunity to describe the numerous and impressive initiatives undertaken by the program
   • Aggregated outputs drawn from an improved tracking, outcomes collections, and analysis process
   • Qualitative review, based on improved use of case studies, as well as success stories and social media
   • Impact assessment, focused on the extent to which NSF meets congressional objectives for the program. Impact assessment should be contracted out to ensure objectivity.
   • Summary conclusions, including prospective views on program activities and improvements for the coming year.

   While caution should be employed when imposing new reporting burdens on the NSF SBIR program, implementation of an improved information management system would provide a cost- and time-effective basis on which to provide better reporting on the program. An annual report would provide much improved transparency and would provide a coherent point of discussion for other stakeholders. It would effectively replace the existing report to SBA which is of limited utility for NSF or other stakeholders, and could be organized to meet some of the new reporting requirements imposed under reauthorization.

6. Additional and better data collection tools and analysis consume staff time and resources. NSF should ensure that its staff is incentivized, and that sufficient funding is available for this purpose.⁹⁹

V. Improving Program Management

Recommendations in this section are designed to improve program operations in ways that should enhance the program’s ability to address some or all of its objectives.

1. NSF should seek to reduce the burden on program directors.¹⁰⁰
   • Review Program Director Workloads: NSF should consider whether the current workload is appropriate. The NSF model calls for a mentoring relationship between Phase II winners and Program directors, and for a closer than normal relationship between Phase I winners and Program directors. This is hard to sustain given current levels of activity with each program director responsible for about 30 Phase II awards and perhaps twice that many Phase I awards.
   • Consider Ways to Make the NSF Model Work Better: While Providing Individual Attention to Projects and Not Overburdening PDs:

___________________

⁹⁹ See Finding VI-C.

¹⁰⁰ See Finding VI-A.

2. • For example, consider adding part-time support staff or an additional PD to make the workloads more manageable, such as a PD to manage proposals in topics outside the announced solicitation.

   • Consider Virtual Meetings: NSF should also consider whether the current approach to identifying candidates to be recommended for awards could be streamlined as suggested by the NSF SBIR/STTR Advisory Committee through the use of more virtual meetings—both for review panels and for review of Phase IIB candidates where currently all have to go through an in-person interview process.
3. NSF should consider ways to improve the average quality of applications.¹⁰¹
   • Encourage firms to submit higher quality proposals: NSF’s use of a pre-selection White Paper approach, which provides guidance and feedback to prospective applicants, appears to be partially effective in improving quality while reducing the number of applications, as is the 2 per company limit. NSF should look to find additional ways to attract fewer and higher quality applications.
   • Eliminate use of applicant success rates to validate program quality. NSF should at a minimum not use low success rates to validate program quality, as this in part simply reflects the receipt of weak proposals. Screening out more weak proposals before application would raise success rates for the remaining applications, while maintaining or even improving the quality of the awardees.
4. NSF should consider the adoption of further mechanisms to reduce funding gaps between Phase I and Phase II.¹⁰²
   Some of the mechanisms used at other agencies include a Fast Track process for merging Phase I/Phase II applications; use of agency options as bridge funding; early identification of projects that are likely to be funded. NSF could explore these and other options.
5. NSF should investigate how open in its solicitations are in practice. It should track the number of proposals by topic and determine the percentage that fit the announced solicitation topic(s). It should explore whether NSF approach to topics is sufficient to keep up with evolving opportunities. The relationship between topics and the number of applications should also be subject to ongoing review.

___________________

¹⁰¹ See Finding VI-C.

¹⁰² See Finding VI-C.