Appendix B

Agendas of Open Committee Meetings

First Meeting
DECEMBER 12-13, 2017

TUESDAY, DECEMBER 12, 2017

Welcome and Introductions

Mary Ellen O’Connell, Executive Director, Division of Behavioral and Social Science and Education

Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation

The Scientific Enterprise

Edward (Ned) Hall, Committee Member; Chair, Department of Philosophy, Harvard University

National Science Foundation’s Interests and Goals for the Study

Joan Ferrini-Mundy, Chief Operating Officer, National Science Foundation

Perspectives on Reproducibility and Replication: Scientific Societies, Part I

Panelists, primarily leaders from U.S. scientific societies and organizations, have been asked to focus on the following topics:

• Within your field of science, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability of research results?
• Are there specific areas within your field of science that are more likely to have issues with reproducing scientific results?
• What reproducibility challenges does your field of science face with cross disciplinary research?

Behavioral and Social Sciences

William G. Jacoby, Department of Political Science, Michigan State University; Editor, American Journal of Political Science

Howard S. Kurtzman, Acting Executive Director, Science Directorate, American Psychological Association

Felice J. Levine, Executive Director and Ethics Officer, American Educational Research Association

Physical Sciences

Kate Kirby, Chief Executive Officer, American Physical Society

David Sholl, John F. Brock III School Chair, School of Chemical and Bio-molecular Engineering, Georgia Institute of Technology

Statistics

Ron Wasserstein, Executive Director, American Statistical Association

Earth Sciences

Brooks Hanson, Senior Vice President Publications, American Geophysical Union

Engineering

Philip DiVietro, Managing Director of Publishing, American Society of Mechanical Engineers

John Baillieul, Distinguished Professor, Department of Mechanical Engineering, Boston University

Public Comments

WEDNESDAY, DECEMBER 13, 2017

Welcome, Day One and Day Two Overviews

Harvey Fineberg, Committee Chair

Perspectives on Reproducibility and Replication: Scientific Societies and Agencies, Part II

1. Panelists, primarily leaders from U.S. scientific societies and organizations, have been asked to focus on the following topics:
   • What reproducibility challenges does your field of science face with cross-disciplinary research?
   • Within your field of science, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability of research results?
   • Are there specific areas within your field of science that are more likely to have issues with reproducing scientific results?

Life Sciences

Yvette Seger, Director of Science Policy, Federation of American Societies for Experimental Biology

Reproducibility of Scientific Research within the Agencies

Patricia Valdez, Extramural Research Integrity Officer, National Institutes of Health

Anne Plant, Biosystems and Biomaterials Division, National Institute of Standards and Technology

2. International panelists have been asked to focus on the following topics:
   • What is the level of awareness, interest, concern, and involvement in reproducibility and replicability of research results within your national scientific societies?
   • Are there specific areas of science that are more likely to have issues with reproducing scientific results?
   • What reproducibility and replicability issues exist for cross-disciplinary research?

Eric-Jan Wagenmakers, Professor of Psychology, University of Amsterdam

Jean Phillipe de Jong, The Dutch Royal Society of Sciences

3. The editor of a major cross-disciplinary journal was asked to focus on the following questions:
   • Can journals assess levels of R&R across science?
   • What R&R challenges does cross-disciplinary research pose that can be addressed by journals?
   • Are cross-disciplinary papers handled differently from “pure” science papers in terms of peer review or publishing decisions?
   • What R&R challenges does cross-disciplinary research pose that can be addressed by journals?

Veronique Kiermer, Executive Editor, Public Library of Science

Reporting of Reproducibility Issues in Science

Richard Harris, Science Correspondent, National Public Radio

Public Comments

Second Meeting
FEBRUARY 22-23, 2018

THURSDAY, FEBRUARY 22, 2018

Welcome and Introductions

Harvey Fineberg , Committee Chair; President, Gordon and Betty Moore Foundation

Perspectives on Reproducibility and Replication: American Economic Association

The speaker has been asked to focus on the following questions:

• Within economics, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability of research results?
• Are there specific areas within economics that are more likely to have issues with reproducing scientific results?
• What reproducibility challenges does economics face with cross-disciplinary research?

Margaret Levenstein, Professor of Economics and Director, Inter-university Consortium for Political and Social Research, University of Michigan

Panel 1: Overview of Extent of Reproducibility Issues in Scientific and Engineering Research

The panelists have been asked to focus on the following session questions:

• How extensive is the lack of reproducibility in research results in science and engineering, in general?
• At what level does a lack of reproducibility become a problem for the wellbeing of science or engineering?
• Does the lack to reproduce scientific results impact the public perception of specific scientific fields and/or science and engineering in general?

John Ioannidis, C.F. Rehnborg Chair in Disease Prevention and Co-Director, Meta-Research Innovation Center, Stanford University

Brian Nosek, Director, Center for Open Science and Professor of Psychology, University of Virginia

Daniel Sarewitz, Co-Director, Consortium for Science, Policy & Outcomes, and Professor of Science and Society, Arizona State University

Panel 2, Part 1: Reproducibility Issues in Computational Sciences and Statistics

The panelists in this session have been asked to address the session questions (above) with a focus on the management of computational code and data.

David Madigan, Executive Vice President and Dean of the Faculty of Arts and Sciences, Columbia University

Arjun Kumar Manrai, Department of Biomedical Informatics, Harvard University

Panel 2, Part 2: Reproducibility Issues in Computational Sciences and Statistics

The panelists in this session have been asked to address the session questions (above) with a focus on the impact of the misuse of statistics in research.

Giovanni Parmigiani, Harvard T.H. Chan School of Public Health and Dana Farber Cancer Institute

Steven Goodman, Professor of Medicine and Epidemiology, Associate Dean for Clinical and Translational Research, and Co-Director, Meta-Research Innovation Center at Stanford (METRICS), Stanford University

Panel 2, Part 3: Reproducibility Issues in Economics and Social Science

The panelists in this session have been asked to address the session questions (above) as they relate to economics, social sciences, and psychology.

Paul L. Joskow, Elizabeth and James Killian Professor of Economics, Emeritus Department of Economics, Massachusetts Institute of Technology

Arthur (Skip) Lupia, Hal R. Varian Collegiate Professor of Political Science, University of Michigan

Joseph Simmons, Professor of Operations, Information, and Decisions, Wharton School, University of Pennsylvania

Public Comments

FRIDAY, FEBRUARY 23, 2018

Welcome and Introductions

Harvey Fineberg, Committee Chair

Panel 3: Reproducibility Issues in Engineering

Gianluca Setti, Politecnico di Torino, Italy, and IEEE editor

Panel 4: Reporting of Reproducibility Issues in Science

Christie Aschwanden, Lead Science Editor, FiveThirtyEight

Laura Helmuth, Science Editor, The Washington Post

Public Comments

Third Meeting
WEDNESDAY, APRIL 18, 2018

Welcome and Introductions

Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation

Perspectives on Scientific Progress and Irreproducibility

Richard Shiffrin, Department of Psychological and Brain Sciences, Indiana University Bloomington

Panel 1: Reproducibility in the Physical and Earth Sciences

Joan Brennecke, Cockrell Family Chair in Engineering #16, McKetta Department of Chemical Engineering, The University of Texas at Austin

Peter Mohr, Atomic Physics Division, National Institute of Standards and Technology

Panel 2: Reproducibility in Industry and Industrial Engineering

Carl Ascoli, Chief Science Officer, Rockland Immunochemicals

William Lyons, Director, Global Research and Development Strategy for the Global Technology Organization, Boeing Research and Technology

Introduction to Economics and Reproducibility

Daniel L. Goroff, Vice President and Program Director, Alfred P. Sloan Foundation

Panel 3: The Economics of Addressing Reproducibility Issues in Science

Heidi Williams, Department of Economics, Massachusetts Institute of Technology

Myron P. Gutmann, Professor of History and Director of the Institute of Behavioral Science, University of Colorado Boulder

Richard Freeman, Herbert Ascherman Chair in Economics, Harvard University

Brent Goldfarb, Management and Organizations Department and Dingman Center for Entrepreneurship, University of Maryland

Public Comments

Fourth Meeting
MAY 9, 2018

Welcome and Introductions

Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation

Panel: Perspectives on Reproducibility and Replication of Results in Climate Science

The panelists have been asked to focus on the following session questions:

• How has the awareness and understanding about reproducibility and replication in climate science evolved over recent years?
• Are there specific challenges regarding reproducibility that you have encountered or are aware of? Identify specific steps that are being taken, either by you or by others, to ameliorate these issues.
• Highlight historical and potential new approaches to reproducing and replicating climate science research using examples such as paleoclimate data to test models and estimate uncertainties.

Michael Evans, Department of Geology, University of Maryland

Gavin Schmidt, Director, Goddard Institute for Space Studies, National Aeronautics and Space Administration

Rich Loft, Director, Technology Development Division, National Center for Atmospheric Research

Andrea Dutton, Department of Geological Sciences, University of Florida

Wrap-Up

Harvey Fineberg, Committee Chair

Fifth Meeting
MAY 31, 2018

Welcome and Call to Order

Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation

Panel: International Perspectives on Reproducibility and Replication in Science and Engineering

The panelists have been asked to focus on the following session questions:

• Are there specific examples in your country/region where a lack of reproducibility and replicability in research results has led to doubt about reported results more broadly? Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?
• Are there particular scientific fields in which lack of reproducibility and replicability is more/less of a concern?
• Are there any concrete actions that organizations (e.g., funders, publishers, societies) in your country or region have taken to address concerns about reproducibility and replicability? What actions should they take?
• Should the research community work regionally and/or globally to address concerns about reproducibility and replicability? If so, what should be the priorities?

Laura Fierce, Environmental and Climate Sciences Department, Brookhaven National Laboratory, and member, Executive Committee, Global Young Academy [in person]

Koen Vermeir, French National Centre for Scientific Research, former Co-Chair, Scientific Excellence and Open Science Programs, and Member, Executive Committee, Global Young Academy [via Zoom]

Harry Xia, President, Alliance for Scientific Editing in China [in person]

Suman Chakraborty, Department of Mechanical Engineering, Indian Institute of Technology Kharagpur, India [via Zoom]