Reproducible Research Working Group

Members

• Michele Dunn, NCI
• Jim Ware, Harvard
• John Kloke, Pittsburgh
• Frank Harrell, Vanderbilt, chair
• Ron Thisted, Chicago
• Chris Lindsell, Cincinnati
• Aniko Szabo, Medical College of Wisconsin
• Phil Schumm, Chicago
• Kathleen Propert, Penn, propert@mail.med.upenn.edu
• Liz DeLong, Duke
• Sandra Taylor, UC Davis
• Michael Berbaum, U Illinois Chicago
• Mary Banach, UC Davis
• Mark Holodniy, Veterans Affairs and Stanford
• Diane Lazzeri, Pfizer, Clinical Operations Manager
• Charles Contant, Tufts, CContant@tuftsmedicalcenter.org
• Kwang-Youn Kim, Northwestern, kykim@northwestern.edu
• Emilia Bagiella, Mount Sinai School of Medicine, emilia.bagiella@mssm.edu
• Jeff Welge, U. Cincinnati, welgeja@uc.edu

CTSA Coordinating Center Project Manager

• Bridget Swindell

Mission Statement

The CTSA BERD Reproducible Research Working Group's mission is to

1. list and discuss all areas related to challenges and solutions to reproducibility of biomedical research and its relationship to questionable research practices and ethical conduct of research
2. compile and create educational materials on reproducible research methodology
3. Find and develop technical solutions for data manipulation, analysis, and reporting and disseminate these to BERD members and beyond
4. Develop and present short courses are web-based materials on how to implement reproducible research practices

The work of the RR Working Group is a scientific collaboration that will not involve the use of any proprietary data.

Suggestions for Working Group Activities

Templates Used by Brian Nosek's Group

Click to see Replication Templates

Discussion with Frank Harrell and Brian Nosek

How do we encourage reproducible research and the building of and sticking to pre-specified analytic plans? How do we assist medical researchers in not switching dependent variables when they do not get the results they wanted? How do we produce quality data analysis?

One suggestion for our work would be to do post-mortem analysis of published studies. Frank suggested that we build an archive of projects or compedium for statistical work. We could analyze the quality, review the study design, analysis, and measurements. In the psychology papers, Brian is looking at pre-specified endpoints and how predictive they are of reproducible studies. These brings us to the question of comparisons that we would like to make in reproducible research. Are we talking about looking at methodological or statistical comparisons of reproducible research?

Brian invited us to join the Open Science Collaboration discussion of Reproducible Research on Google (https://groups.google.com/forum/#!forum/openscienceframework). This is a collaboration with individuals from a variety of fields, including non-scientists and non-psychologists, social scientists, cognitive investigators, methodologists. Frank suggested that our work should be more targetted. Should we be looking at biomedical or psychology studies and addressing a particular question, such as confirmation bias? Brian thought another good focus for BERD Reproducible Research projects would be methodology quality and observer variability.

Bottom line - what qualifies as reproducible? We would like to downplay point estimates. Shouldn't we be looking at the smaller confidence intervals in two studies?

Brian spent the year at Stanford and will be returning to University of Virginia in July.

Teleconferences

24 Nov 2010

Attending: Dunn, Ware, Kloke, Thisted, Lindsell, Harrell

• Went over initial ctspedia skeleton for RR
• Discussed other topics needed to be added such as version control, which will be led by Ron Thisted
• Jim Ware proposed a "how to" at the next BERD face-to-face meeting to bring BERD members up to speed with RR tools
• We need to create a section in the RR part of ctspedia listing ways we can have impact, e.g., as journal referees
• A library of templates of RR scripts would be very helpful to students
• We need some text on archiving entire projects, including data

12 Apr 2011

Attending: Lindsell, Szabo, Thisted, Schumm, Harrell

• Discussed the importance of focus and scientific method to a successful research project
• Discussed pre-specified statistical analysis plans (SAP) including sketches of tables and figures
  • Do deviations from the plan require an IRB amendment?
• Need for templates, example SAPs
• Referred to a new paper in The Political Methodologist
• Fleshed out the Methodology section (new skeleton edited on ctspedia)
• Favor a broad publication such as in The Scientist which would also influence basic scientists

16 Nov 2011

• Welcome new members
• Roundup of RR-related activities
• Pre-specified statistical analysis plans: https://biostat.app.vumc.org/PreStatAnalysisPlan
• Version control
• Review of initial material on https://ctspedia.org
• Homework assignment: fill in topics started on https://ctspedia.org

16 Nov 2011 and Later

• See https://www.ctsacentral.org/committees/available-documents/273 summaries from CTSAcentral

This topic: CTSpedia > WebHome > ReproducibleResearch > ReproducibleResearchGroup
Topic revision: 27 Jan 2022, WikiAdmin

---