2. Introduction to the Reproducible analysis and Research Transparency workshop
2.1. Learning objectives for this workshop:
- Understand the concept, importance, and components of reproducible research.
- In practice, showcase an example of transparent and reproducible research, using existing tools.
The sciences have with reproducibility problem (Nature article
Many published studies cannot be reproduced)
2.2. Replication and reproduction
Peng (2009) Reproducible research and Biostatistics. Biostatistics 10: 405-408
- Replication - when independent investigators use methods, protocols, data, and equipment to confirm scientific claims.
- Reproduction - when data sets and computer code are made available for researchers to verify results.
2.3. Science retracts paper without agreement of lead author.
- Journal retracted a study of how canvassers can sway people’s opinions about gay marriage. Issues raised:
- Original survey data not made available for independent reproduction of results
- Survey incentives misrepresented
- Sponsorship statement false
- Two grad students attempted to reproduce the study and could not.
- they concluded the data must have been fabricated story
2.4. Retracted, but not fraud
- One researcher had seven papers retracted because of irreproducibility
- Couldn’t find the [data] (http://onlinelibrary.wiley.com/doi/10.1111/j.1468-1331.2011.03524.x/abstract)
- “Extensive” errors force retraction of lymphoma paper
- Many, many more irreproducible examples
- Seizure study retracted after authors realized data were “Terribly mixed”
- From the authors of Low Dose Lidocaine for Refractory Seizures in Preterm Neonates: “The article has been retracted at the request of the authors. After carefully re-examining the data presented in the article, they identified that data of two different hospitals got terribly mixed. The published results cannot be reproduced in accordance with scientific and clinical correctness.”
- Bad spreadsheet merge kills depression paper, quick fix resurrects it
- The authors informed the journal that the merge of lab results and other survey data used in the paper resulted in an error regarding the identification codes.
- Original conclusion: Lower levels of CSF IL-6 were associated with current depression and with future depression.
- Revised conclusion: Higher levels of CSF IL-6 and IL-8 were associated with current depression.
2.5. The Four Facets of reproducibility:
- Documentation: note the difference between binary files (e.g. docx) and .txt files and why text files are preferred for documentation.
- Organization: tools to organize your projects so that you don’t have a single folder with hundreds of files.
- Automation: the power of scripting to create automated data analyses.
- Dissemination: publishing is not the end of your analysis, rather it is a way station towards your future research and the future research of others.
2.6. References / Sources
The information presented here is based on information found in these sources: