Code Review Demo
Reproducibility Audit
Eric R. Scott
Mar 27, 2025
Why Reproducibility?
- Many reasons!
- Today:
- reliability/correctness of code
- reusability of code (by future you or other researchers)
Reproducibility Audit
- Broad & shallow review of code
- Often, (not ideally), in pre-print stage
- Focus on:
- Is there sufficient documentation?
- Does the code run?
- Is the code readable?
- Does the code produce the expected outputs?
Reproducibility Auditor
Ideal reviewer depends on target audience of code.
If you want it to be reproducible by a broad group, a reviewer outside of your specific domain of research and with little experience in your tools is a good choice!
Some scaffolding is helpful. Give reviewer a rubric or guidelines on what you specifically want feedback on.
ReproHack.org
- Provides resources to organize reproduciblity hackathons (“reprohacks”)
- Authors submit code from manuscripts
- Reviewers provide feedback and reproduciblitiy scores
- Provides guidelines for participants and authors
- Database of submitted code and audits for practice & learning
Lab-level reproducibility audits
- Guidelines similar to those on reprohack.org and code of conduct are essential!
- Code review anxiety shoudl be discussed as a group
- A rubric I came up with
- Existing materials?
Reproducibility Audit Rubric
Demo
