We’re excited to announce the publication of the REPETO Report on Challenges of Practical Reproducibility for Systems and HPC Computer Science, a culmination of our Community Workshop on Practical Reproducibility in HPC, held in November 2024 in Atlanta, GA (reproduciblehpc.org).

View or contribute to the experiment packaging and style checklists (appendix A and B) on our GitHub repository here.

Download the report here.

Many of our community members participated in the workshop as well as contributed thoughtful comments and feedback on the report. We are grateful for the chance to share this knowledge back with this community.

This comprehensive report emerged from the Community Workshop on Practical Reproducibility in HPC held on November 18, 2024, in Atlanta, Georgia, alongside SC24. The workshop brought together reproducibility practitioners, experiment authors, reviewers, and chairs of reproducibility initiatives to share experiences and formulate a path forward for community adoption of reproducibility practices. The report details key challenges in practical reproducibility for systems and HPC research, including:

• Technical difficulties in packaging experiments and reviewing them
• Finding specialized hardware for reproduction
• Establishing experimental environments
• Defining reproducibility conditions

Most importantly, the report provides actionable recommendations for authors, reviewers, organizations, and the broader community, including detailed checklists for experiment packaging and style guidelines that can immediately improve artifact quality. This work was supported by the National Science Foundation REPETO project (NSF award 2226406) and indirectly by the Chameleon project (NSF award 2027170).