The Chameleon team is excited to hold our first Chameleon Hackathon event sometime in the 4th week of August or 1st week of September.  This year’s hackathon will focus on reproducing and packaging experiments on the Chameleon platform.  In this Call for Participations, we would like to survey Chameleon users who are interested in joining this hackathon. Please continue reading and fill out the Google form at the very end.

Goals and benefits:

• The goal of this hackathon is to reproduce and package participants’ or existing published experiments. 
• The chameleon staff will help participants package their experiments. 
• Participants who join this hackathon will get full technical support from the Chameleon operators during the hackathon (via Zoom, Slack, etc.).
• After the hackathon, participants can share their packaged experiments to their own community so that other researchers can reproduce their experiments (and potentially cite their papers). 
• Participants can also use this opportunity to package their ongoing works to be ready for future artifact review and bading (e.g., https://www.acm.org/publications/policies/artifact-review-badging)
• Participants will have the opportunity to give feedback on how the staff can improve Chameleon features.

Time and Duration:

• The hackathon is tentatively to be held in the 3rd or 4th week of August or the 1st week of September.
• The duration of the hackathon will be either 1 full day or two half days. 
• Each participant only needs to attend a 1- to 2-hour slot.

Eligibility:

• Participants should be existing users of Chameleon and are familiar with the Chameleon interface.
• Participants should be part of an existing project that is already approved.  (During the hackathon, the participants will use their own project IDs).

Phases of the hackathon:

• The hackathon will be broken into three phrases: (1) Preparation before the hackathon, (2) Direct communication during the hackathon, and (3) Discussion and reporting phase. 

Phase 1: Preparation Before the Hackathon 

• Participants may use their personal experiments, either ongoing or recently published.
• Participants may also choose at least 3 papers of their own choice (e.g. papers in their research areas) that already have significant artifact badges such as the ACM “Artifacts Available,” “Results Reproduced,” and/or “Results Replicated” badges.   Any other badges such as those from IEEE or USENIX are also recommended.   Choosing papers with such badges increases the probability that the experiments will be reproduced successfully.  We recommend 3 papers to anticipate a situation where some of the papers are not reproducible.   It is expected that participants already read the papers and download the code/data that they picked so that during the hackathon there is no delay.
• Participants are also advised to make sure that the Chameleon Cloud has the minimum hardware/software requirements to reproduce the artifacts in the papers.  Please consult the Chameleon Hardware Discovery page (https://chameleoncloud.org/hardware).
• Every participant will sign up for a particular 2-hourly slot.  Please fill the Google form at the end of this page, if you would like to join the hackathon.  We will send you more information later on.  
• Participants should get themselves familiar with Chameleon reproducibility features such as Trovi and Jupiter via the webinars:
  • Reproducing Figure 1 of DAWNBench experiment with Tensorflow on Chameleon with Jupyter
  • Reproducing the LinnOS experiment on Chameleon with Jupyter
  • Reproducing AlexNet with Stanford Dogs Dataset on Chameleon using Jupyter
  • Reproducing the DTRS experiment from tinyTailFlash on Chameleon with Jupyter
• Participants are encouraged to reproduce and package their experiments before the hackathon and collect questions to be answered by the Chameleon staff during the hackathon.  Participants may also submit issue tickets before the hackathon.   If the participants successfully reproduce and package their chosen experiments, they can still join the hackathon and directly jump to the reporting (see Phase 3).

Phase 2: Communication During the Hackathon

• Participants will sign in (via Zoom and Slack, which will be announced later after you sign up) during the hour that they already signed up for.
• The participants communicate any technical issue with the hackathon organizers (e.g. any command errors), especially if the participants believe that the issue is the incompatibility between the being-reproduced artifact and the Chameleon platform.
• After the artifact runs without any error, the participant should package and share the whole experiment with Trovi.  This way all the effort that was done is not lost and now “monumentalized” via the Trovi platform for other members in the community to use.
   

Phase 3: Discussion and reporting phase

• After participants successfully/unsuccessfully reproduce and package their experiments, they are expected to discuss and report their experience (e.g. a 5-minute discussion at the end of their slot).
• To concretely measure this reproducibility and packaging experience, participants will fill a simple spreadsheet (to be shared later) that asks questions such as:
  • Is the public link to the artifact (code/data/experiment) available? Y/N  (In some cases, the links provided by the authors are already broken). 
  • Is the artifact downloadable? Y/N (In some cases, the links are available but the download breaks).
  • Is the artifact compilable? Y/N (In some cases, they are downloadable but cannot be compiled).
  • Is the artifact runnable on the Chameleon platform? Y/N (In some cases, the artifact is compilable but cannot run easily on the Chameleon platform).  In this case, participants can  try to resolve the problem by themself and/or with the help of Chameleon staff.
  • If the problem is eventually fixable and how long to fix the problem? 
  • Does the artifact include graph making scripts that help you generate the result graphs? Y/N
  • Are the experimental results reproducible? Y/N --- i.e. do you see the same pattern in the results (e.g. approach X is better than approach Y), although the results might be different
  • Is the articant packageable with Trovi? Y/N
• With many data points, we hope that we can write a workshop paper together with you to report your experience.
   

Are you interested?  If so, please fill this Google form:

• https://docs.google.com/forms/d/e/1FAIpQLScNigiZ8go3Uqoe7mXvvDZvDHSO98xsD31CTfM21b7ZSCx46A/viewform?usp=sf_link

Any questions or feedback?

• Please email: chameleon-hackathon at googlegroups.com