Dear Chameleon users,

 

I think it's fair to say most everybody had a truly awful year at some level. Without minimizing that, I wanted to take a brief moment to reflect with gratitude on what went right this year, at least in the (small) realm of our work on this research testbed. So here’s one year in review, representing combined “changelog” information from the entire year.

 

Four more years of Chameleon! First things first: Chameleon wrapped up the end of Phase 2 this year, and we are so thankful that we are able to continue developing and operating this infrastructure for you all. Phase 3 began just a few months ago and we're already hard at work designing new capabilities and improvements, which have been shaped by your valuable feedback, requirements and use cases over Phase 2.

 

Support for identity federation. As many of you are aware, we transitioned all Chameleon legacy accounts to federated identity over the last months (thanks again for bearing with us on that--that was a bear of a project!) One reason for doing this is that it makes it easier for you to use a variety of existing and emergent research infrastructure, such as CloudBank or FABRIC with a single identity--allowing you to move across these infrastructures with greater ease when constructing experiments that leverage resources distributed across them. We're looking forward to building stronger ties between Chameleon and these exciting new resources!

 

New hardware. Most of the phase 2 hardware at the CHI@UC site got GPUs installed, with 40 RTX 6000 cards installed into existing nodes and 3 new V100 nodes coming online in November. At TACC, we started adding new Cascade Lake nodes equipped with Intel Xeon Gold 6240R processors, 192GB of RAM, and a 480GB SSD; ultimately 56 of those will be added at TACC and they will replace our aging Haswell nodes. As always, you can find out more about their specs in our resource catalogue.

 

Support for AWS DirectConnect in experiments. This was a fun one… in February we finished work that allows Chameleon experiments to peer directly with AWS via Internet2's CloudConnect. Chameleon eliminates the need to involve campus IT staff and enables access to direct public cloud network connections to all Chameleon users. Thus, the identity federation with CloudBank is not just about theoretical potential -- any Chameleon user can now easily use resources from both of these NSF-sponsored infrastructures without the need for complicated campus networking configuration! Try it out in the Jupyter Notebook here.

 

Expanded SDN experiment capabilities. In January we also made significant improvements to the SDN capabilities on Chameleon, making it possible to deploy SDN experiments using any of our nodes (prior to that, it was only possible to run SDN experiments using our Skylake nodes). Furthermore, these improvements also unlocked interesting new experiments leveraging the fact that many of our Haswell nodes have two configurable network interfaces.

 

Trovi: an experiment sharing platform. In November we officially announced Trovi, the Chameleon reproducibility platform that allows Chameleon users to discover and share packaged experiments. You can even publish your packaged experiment to Zenodo for long-term storage and citation. Trovi is integrated with Chameleon’s Jupyter installation, so if you're familiar with Jupyter Notebooks, you'll be right at home. If you're new to notebooks, check out this webinar from last year, which illustrates how Notebooks can be used to organize and orchestrate CISE experiments. We have had several users already taking advantage of this new capability to organize their experimental workflow and leverage the artifacts to improve the quality of their paper submissions. In particular, last summer we worked with a group of talented students who reproduced and packaged experiments from foundational papers (such as DAWNBench) that can now be used for further research or education. They have written about their experiences here and here--check it out!

 

General system upgrades. Much of this year was focused on “behind the scenes” work  geared towards making the system better. We upgraded all our installations to the latest OpenStack release (Train) to leverage bug fixes and performance improvements around instance launch times. This is true of our CHI installations on both sites as well as our KVM platform. We also made various usability improvements in the reservation system as well as user interface, improved the system defaults, and generally worked on making the system more reliable and responsive.

 

Appliances. We added multiple appliances this year (CC-CentOS8 and CC-CentOS8-CUDA10 being the most important). We also installed Python 3 in all Chameleon-supported images and set as default (due to Python 2 end-of-life in early 2020), made improvements to the cc-snapshot tool, updated the DevStack appliance for the OpenStack Rocky release, and improved default security rules for all our base images, to reduce chance of accidental compromise.

 

Cloud trace data. We have updated both our open-source trace generator and the trace format specification to serve for both the virtual machine and bare-metal clouds. We also published a new KVM trace from our KVM site and bare-metal traces from the CHI@UC site and CHI@TACC site using the updated trace format version.

 

More IPs for more experiments. We obtained and added another 100+ IP addresses to the CHI@UC site in response to increased demand. If you are still having issues reserving multiple IP addresses for your experiment, check out this blog post for some ideas on how to make use of fewer. Reduce, re-use, recycle!

 

Reaching out. We sought to broaden our communication with users via webinars (most of which are recorded on our YouTube channel), social media, and our blog. We also increasingly invest in tools that allow our users to use Chameleon as an education platform; this year the CSInParallel and the ANL Big Data Camp took advantage of those capabilities. 

---

We look forward to starting anew in January: we are cooking up a lot of exciting things and will share more details as the year unfolds! We will be releasing monthly changelogs again in the new year so you can keep abreast of the latest updates.

 

Finally, I wanted to extend a thank you to the Chameleon ticketmasters, who have continued to provide same-day responses to your questions and difficulties throughout the year. And, thanks to all of you who wrote in to give feedback on the testbed, reporting issues and also letting us know how your experience is overall. Looking forward to helping you out in the next year!

 

From all of us at Chameleon,

 

Jason