Reported Outages

Dear Chameleon Users,

CHI@TACC will undergo a scheduled outage on Tuesday, August 18, 2026, lasting the day while we perform system upgrades.

• Existing instances and leases will continue to run
• Network access may be intermittent as upgrades are applied
• Due to connectivity interruptions, consider CHI@UC or CHI@NCAR for time-sensitive or remotely monitored workloads

Thank you for your patience as we work to improve the platform. Please direct questions to the helpdesk.

We observed instability in the ironic provisioning service at CHI@UC following the openstack version upgrade on Tuesday Jul 28th, lasting until early on Thursday the 30th.

Users may have observed instances moving to an "Error" state instead of provisioning successfully, failures to access the serial console, and a brief window where nodes were marked as unavailable for reservation.

This issue is now resolved.

On Tuesday, July 28th, we're planning downtime at CHI@UC in order to perform an openstack version upgrade and maintenance on the controller nodes. From 9am-5pm central time on the 28th, new instances won't be able to be launched, and existing ones won't be accessible via the network.

If this will cause issues for you, please get in touch with us at help@chameleoncloud.org

The chameleoncloud.org website was unavailable from 10:00 AM to 11:30 AM on Thursday May 7th. The root cause was a storage driver issue impacting the cluster serving the website's services.

During this period, users may have experienced failures when creating new leases, but no running experments were impacted.

From approximately 1:50 AM to 8:30 AM on the Monday of May 6, 2026 the ChameleonCloud portal, JupyterHub, and Trovi systems were unavailable.

4:00 PM CDT: This issue is now resolved.

After last night's network outage, some remaining issues mean that launching new instances, and reconfiguring network interfaces are currently failing.

The suspected cause is a failure between the openstack control-plane services and the management control plane for our baremetal switches. Datacenter staff are investigating.

Connectivity to existing instances should not be impacted.

CHI@UC will be offline due to datacenter-wide power and cooling maintenance from 12:00 PM on Monday, May 18th until 2:00 PM on Tuesday, May 19th.

All instances, APIs, and services will be unavailable.

Please shut down your instances before 5:00 PM on the 18th — any remaining instances will be hard-powered off. Extend any leases expiring during this window beforehand.

Due to a network outage at ANL, CHI@UC was experiencing intermitent connectivity issues starting around 10PM, Monday May 4th, and completely unavailable from 2AM to 8AM central time on Monday May 5th.

All services are now back to normal.

The root cause was an issue triggered during replacement of border routers on the Argonne Campus where CHI@UC is hosted.

The Chameleon user forum is down. We are working on bringing this back.

Resolved: The issue with the provisioning network has been resolved.

As of ~11:40AM on April 30th, the provisioning network at CHI@TACC appears down. This prevents launching new instances, shutting down or hard-rebooting existing ones, and making new leases.

Existing instances should *not* be affected.

Datacenter staff are investigating.