Reported Outages

RESOLVED: All compute_zen5 and tharm nodes at CHI@NCAR are back in production. We are waiting for a replacement SSD for our switch, but have moved the primary links for the affected nodes to a different switch. The secondary links will be unavailable until we receive the replacement hardware. All existing instances on the affected nodes have been migrated and should be accessible. The impacted leases have been extended to account for the unplanned downtime. Please let us know if you experience any issues or have any questions at help@chameleoncloud.org. Thank you for your patience!

Dear Chameleon Users,

CHI@NCAR will undergo a scheduled facility outage on Tuesday, September 1, 2026, to upgrade our network.

• Existing instances and leases will continue to run

• Network access will be intermittent as upgrades are applied

• Due to connectivity interruptions, consider CHI@UC or CHI@TACC for time-sensitive or remotely monitored workloads

Thank you for your patience as we work to improve the platform. Please direct questions to the helpdesk.

Resolved: Maintenance window for CHI@NCAR

The announced outage is now resolved, and CHI@NCAR is fully operational on the new openstack release.

All service is restored.

On August 25, Chameleon's portal will be upgraded. During the window, the service will be stopped and restarted. We expect total downtime to be less than 15 minutes.

Users will experience degraded ability to create/update leases at any site, access project management pages, and access to the help desk (except via email). No running experiments will be impacted.

On Monday, August 17th, CHI@UC will have a maintenance window to upgrade remaining services that could not be upgraded in the prior window.

This will be visible as intermittent disruption to services and networking for several hours. Although we are holding a 9-5 window, we expect the work to take only 1-2 hours in the morning.

Please get in touch if this will cause issues for your experiments.

Dear Chameleon Users,

CHI@NCAR will undergo a scheduled outage on Tuesday, August 11, 2026, lasting the day while we perform system upgrades.

• Existing instances and leases will continue to run
• Network access may be intermittent as upgrades are applied
• Due to connectivity interruptions, consider CHI@UC for time-sensitive or remotely monitored workloads

Thank you for your patience as we work to improve the platform. Please direct questions to the helpdesk.

Resolved: Maintenance for CHI@TACC is complete.

We observed instability in the ironic provisioning service at CHI@UC following the openstack version upgrade on Tuesday Jul 28th, lasting until early on Thursday the 30th.

Users may have observed instances moving to an "Error" state instead of provisioning successfully, failures to access the serial console, and a brief window where nodes were marked as unavailable for reservation.

This issue is now resolved.

On Tuesday, July 28th, we're planning downtime at CHI@UC in order to perform an openstack version upgrade and maintenance on the controller nodes. From 9am-5pm central time on the 28th, new instances won't be able to be launched, and existing ones won't be accessible via the network.

If this will cause issues for you, please get in touch with us at help@chameleoncloud.org

The chameleoncloud.org website was unavailable from 10:00 AM to 11:30 AM on Thursday May 7th. The root cause was a storage driver issue impacting the cluster serving the website's services.

During this period, users may have experienced failures when creating new leases, but no running experments were impacted.