Reported Outages

RESOLVED: CHI@TACC maintenance is complete.

UPDATE: The OpenStack upgrade is complete. Staff are aware of some potential issues launching new instances, and are working to resolve them. Existing instances are unaffected, and once again available.

At 8am central time, on August 21st, CHI@TACC will be unavailable for use, as we upgrade the core OpenStack services on the controller hosts. During this time, the chi.tacc.chameleoncloud.org webpage and APIs will be inaccessible, as will network connectivity to any running instances.

We're observing intermittent provisioning failures for a small set of nodes at CHI@UC, all of which are "phase 2" nodes, mostly rtx_6000s.

The nodes currently known to be affected are listed below, and have been placed into a non-reservable maintenance mode until we can resolve the issue.

Resolved outage

At CHI@UC, when submitting a request for a new lease, users are intermittently receiving an error message about "enforcement failed". After restarting a relevant service, lease requests are now succeeding again.

We suspect this issue is related to a service token expiring and not being correctly renewed, and are investigating a proper fix.

On the morning of Tuesday, July 23rd, there will be a brief outage affecting login to all Chameleon sites and services at 9:30 AM central time.

We expect a 5-10 minute outage while we apply updates to the service that handles federated login.
This won't affect any running workloads or nodes, but you may need to refresh your browser once the outage ends.

As of 1pm, July 23rd, reservation enformenents are fixed, and we're declaring the CHI@UC outage over.
You may notice some minor changes to the horizon dashboard as we fix some remaining UI issues in the instance launch dialog, but these should not affect any functionality.

UPDATE: All services are updated and should be working as expected

On the morning of Tuesday, June 18th, there will be a brief outage affecting login to all Chameleon sites and services at 9:30 AM central time.

We expect a 5-10 minute outage while we apply updates to the service that handles federated login.
This won't affect any running workloads or nodes, but you may need to refresh your browser once the outage ends.

CHI@UC had a brief interruption in connectivity, preventing access to chi.uc.chameleoncloud.org. This issue manifested as a failure for the UC control-plane servers to contact the chameleon authentication server, triggered by what was planned to be unrelated work elsewhere in the network.

Upon discovering the issue, the other network changes were backed out, and service has been restored.

CHI@TACC was unavailable from May 25 through the morning of May 28, 2024. This effected the API services and web interface, but did not impact running instances.

We have corrected the problem, and service is restored.

UPDATE: All services are updated and should be working as expected

On the morning of Tuesday, June 4th, there will be a brief outage affecting login to all Chameleon sites and services at 9:30 AM central time.

We expect a 5-10 minute outage while we apply updates to the service that handles federated login.
This won't affect any running workloads or nodes, but you may need to refresh your browser once the outage ends.