Reported Outages

The outage should now be resolved, but we're continuing to monitor.

Please let us know if you observe new issues connecting to baremetal nodes, or accessing shared storage.

As of around 2pm on Wednesday, CHI@Edge is back online, and device reservation and container launch are working.

We're continuing to monitor the system.

Update: Email services should be working as normal again. 

Email notifications to Gmail addresses are not reliably being received from our services. In particular this affects lease end reminder emails. We are working on improving this service in the near future.

Please take care noting the end time of your lease to ensure your experiment isn't terminated unexpectedly.

On the morning of Tuesday, October 29th, there will be a brief outage affecting login to all Chameleon sites and services at 9:30 AM central time.

We expect a 5-10 minute outage while we apply updates to the service that handles federated login.
This won't affect any running workloads or nodes, but you may need to refresh your browser once the outage ends.

On the morning of Tuesday, September 24th, there will be a brief outage affecting login to all Chameleon sites and services at 9:30 AM central time.

We expect a 5-10 minute outage while we apply updates to the service that handles federated login.
This won't affect any running workloads or nodes, but you may need to refresh your browser once the outage ends.

The CHI@NU site is currently inaccessible due to an issue with its proxy server interfering with federated login.

You will observe symptoms including the message "No active allocations" from the dashboard, or an HTTP 403 error from the API. Don't worry, nothing is wrong with your account, the failure just happens to be prevending the site from fetching your list of projects correctly.

We've notified the site operator, and will post updates here.

On the morning of Tuesday, August 27rd, there will be a brief outage affecting login to all Chameleon sites and services at 9:30 AM central time.

We expect a 5-10 minute outage while we apply updates to the service that handles federated login.
This won't affect any running workloads or nodes, but you may need to refresh your browser once the outage ends.

At 2:12 PM a DNS issue took down access to chi.uc.chameleoncloud.org. We have rolled back the responsible change, and the site should come back online in the next 30 minutes.

RESOLVED: CHI@TACC maintenance is complete.

UPDATE: The OpenStack upgrade is complete. Staff are aware of some potential issues launching new instances, and are working to resolve them. Existing instances are unaffected, and once again available.

At 8am central time, on August 21st, CHI@TACC will be unavailable for use, as we upgrade the core OpenStack services on the controller hosts. During this time, the chi.tacc.chameleoncloud.org webpage and APIs will be inaccessible, as will network connectivity to any running instances.

We're observing intermittent provisioning failures for a small set of nodes at CHI@UC, all of which are "phase 2" nodes, mostly rtx_6000s.

The nodes currently known to be affected are listed below, and have been placed into a non-reservable maintenance mode until we can resolve the issue.