Reported Outages

The Jan 28th maintenance window has concluded, and KVM@TACC is back to normal.

We were able to migrate the loadbalancer and database services to a redundant configuration, but had to roll back the migration of other control-plane services due to unforseen complications. We'll be investigating a path forward, and a subsequent (shorter) maintenance window to finish the migration for each service.

On Thursday, December 19 at 9:30 am CT, Chameleon's authentication service and Trovi will be down for network maintenance. During this time, users will not be able to access the Chameleon sites (CHI@UC, CHI@TACC, KVM), or authenticate to the user portal. Running nodes will be unaffected.

We're receiving reports of issues reaching CHI@UC, but the issues seem depend on the source address, so we have not been able to reproduce them consistently.

This manifests as timeouts for horizon dashboard at https://chi.uc.chameleoncloud.org, or disconnections when connecting to instances over ssh.

We're working with our network provider to troubleshoot the issue.

The outage should now be resolved, but we're continuing to monitor.

Please let us know if you observe new issues connecting to baremetal nodes, or accessing shared storage.

As of around 2pm on Wednesday, CHI@Edge is back online, and device reservation and container launch are working.

We're continuing to monitor the system.

Update: Email services should be working as normal again. 

Email notifications to Gmail addresses are not reliably being received from our services. In particular this affects lease end reminder emails. We are working on improving this service in the near future.

Please take care noting the end time of your lease to ensure your experiment isn't terminated unexpectedly.

On the morning of Tuesday, October 29th, there will be a brief outage affecting login to all Chameleon sites and services at 9:30 AM central time.

We expect a 5-10 minute outage while we apply updates to the service that handles federated login.
This won't affect any running workloads or nodes, but you may need to refresh your browser once the outage ends.

On the morning of Tuesday, September 24th, there will be a brief outage affecting login to all Chameleon sites and services at 9:30 AM central time.

We expect a 5-10 minute outage while we apply updates to the service that handles federated login.
This won't affect any running workloads or nodes, but you may need to refresh your browser once the outage ends.

The CHI@NU site is currently inaccessible due to an issue with its proxy server interfering with federated login.

You will observe symptoms including the message "No active allocations" from the dashboard, or an HTTP 403 error from the API. Don't worry, nothing is wrong with your account, the failure just happens to be prevending the site from fetching your list of projects correctly.

We've notified the site operator, and will post updates here.

On the morning of Tuesday, August 27rd, there will be a brief outage affecting login to all Chameleon sites and services at 9:30 AM central time.

We expect a 5-10 minute outage while we apply updates to the service that handles federated login.
This won't affect any running workloads or nodes, but you may need to refresh your browser once the outage ends.