Reported Outages

CHI@NU was down due to a hardware failure in the controller node. This has now been replaced, and the site is back online.

05/12/23: CHI@UC is back online. All systems look stable, but we'll be keeping an eye on it over the weekend to be sure. 

Pleas let us know if you encounter issues!

Update 11:01am: The upstream issue seems to have resolved, and sites are again accessible. We'll be monitoring the situation to see if it remains stable.

Authentication to all sites is currently down due to a failure in our central authentication system.

Existing instances should be unaffected, but users won’t be able to create or modify existing instances.

This seems to be caused by a failure in our upstream DNS provider, we are working with them to get an ETA for resolution.

Update: Maintenance was completed a 1:12 PM (CDT) yesterday Sunday, 26 March 2023.

Network maintenance will be carried out between 8:00 AM and 2:00 PM (CDT) on Sunday, 26 March 2023. Access to all TACC systems will be unavailable during this time, including CHI@TACC, KVM@TACC, and the Chameleon Portal. Instances will continue to run, but users will have no access to TACC services and systems until the upgrade is complete.

Please submit any questions you may have via the TACC User Portal.

On April 3rd, CHI@EVL will be down while we replace the controller node. Conservatively, this should take about 4 hours before services are restored. Running instances and leases won't be modified, but will not be accessible during the outage window.

Update 5PM CT: Object store is back online via a workaround.
There will be a blip tomorrow at 1PM so we can test a permanent fix for this issue that triggered this outage.

The user portal will be down for about an hour for scheduled maintenance on 02/28.

CHI@IIT is back up, but we're still waiting for the arrival of replacement hardware. Currently, bringing the site back online requires in-person actions, and so you'll observe instability until said hardware is installed. We plan for this work to be completed by the end of this week, subject to parts availability.

COMPLETE: The work is complete. Please let us know via the helpdesk if you encounter any ongoing issues.

In order to perform some necessary network maintenance, there will be brief interruptions to compute instances for CHI@TACC and KVM@TACC.

Current status:
Stability issues are resolved, we had observed and fixed deadlocks in both container launches and lease creation, due to an upstream eventlet bug.
Public Floating IPs are now functional again. The network operated by our infrastructure provider was filtering some mac-addresses, which bridged networks from working.