Reported Outages

TACC Network Maintenance 28 November 2021

Resolved Posted by Cody Hammock on October 25, 2021
Outage start Sunday, November 28, 2021 10 a.m.
Expected end Sunday, November 28, 2021 4 p.m.

Resolved: The network maintenance at TACC was completed at 1:19 PM (CST) on Sunday, November 28

Network maintenance will be carried out between 10:00 AM and 4:00 PM (CST) on Sunday, November 28. Access to all TACC systems (CHI@TACC, KVM@TACC, and Chameleon Portal) will be unavailable during this time. Instances will continue to run, but users will have no access to TACC-based services and systems until the upgrade is complete.

Power Maintenance at CHI@UC November 19th - 22nd

Resolved Posted by Michael Sherman on October 25, 2021
Outage start Friday, November 19, 2021 6 p.m.
Expected end Monday, November 22, 2021 2:10 p.m.

Update: The outage is now over. Jupyterhub, Trovi, and CHI@UC are now available.

Important Note: Instances created prior to the outage are available, but must be powered back on manually. You can do so via the web ui, or other CLI / API as you prefer. They were shut down prior to the outage to avoid any data-loss.


There will be a total outage for the CHI@UC site 6pm, Friday November 19th - 12pm, Monday November 22nd, due to maintenance on the power system in the data center.

Oct 24 login system outage

Resolved Posted by Jason Anderson on October 24, 2021
Outage start Sunday, October 24, 2021 7 a.m.
Expected end Sunday, October 24, 2021 2:05 p.m.

Due to a certificate expiration, the authentication system for the Chameleon sites is effectively online, preventing web and CLI-based logins. This affects all Chameleon sites, the user portal, and the Jupyter environment.

We are working with the team providing the certificate service and will update when it is back online.

Jupyter.chameleoncloud.org not accessible

Resolved Posted by Michael Sherman on October 20, 2021
Outage start Wednesday, October 20, 2021 3 p.m.
Expected end Wednesday, October 20, 2021 3:42 p.m.

Update 3:35pm: Access has been restored. Please let us know if you continue to see issues.


The chameleon Jupyterhub is currently not accessible, as of ~2:40 PM today.

We believe the cause to be related to a network configuration change, and are investigating.

We'll update here with an ETA or resolution.

 

Ticketing system e-mail outage

Resolved Posted by Francois Halbach on October 15, 2021
Outage start Wednesday, October 13, 2021 12 a.m.
Expected end Friday, October 15, 2021 6 p.m.

The helpdesk email help@chameleoncloud.org is currently non-operational. New tickets and ticket replies submitted by e-mail are not being delivered.

New tickets and replies can still be sent via the 'My Tickets' portal page:
https://chameleoncloud.org/user/help/

It is unclear when the ticketing system will be restored, we apologize for the inconvenience.

Swift Object store still down for CHI@UC

Resolved Posted by Michael Sherman on October 05, 2021
Outage start Monday, October 04, 2021 9:30 a.m.
Expected end Tuesday, October 05, 2021 1:33 p.m.

Update: 1:30 PM - The object store is now available.

The swift object store for CHI@UC is still down as of 9:30 AM October 5th, we expect resolution by mid-day.

Storage outage at TACC

Resolved Posted by Cody Hammock on September 29, 2021
Outage start Wednesday, September 29, 2021 11:12 p.m.
Expected end Thursday, October 07, 2021 8:36 a.m.

The storage system at TACC is currently unavailable. This outage affects launching new instances for CHI@TACC, the object storage for CHI@TACC, and currently running, and starting new instances in KVM@TACC. 

It is unclear when services can be restored, and we apologize for the inconvenience.

Planned maintainance at CHI@UC for transition to new hardware

Resolved Posted by Michael Sherman on September 27, 2021
Outage start Monday, October 04, 2021 9 a.m.
Expected end Tuesday, October 05, 2021 8:32 p.m.

Update: 7:30pm CST:   Existing user instances are reachable, and baremetal nodes can be deployed.

Update: 8:30pm CST: Regular functionality is restored, with the following exceptions
- Heat Orchestration may encounter issues.
- Bring-your-own-controller SDN experiments at UC are not available.
- Gnocchi at UC is not available, but was also completely unused.

This page will be updated as services are tested.

Launch issues at CHI@TACC

Resolved Posted by Francois Halbach on September 02, 2021
Outage start Thursday, September 02, 2021 12:30 p.m.
Expected end Friday, September 03, 2021 12:30 p.m.

Update: The issue has been resolved.

CHI@TACC is experiencing launch issues for new instances since today at 12:30 PM. We are investigating and will provide an update once functionality is restored. CHI@UC is still available at https://chi.uc.chameleoncloud.org.

KVM@TACC Urgent Outage 26 Aug 2021

Resolved Posted by Cody Hammock on August 26, 2021
Outage start Thursday, August 26, 2021 2 p.m.
Expected end Thursday, August 26, 2021 3:33 p.m.

Update: Maintenance is complete

In order to address recent stability concerns on KVM@TACC, maintenance will be performed today, August 26, 2021, from 2:00PM (CDT) to 4:00PM (CDT). During this time all instances will continue to run, but will be inaccessbile until the work is complete.