Scheduled downtime for authentication systems
| Outage start | Tuesday, September 06, 2022 11 a.m. |
| Expected end | Tuesday, September 06, 2022 12 p.m. |
On Tuesday, September 6th, our authentication services will be down from 11 AM to 12 PM US Central Time.
Reported Outages
http requests being dropped for instances at CHI@UC
| Outage start | Wednesday, August 17, 2022 12 p.m. |
| Expected end | Thursday, August 18, 2022 3:29 p.m. |
3:30PM: The issue has been resolved.
We're observing that http requests to non-https endpoints from nodes at CHI@UC to the internet are being dropped, and are working with our upstream network provider to investigate.
As a workound, you can make requests to an https endpoint, if available.
Jupyterhub storage slowness
| Outage start | Friday, August 05, 2022 12 p.m. |
| Expected end | Friday, August 05, 2022 2 p.m. |
Update 4PM: https://jupyter.chameleoncloud.org/ is back online, and the storage system has stabilized.
Resolving this required more changes than expected, so there may be some remaining instability over the weekend.
Please reach out to the helpdesk if you encounter issues.
Datacenter Cooling failure for CHI@UC
| Outage start | Monday, August 01, 2022 4:34 p.m. |
| Expected end | Monday, August 01, 2022 9 p.m. |
Update 9pm: The chillers have been repaired, and we've received the all clear from ANL staff. CHI@UC is now back to normal operation.
The datacenter hosting CHI@UC has experienced a failure in its cooling system. To reduce load on the remaining cooling, we are blocking new node reservations at UC until the failure is resolved.
maintenance window for CHI@NU
| Outage start | Tuesday, August 02, 2022 10 a.m. |
| Expected end | Tuesday, August 02, 2022 2 p.m. |
On August 2nd, CHI@NU will be down for a major version upgrade.
Running instances will be inaccessible for the duration, but not otherwise impacted.
Other sites are not affected.
Elevated error rates at CHI@UC
| Outage start | Wednesday, July 13, 2022 2:41 p.m. |
| Expected end | Monday, July 18, 2022 2:41 p.m. |
We're currently seeing elevated error rates when provisoning nodes at CHI@UC.
When deploying an instance at CHI@UC, if it moves to state "error", with a message about "provisioning timed out", please reach out to the helpdesk, and include the UUID of your instance and node(s).
We're currently working to narrow down which nodes are affected and reproduce the issue.
KVM@TACC API and WebUI down
| Outage start | Monday, July 11, 2022 12:40 p.m. |
| Expected end | Monday, July 11, 2022 2:14 p.m. |
2:14PM: The issue is now resolved, and KVM is accessible again.
Provisioning issues with some configurations at CHI@TACC
| Outage start | Tuesday, July 05, 2022 8 a.m. |
| Expected end | Wednesday, July 06, 2022 6 p.m. |
All node_types are now working as expected.
Upstream Network maintenance affecting CHI@UC and Jupyter
| Outage start | Monday, July 11, 2022 8 p.m. |
| Expected end | Monday, July 11, 2022 9 p.m. |
On July 11 at 8pm Central, the routers upstream from CHI@UC will undergo emergency maintenance, which should last between 15 minutes and 1 hour. During this period, all external connectivity to CHI@UC and Jupyterhub will be down. Your instances will keep running, but you won't be able to access them during this period.
Other sites will not be affected.
Floating IP Reservation Outage for CHI@TACC
| Outage start | Friday, July 01, 2022 5 p.m. |
| Expected end | Wednesday, July 06, 2022 12:56 p.m. |
Update July 6th: Reservations for floating IPs at CHI@TACC should work as normal now.
Update: July 5th
Reservations for Floating IPs at CHI@TACC are still failing, and we are working on a fix.
This only affects reservation for a floating IP. You can work-around this issue by not reserving IP addresses, and using ad-hoc IPs instead. You can find instructions for this method here: https://chameleoncloud.readthedocs.io/en/latest/getting-started/index.html#associating-an-ip-address