ResolvedPosted by Michael Sherman on August 24, 2022
Outage start
Wednesday, August 24, 2022 5:22 p.m.
Expected end
Thursday, August 25, 2022 10:52 a.m.
The outage is resolved, and the object store is available again.
A rebalancing operation in the underlying storage cluster caused higher than normal I/O, and pushed some of the storage nodes into an unstable state. The node configuration has been tuned to prevent this instability from recurring.
ResolvedPosted by Michael Sherman on August 18, 2022
Outage start
Wednesday, August 17, 2022 12 p.m.
Expected end
Thursday, August 18, 2022 3:29 p.m.
3:30PM: The issue has been resolved.
We're observing that http requests to non-https endpoints from nodes at CHI@UC to the internet are being dropped, and are working with our upstream network provider to investigate.
As a workound, you can make requests to an https endpoint, if available.
ResolvedPosted by Michael Sherman on August 05, 2022
Outage start
Friday, August 05, 2022 12 p.m.
Expected end
Friday, August 05, 2022 2 p.m.
Update 4PM: https://jupyter.chameleoncloud.org/ is back online, and the storage system has stabilized.
Resolving this required more changes than expected, so there may be some remaining instability over the weekend.
Please reach out to the helpdesk if you encounter issues.
ResolvedPosted by Michael Sherman on August 01, 2022
Outage start
Monday, August 01, 2022 4:34 p.m.
Expected end
Monday, August 01, 2022 9 p.m.
Update 9pm: The chillers have been repaired, and we've received the all clear from ANL staff. CHI@UC is now back to normal operation.
The datacenter hosting CHI@UC has experienced a failure in its cooling system. To reduce load on the remaining cooling, we are blocking new node reservations at UC until the failure is resolved.
ResolvedPosted by Michael Sherman on July 13, 2022
Outage start
Wednesday, July 13, 2022 2:41 p.m.
Expected end
Monday, July 18, 2022 2:41 p.m.
We're currently seeing elevated error rates when provisoning nodes at CHI@UC.
When deploying an instance at CHI@UC, if it moves to state "error", with a message about "provisioning timed out", please reach out to the helpdesk, and include the UUID of your instance and node(s).
We're currently working to narrow down which nodes are affected and reproduce the issue.
ResolvedPosted by Michael Sherman on July 05, 2022
Outage start
Monday, July 11, 2022 8 p.m.
Expected end
Monday, July 11, 2022 9 p.m.
On July 11 at 8pm Central, the routers upstream from CHI@UC will undergo emergency maintenance, which should last between 15 minutes and 1 hour. During this period, all external connectivity to CHI@UC and Jupyterhub will be down. Your instances will keep running, but you won't be able to access them during this period.