CHI@UC Maintenance window for Openstack version upgrade

Resolved Posted by Michael Sherman on July 02, 2024
Outage start Monday, July 22, 2024 9 a.m.
Expected end Tuesday, July 23, 2024 1 p.m.

As of 1pm, July 23rd, reservation enformenents are fixed, and we're declaring the CHI@UC outage over.
You may notice some minor changes to the horizon dashboard as we fix some remaining UI issues in the instance launch dialog, but these should not affect any functionality.


As of 8pm, July 22nd, the CHI@UC object store is available again. However, we have observed issues with the creation of new blazar reservations, with an error message about enforcement checks. We will update here when that issue is resolved.

All other features should be back to normal, and again, all non-CHI@UC sites are unaffected.


7pm, July 22: Most compoents of the upgrade are now complete, and CHI@UC is now available for use again.
All of your instances and reserved resources should be reachable and usable.

However, we are maintaining a sev-2 outage, as the CHI@UC object store is still under maintenance, we expect to have it restored by tomorrow, the 23rd. An unexpected interaction betwen ceph and keystone configuraton updates has required more troubleshooting than arose in the staging environments.


Update: Due to unexpected blockers, we were not able to perform the upgrade on the 15th, and are rescheduling it to Monday, July 22nd. No services were taken down on the 15th.

At 8am central time, on July 22nd, CHI@UC will be unavailable for use, as we upgrade the core openstack services on the controller hosts. During this time, the chi.uc.chameleoncloud.org webpage and APIs will be inaccessible, as will network connectivity to any running instances.

All instances and leases will be unchanged by the upgrade, and you'll be able to access your resources after the maintenance window ends.

All other sites and services are unaffected by this work.