| Outage start | Monday, May 12, 2025 4:30 p.m. |
| Expected end | Monday, May 12, 2025 6:56 p.m. |
Update: 7:00 pm Monday:
Reservations are workign again, and tests to launch a container and access it via floating IP are succeeding, implying that the network underlay is also healthy once again.
Please let us know if you continue to observe issues.
Update: 6:15pm Monday:
Container scheduling and wireguard tunnels for most devices has been restored, but end-to-end tests are not yet passing.
The ability to reserve devices and launch containers on CHI@Edge is currently down.
During prep of some staging infrastructure, the production instance for CHI@Edge was mistakenly hit with a hard reboot. This seems to have caused an inconsistent state in several services, including the openstack-kubernetes API bridge, and underlay network.
We're working to bring things back up, but it may take until mid-day tomorrow (Tuesday) to resolve.