| Outage start | Tuesday, July 28, 2026 6 p.m. |
| Expected end | Thursday, July 30, 2026 6 p.m. |
We observed instability in the ironic provisioning service at CHI@UC following the openstack version upgrade on Tuesday Jul 28th.
Some users observed instances moving to an "Error" state instead of provisioning successfully, failures to access the serial console, and a brief window where nodes were marked as unavailable for reservation.
Aside from user access to the serial console, already running experiments were not impacted.
The root cause was a new limit on limit on the number of processes the provisioning service could spawn, accidentally introduced in an upstream change. When the provisioning service spanwed additional proccesses to provide serial console access, it tripped this limit and failed to read or change node power state, moving instances and nodes to an error state.
The issue was resolved by reverting this limit, as well as several fixes to code paths that didn't recover automatically from the failure.