Update: the storage capacity is back to normal, rebalancing is happening in the background now.

The effected hosts are being phased out, systems are running normally at the moment, but we expect some slowness from time to time for the next 48h during rebalancing to other nodes.

The storage cluster is currently recovering, but we detected multiple defects in multiple servers. We are currently removing the data from these servers and replace them with new ones.

All service should be back to normal, if you continue to experience issues please contact [email protected]

Axpo verified - they made a mistake on synchronising a machine on the 220kV line and thus causing an emergency switch to turn off power. This affected the whole back valley (place10, place6, place7)

Storage cluster for VMs is back in write mode, I/O performance currently degraded.

Systems that failed are currently rebooting, storage cluster for VMs still in emergency state.

Update 10:27 Axpo switched over to emergency - we got power back.

All kubernetes based services are still online (separate Ceph / hardware cluster) However the cluster is also running on battery power at the moment, UPS state around 80%.

We got an update that the problem comes from Axpo affecting the grid to which we are also connected. * Affected: place10 (VM, kubernetes based services) * Not affected: place5 (colocation)

Hello @znurree the whole region of Glarus is having a power outage and we are affected by it. We are in touch with the grid and local power plant to get insight on what is going on.

2024-11-26 power outage * at the moment we experience a power outage * the whole region of Glarus is having a power outage * all systems are still up, running on battery power at the moment * we are in touch with the grid and local power plant to get insight on what is going on

All VMs are back online.

Most virtual machines are back online. ETA for the rest is 20 minutes.

All virtual machines are restarting.

Majority of storage cluster is back online.

Storage cluster is rebooting.

There was a power outage in place10. We are investigating the cause and now restarting all affected systems.

All issues are resolved now. All services are back online.