Cooling issues in the UPPMAX compute room closed

We have an ongoing issue with cooling in the server hall. All systems have been closed down to prevent hardware damage.

Final ticket report

All systems are up and running after the ångström wide cooling failure.

At around 08:09 our computer room lost cooling due to failure of the Ångström cooling curcuit.

According to to Akademiska Hus the cooling failed because of two expansion tanks beeing empty. At this time they dont know why and 800L of water have mysteriously dissapeared. This made the pumps that drive the cooling curcuit to stop.

The temperature then started to rise about 1°C per minute and the situation quickly became critical.

Here is a short outline of what happened:

Update 2018-12-20 13:30

The UPPMAX Cloud is back up.

Update 2018-12-19 14:30

Bianca is back up.

Update 2018-12-19 11:50

Rackham, Snowy and Irma are up and the queues are running.

Update 2018-12-19 09:30

The problem has been fixed according to Akademiska Hus. It was related to an expansion tank losing pressure, which made the circulation stop. We will begin restoring our systems.