Crex is slow closed

Crex is not behaving well att the moment and file system access is very slow which means that it is in practice not working. We upgraded Crex yesterday to the latest recommended version from the vendor. We are opening a case at the vendor.

Update 2022-03-03 14:45

We have restarted one of the matadata servers for Crex and it seems like Crex is working fine at the moment. We will follow this up with the vendor.

Update 2022-03-03 16:00

Crex worked fine for a while but then stopped again. We are in contact with the vendor support again.

Update 2022-03-03 19:00

We are still working with Crex together with the vendor.

Update 2022-03-03 21:30

We stopped the queues earlier this afternoon so that no new jobs started.

We have worked with the vendor during the day and resolved issues with the configuration after the upgrade of Crex during the service window. Crex is now up and running again.

We have not started the queues yet. We will first check that the cluster is running okay tomorrow. We will continue working with the vendor to analyze what went wrong.

Update 2022-03-04 09:00

Crex has been running fine during the night. We have started the queues again on Rackham and Snowy.

Update 2022-03-07 17:00

Crex stopped working about one hour ago. The queues on Rackham and Snowy has been stopped. We have reopened our case with the vendor.

Update 2022-03-08 08:00

Crex worked fine yesterday evening so we started up the queues again.

Update 2022-03-08 13:15

Crex has been working fine today.

Update 2022-03-09 15:30

Crex is slow again. We are in contact with the vendor.

Update 2022-03-10 16:45

Crex has behaved reasonably well today. It has been a bit slow a few times and we have updated the vendor.

Update 2022-03-21 11:00

Crex have behaved well since last update. Status closed.