Issues with Crex (file system for Rackham) closed

We are experiencing problems with Snowy’s and Rackham’s storage system “Crex”. Many disks in the same enclosure is reported as failed. The disks are now rebuilding and the manufacturer is informed and working on the issue.

Rebuilding many disks simultaneously has significant impact on the performance of the filesystem. The whole system might appear slow and all operations against the filesystem may take much more time than usually do. This is why commands such as “ls” or “ll” takes so long to complete. Navigating inside your project directory may also take significantly more time than normal.

It is only the filesystem that houses the projects directories that is affected. Home directories are not affected and users should be able to login and access their files there. GUI programs like MobaXTerm and WinSCP will be slow or even crash when their connection is set up to access project directories.

Update 2019-02-18 12:00

The rebuilds of Crex disks are now finished and the system does not report any errors. Rackham and Snowy should be in normal operations now. Please, do not hesitate to submit a support ticket if something is still not functioning properly. Any such feedback is highly appreciated.

Update 2019-02-15 14:30

Rebuilds are starting to wrap up, but have not yet finished. We will need to do a little bit of physical work next week but believe we’re past the major disturbances we saw earlier (e.g. yesterday).

Update 2019-02-14 10:00

We are in continous contact with the manufacturer to get this solved. The large amount of disks that simultaneous gets reported as false makes it unlikely for it to be an hardware issue related to the disks. We expect Rackham and Snowy to be fully back in normal operations by early next week, but performance will improve as more disks gets rebuilt, so it is difficult to give an ETA.