Poor /proj performance on Rackham and Snowy closed

We have an issue with the project storage system Crex which houses the project directories for Rackham and Snowy. If you attempt to browse or list files from /proj you might experience severe performance degradation, uninterruptable terminals, and other non-responsive behavior. We are investigating this issue.

Update 2020-10-21 11:20

Crex performance recovered monday evening and so far seems to have stabilized at normal responsiveness levels. We have not yet gather convincing evidence of the root cause. We did however discover a few I/O intensive jobs that we could optimize and reduce load on the file system. We have asked for input from the storage vendor, who has examined the system, but has not been able to identify any issues on the server side.