Rackham and Snowy /proj was out of inodes this morning closed

One of the MDTs in the storage system Crex that attaches /proj to Rackham and Snowy ran out of inodes this morning. An inode is created for every regular new file. We expect this problem was caused due to one or many jobs creating a large amount of files in a short period of time. If you ran into this issue you would have received I/O-errors such as seen below when trying to create a new file inside your project directory:

$ touch /proj/snic2020-1-1/newfile
touch: cannot touch ‘.fil’: No space left on device

The problem was quickly noted and we are in the process of balancing the MDT by spreading the inodes onto more available MDTs. Creating new files is possible again, but we ask users to hold off starting inodes intensive jobs for a few hours so that we can free up even more inodes.

If you have any questions please contact the support at support@uppmax.uu.se.

Update 2020-12-16 13:50

At the time of writing we have 31M inodes available and we will be able to add up to ~50M. The immediate problem is thus solved. We still urge users to clean up your project directory and remove files you no longer have a need for.