Issue with castor (filesystem for bianca) closed

We’re currently seeing an issue on one of the nodes providing castor (the file system for bianca). This may lead to failed read/writes and possibly crashed jobs. We’re working to get this fixed as soon as possible.

Final ticket report

All files should be restored now.

While we have no reason to suspect any files should have disapperaed/been corrupted, we do not have data so we can say for certain nothing such has happened. Be extra vigilant and let us know if you see anything unexpected.

Update 2018-10-29 11:56

This is fixed now, but read/writes may have failed, causing jobs to crash.

Update 2018-10-29 12:26

We’re seeing additional issues with files being out of place and are investigating this. We will post updates as we learn more.

Update 2018-10-30 13:09

We’re still working with this in several ways and are still not finished restoring access (some files may still be unavailable),

Update 2018-10-31 15:55

This is still ongoing. Since we’ve seen a few oddities from the hardware, we’re trying not to rush things to much while at the same time restoring full access in a reasonable time fram.