Files may still appear hidden on Bianca open
We have received reports and seen a few cases from Bianca and Castor that files sadly may still appear as hidden. We are investigating this issue again, and advice users to report any missing file issues to our support at email@example.com
A bit of backstory:
As many Bianca users are aware, in the months leading up to summer we were troubled by an issue with hidden or invisible files on Bianca. The files were still there, no data loss had occurred, but using “ls” or “stat” inside a directory would not show all files (!). If you however used the path to a known file, or made a copy of it, it was possible to make the file show again. Very annoying - and troublesome - since many jobs use wildcards (e.g. (/dir/*.fasta) that may not include all intended files.
The problem was quickly identified as a problem with the storage server Castor and the internal working of its filesystem (GlusterFS). We were able to mitigate the issue by “manually” repairing the filesystem, while at the same time trying to update both client and server versions of GlusterFS.
Update 2018-08-24 09:05
We believe we roughly understand the situations in which issues occur, and while we can’t do much to avoid them, we can give useful advice to avoid them and detect automatically and repair (automatically in most cases).
While there are newer releases of the software used to provide the storage system castor and they seem to address some of the cases we’ve seen, they do come with new issues, even ignoring the fact that our testing probably misses out on a lot of edge cases the system is subjected to in real life.
This means that while we of course continue to monitor and review the situation, we do not expect the situation to change for the time being.
We traverse the file system daily and perform automatic repairs or raise an alert for manual intervention when that’s required.
If you see cases where you believe files are not showing as they should, please let us know and we’ll look into it.