Files and directories may be hidden on Bianca closed

We have received reports of the issue of missing files and directories inside the /proj and /proj/nobackup directories on Bianca again.

It seems the software update of the Gluster file system we did last week didn’t have the desired effect to the full extent we were hoping.

We’ve reverted to previous settings that should address this issue (although with a performance penalty), some settings will not change until the file system is mounted again (typically when a login node is removed and then recreated when needed).

Final ticket report

We’ve reverted to an older client version that solves the issues we’ve seen.

Update 2018-04-27 08:57

For clarity; the reverted client version will be active for new VMs started since yesterday, i.e. for new nodes given to your virtual cluster for jobs when existing nodes are not enough or when a new virtual login node is created because one is not running upon login.

Update 2018-04-26 14:25

We have reverted to a older client version and verified that we can’t trigger the incorrect behaviour.

Update 2018-04-25 10:23

The base problem (gluster not always updating it’s meta structure as it should) is still lingering. We’re still evaluatuing our options and run operations to repair these often in the meantime.

Update 2018-04-19 11:33

Unfortunately, one of the settings we reverted to turned out to behave differently with the newer file system version we’re now running and caused a lot more to be hidden. We’ve removed this setting.

Nodes created with the incorrect setting need to be handled, and we’re working through them.

Update 2018-04-17 15:15

Some projects seem to have issues related to this, we’re investigating.