Issues with singularity since the latest maintenance window closed

Final ticket report

We believed this was fixed by the latest build installed.

Update 2018-11-14 15:26

A build that should contain fixes for the issue

container creation failed: mount error: can't mount image /proc/self/fd/13: failed to find loop device: file exists

is now installed and will be used for virtual compute nodes created from this morning. It will take some time before this is used for all jobs, but if you see it still, let us know.

Original report

During the latest maintenance window, we upgraded singularity to the latest version (3.0). Some users have seen failed jobs since then.

We’re working on figuring out what goes wrong, but so far, our best suggestion is to try rerunning and see if that helps.