Issues with Grus closed

There is currently an issue with the Grus system that is used to download NGI delivery projects. We are investigating this issue.

Around 15:00 on 16th of September one of the storage servers reported that four disks at the same time left the storage array - rendering the server storage unavailable. We use a distributed file system in Grus called Gluster which is spread across all storage servers. When one of its storage servers becomes unavailable the filesystem is unable to handle any useful IO. We are investigating the hardware and firmware for the cause.

Update 2020-09-21 14:50

Grus is now open for production. Deliveries that are delayed for delivery have been resumed. Grus is accessible both using SFTP and from within Irma.

Update 2020-09-21 12:00

The disks show no signs of hardware issues (eg. bad sectors). The parity check reports no issues and the filesystem (XFS) reports no inconsistencies. The filesystem can be mounted and the data appears as expected. We have opened an issue at the vendor and sent logs for analysis. We see no signs of hardware problems. This might indicate that the disks left the array due to firmware bugs. We have a more recent firmware and we are performing the upgrade today. Once the firmware upgrade completes, expected later today if everything goes as expected, we will return Grus to production.

Update 2020-09-17 10:00

Yesterday afternoon we limited access to Grus for all users. At this time no new delivery projects will be delivered from Grus either using SFTP or using the delivery button to Bianca on SUPR. Moving data to Grus from Irma is also placed on hold.