Slow slurm on rackham closed
Slurm (the workload manager we use to schedule jobs) does not always like it then it has too many jobs to keep track of. This has happened quite a few times lately.
When this happens, squeue or sbatch/salloc may be very slow or even time out with an error like
slurm_load_jobs error: Socket timed out on send/recv operation
we’re aware of this and do what we can to improve the situation, but a lot of jobs means a lot of work which takes time.
Final ticket report
We have not seen this issue for some time now.