So you’re trying to set up an advance reservation for someone, something fairly simple. However, due to the cluster being busy you get an error:

1
2

qrsub -u bob -N openmpi_test -pe openmpi 8 -l h_vmem=1G -d 500:0:0
error: no suitable queues

This is because for an advance reservation to be successfully submitted, GridEngine must be able to guarantee the requested resources will be available as soon as the reservation starts. Since we didn’t supply a start time, and the resources aren’t currently available, the request fails. One option is to figure out the walltimes of the currently running jobs, particularly those which are using the resources we want to reserve, and submit the reservation to start after they would be available. For example, if we knew GridEngine could guarantee the resources be available 10 hours from now, we’d use something like (assuming it was currently 08:08 on October 10th):

1

qrsub -u bob -N openmpi_test -pe openmpi 8 -l h_vmem=1G -d 500:0:0 -a 10211808

But what if we don’t know when the resources will be available, or more likely, we don’t care? We just want the reservation to be made as soon as possible. Well, the solution is to submit the reservation as a job which requires the same resources. The job won’t run until the resources are available and since we can set it up with a known walltime we can have it submit the reservation for just after it completes. The most difficult bit is calculating the value for the -a switch. For that, I use a little bit of Python. Here’s my script:

1
2
3
4
5
6
7
8

#!/bin/bash
#$ -pe openmpi 8
#$ -N openmpi_test_submit
#$ -l h_vmem=1G,h_rt=0:5:0
#$ -m e
#$ -M kamil@example.com,bob@example.com
after=$( python -c 'import datetime; print (datetime.datetime.now() + datetime.timedelta(minutes=10)).strftime("%y%m%d%H%M")' )
qrsub -u bob -N openmpi_test -pe openmpi 8 -l h_vmem=1G -d 500:0:0 -a $after

When submitted, the script will sit in the queue until the resources are available and then run with a walltime of 5 minutes. It will submit the advance reservation to start 10 minutes in the future. Upon completion it will email both myself and the user for whom the reservation is being created. Maybe a little convoluted, but it works.

Two ways I think GridEngine could make the reservation system better to support this use case:

1. Allow relative values for the -a flag. For example -a +0:10:0 to start the reservation 10 minutes in to the future.

2. Even better, allow the reservations to be queued as a flag to qrsub. This would eliminate the point for such a script entirely.