AWS Batch Queue Switching Based on Number of Retries

[kirkgrubbs1]

Hello,
I have a pipeline running on AWS Batch via nextflow. This pipeline starts several parallel jobs some of which are longer than others. I currently have them being submitted to a spot based queue ("default-gwfcore"). Unfortunately, some of them run quite long and have a tendency get killed after exhausting the number of retries I have set. To remedy this I would like to have the jobs switch to our on-demand queue ("priority-gwfcore") on the 5th try. I added the following to the nextflow.config but it doesn't seem to accomplish what I want:

process.queue = { task.attempt > 4 ? 'priority-gwfcore' : 'default-gwfcore' }

Any help is greatly appreciated.

[bentsherman]

Hi @kirkgrubbs1 , since version 22.04, Nextflow uses an AWS-specific mechanism to handle spot reclamations automatically. As a result, these retries are not "seen" by Nextflow. By default, AWS Batch will automatically retry the task 5 times before giving up, which means in your example the task would have to fail 20 times in order to be routed to the on-demand queue. 😅

You can control this setting with the aws.batch.maxSpotAttempts config option (docs). Either set it to 0 and use your queue setting as is, or set the max spot attempts to 4 and change your queue setting to use task.attempt > 1.

[kirkgrubbs1]

in a similar situation. would:

aws.batch.retryDelay = 1800s

make sure that a job waits 30 minutes before retrying?
Or would that be more like:

retryDelay = 1800s

as the aws batch retries are now being used

[bentsherman]

I don't think that option exists, but there is aws.batch.retryMode, which by default uses an exponential backoff up to 20s.

It looks like you can use executor.submitter.retryDelay to set the retry delay, it's just not documented.

[kirkgrubbs1]

I added the following to my config file, and it does not seem to have worked. I'm guessing I'm missing something.

process {
    
    cpus   = { check_max( 16, 'cpus'   ) }
    memory = { check_max( 64.GB + (8.GB * task.attempt), 'memory' ) }
    time   = { check_max( 12.h  * task.attempt, 'time'   ) }
    errorStrategy = 'retry'
    maxRetries = 6
    maxSpotAttempts = 0
    queue = { task.attempt > 2 ? 'priority-gwfcore' : 'default-gwfcore' }
    executor.submitter.retryDelay = 1800

}

[bentsherman]

Some of those config options belong in other scopes:

process { /* ... */ }
aws.batch.maxSpotAttempts = 0
executor.submitter.retryDelay = 1800

[kirkgrubbs1]

awesome, thank you! I'm giving it a test now

[dxu104]

I tested it with below config, it work very well

queue     = { task.attempt > 1 ? 'MMBatch-GPU-8xlarger-onDemand' : 'MMBatch-GPU-8xlarger-spot' } 
 
 maxSpotAttempts = 0