AWS Batch Fargate requires exact CPU values?

[Shians]

When running wf-human-variation from ONT on AWSBatch Fargate, I get this error

ERROR ~ Error executing process > 'ingress:minimap2_alignment (1)'

Caused by:
  Requirement of 12 CPUs is not allowed by Fargate -- Check process with name 'ingress:minimap2_alignment (1)'

From what I can tell it's because the requested CPU value does not match one from the list found here:

nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchTaskHandler.groovy

Lines 923 to 943 in 944f48f

	static final Map<Integer, MemSlot> FARGATE_MEM = [1 : MemSlot.ofGiga(2,8,1),
	2 : MemSlot.ofGiga(4, 16, 1),
	4 : MemSlot.ofGiga(8, 30, 1),
	8 : MemSlot.ofGiga(16,60, 4),
	16: MemSlot.ofGiga(32, 120, 8) ]
	protected long normaliseFargateMem(Integer cpus, MemoryUnit mem) {
	final mega = mem.toMega()
	final slot = FARGATE_MEM.get(cpus)
	if( slot==null )
	throw new ProcessUnrecoverableException("Requirement of $cpus CPUs is not allowed by Fargate -- Check process with name '${task.lazyName()}'")
	if( mega <slot.min ) {
	log.warn "Process '${task.lazyName()}' memory requirement of ${mem} is below the minimum allowed by Fargate of ${MemoryUnit.of(mega+'MB')}"
	return slot.min
	}
	if( mega >slot.max ) {
	log.warn "Process '${task.lazyName()}' memory requirement of ${mem} is above the maximum allowed by Fargate of ${MemoryUnit.of(mega+'MB')}"
	return slot.max
	}
	return ceilDiv(mega, slot.step) * slot.step
	}

It seems a bit restrictive to require specific CPU values, am I doing something wrong or is that in fact the case?

[bentsherman]

It looks like 12 CPUs is not a valid request for Fargate. This cpu-memory map is derived from the table here

[Shians]

Fair enough, would you consider implementing rounding up to the next valid CPU quantity, this would be immensely helpful when adapting workflows to work on AWS Fargate.

[pditommaso]

Consider submitting a PR for that

[Shians]

I don't have enough experience with Groovy or Java to feel comfortable doing that, but the Copilot answer is to add a function like

Integer roundUpCpu(Integer cpus) {
    def maxCpus = FARGATE_MEM.keySet().max()
    if (cpus > maxCpus) {
        throw new ProcessUnrecoverableException("Requirement of $cpus CPUs is not allowed by Fargate. The maximum allowed is $maxCpus CPUs.")
    }

    def validCpus = FARGATE_MEM.keySet().findAll { it >= cpus }
    return validCpus.min()
}

That that seem like a valid solution?