Preemption during flavor assignment ambiguous

[gabesaba]

What would you like to be cleaned:
It is possible that we assign mode Preempt, even when preemption is never possible. E.g. imagine the case that the CQ never allows preemption, and yet the workload's requests are below nominal quota.

kueue/pkg/scheduler/flavorassigner/flavorassigner.go

Lines 622 to 631 in 14e2ca7

	// Check if preemption is possible
	mode := noFit
	if val <= rQuota.Nominal {
	mode = preempt
	if a.oracle.IsReclaimPossible(log, a.cq, *a.wl, fr, val) {
	mode = reclaim
	}
	} else if a.canPreemptWhileBorrowing() {
	mode = preempt
	}

However, the fact that we chose Preempt and not NoFit is later used to reserve resources that belong to the CQ - see #4105.

Nevertheless, this is confusing. Something like the following categories may make more sense:

NeverFit - current NoFit, where workload is too big even if cluster is empty
NoFit - workload can fit if no workloads are running, but it has to wait, as preemption is not an option
Preemption
Fit

/cc @mimowo

[gabesaba]

/assign @gabesaba

[mimowo]

The fact that NeverFit workloads are handled as NoFit might be even considered a bug currently, because such a workload blocks indefinitely other "valid" workloads without a clear response saying "why".