Fair sharing mechanism without preemptions

[mwielgus]

What would you like to be added:

An admission-time mechanism to fair share unused resources without preemptions.

Why is this needed:

The current fair sharing approach enforces fair sharing via preemptions. Workloads from CQ that is consuming more shared resources are preempted in favor of workloads that is consuming less. This however has couple of drawbacks:

• Doesn't work in a situation when preemptions are not recommended.
• Doesn't allow stable admission of large (like consuming more than 1/Nth) amount of shared resources.

A new, complementary mechanism is needed to address these needs.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[tenzen-y]

@mwielgus What is this expectation instead of preemption? Do we want to scale out cluster to have fair sharing instead of preemption? Or do we want to restrict Job admission based on the fair share policy even if the cluster has unused resources?