Operator set wave 3

[fdwr]

Adds the following operators, per #375 (comment) Support for transformers:

• Resolves Add QuantizeLinear and DequantizeLinear for mixed precision #93
• Resolves Where is scatter and gather op? #467
• Resolves Support strides option for slice operator #772
• Resolves Support reverse operator #773
• Resolves Support block-wise quantization #779

Todos

Remaining Either done now, or deferred to separate CR/issue:

• finish algorithm steps in gatherElements/scatterElements
• finish algorithm steps in gatherNd/scatterNd
• update slice output shape
• add blockwise-compatible definition
• show emulation of DQ/Q
• examples to the scattering/gathering operators
• data type tables
• all remaining reviewer feedback
• ⏳moved to separate change - u/int4 addition
• ⏳considered separately - simplify DQ decomposition by expanding expand to support blockwise
• ⏳considered separately - Request the decomposition for gatherElements, scatterElements and scatterND #767
• ⏳considered separately - (de)quantization behaviors on CoreML #822

API

partial interface MLGraphBuilder
{
    ...
    MLOperand cumulativeSum(MLOperand input, unsigned long axis, optional MLCumulativeSumOptions options = {});
    MLOperand sign(MLOperand input, optional MLOperatorOptions options = {});
    MLOperand tile(MLOperand input, sequence<unsigned long> repetitions, optional MLOperatorOptions options = {});

    // Extends the family beyond the existing gather.
    MLOperand gatherElements(MLOperand input, MLOperand indices, optional MLGatherOptions options = {});
    MLOperand scatterElements(MLOperand input, MLOperand indices, MLOperand updates, optional MLScatterOptions options = {});
    MLOperand gatherND(MLOperand input, MLOperand indices, optional MLOperatorOptions options = {});
    MLOperand scatterND(MLOperand input, MLOperand indices, MLOperand updates, optional MLOperatorOptions options = {});

    MLOperand dequantizeLinear(MLOperand input, MLOperand scale, MLOperand zeroPoint, optional MLOperatorOptions options = {});
    MLOperand quantizeLinear(MLOperand input, MLOperand scale, MLOperand zeroPoint, optional MLOperatorOptions options = {});

    MLOperand logicalAnd(MLOperand a, MLOperand b, optional MLOperatorOptions options = {});
    MLOperand logicalOr(MLOperand a, MLOperand b, optional MLOperatorOptions options = {});
    MLOperand logicalXor(MLOperand a, MLOperand b, optional MLOperatorOptions options = {});
    MLOperand notEqual(MLOperand a, MLOperand b, optional MLOperatorOptions options = {});

    MLOperand reverse(MLOperand input, optional MLReverseOptions options = {});
    MLOperand slice(
        MLOperand input,
        sequence<[EnforceRange] unsigned long> starts,
        sequence<[EnforceRange] unsigned long> sizes,
        optional MLSliceOptions options = {} // Now includes steps
    );
    ...
}

dictionary MLCumulativeSumOptions : MLOperatorOptions
{
    boolean exclusive = false; // Post-sum addition rather than inclusive pre-sum. https://en.wikipedia.org/wiki/Prefix_sum
    boolean reversed = false; // Reverse the summation direction
}

// Already exists for `gather`. Reuse for `gatherElements` too.
dictionary MLGatherOptions : MLOperatorOptions
{
    unsigned long axis = 0;
};

dictionary MLScatterOptions : MLOperatorOptions
{
    unsigned long axis = 0;
};

dictionary MLReverseOptions : MLOperatorOptions
{
    sequence<[EnforceRange] unsigned long> axes;
};

dictionary MLSliceOptions : MLOperatorOptions
{
    sequence<[EnforceRange] unsigned long> strides;
};

---

Preview | Diff

[inexorabletash]

Another "TODO" - the new ops need "constraints" tables

[inexorabletash]

Note from discussion w/ @a-sully - CoreML has restrictions on the dequantize op that we'll need to think about.

• scale and bias must be constant
• the input must be int8 or uint8 (note: CoreML has a different operator for (u)int4, but it requires everything - including the input - to be constant)
• scale and bias must be either scalars or 1D
• scale and bias must be the same rank (so, both scalars or both 1D)
• scale must be the same data type as the output (note: the existing WPTs appear to assert the scale is always float32 (and the description of that test appears to have a typo))
• scale must be positive

Re-emphasizing that dequantizing (u)int4 in CoreML is extremely limited (input must be const). @mwyrzykowski - any thoughts about how we can handle the proposed ops efficiently?

[inexorabletash]

Initial pass.

[fdwr]

Another "TODO" - the new ops need "constraints" tables

Added data type tables.

Initial pass.

Thanks - will address more tomorrow after the weekend.

[inexorabletash]

Add "Resolves #779" to the summary, so that issue will get linked to this PR and auto-closed when this merges?

... and "Resolves #773"
... and "Resolves #772"
... and "Resolves #467"
... and "Resolves #93" (I think?)

Maybe #767 too

[fdwr]

Addressed most feedback (not output shape calculation and blockwise compatible definition).

[fdwr]

🤔 What about "atop" in-place of inplace? Or in-place of a copy of?

[inexorabletash]

Another batch of feedback; I didn't make it all the way through the PR though.

[inexorabletash]

LGTM!

[anssiko]

🌊 Huge thank you to @fdwr for this significant op set update and @inexorabletash @huningxin, other participants for contributions and comments, in total close to 150 over 200!

As discussed, we'll periodically seek TAG review for significant changes to the spec. This spec update complemented with a demonstration of implementation experience across multiple backends and OSes will be brought to the TAG's attention alongside other significant changes.

[inexorabletash]

Sorry, noted a few more places where an arg reference should be linkified rather than just styled.

[fdwr]

That should address all of Ningxin and Joshua's last feedback. 🤞

[huningxin]

LGTM!

[fdwr]

@inexorabletash Want to take one more pass?
If good, will merge.

[inexorabletash]

I've been following along, still LGTM. Merge away!