[KleidiAI] Always attempt activation packing

[mcr229]

Currently, we only leverage KleidiAI kernels for Dynamically Quantized Activations with 4 bit blockwise weights on linear layers. This has seen a lot of success in our LLM Prefill performance.

However KleidiAI has also integrated into other kernels for XNNPACK. Specifically 4 bit channelwise weights, and 8 bit channelwise weights. We should attempt to use their kernels for these Linear schemes as well. This should have an effect on some example models we have like:

• ViT
• MobileBert
• W2L
• Emformer

And in general other models that can do 8 bit channelwise quantization. (We don't support 4 bit channelwise quantization atm).

Performance

Android S24 (6 Threads) (10 Runs)

On Android S24, we see a nice perf uplift using the KleidiAI's activation packing and QD8_QC8W gemm kernels. Specifically on the ViT model we see ~8% (58.61ms --> 53.6948ms). You can see the difference in the GEMM performance by looking at the operator profiling below.
Consider event 834. This is a Fully Connected Layer:

• Without Kleidi we do QD8 (no activation packing), and the p50 timing is around 0.75ms.
• With Kleidi we do QP8 (activation packing), and the p50 timing is around 0.6048ms. This is ~20% uplift on GEMMs!

Profiles:

• kleidi_android.csv
• no_kleidi_android.csv

Macbook (6 Threads) (10 Runs)

On macbook, we see a different story. With KleidiAI, we see a dip in perf: (49.32ms --> 56.53ms) which is around a ~14% drop.

Let's take a look at the Fully Connected Layers again specifically event 834 again:

• Without Kleidi the p50 timing is 0.4935ms
• With Kleidi the p50 Timing is 0.648ms

This is a 24% dip in GEMM Performance!

Profiles

• kleidi_mac.csv
• no_kleidi_mac.csv

[mcr229]

Stack from ghstack (oldest at bottom):

• [KleidiAI] turn on SME2 by default #13233
• -> [KleidiAI] Always attempt activation packing #13232
• [Profiling] Add python scripts to generate per-op profiling from etdumps #13302
• Add csv and etdump files to gitignore #13301

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13232

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 1c33efb with merge base a84b3c9 ():

NEW FAILURE - The following job has failed:

• Build documentation / build (buck2) / Build doc (gh)
  At least one of the pre-conditions you specified did not hold

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[digantdesai]

Sounds good. Perf numbers? IIRC you said this didn't result in perf uplift? Stamping if I am remembering it wrong.

[mcr229]

Sounds good. Perf numbers? IIRC you said this didn't result in perf uplift? Stamping if I am remembering it wrong.

collecting them now. I realized i was running with debug mode on, so the perf numbers weren't representative.