Support WebNN EP #15698

Honry · 2023-04-26T09:12:25Z

Description:

This PR intends to enable WebNN EP in ONNX Runtime Web. It translates the ONNX nodes by WebNN API, which is implemented in C++ and uses Emscripten Embind API. Temporarily using preferred layout NHWC for WebNN graph partitions since the restriction in WebNN XNNPack backend implementation and the ongoing discussion in WebNN spec that whether WebNN should support both 'NHWC' and 'NCHW' layouts. No WebNN native EP, only for Web.

Motivation and Context:
Allow ONNXRuntime Web developers to access WebNN API to benefit from hardware acceleration.

WebNN API Implementation Status in Chromium:

Tracked in Chromium issue: #1273291
CPU device: based on XNNPack backend, and had been available on Chrome Canary M112 behind "#enable-experimental-web-platform-features" flag for Windows and Linux platforms. Further implementation for more ops is ongoing.
GPU device: based on DML, implementation is ongoing.

Open:

GitHub CI: WebNN currently is only available on Chrome Canary/Dev with XNNPack backend for Linux and Windows. This is an open to reviewers to help identify which GitHub CI should involved the WebNN EP and guide me to enable it. Thanks!

This PR enables WebNN EP in ONNX Runtime Web. It translates the ONNX nodes by WebNN API, which is implemented in C++ and uses Emscripten Embind API. Temporarily using preferred layout NHWC for WebNN graph partitions since the restriction in WebNN XNNPack backend implementation and the ongoing discussion in WebNN spec that whether WebNN should support both 'NHWC' and 'NCHW' layouts. No WebNN native EP, only for Web.

Honry · 2023-04-26T09:16:24Z

@fdwr, @guschmue, PTAL, thanks! Any other reviewer should I invite?

cc/ @huningxin @zesongw

snnn · 2023-04-26T15:12:35Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Windows ARM64 QNN CI Pipeline

snnn · 2023-04-26T15:12:42Z

/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline

snnn · 2023-04-26T15:12:48Z

/azp run Windows GPU TensorRT CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

azure-pipelines · 2023-04-26T15:12:54Z

Azure Pipelines successfully started running 2 pipeline(s).

azure-pipelines · 2023-04-26T15:13:12Z

Azure Pipelines successfully started running 5 pipeline(s).

azure-pipelines · 2023-04-26T15:13:13Z

Azure Pipelines successfully started running 9 pipeline(s).

guschmue · 2023-04-26T15:35:28Z

@Honry, thanks for the PR! Going to look at it today.

guschmue · 2023-04-26T18:04:44Z

ci complains about a minor lint issue.
You can run the lint pass local with "cs js; npm run lint".

https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=978665&view=logs&j=96bb6f8b-177f-5c87-f240-68050a28686f&t=5f53931c-9b29-5372-7076-43dd32f36bb8

guschmue · 2023-04-26T22:32:42Z

@Honry, got it to build and I can run mobilenet-v2 on canary - awesome!

snnn · 2023-04-27T02:22:43Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Windows ARM64 QNN CI Pipeline

snnn · 2023-04-27T02:22:50Z

/azp run Windows GPU TensorRT CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

snnn · 2023-04-27T02:22:57Z

/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline

azure-pipelines · 2023-04-27T02:23:08Z

Azure Pipelines successfully started running 2 pipeline(s).

azure-pipelines · 2023-04-27T02:23:11Z

Azure Pipelines successfully started running 5 pipeline(s).

azure-pipelines · 2023-04-27T02:23:18Z

Azure Pipelines successfully started running 9 pipeline(s).

snnn · 2023-04-27T04:33:26Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Windows ARM64 QNN CI Pipeline

snnn · 2023-04-27T04:33:43Z

/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

azure-pipelines · 2023-04-27T04:34:07Z

Azure Pipelines successfully started running 9 pipeline(s).

azure-pipelines · 2023-04-27T04:34:15Z

Azure Pipelines successfully started running 7 pipeline(s).

snnn · 2023-04-27T04:34:24Z

/azp run Post Merge

azure-pipelines · 2023-04-27T04:34:30Z

No pipelines are associated with this pull request.

snnn · 2023-04-28T14:32:22Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Windows ARM64 QNN CI Pipeline

snnn · 2023-04-28T14:32:31Z

/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

azure-pipelines · 2023-04-28T14:32:58Z

Azure Pipelines successfully started running 7 pipeline(s).

azure-pipelines · 2023-04-28T14:32:59Z

Azure Pipelines successfully started running 9 pipeline(s).

guschmue · 2023-04-27T15:34:22Z

cmake/adjust_global_compile_flags.cmake

+    # Avoid unboundTypeError for WebNN EP since unbound type names are illegal with RTTI disabled
+    # in Embind API, relevant issue: https://github.com/emscripten-core/emscripten/issues/16911
+    if(NOT onnxruntime_USE_WEBNN)
+      add_compile_options("$<$<COMPILE_LANGUAGE:CXX>:-fno-rtti>")


wonder what the size implications for our release builds are

guschmue · 2023-04-27T15:57:18Z

js/web/lib/wasm/session-options.ts

@@ -64,6 +64,29 @@ const setExecutionProviders =
          case 'xnnpack':
            epName = 'XNNPACK';
            break;
+          case 'webnn':
+            epName = 'WEBNN';


I wonder if we should throw if proxy is not set; might save devs some time to debug.

size increase:
ort-wasm-simd.wasm / no webnn: 8870924
ort-wasm-simd.wasm / with webnn: 9535743
~650KB, should be manageable.

I wonder if we should throw if proxy is not set; might save devs some time to debug.

Indeed, I will set proxy to true once webnn backend is used.

fs-eire · 2023-05-02T07:19:23Z

onnxruntime/core/session/provider_registration.cc

@@ -86,8 +86,8 @@ ORT_API_STATUS_IMPL(OrtApis::SessionOptionsAppendExecutionProvider,
 #endif
  } else if (strcmp(provider_name, "WEBNN") == 0) {
 #if defined(USE_WEBNN)
-    std::string deviceType = options->value.config_options.GetConfigOrDefault("deviceType", "2");
-    std::string powerPreference = options->value.config_options.GetConfigOrDefault("powerPreference", "0");
+    std::string deviceType = options->value.config_options.GetConfigOrDefault("deviceType", "cpu");


we have 2 different places in the code to set default value of the configs. this may cause potential inconsistency.

is there a way to put default value to only one place?

The other place is Line 29 in webnn_provider_factory.cc , in case you don't find it.

Good point! These should be duplicated. I will remove one of them.

fdwr · 2023-05-03T21:49:38Z

onnxruntime/core/providers/webnn/builders/impl/concat_op_builder.cc

+    return false;
+
+  const auto input_size = input_shape.size();
+  if (input_size > 4 || input_size == 0) {


🤔 I don't see anything in the spec about concat (https://www.w3.org/TR/webnn/#api-mlgraphbuilder-concat) being limited to 4D. Concat should support an arbitrary number of dimensions because the backend implementation can always flatten/reshape adjacent dimensions. DML supports up to 8D directly, and XNNPack can always just flatten dimensions. I know this because that's what we did for many DML ops, which were once limited to only 4D in the DML API, reshaping adjacent dimensions so that a shape like [2,3,4,5,6] with concat axis = 2 became [6,4,30] with concat axis = 1.

Good catch! Thanks @fdwr! I will remove this restriction.

fdwr · 2023-05-04T06:58:12Z

onnxruntime/core/providers/webnn/builders/op_builder_factory.cc

+  }
+
+  {  // Gemm
+    CreateGemmOpBuilder("Gemm", op_registrations);


CreateGemmOpBuilder("MatMul", op_registrations); too? I see MatMul checked here: https://github.com/microsoft/onnxruntime/pull/15698/files#diff-80ffe78c84984a47483ee44069d532504c82ccc174aaa5a36dfa8900c3530662R40

Oops, I forgot to remove MatMul check, as which hasn't been implemented in Chromium.

Doh. Too bad because I have MatMul implemented in my fork :b. https://github.com/fdwr/chromium-src-webnn-dml/pull/1/files#diff-48bafbfc616dd59a5134cc609ccd6936f00b36e8ad814e950f8f235a9980bf21R2279

Honry · 2023-05-04T08:58:23Z

@fdwr, @fs-eire, @guschmue, thanks for your comments, PTAL again! :)

guschmue · 2023-05-08T15:20:57Z

/azp run Linux CPU CI Pipeline

azure-pipelines · 2023-05-08T15:21:08Z

Azure Pipelines successfully started running 1 pipeline(s).

guschmue · 2023-05-08T19:13:37Z

/azp run Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Windows ARM64 QNN CI Pipeline

azure-pipelines · 2023-05-08T19:14:09Z

Azure Pipelines successfully started running 8 pipeline(s).

guschmue · 2023-05-08T21:36:58Z

/azp run

azure-pipelines · 2023-05-08T21:37:03Z

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

guschmue · 2023-05-08T21:38:19Z

/azp run Windows CPU CI Pipeline

azure-pipelines · 2023-05-08T21:38:27Z

Azure Pipelines successfully started running 1 pipeline(s).

guschmue · 2023-05-08T23:11:43Z

/azp run Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline,onnxruntime-binary-size-checks-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,

azure-pipelines · 2023-05-08T23:12:08Z

Azure Pipelines successfully started running 6 pipeline(s).

fdwr

Although my comments remain, I'm conditionally approving this to unblock the demo and Guenther, so long as they are addressed in a subsequent PR (MatMul and Concat being 4D). Thanks much for this substantial work and contribution.

guschmue · 2023-05-09T04:27:49Z

Awesome PR @Honry !

Honry · 2023-05-09T05:08:19Z

@guschmue, @fdwr, @fs-eire, thanks you so much for your review!

Although my comments remain, I'm conditionally approving this unblock the demo and Guenther, so long as they are addressed in a subsequent PR (MatMul and Concat being 4D).

@fdwr, Concat had been fixed at this commit, and MatMul will not be too late. :)

guschmue · 2023-05-09T15:22:47Z

no worry, can always send a new PR with fixes and new ops.

**Description**: This PR intends to enable WebNN EP in ONNX Runtime Web. It translates the ONNX nodes by [WebNN API](https://webmachinelearning.github.io/webnn/), which is implemented in C++ and uses Emscripten [Embind API](https://emscripten.org/docs/porting/connecting_cpp_and_javascript/embind.html#). Temporarily using preferred layout **NHWC** for WebNN graph partitions since the restriction in WebNN XNNPack backend implementation and the ongoing [discussion](webmachinelearning/webnn#324) in WebNN spec that whether WebNN should support both 'NHWC' and 'NCHW' layouts. No WebNN native EP, only for Web. **Motivation and Context**: Allow ONNXRuntime Web developers to access WebNN API to benefit from hardware acceleration. **WebNN API Implementation Status in Chromium**: - Tracked in Chromium issue: [#1273291](https://bugs.chromium.org/p/chromium/issues/detail?id=1273291) - **CPU device**: based on XNNPack backend, and had been available on Chrome Canary M112 behind "#enable-experimental-web-platform-features" flag for Windows and Linux platforms. Further implementation for more ops is ongoing. - **GPU device**: based on DML, implementation is ongoing. **Open**: - GitHub CI: WebNN currently is only available on Chrome Canary/Dev with XNNPack backend for Linux and Windows. This is an open to reviewers to help identify which GitHub CI should involved the WebNN EP and guide me to enable it. Thanks!

snnn requested review from fs-eire and guschmue and removed request for fs-eire April 26, 2023 15:11

snnn requested a review from fs-eire April 26, 2023 16:40

hanbitmyths requested a review from shalvamist April 26, 2023 17:42

Fixed lint errors

74ee55a

Did npm run format

6db9969

guschmue reviewed Apr 28, 2023

View reviewed changes

fs-eire reviewed May 2, 2023

View reviewed changes

fdwr reviewed May 3, 2023

View reviewed changes

fdwr reviewed May 4, 2023

View reviewed changes

Addressed further comments

4653246

Fixed nit

58a8bec

guschmue approved these changes May 9, 2023

View reviewed changes

guschmue removed the request for review from shalvamist May 9, 2023 03:54

fdwr approved these changes May 9, 2023

View reviewed changes

guschmue merged commit 00b1e79 into microsoft:main May 9, 2023

Support WebNN EP #15698

Support WebNN EP #15698

Conversation

Honry commented Apr 26, 2023

Honry commented Apr 26, 2023

snnn commented Apr 26, 2023

snnn commented Apr 26, 2023

snnn commented Apr 26, 2023

azure-pipelines bot commented Apr 26, 2023

azure-pipelines bot commented Apr 26, 2023

azure-pipelines bot commented Apr 26, 2023

guschmue commented Apr 26, 2023 • edited Loading

guschmue commented Apr 26, 2023

guschmue commented Apr 26, 2023

snnn commented Apr 27, 2023

snnn commented Apr 27, 2023

snnn commented Apr 27, 2023

azure-pipelines bot commented Apr 27, 2023

azure-pipelines bot commented Apr 27, 2023

azure-pipelines bot commented Apr 27, 2023

snnn commented Apr 27, 2023

snnn commented Apr 27, 2023

azure-pipelines bot commented Apr 27, 2023

azure-pipelines bot commented Apr 27, 2023

snnn commented Apr 27, 2023

azure-pipelines bot commented Apr 27, 2023

snnn commented Apr 28, 2023

snnn commented Apr 28, 2023

azure-pipelines bot commented Apr 28, 2023

azure-pipelines bot commented Apr 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guschmue May 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fdwr May 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Honry commented May 4, 2023

guschmue commented May 8, 2023

azure-pipelines bot commented May 8, 2023

guschmue commented May 8, 2023

azure-pipelines bot commented May 8, 2023

guschmue commented May 8, 2023

azure-pipelines bot commented May 8, 2023

guschmue commented May 8, 2023

azure-pipelines bot commented May 8, 2023

guschmue commented May 8, 2023

azure-pipelines bot commented May 8, 2023

fdwr left a comment • edited Loading

Choose a reason for hiding this comment

guschmue commented May 9, 2023

Honry commented May 9, 2023

guschmue commented May 9, 2023

guschmue commented Apr 26, 2023 •

edited

Loading

guschmue May 1, 2023 •

edited

Loading

fdwr May 3, 2023 •

edited

Loading

fdwr left a comment •

edited

Loading