WebNN Samples Test Framework

An automation test framework for testing W3C WebNN Samples and Microsoft WebNN Demos.

Install

$ npm install
$ cp env.example.json env.json

Set env.json

• env: 'debug' for local development. 'production' for production model and the test result will be sent via email.
• hostname: Identify yourself.
• proxy: Set the proxy server, which is usually needed.
• emailService: Edit the email config according to your server.

Set config.json

Generate config

The config.json file can be automatically generated using the command:

npm run generate-config

Run with -h to see the help message.

$ npm run generate-config -- -h
Usage: npm run generate-config -- [options]

Generate config for webnn sample tests

Options:
  -b, --browser <browser>     The browser to use (choices: "chrome_canary", "chrome_dev", "chrome_beta", "chrome_stable", "edge_canary", "edge_dev", "edge_beta", "edge_stable", default: "chrome_canary")
  -d, --devices <devices...>  The devices to use (choices: "cpu", "gpu", "npu", default: ["cpu","gpu","npu"])
  -e, --backend <backend>     The backend to use (choices: "ort", "tflite", default: "ort")
  -o, --output <path>         The output config file path (default: "config.json")
  -h, --help                  display help for command

Change config

You can also manually adjust the parameters in config.json to fit your specific needs.

• browser: Choose the browser to run the test. Options include chrome_canary, chrome_dev, chrome_beta, chrome_stable, edge_canary (except on Linux), edge_dev, edge_beta, edge_stable.
• browserUserData: Browser user data includes browser cache. When webnn flag is enabled and browserUserData is used, script won't test wasm or webgl backends.
• browserUserDataPath: The browser User Data folder path supports manual setting by users and is empty by default.
• browserAppPath: The browser Application folder path supports manual setting by users and is empty by default.
• headless: Display chrome UI.
• timeout: Browser page waiting time (ms), if the test device performance is poor, please set a larger value.
• imageCompareThreshold: Images matching threshold, ranges from 0 to 1. Smaller values make the comparison more sensitive. 0.1 by default.
• samplesBasicUrl: URL of WebNN samples.
• samplesUrl: Each sample URL: samplesBasicUrl + samplesUrl.
• samples: WebNN samples config.
• developerPreview: WebNN developer preview demo config.

Start

Test all samples

$ npm test

Test a specific single sample

Get all supported samples from help message

$ npm test -- -f
Using config file: config.json
Available filters:
┌─────────┬──────────────────────────────────────────────────────────────────────────┐
│ (index) │ Values                                                                   │
├─────────┼──────────────────────────────────────────────────────────────────────────┤
│ 0       │ 'samples-image-classification-cpu-fp32-mobileNetV2'            │
| ...     | ...                                                                      │
└─────────┴──────────────────────────────────────────────────────────────────────────┘

Specify the sample with -f or --filter option. Multiple samples can be specified by separating them with spaces.

$ npm test -- -f samples-image-classification-cpu-fp32-mobileNetV2

Support

Dependency

Node

NOTE: The command npm run [command] -- [options | args] may not work correctly in PowerShell on Windows with certain Node.js versions. It has been tested successfully on Node.js 22.11.0, but may fail on later versions until fixing issue 7375. If you encounter issues, try running it in a different terminal, such as Git Bash.

Platform

Windows / Linux / MacOS

Browser

chrome_canary / chrome_dev / chrome_beta / chrome_stable / edge_canary / edge_dev / edge_beta / edge_stable

• edge_canary is not available on Linux.