[Example] Yolo12 Detection sample with OpenVINO/XNNPACK backend #10156

daniil-lyakhov · 2025-04-14T21:05:39Z

Summary

Ultralytics Yolo12 Detection sample to expand example models with the detection task:

Export script to get executorch yolo model lowered to OpenVINO/XNNPACK
Demo app which gets a model and a video as an input and outputs an annotated video

Changes in the OpenVINO backend:

OpenVINO repo branch is updated to support yolo12 models (Commit to Openvino master WIP)
Minor fixes in quantizer.py

Test plan

Warning:

OpenCV should be installed in the environment.

OpenVINO:

./.ci/scripts/test_yolo12.sh  -model yolo12n -mode openvino -pt2e_quantize OFF -upload tmp_ov_run -video_path  path/to/mp4/file

XNNPACK:

./.ci/scripts/test_yolo12.sh  -model yolo12n -mode xnnpack -pt2e_quantize OFF -upload tmp_xnnpack_run -video_path  path/to/mp4/file

Issues:

Quantization does not work in both backends, issues WIP. Marked by NonImplemented error by now
OpenVINO is being build twice because it is not present in the executorch-config.cmake file. Perhaps we could add the OpenVINO backend to the config file?

CC: @ynimmaga @suryasidd @alexsu52

pytorch-bot · 2025-04-14T21:05:43Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10156

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 9e1d95f with merge base f0a7d10 ():

NEW FAILURE - The following job has failed:

pull / unittest-editable / linux / linux-job (gh)
backends/xnnpack/test/ops/test_gelu.py::TestGelu::test_fp16_gelu

This comment was automatically generated by Dr. CI and updates every 15 minutes.

GregoryComer · 2025-04-14T21:13:52Z

FYI On the mlock error, you can try switching the load mode in the Module constructor (or data loader) to Mmap (from Mlock). IMO that should be the default.

alexsu52

I think some sanity checking needs to be added to this example.

alexsu52 · 2025-04-15T05:24:25Z

backends/openvino/quantizer/quantizer.py

@@ -88,7 +88,7 @@ def set_ignored_scope(
        names: Optional[List[str]] = None,
        patterns: Optional[List[str]] = None,
        types: Optional[List[str]] = None,
-        subgraphs: Optional[List[Tuple[List[str], List[str]]]] = None,
+        subgraphs: Optional[List[nncf.Subgraph]] = None,


Could you manage subgraphs without changing the function signature?

daniil-lyakhov · 2025-04-15T16:02:41Z

FYI On the mlock error, you can try switching the load mode in the Module constructor (or data loader) to Mmap (from Mlock). IMO that should be the default.

This helped, thanks!

daniil-lyakhov · 2025-04-25T09:14:29Z

@lucylq, @jackzhxng, could you please take a look?

kimishpatel · 2025-06-03T16:43:00Z

.ci/scripts/test_yolo12.sh

@@ -0,0 +1,198 @@
+#!/bin/bash


Does this need to be running in CI?

This is copy-paste from the other similar .sh files like this https://github.com/pytorch/executorch/blob/main/.ci/scripts/test_openvino.sh

No my question is this script is not actually running in ci. If so why is it inside .ci folder

kimishpatel · 2025-06-03T16:44:23Z

backends/openvino/README.md

+git clone https://github.com/daniil-lyakhov/openvino.git
+cd openvino && git checkout dl/executorch/yolo12


is this meant to be left like this?

We are working on a fix in the openvino itegration, I'll update the branch as soon as we will have it merged

kimishpatel · 2025-06-03T16:48:17Z

examples/models/yolo12/main.cpp

@@ -0,0 +1,168 @@
+#include "inference.h"


@larryliu0820 have we setup examples repo already? I am wondeirng if we should move it there.

kimishpatel · 2025-06-03T16:49:31Z

examples/models/yolo12/inference.h

@@ -0,0 +1,151 @@
+#ifndef INFERENCE_H


nit: maybe orgnaize folders into export and inference

kimishpatel

Left some comments. Didnt review in details since this is largely in examples folder. At high level can you include some performance numbers in the summary

kimishpatel · 2025-06-03T17:11:13Z

@guangy10 to see if this can be done also via optimum-et

guangy10 · 2025-06-03T17:58:36Z

@guangy10 to see if this can be done also via optimum-et

HF Transformers seem to have a different one yolos https://github.com/huggingface/transformers/tree/main/src/transformers/models/yolos. Should we try that in optimum-et?

daniil-lyakhov · 2025-06-04T17:40:59Z

Left some comments. Didnt review in details since this is largely in examples folder. At high level can you include some performance numbers in the summary

Please take a look https://github.com/pytorch/executorch/pull/10156/files#diff-a2182fd228e1b70702adceb8eacfcb42462b65dab17a66113a43ba511ee6988aR15-R22

kimishpatel · 2025-06-05T01:12:02Z

WIll take a look. Please do ping in case this slips through. I apologize that this has been waiting for so long

kimishpatel · 2025-06-06T13:52:38Z

examples/models/yolo12/export_and_validate.py

+from torch.ao.quantization.quantize_pt2e import convert_pt2e, prepare_pt2e
+from torch.export.exported_program import ExportedProgram
+from torch.fx.passes.graph_drawer import FxGraphDrawer
+from ultralytics import YOLO


is this different from hf yolo?

Could you please share a link to the HF yolo? I believe that Ultralitics is the primary source of the Yolo12 models

Aah. I see. Yeah while looking for yolo12 + hf I am also coming across mentions of ultralitics.

I was trying to find the source code of it. If you can point me to that, it would be great. I am trying to understand how bounding box related logic is handled in the model and its exportability

The ultralytics Yolo repo is quite complex due to the fact that a single class (YOLO) is handling all available yolo versions + training/evaluation/inference code as well.
The inference pipeline could be found here
https://github.com/ultralytics/ultralytics/blob/main/ultralytics/engine/validator.py#L202-L226
With preprocessing https://github.com/ultralytics/ultralytics/blob/main/ultralytics/models/yolo/detect/val.py#L66-L81
and postprocessing
https://github.com/ultralytics/ultralytics/blob/main/ultralytics/models/yolo/detect/val.py#L113-L133

The example code is based on the YOLOv8-CPP-Inference example (the bounding box logic could be found here https://github.com/ultralytics/ultralytics/blob/main/examples/YOLOv8-CPP-Inference/inference.cpp#L16)

Summarizing my work, I've replaced the ONNX backend with the ExecuTorch and made sure everything is working

kimishpatel · 2025-06-06T13:57:10Z

backends/openvino/README.md

@@ -45,8 +45,9 @@ Before you begin, ensure you have openvino installed and configured on your syst
 ### Build OpenVINO from Source

 ```bash
-git clone https://github.com/openvinotoolkit/openvino.git
-cd openvino && git checkout releases/2025/1
+git clone https://github.com/daniil-lyakhov/openvino.git


Lets wait till the fix is landed? I prefer that we dont have to do this

Fix is finally landed on the master branch! The PR is updated

kimishpatel

lets wait till the fix is landed so that we dont have to patch to your personal repo + branch

daniil-lyakhov · 2025-06-25T13:04:37Z

lets wait till the fix is landed so that we dont have to patch to your personal repo + branch

The fix is finally merged to the openvino master! The README and the test script are updated accordingly

kimishpatel · 2025-06-25T23:26:15Z

lets wait till the fix is landed so that we dont have to patch to your personal repo + branch

The fix is finally merged to the openvino master! The README and the test script are updated accordingly

Thank you. Accepted. Will wait for rest of the CI and merge

kimishpatel · 2025-06-26T23:11:29Z

Just realized, i didnt approve the workflows. Just did

daniil-lyakhov · 2025-06-27T10:47:25Z

Linters and some other jobs were failed most likely due to the old base commit. Rebased the PR and fixed the linter

kimishpatel · 2025-06-27T13:39:57Z

kicked off the run

kimishpatel · 2025-06-28T03:18:32Z

some tests are failing. not sure if they are realted

daniil-lyakhov · 2025-06-30T09:27:09Z

some tests are failing. not sure if they are realted

Looks like some sporicidal problems with the XNNPACKQuantizer

FAILED backends/xnnpack/test/ops/test_exp.py::TestExp::test_fp16_exp - AssertionError: Output 0 does not match reference output.
FAILED backends/xnnpack/test/ops/test_conv1d.py::TestConv1d::test_qs8_conv1d_batchnorm_seq - AssertionError: Output 0 does not match reference output.

and a problem with access to a json file for the llama tests on a mac os

Starting to run llama runner at 23:25:16
+ cmake-out/examples/models/llama/llama_main --model_path=tinyllama_coreml_fp32.pte --tokenizer_path=tokenizer.bin --prompt=Once --temperature=0 --seq_len=10 --warmup=1
I tokenizers:regex.cpp:27] Registering override fallback regex
Error parsing json file: [json.exception.parse_error.101] parse error at line 1, column 1: attempting to parse an empty input; check that your input string or stream contains the expected JSON
invalid charcouldn't load tokenizer.bin. 
Error message: 
could not parse ModelProto from tokenizer.bin
It is likely that the tokenizer artifact is broken or of a different format.++ date +%H:%M:%S

Most likely this commits will fix the problems #12096 #12089. Merging the main branch to my PR

kimishpatel · 2025-07-01T02:29:53Z

Unrelated failures. merging

…nd (#10156)" This reverts commit 00df64c.

@daniil-lyakhov

…nd" (#12136) Reverts #10156 @daniil-lyakhov I'm reverting this PR because the gif file is large. Can you please remove the gif file and re-land?

@ynimmaga

…rch#10156) ### Summary Ultralytics Yolo12 Detection sample to expand example models with the detection task: 1) Export script to get executorch yolo model lowered to OpenVINO/XNNPACK 2) Demo app which gets a model and a video as an input and outputs an annotated video Changes in the OpenVINO backend: * OpenVINO repo branch is updated to support yolo12 models (Commit to Openvino master WIP) * Minor fixes in quantizer.py ### Test plan #### Warning: OpenCV should be installed in the environment. OpenVINO: ``` ./.ci/scripts/test_yolo12.sh -model yolo12n -mode openvino -pt2e_quantize OFF -upload tmp_ov_run -video_path path/to/mp4/file ``` XNNPACK: ``` ./.ci/scripts/test_yolo12.sh -model yolo12n -mode xnnpack -pt2e_quantize OFF -upload tmp_xnnpack_run -video_path path/to/mp4/file ``` ### Issues: * Quantization does not work in both backends, issues WIP. Marked by NonImplemented error by now * OpenVINO is being build twice because it is not present in the [executorch-config.cmake](https://github.com/pytorch/executorch/blob/main/tools/cmake/executorch-config.cmake) file. Perhaps we could add the OpenVINO backend to the config file? CC: @ynimmaga @suryasidd @alexsu52

@kimishpatel

### Summary Follow up #12136 Revert of the revert #10156 The big gif is removed from the PR CC: @kimishpatel

@ynimmaga

…rch#10156) ### Summary Ultralytics Yolo12 Detection sample to expand example models with the detection task: 1) Export script to get executorch yolo model lowered to OpenVINO/XNNPACK 2) Demo app which gets a model and a video as an input and outputs an annotated video Changes in the OpenVINO backend: * OpenVINO repo branch is updated to support yolo12 models (Commit to Openvino master WIP) * Minor fixes in quantizer.py ### Test plan #### Warning: OpenCV should be installed in the environment. OpenVINO: ``` ./.ci/scripts/test_yolo12.sh -model yolo12n -mode openvino -pt2e_quantize OFF -upload tmp_ov_run -video_path path/to/mp4/file ``` XNNPACK: ``` ./.ci/scripts/test_yolo12.sh -model yolo12n -mode xnnpack -pt2e_quantize OFF -upload tmp_xnnpack_run -video_path path/to/mp4/file ``` ### Issues: * Quantization does not work in both backends, issues WIP. Marked by NonImplemented error by now * OpenVINO is being build twice because it is not present in the [executorch-config.cmake](https://github.com/pytorch/executorch/blob/main/tools/cmake/executorch-config.cmake) file. Perhaps we could add the OpenVINO backend to the config file? CC: @ynimmaga @suryasidd @alexsu52

@daniil-lyakhov

…nd" (pytorch#12136) Reverts pytorch#10156 @daniil-lyakhov I'm reverting this PR because the gif file is large. Can you please remove the gif file and re-land?

@kimishpatel

### Summary Follow up pytorch#12136 Revert of the revert pytorch#10156 The big gif is removed from the PR CC: @kimishpatel

daniil-lyakhov requested review from lucylq and jackzhxng as code owners April 14, 2025 21:05

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 14, 2025

ynimmaga added partner: intel For backend delegation, kernels, demo, etc. from the 3rd-party partner, Intel release notes: openvino OpenVino backend related issues, features, bugs etc. labels Apr 14, 2025

alexsu52 reviewed Apr 15, 2025

View reviewed changes

daniil-lyakhov force-pushed the dl/yolo12_main branch 2 times, most recently from 4c302d6 to 60d9cfb Compare April 22, 2025 13:23

daniil-lyakhov requested a review from alexsu52 April 23, 2025 09:15

kimishpatel reviewed Jun 3, 2025

View reviewed changes

kimishpatel requested changes Jun 3, 2025

View reviewed changes

Yolo12 OpenVINO/XNNPACK sample

71f9551

daniil-lyakhov force-pushed the dl/yolo12_main branch from b3e69c5 to 794cf64 Compare June 4, 2025 17:39

daniil-lyakhov requested review from jathu, larryliu0820 and kirklandsign as code owners June 4, 2025 17:39

daniil-lyakhov requested a review from kimishpatel June 4, 2025 17:41

Quantization support

546c939

daniil-lyakhov force-pushed the dl/yolo12_main branch from 794cf64 to 546c939 Compare June 4, 2025 17:43

kimishpatel reviewed Jun 6, 2025

View reviewed changes

daniil-lyakhov mentioned this pull request Jun 10, 2025

[XNNPACK] Yolo12 model quantization #11523

Open

Readme is updated/ minor refactoring

6a855db

daniil-lyakhov requested a review from kimishpatel June 25, 2025 13:03

daniil-lyakhov branch is removed from scripts and README / minor typo

2b33a5e

kimishpatel approved these changes Jun 25, 2025

View reviewed changes

torch.ao -> torchao

d5ac01f

Merge remote-tracking branch 'origin/main' into dl/yolo12_main

8e1e9bf

Minor

4f3a30d

Merge branch 'main' into dl/yolo12_main

9e1d95f

kimishpatel merged commit 00df64c into pytorch:main Jul 1, 2025
189 of 190 checks passed

metascroy added a commit that referenced this pull request Jul 1, 2025

Revert "[Example] Yolo12 Detection sample with OpenVINO/XNNPACK backe…

166b158

…nd (#10156)" This reverts commit 00df64c.

metascroy mentioned this pull request Jul 1, 2025

Revert "[Example] Yolo12 Detection sample with OpenVINO/XNNPACK backend" #12136

Merged

daniil-lyakhov mentioned this pull request Jul 2, 2025

[Yolo12] Revert of the revert of the Yolo12 Sample #12163

Merged

kimishpatel pushed a commit that referenced this pull request Jul 8, 2025

[Yolo12] Revert of the revert of the Yolo12 Sample (#12163)

1decf7a

### Summary Follow up #12136 Revert of the revert #10156 The big gif is removed from the PR CC: @kimishpatel

		git clone https://github.com/daniil-lyakhov/openvino.git
		cd openvino && git checkout dl/executorch/yolo12

[Example] Yolo12 Detection sample with OpenVINO/XNNPACK backend #10156

[Example] Yolo12 Detection sample with OpenVINO/XNNPACK backend #10156

Uh oh!

Conversation

daniil-lyakhov commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Warning:

Issues:

Uh oh!

pytorch-bot bot commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10156

❌ 1 New Failure

Uh oh!

GregoryComer commented Apr 14, 2025

Uh oh!

alexsu52 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniil-lyakhov commented Apr 15, 2025

Uh oh!

daniil-lyakhov commented Apr 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kimishpatel left a comment

Choose a reason for hiding this comment

Uh oh!

kimishpatel commented Jun 3, 2025

Uh oh!

guangy10 commented Jun 3, 2025

Uh oh!

daniil-lyakhov commented Jun 4, 2025

Uh oh!

kimishpatel commented Jun 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kimishpatel left a comment

Choose a reason for hiding this comment

Uh oh!

daniil-lyakhov commented Jun 25, 2025

Uh oh!

kimishpatel commented Jun 25, 2025

Uh oh!

kimishpatel commented Jun 26, 2025

Uh oh!

daniil-lyakhov commented Jun 27, 2025

Uh oh!

daniil-lyakhov commented Apr 14, 2025 •

edited

Loading

pytorch-bot bot commented Apr 14, 2025 •

edited

Loading

daniil-lyakhov commented Jun 30, 2025 •

edited

Loading