NXP Backend: Add eIQ Neutron Backend #10196

robert-kalmar · 2025-04-15T13:07:15Z

Summary

Initial implementation for the NXP eIQ Neutron Backend for Neutron-N3-64 (i.MX RT700)

Test plan

Functionality tested by python unit tests:

pytest -c /dev/null/ backend/nxp/tests

cc @digantdesai @JakeStevens , @JakeStevens , @skywall

pytorch-bot · 2025-04-15T13:07:19Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10196

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit b102be2 with merge base 4559a61 ():

NEW FAILURE - The following job has failed:

Lint / lintrunner / linux-job (gh)
>>> Lint for setup.py:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

robert-kalmar · 2025-04-15T13:08:30Z

@pytorchbot label "module: nxp" , label "release notes: nxp"

pytorch-bot · 2025-04-15T13:08:34Z

Didn't find following labels among repository labels: ,,label

robert-kalmar · 2025-04-15T13:25:17Z

backends/nxp/backend/neutron_converter_manager.py

+    def convert(self, tflite_model: bytes, target: str, neutron_converter_flavor: str) -> bytes:
+        # Neutron converter crashes if we provide invalid target -> verify.
+        if target not in self._supported_target_names:
+            raise RuntimeError(f"Target '{target}' is not supported by NeutronConverterManager.")


@digantdesai it requires to install the Neutron Converter from eiq.nxp.com/repository:

pip install --index-url https://eiq.nxp.com/repository neutron_converter_SDK_25_03

do you prefer to add requirements.txt into the /backends/nxp directory or rather follow Arm's approach with setup.sh script?

I guess we can start with setup.sh, I need to discuss with the team if we want to do optional pip installs or setup.sh is the right approach

digantdesai · 2025-04-15T15:14:47Z

wow this is a huge PR! I will try to go through this bit by bit and leave comments, expect a few days before we can finish reviewing this. Thanks though :)

install_executorch.py

robert-kalmar · 2025-04-17T14:53:05Z

wow this is a huge PR! I will try to go through this bit by bit and leave comments, expect a few days before we can finish reviewing this. Thanks though :)

Unfortunately, it was hard to break it down as most of it is the infrastructure for conversion from Edge Dialect to format suitable for Neutron Converter (LiteRT/tflite flatbuffer).
But let me give you some insight into the code, what hopefully help you with review:

the backends/nxp/backend/neutron_partitioner.py and backends/nxp/backend/nxp_backend.py are the main files for the eIQ Neutron Backend
the backends/nxp/backend/neutron_node_extraction.py contains logic to extract the microcode for the Neutron NPU from the Neutron Converter invocation.
the backends/nxp/backend/ir/lib/tflite is automatically generated from LiteRT flatbuffers schema
the backends/nxp/backend/ir/converter/builder is the LiteRT flatbuffer generation logic
the backend/nxp/backedns/ir/converter/node_conversion/ contains the conversion logic from Edge Dialect to LiteRT

JakeStevens · 2025-04-22T13:30:33Z

@robert-kalmar please make sure to run the linter and re-submit

JakeStevens · 2025-04-22T20:22:44Z

@robert-kalmar please make sure to run the linter and re-submit

Still failing after latest push.

backends/nxp/backend/node_format_inference.py

JakeStevens · 2025-04-15T13:34:43Z

backends/nxp/backend/node_format_inference.py

@@ -0,0 +1,250 @@
+# Copyright 2024 NXP


@digantdesai is this functionality not provided elsewhere in ET already?

JakeStevens · 2025-04-15T14:14:09Z

backends/nxp/tests/test_node_format_inference.py

+
+    # We need to create custom model verifier with max_pool2d added as exception.
+    # Otherwise, we get violation that this op is not part of ATen Core ops.
+    edge_program._verifiers = [EXIREdgeDialectVerifier(


Interesting. For my own understanding, why is this the case? Seems like max_pool is in aten from the path for the op torch.ops.aten

JakeStevens · 2025-04-22T13:35:13Z

backends/nxp/neutron_node_extraction.py

+    model = Model.GetRootAs(flatbuffer, 0)
+    assert model.SubgraphsLength() == 1, f'The model has `{model.SubgraphsLength()}` SubGraphs instead of `1`.'
+
+    sub_graph = model.Subgraphs(0)


What happens with graph breaks? Can there only be one subgraph for Neutron?

at least add an assert?

Here is the workflow:
The Neutron Converter currently supports tflite format as input and returns the tflite as output. That is the returned tflite flatbuffers in general contains combination of Neutron Nodes (which are parts of the original compute graph to be computed on Neutron) and leftover tflite ops, which will be computed on CPU.

In case of ExecuTorch's NeutronBackend, we obviously do not have any TFLite runtime, so the partitioning logic is in the NeutronPartitioner. The Neutron Partitioner already identifies subgraphs that are supported on Neutron, and so the NeutronBackend invoking the Neutron Converter shall generate a microcode for whole such a subgraph. That is no tflite ops shall stay in the tflite graph passed to the Neutron Converter. If that happens, that is a RuntimeError, as we do not convert the leftovers tflite ops back to executorch ops. The Partitioner shall ensure that not happens.
In short - the return value from Neutron converter shall contain exactly 1 op/node, which is to be the "NeutronNode". This is checked on #44 and #47 and the node to be NeutronNode on line #76.

There can be multiple subraphs for Neutron, but this is controlled by the ExecuTorch, which for every identified partition creates a subraph - ExportedProgram, calls the NeutronBackend::preprocess() function, which converts this subgraph and respond with the payload (microcode) for Neutron NPU. We convert 1 subgraph at a time.

digantdesai

part 1 / n

digantdesai · 2025-04-17T15:17:34Z

setup.py

@@ -144,6 +145,10 @@ def openvino(cls) -> bool:
    def xnnpack(cls) -> bool:
        return cls._is_cmake_arg_enabled("EXECUTORCH_BUILD_XNNPACK", default=True)

+    @classmethod
+    def neutron(cls) -> bool:
+        return cls._is_cmake_arg_enabled("EXECUTORCH_BUILD_NEUTRON", default=True)


Suggested change

return cls._is_cmake_arg_enabled("EXECUTORCH_BUILD_NEUTRON", default=True)

return cls._is_cmake_arg_enabled("EXECUTORCH_BUILD_NEUTRON", default=False)

We can change this in the future, let's start conservatively to no disrupt existing workflows.

digantdesai · 2025-04-17T15:27:52Z

install_executorch.py

@@ -53,7 +53,7 @@ def clean():


 # Please keep this insync with `ShouldBuild.pybindings` in setup.py.
-VALID_PYBINDS = ["coreml", "mps", "xnnpack", "training", "openvino"]
+VALID_PYBINDS = ["coreml", "mps", "xnnpack", "neutron", "training", "openvino"]


This broadly implies we can run the .PTE with neutron delegates in it on the dev machine like x86.

digantdesai · 2025-04-17T15:33:01Z

backends/nxp/nxp_backend.py

@@ -0,0 +1,297 @@
+# Copyright 2024 NXP


filename nit
s/nxp_backend.py/neutron_backend.py

If don't mind, let do the file renaming as we push all our changes from internal repository.

digantdesai · 2025-04-23T19:55:09Z

backends/nxp/nxp_backend.py

+    extract_artifacts_from_neutron_node,
+    NeutronNodeArtifacts,
+)
+from executorch.backends.xnnpack._passes import RemoveGetItemPass, XNNPACKPassManager


Fork the pass manager?

Move the RemoveGetItemPass to backends/transforms?

✅

Introduced NeutronPassManager.

The RemoveGetItemPass moved to backend/transform in separate commit for better visibility

@digantdesai Is there some specific reason PassManager is built on top of GraphModule and not ExportedProgram? I see basically every backend has some custom managers that accept ExportedProgram as input type and also produce ExportedProgram as an output of run/__call__ function. I guess it is done that way because wrapping of modified graph_module back into ExportedProgram seems to be reponsibility of PassManager itself and not PassManager call site.

PassManager is FX concept, while ExportedProgram is an export concept. FX has more users than export. The ExportedProgram deals with GraphSignature etc. which is not in the scope for FX. So XNNPACKPassmanager is that wrapper. EdgeProgramManager.transform is kind of similar which is also built on top of _transform.

digantdesai · 2025-04-23T19:56:37Z

backends/nxp/nxp_backend.py

+        Args:
+            config: Neutron accelerator configuration, e.g. "imxrt700"
+            extra_flags: Extra flags for the Neutron compiler
+            operators_not_to_delegate: List of operators that should not be delegated


incomplete args docs

digantdesai · 2025-04-24T03:10:02Z

backends/nxp/neutron_partitioner.py

+        qdq_clusterer.tag_qdq_clusters(nodes)
+
+        graph_module.recompile()
+        target = self.delegation_spec[1][2].value


can we use CompileSpec.key : str to make this more readable?

digantdesai · 2025-04-24T03:12:20Z

backends/nxp/neutron_partitioner.py

+        graph_module = exported_program.graph_module
+        nodes = list(graph_module.graph.nodes)
+
+        qdq_clusterer = QDQClusterRecognizer()


nit

Suggested change

qdq_clusterer = QDQClusterRecognizer()

qdq_cluster_recognizer = QDQClusterRecognizer()

digantdesai · 2025-04-24T03:22:15Z

backends/nxp/neutron_partitioner.py

+    QUANTIZE_OPERATORS = [
+        exir_ops.edge.quantized_decomposed.quantize_per_channel.default,
+        exir_ops.edge.quantized_decomposed.quantize_per_tensor.default,
+        exir_ops.edge.quantized_decomposed.quantize_per_tensor.tensor,


This is for dynamic (at runtime) quantization of the activation for conv, linear etc. Do we support that?

✅
Ouh. we do not.
For our education, how this is defined? I see ops definition in kernels/quantized/quantized.yaml There are out variants and tensor variants. In executorch.exir.dialects._ops there are variants default and tensor. Is there some definition what is the difference btw. "out"/"default" and "tensor". For quantize ops is starts make sense - static quantization vs. dynamic quantization. But any general rule of thumb?

I guess the best way to check for the dynamic quant is presence of the choose_qparam node to dynamically populate scale and zp. I am not sure if we are being consistent with .tensor variant TBH.

digantdesai · 2025-04-24T03:24:00Z

backends/nxp/neutron_node_extraction.py

+    model = Model.GetRootAs(flatbuffer, 0)
+    assert model.SubgraphsLength() == 1, f'The model has `{model.SubgraphsLength()}` SubGraphs instead of `1`.'
+
+    sub_graph = model.Subgraphs(0)


at least add an assert?

digantdesai · 2025-04-24T03:28:45Z

backends/nxp/neutron_node_extraction.py

+            opcode.BuiltinCode() == BuiltinOperator.CUSTOM
+            and opcode.CustomCode() == b"NeutronGraph"
+        ):
+            # Found the NeutronNode.


I guess this the only thing in the tflite flatbuffer of relevance?

Co-authored-by: Lukas Sztefek <[email protected]> Co-authored-by: Martin Pavella <[email protected]> Co-authored-by: Jiri Ocenasek <[email protected]> Co-authored-by: Roman Janik <[email protected]> Co-authored-by: Simon Strycek <[email protected]>

Multiple backends are using this pass - Arm, Neutron, XNNPACK.

robert-kalmar · 2025-04-25T11:09:27Z

Linting errors should be fixed now.

JakeStevens · 2025-04-25T17:58:45Z

Think you need just one more run of the linter, misplaced import

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 15, 2025

pytorch-bot bot added module: nxp Issues related to NXP Neutron NPU delegation and code under backends/nxp/ release notes: nxp Changes to the NXP Neutron backend delegate labels Apr 15, 2025

robert-kalmar commented Apr 15, 2025

View reviewed changes

skywall reviewed Apr 17, 2025

View reviewed changes

install_executorch.py Outdated Show resolved Hide resolved

robert-kalmar force-pushed the upstream/release-mcux-25.03-full/neutron-backend_2_upstream branch from d58bec5 to 32253f6 Compare April 22, 2025 13:48

JakeStevens reviewed Apr 23, 2025

View reviewed changes

digantdesai reviewed Apr 24, 2025

View reviewed changes

robert-kalmar and others added 2 commits April 25, 2025 12:53

NXP Backend: Add eIQ NeutronBackend

d403958

Co-authored-by: Lukas Sztefek <[email protected]> Co-authored-by: Martin Pavella <[email protected]> Co-authored-by: Jiri Ocenasek <[email protected]> Co-authored-by: Roman Janik <[email protected]> Co-authored-by: Simon Strycek <[email protected]>

Move RemoveGetItemPass (remove_getitem_op.py) to backends.transform

b102be2

Multiple backends are using this pass - Arm, Neutron, XNNPACK.

robert-kalmar force-pushed the upstream/release-mcux-25.03-full/neutron-backend_2_upstream branch from 32253f6 to b102be2 Compare April 25, 2025 10:55

robert-kalmar requested review from mcr229 and kimishpatel as code owners April 25, 2025 10:55

	return cls._is_cmake_arg_enabled("EXECUTORCH_BUILD_NEUTRON", default=True)
	return cls._is_cmake_arg_enabled("EXECUTORCH_BUILD_NEUTRON", default=False)

	qdq_clusterer = QDQClusterRecognizer()
	qdq_cluster_recognizer = QDQClusterRecognizer()

NXP Backend: Add eIQ Neutron Backend #10196

Are you sure you want to change the base?

NXP Backend: Add eIQ Neutron Backend #10196

Conversation

robert-kalmar commented Apr 15, 2025 • edited by pytorch-bot bot Loading

Summary

Test plan

pytorch-bot bot commented Apr 15, 2025 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10196

❌ 1 New Failure

robert-kalmar commented Apr 15, 2025

pytorch-bot bot commented Apr 15, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

digantdesai commented Apr 15, 2025

robert-kalmar commented Apr 17, 2025

JakeStevens commented Apr 22, 2025

JakeStevens commented Apr 22, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robert-kalmar Apr 24, 2025 • edited Loading

Choose a reason for hiding this comment

digantdesai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robert-kalmar commented Apr 25, 2025 • edited Loading

JakeStevens commented Apr 25, 2025

robert-kalmar commented Apr 15, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Apr 15, 2025 •

edited

Loading

robert-kalmar Apr 24, 2025 •

edited

Loading

robert-kalmar commented Apr 25, 2025 •

edited

Loading