When the batch-size is not 128, 'Failed QNN validation for /layers.0/attn/core/Transpose with error' will occur.

[Ulthunzy]

When I ran the following code in the terminal:

python converter.py                                 \
    --model-folder ./SmallThinker-3B-Preview            \
    --model-name smallthinker_3b                        \
    --system-prompt-file prompt/system_prompt_qwen.txt    \
    --prompt-file prompt/lab_intro_qwen.md                \
    --batch-sizes 64                             \
    --artifact-name smallthinker_3b                     \
    --n-model-chunk 4                               \
    --output-folder ./smallthinker_3b-batch1-QNN               \
    --build-folder ./smallthinker_3b-batch1-QNN-tmp            \
    --silent \
    --clear-build-files \
    --soc 8650

The terminal had printed error as following:

AssertionError
> python /PowerServe/tools/qnn_converter/build_shared_object.py
> /qnn_sdk/src/backends/qnn/2.28.2.241116/bin/x86_64-linux-clang/qnn-onnx-converter
> /qnn_sdk/src/backends/qnn/2.28.2.241116/bin/x86_64-linux-clang/qnn-model-lib-generator
> python /PowerServe/tools/qnn_converter/build_shared_object.py
> /qnn_sdk/src/backends/qnn/2.28.2.241116/bin/x86_64-linux-clang/qnn-onnx-converter
Traceback (most recent call last):
  File "/PowerServe/tools/qnn_converter/build_shared_object.py", line 123, in <module>
    convert_model()
  File "/PowerServe/tools/qnn_converter/build_shared_object.py", line 99, in convert_model
    run(cmd_args)
  File "/PowerServe/tools/qnn_converter/build_shared_object.py", line 47, in run
    assert ret == 0
AssertionError
> python /PowerServe/tools/qnn_converter/build_shared_object.py
> /qnn_sdk/src/backends/qnn/2.28.2.241116/bin/x86_64-linux-clang/qnn-onnx-converter
Traceback (most recent call last):
  File "/PowerServe/tools/qnn_converter/build_shared_object.py", line 123, in <module>
    convert_model()
  File "/PowerServe/tools/qnn_converter/build_shared_object.py", line 99, in convert_model
    run(cmd_args)
  File "/PowerServe/tools/qnn_converter/build_shared_object.py", line 47, in run
    assert ret == 0
AssertionError
Traceback (most recent call last):
  File "/PowerServe/tools/qnn_converter/build_shared_object.py", line 123, in <module>
    convert_model()
  File "/PowerServe/tools/qnn_converter/build_shared_object.py", line 99, in convert_model
    run(cmd_args)
  File "/PowerServe/tools/qnn_converter/build_shared_object.py", line 47, in run
    assert ret == 0
AssertionError
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/anaconda3/envs/powerserver/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/anaconda3/envs/powerserver/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/PowerServe/tools/qnn_converter/build_all_layers.py", line 67, in build_shared_object
    run(command)
  File "/PowerServe/tools/qnn_converter/build_all_layers.py", line 29, in run
    assert ret == 0
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/PowerServe/tools/qnn_converter/build_all_layers.py", line 104, in <module>
    pool.map(build_shared_object, chunk_ids)
  File "/anaconda3/envs/powerserver/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/anaconda3/envs/powerserver/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
AssertionError
Traceback (most recent call last):
  File "/PowerServe/tools/qnn_converter/converter.py", line 138, in <module>
    main(args)
  File "/PowerServe/tools/qnn_converter/converter.py", line 89, in main
    run_shell_command(generate_so_command)
  File "/PowerServe/tools/qnn_converter/converter.py", line 16, in run_shell_command
    assert ret == 0
AssertionError

The file "build_so.log" saved the error as following:

Traceback (most recent call last):
  File "/qnn_sdk/src/backends/qnn/2.28.2.241116/bin/x86_64-linux-clang/qnn-onnx-converter", line 78, in main
    backend.save(optimized_graph)
  File "/qnn_sdk/src/backends/qnn/2.28.2.241116/lib/python/qti/aisw/converters/backend/ir_to_qnn.py", line 533, in save
    ir_graph = self.get_ir_graph(graph)
  File "/qnn_sdk/src/backends/qnn/2.28.2.241116/lib/python/qti/aisw/converters/qnn_backend/qnn_backend_base.py", line 555, in get_ir_graph
    raise e
  File "/qnn_sdk/src/backends/qnn/2.28.2.241116/lib/python/qti/aisw/converters/qnn_backend/qnn_backend_base.py", line 546, in get_ir_graph
    QnnTranslations.apply_method_to_all_ops(BackendTranslationBase.ADD_OP_TO_BACKEND, graph, self)
  File "/qnn_sdk/src/backends/qnn/2.28.2.241116/lib/python/qti/aisw/converters/common/converter_ir/translation.py", line 71, in apply_method_to_all_ops
    self.apply_method_to_op(node.op.type, method_name, node, graph, *args, **kwargs)
  File "/qnn_sdk/src/backends/qnn/2.28.2.241116/lib/python/qti/aisw/converters/common/converter_ir/translation.py", line 51, in apply_method_to_op
    return translation.apply_method(method_name, *args, **kwargs)
  File "/qnn_sdk/src/backends/qnn/2.28.2.241116/lib/python/qti/aisw/converters/common/converter_ir/translation.py", line 18, in apply_method
    return self.indexed_methods[method_name](*args, **kwargs)
  File "/qnn_sdk/src/backends/qnn/2.28.2.241116/lib/python/qti/aisw/converters/qnn_backend/qnn_translations.py", line 2950, in add_op_to_backend
    backend.add_node(node.op.name, ir_graph.QNN_OP_TRANSPOSE,
  File "/qnn_sdk/src/backends/qnn/2.28.2.241116/lib/python/qti/aisw/converters/backend/ir_to_qnn.py", line 249, in add_node
    if not self.model.add_node(node_name, node_type, node_package_name, tensor_params, scalar_params,
RuntimeError: validateQnnOpConfig: Failed QNN validation for /layers.0/attn/core/Transpose with error
Validating Output[0] of ID 290.
Validating tensor 286 and 290 have the same Datatype.
Validating tensor 286 and 290 have the same Rank.
Incorrect output dimension at index 1. Expected 32 but got 128.
Op specific validation failed.

How to solve this problem？？？？？

[SorryMaker2022]

@Ulthunzy All the graph sizes are predefined here https://github.com/powerserve-project/PowerServe/blob/main/tools/qnn_converter/graph_params.py

We did not offer choice of batch 64, I guess you have modified the file above and added it. If you didn't do so, maybe the argparse went wrong and it accidentally generated a wrong setting, you can add a set of settings and try again. If it doesn't work, can you post more info by disabling --silent to help address the problem?

I added the silent option in the last version to reduce redundant print for experienced users, but here we're debugging :D

My suggestions:

First check the graph_params.py and add your settings
Disable --silent option and check what graph_names was actrally processed
If the problem holds, post it here