Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

UINT8-to-FLOAT cast after transpose breaks the graph. #3985

Open
srasmussenvl opened this issue Jul 5, 2024 · 10 comments
Open

UINT8-to-FLOAT cast after transpose breaks the graph. #3985

srasmussenvl opened this issue Jul 5, 2024 · 10 comments
Labels
Feature Request Request for new functionality triaged Issue has been triaged by maintainers

Comments

@srasmussenvl
Copy link

srasmussenvl commented Jul 5, 2024 •
edited
Loading

Description

Casting an UINT8 to FLOAT16/32 after a transpose operation breaks the graph (network.num_outputs = 0).
Casting before the transpose works fine.

Environment

TensorRT Version: 10.1.0

NVIDIA GPU: A100-SXM4-80GB

NVIDIA Driver Version: 550.90.07

CUDA Version: 12.5

CUDNN Version: 9.1.0

Operating System: Ubuntu 22.04.4 LTS

Python Version: Python 3.10.12

PyTorch Version: 2.3.1+cu121

ONNX version: 1.16.1

Container: nvcr.io/nvidia/tensorrt:24.06-py3

Steps To Reproduce

Minimal example

import tensorrt as trt
import torch as th

class TransposeCast(th.nn.Module):
    def forward(self, x):
        x = x.permute(1,0)
        x = x.float()
        return x

class CastTranspose(th.nn.Module):
    def forward(self, x):
        x = x.float()
        x = x.permute(1,0)
        return x

TC_model = TransposeCast()
CT_model = CastTranspose()

data = th.zeros((1,1), dtype=th.uint8)

th.onnx.export(TC_model, data, "TC.onnx", opset_version=17)
th.onnx.export(CT_model, data, "CT.onnx", opset_version=17)

logger = trt.Logger()
#logger.min_severity = trt.Logger.Severity.VERBOSE
builder = trt.Builder(logger)
for file in ("TC.onnx", "CT.onnx"):
    network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
    parser = trt.OnnxParser(network, logger)
    with open(file, 'rb') as fd:
        parser.parse(fd.read())
    print(f"{file}: {network.num_outputs=}")

Output

TC.onnx: network.num_outputs=0
CT.onnx: network.num_outputs=1

Both should have one output.

Logs:

TransposeCast

[07/05/2024-14:25:29] [TRT] [I] [MemUsageChange] Init CUDA: CPU +19, GPU +0, now: CPU 111, GPU 26482 (MiB)
[07/05/2024-14:25:29] [TRT] [V] Trying to load shared library libnvinfer_builder_resource.so.10.1.0
[07/05/2024-14:25:29] [TRT] [V] Loaded shared library libnvinfer_builder_resource.so.10.1.0
[07/05/2024-14:25:31] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +1931, GPU +354, now: CPU 2189, GPU 26836 (MiB)
[07/05/2024-14:25:31] [TRT] [V] CUDA lazy loading is enabled.
[07/05/2024-14:25:31] [TRT] [V] Adding network input: onnx::Transpose_0 with dtype: uint8, dimensions: (1, 1)
[07/05/2024-14:25:31] [TRT] [V] Registering tensor: onnx::Transpose_0 for ONNX tensor: onnx::Transpose_0
[07/05/2024-14:25:31] [TRT] [V] Static check for parsing node: /Transpose [Transpose]
[07/05/2024-14:25:31] [TRT] [V] Parsing node: /Transpose [Transpose]
[07/05/2024-14:25:31] [TRT] [V] Searching for input: onnx::Transpose_0
[07/05/2024-14:25:31] [TRT] [V] /Transpose [Transpose] inputs: [onnx::Transpose_0 -> (1, 1)[UINT8]],
[07/05/2024-14:25:31] [TRT] [V] Registering layer: /Transpose for ONNX node: /Transpose
[07/05/2024-14:25:31] [TRT] [V] Registering tensor: /Transpose_output_0 for ONNX tensor: /Transpose_output_0
[07/05/2024-14:25:31] [TRT] [V] Static check for parsing node: /Cast [Cast]
TC.onnx: network.num_outputs=0

CastTranspose

[07/05/2024-14:25:31] [TRT] [V] Adding network input: onnx::Cast_0 with dtype: uint8, dimensions: (1, 1)
[07/05/2024-14:25:31] [TRT] [V] Registering tensor: onnx::Cast_0 for ONNX tensor: onnx::Cast_0
[07/05/2024-14:25:31] [TRT] [V] Static check for parsing node: /Cast [Cast]
[07/05/2024-14:25:31] [TRT] [V] Parsing node: /Cast [Cast]
[07/05/2024-14:25:31] [TRT] [V] Searching for input: onnx::Cast_0
[07/05/2024-14:25:31] [TRT] [V] /Cast [Cast] inputs: [onnx::Cast_0 -> (1, 1)[UINT8]],
[07/05/2024-14:25:31] [TRT] [V] Casting to type: float32
[07/05/2024-14:25:31] [TRT] [V] Registering layer: /Cast for ONNX node: /Cast
[07/05/2024-14:25:31] [TRT] [V] Registering tensor: /Cast_output_0 for ONNX tensor: /Cast_output_0
[07/05/2024-14:25:31] [TRT] [V] /Cast [Cast] outputs: [/Cast_output_0 -> (1, 1)[FLOAT]],
[07/05/2024-14:25:31] [TRT] [V] Static check for parsing node: /Transpose [Transpose]
[07/05/2024-14:25:31] [TRT] [V] Parsing node: /Transpose [Transpose]
[07/05/2024-14:25:31] [TRT] [V] Searching for input: /Cast_output_0
[07/05/2024-14:25:31] [TRT] [V] /Transpose [Transpose] inputs: [/Cast_output_0 -> (1, 1)[FLOAT]],
[07/05/2024-14:25:31] [TRT] [V] Registering layer: /Transpose for ONNX node: /Transpose
[07/05/2024-14:25:31] [TRT] [V] Registering tensor: 2_0 for ONNX tensor: 2
[07/05/2024-14:25:31] [TRT] [V] /Transpose [Transpose] outputs: [2 -> (1, 1)[FLOAT]],
[07/05/2024-14:25:31] [TRT] [V] Marking 2_0 as output: 2
CT.onnx: network.num_outputs=1
@lix19937
Copy link

lix19937 commented Jul 6, 2024 •
edited
Loading

In my env it runs normal.

(torch113) lix@lix19937:~/lix$ python  ./test.py
export done!
[07/06/2024-13:13:18] [TRT] [I] [MemUsageChange] Init CUDA: CPU +655, GPU +0, now: CPU 738, GPU 1216 (MiB)
[07/06/2024-13:13:19] [TRT] [V] Trying to load shared library libnvinfer_builder_resource.so.8.5.10
[07/06/2024-13:13:19] [TRT] [V] Loaded shared library libnvinfer_builder_resource.so.8.5.10
[07/06/2024-13:13:27] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +476, GPU +116, now: CPU 1241, GPU 1332 (MiB)
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::BatchedNMS_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::BatchTilePlugin_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::Clip_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::CoordConvAC version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::CropAndResizeDynamic version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::CropAndResize version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::DecodeBbox3DPlugin version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::DetectionLayer_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::EfficientNMS_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::FlattenConcat_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::GenerateDetection_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::GridAnchor_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::GridAnchorRect_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::InstanceNormalization_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::InstanceNormalization_TRT version 2
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::LReLU_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::NMSDynamic_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::NMS_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::Normalize_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::PillarScatterPlugin version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::PriorBox_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::ProposalDynamic version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::ProposalLayer_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::Proposal version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::Region_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::Reorg_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::ResizeNearest_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::ROIAlign_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::RPROI_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::ScatterND version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::SpecialSlice_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::Split version 1
[07/06/2024-13:13:27] [TRT] [V] Registered plugin creator - ::VoxelGeneratorPlugin version 1
[07/06/2024-13:13:27] [TRT] [V] Adding network input: onnx::Transpose_0 with dtype: uint8, dimensions: (1, 1)
[07/06/2024-13:13:27] [TRT] [V] Registering tensor: onnx::Transpose_0 for ONNX tensor: onnx::Transpose_0
[07/06/2024-13:13:27] [TRT] [V] Parsing node: /Transpose [Transpose]
[07/06/2024-13:13:27] [TRT] [V] Searching for input: onnx::Transpose_0
[07/06/2024-13:13:27] [TRT] [V] /Transpose [Transpose] inputs: [onnx::Transpose_0 -> (1, 1)[UINT8]],
[07/06/2024-13:13:27] [TRT] [V] Registering layer: /Transpose for ONNX node: /Transpose
[07/06/2024-13:13:27] [TRT] [V] Registering tensor: /Transpose_output_0 for ONNX tensor: /Transpose_output_0
[07/06/2024-13:13:27] [TRT] [V] /Transpose [Transpose] outputs: [/Transpose_output_0 -> (1, 1)[UINT8]],
[07/06/2024-13:13:27] [TRT] [V] Parsing node: /Cast [Cast]
[07/06/2024-13:13:27] [TRT] [V] Searching for input: /Transpose_output_0
[07/06/2024-13:13:27] [TRT] [V] /Cast [Cast] inputs: [/Transpose_output_0 -> (1, 1)[UINT8]],
[07/06/2024-13:13:27] [TRT] [V] Casting to type: float32
[07/06/2024-13:13:27] [TRT] [V] Registering layer: /Cast for ONNX node: /Cast
[07/06/2024-13:13:27] [TRT] [V] Registering tensor: 2_0 for ONNX tensor: 2
[07/06/2024-13:13:27] [TRT] [V] /Cast [Cast] outputs: [2 -> (1, 1)[FLOAT]],
[07/06/2024-13:13:27] [TRT] [V] Marking 2_0 as output: 2
TC.onnx: network.num_outputs=1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::BatchedNMSDynamic_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::BatchedNMS_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::BatchTilePlugin_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::Clip_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::CoordConvAC version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::CropAndResizeDynamic version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::CropAndResize version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::DecodeBbox3DPlugin version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::DetectionLayer_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::EfficientNMS_Explicit_TF_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::EfficientNMS_Implicit_TF_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::EfficientNMS_ONNX_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::EfficientNMS_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::FlattenConcat_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::GenerateDetection_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::GridAnchor_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::GridAnchorRect_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::InstanceNormalization_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::InstanceNormalization_TRT version 2
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::LReLU_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::MultilevelCropAndResize_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::MultilevelProposeROI_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::MultiscaleDeformableAttnPlugin_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::NMSDynamic_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::NMS_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::Normalize_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::PillarScatterPlugin version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::PriorBox_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::ProposalDynamic version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::ProposalLayer_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::Proposal version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::PyramidROIAlign_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::Region_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::Reorg_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::ResizeNearest_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::ROIAlign_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::RPROI_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::ScatterND version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::SpecialSlice_TRT version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::Split version 1
[07/06/2024-13:13:27] [TRT] [V] Plugin creator already registered - ::VoxelGeneratorPlugin version 1
[07/06/2024-13:13:27] [TRT] [V] Adding network input: onnx::Cast_0 with dtype: uint8, dimensions: (1, 1)
[07/06/2024-13:13:27] [TRT] [V] Registering tensor: onnx::Cast_0 for ONNX tensor: onnx::Cast_0
[07/06/2024-13:13:27] [TRT] [V] Parsing node: /Cast [Cast]
[07/06/2024-13:13:27] [TRT] [V] Searching for input: onnx::Cast_0
[07/06/2024-13:13:27] [TRT] [V] /Cast [Cast] inputs: [onnx::Cast_0 -> (1, 1)[UINT8]],
[07/06/2024-13:13:27] [TRT] [V] Casting to type: float32
[07/06/2024-13:13:27] [TRT] [V] Registering layer: /Cast for ONNX node: /Cast
[07/06/2024-13:13:27] [TRT] [V] Registering tensor: /Cast_output_0 for ONNX tensor: /Cast_output_0
[07/06/2024-13:13:27] [TRT] [V] /Cast [Cast] outputs: [/Cast_output_0 -> (1, 1)[FLOAT]],
[07/06/2024-13:13:27] [TRT] [V] Parsing node: /Transpose [Transpose]
[07/06/2024-13:13:27] [TRT] [V] Searching for input: /Cast_output_0
[07/06/2024-13:13:27] [TRT] [V] /Transpose [Transpose] inputs: [/Cast_output_0 -> (1, 1)[FLOAT]],
[07/06/2024-13:13:27] [TRT] [V] Registering layer: /Transpose for ONNX node: /Transpose
[07/06/2024-13:13:27] [TRT] [V] Registering tensor: 2_0 for ONNX tensor: 2
[07/06/2024-13:13:27] [TRT] [V] /Transpose [Transpose] outputs: [2 -> (1, 1)[FLOAT]],
[07/06/2024-13:13:27] [TRT] [V] Marking 2_0 as output: 2
CT.onnx: network.num_outputs=1 

(torch113) lix@lix19937:~/lix$ pip list |grep torch
torch                     1.13.1+cu116
torchaudio                0.13.1+cu116
torchvision               0.14.1+cu116

@lix19937
Copy link

lix19937 commented Jul 6, 2024

trt version 8510

@srasmussenvl
Copy link
Author

trt version 8510

It also fails on my 8.6.1 env.

@lix19937
Copy link

lix19937 commented Jul 6, 2024

You can upload the two onnx here.

@srasmussenvl
Copy link
Author

You can upload the two onnx here.

CT_TC.zip

@lix19937
Copy link

Also run passed.

@lix19937
Copy link

lix19937 commented Aug 7, 2024

Can you try the latest trt ?

@srasmussenvl
Copy link
Author

srasmussenvl commented Aug 20, 2024 •
edited
Loading

Can you try the latest trt ?

Same thing happens with TRT 10.2.0 and 10.3.0

@braindevices
Copy link

braindevices commented Oct 17, 2024 •
edited
Loading

also has the same problem:

tensorrt 10.1.0
tensorrt-cu12 10.5.0
tensorrt-cu12-bindings 10.1.0
tensorrt-cu12-libs 10.1.0

@kevinch-nv kevinch-nv added Feature Request Request for new functionality triaged Issue has been triaged by maintainers labels Feb 11, 2025
@kevinch-nv
Copy link
Collaborator

This is expected behavior. All UINT8 inputs must directly feed into a cast layer in TensorRT. No other layers support UINT8 inputs, even just data-movement ops like reshape / transpose.

For current users of TensorRT, the recommendation is to change your network structure for UINT8 inputs to feed directly into a cast.

Marking this a feature request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Feature Request Request for new functionality triaged Issue has been triaged by maintainers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@braindevices @lix19937 @kevinch-nv @srasmussenvl