Skip to content

Latest commit

 

History

History
419 lines (354 loc) · 36.1 KB

perf-overview.md

File metadata and controls

419 lines (354 loc) · 36.1 KB

(perf-overview)=

Important

As of TensorRT-LLM v0.10, these performance benchmarks have changed methodology to utilize in-flight batching and no longer utilize static benchmarking. These numbers are initial measurements and are expected to improve in future releases.

Overview

This document summarizes performance measurements of TensorRT-LLM on a number of GPUs across a set of key models.

The data in the following tables is provided as a reference point to help users validate observed performance. It should not be considered as the peak performance that can be delivered by TensorRT-LLM.

Known Issues

The following issues are being addressed to improve the efficiency of TensorRT-LLM.

Known AllReduce performance issue on AMD-based CPU platforms

We observed a performance issue on NCCL 2.23.4, which can be workarounded by setting NCCL_P2P_LEVEL to SYS:

export NCCL_P2P_LEVEL=SYS

Multi-GPU cases could be affected due to the issue, which is being addressed.

Fused Matmul + Gated-SiLU (LLaMA)

The current implementation combines two Matmul operations into one Matmul followed by a separate SwiGLU kernel (when --use_fused_mlp=enable is enabled). There is also a more efficient implementation that runs single Matmul + SwiGLU fused kernel for FP8 on Hopper (when --use_fused_mlp=enable --gemm_swiglu_plugin fp8 is enabled). The gemm_swiglu_plugin will support more data types and GPU architectures in the future release.

Use gptManagerBenchmark for GH200

For release v0.17, on GH200 systems, we recommend using the legacy flow based on gptManagerBenchmark to measure performance.

Throughput Measurements

The below table shows performance data where a local inference client is fed requests at an infinite rate (no delay between messages), and shows the throughput client-server scenario under maximum load.

The performance numbers below were collected using the steps described in this document.

All data in the table below was generated using version 0.17 and presents token throughput in tokens/second.

Throughput (tokens / sec) GPU B200 180GB H200 141GB HBM3 H100 80GB HBM3 GH200 480GB L40S A100-SXM4-80GB
Precision FP4 FP8 FP8 FP8 FP8 FP16
Model Tensor Model Parallel Size Runtime Input/Output Lengths
LLaMA v3.1 8B 1 128, 128 59068.44 29526.04 28836.77 29852.96 9104.61 6627.27
128, 2048 44020.83 25398.86 21109.38 21769.55 5365.81 5255.99
128, 4096 27897.86 17370.8 13593.65 14189.89 3025.92 3453.79
500, 2000 36802.05 21020.81 16500.69 17137.29 4273.75 4276.58
1000, 1000 31948.27 17537.96 15244.78 16482.77 4054.71 3786.83
2048, 128 8976.63 3794.14 3556.73 3843.95 1066.52 799.61
2048, 2048 19991.52 11968.5 9488.42 10265.9 2225.27 2424.16
5000, 500 8042.82 3987.79 3559.36 3932.58 981.2 825.13
20000, 2000 2990.57 1804.1 1401.31 1560.2 327.97 330.04
LLaMA v3.1 70B 1 128, 128 11304.22 4020.75 3378.03 3636.91
128, 2048 9325.33 4165.68 911.62 2082.74
128, 4096 5922.41 2651.75 426.32 1263.98
500, 2000 7675.61 3018.39 775.57 1973.86
1000, 1000 6733.46 2823.45 839.97 1746.12
2048, 128 1376.73 465.99 343.29 424.96
2048, 2048 4481.76 1913.8 1086.93
5000, 500 1484.42 560.16 245.34 422.36
20000, 2000 623.92 279.52
2 128, 128 16550.07 6823.01 6645.12 1313.96
128, 2048 11690.11 8290.35 6169.58 531.26
128, 4096 8711.55 6526.67 3897.06
500, 2000 10780.12 6848.02 4972.57 439.41
1000, 1000 9931.41 5164.76 4390.53 472.94
2048, 128 2045.23 809 772.66 148.96
2048, 2048 6807.26 4183.88 2898.16 261.1
5000, 500 2418.06 1025.38 919.73 121.47
20000, 2000 1169.33 640.62 443.01
4 128, 128 22842.4 11098.63 11127.53 1523.52 2733.48
128, 2048 23206.76 14156 11511.93 1942.66 2811.27
128, 4096 17014.24 10574.06 7439.41 1440.23 1976.49
500, 2000 20293.55 12452.79 9836.7 1634.72 2275.79
1000, 1000 16376.74 8911.29 7430.99 1209.25 1921.77
2048, 128 3072.04 1358.06 1302.6 177.72 325.15
2048, 2048 11700.58 7130.44 5480.03 969.68 1393.64
5000, 500 3714.05 1811.55 1602.78 249.52 392.62
20000, 2000 1992.45 1199.68 920.19 162.25 212.08
8 128, 128 27023.65 15355.84 14730.69 1464.03 4717.62
128, 2048 33523.64 21195.88 17061.82 2303.31 5241.5
128, 4096 26144.78 16941.52 14171.43 2018.22 3724.67
500, 2000 28589.19 17278.4 14679.33 1971.96 4445.37
1000, 1000 21870.92 13181.24 11451.16 1333.62 3320.41
2048, 128 3776.53 1983.03 1923.41 176.16 542.38
2048, 2048 16481.62 11142.47 8801.95 1200.16 2553.71
5000, 500 2717.83 2457.42 259.71 696.34
20000, 2000 2967.99 1920.45 1512.6 209.87 413.38
LLaMA v3.1 405B 8 128, 128 8830.69 3874.19
128, 2048 11859.61 5938.09
128, 4096 9516.52 5168.37
500, 2000 10021.79 5084.29
1000, 1000 7370.54 3399.69
2048, 128 1092.53 463.42
2048, 2048 5946.17 2940.62
5000, 500 1522.11 669.13
20000, 2000 1049.55 535.31
Mistral 7B 1 128, 128 65091.47 31938.12 31674.49 32498.47 9664.13 6982.53
128, 2048 47441.2 27409.3 23496.42 23337.29 5720.65 5630.62
128, 4096 29388.45 18505.03 14350.99 15017.88 3136.33 3591.22
500, 2000 39316.32 22354.67 18026.27 18556 4521.77 4400.48
1000, 1000 33610.6 18426.16 16035.66 17252.11 4177.76 3896.58
2048, 128 9112.21 3834.43 3642.48 3813.13 1076.74 808.58
2048, 2048 20785.3 12347.37 9958.17 10755.94 2286.71 2489.77
5000, 500 8156.29 4041.59 3591.33 3949.66 1001.02 844.64
20000, 2000 3023.43 1822.69 1373.24 1601.28 337.83 332.3
Mixtral 8x7B 1 128, 128 40420.54 17157.72 15962.49 16859.18
128, 2048 33687.5 15095.21 8290.13 11120.16
128, 4096 21279.96 9534.62 4784.86 6610.47
500, 2000 26616.79 12105.27 6800.6 9192.86
1000, 1000 22948.78 10371.36 6868.52 8849.18
2048, 128 5299.93 2009.67 1892.81 1994.31
2048, 2048 15123.66 6940.32 3983.1 5545.46
5000, 500 5442.19 2309.1 1764.7 2078.27
20000, 2000 2337.08 1151.78 673.7 860.68
2 128, 128 56552.59 27825.34 27451.13 5541.47
128, 2048 50058.07 29584.05 22830.08 4169.78
128, 4096 30718.57 21564.68 14237.01 2608.05
500, 2000 41875.21 23410.63 17036.04 3446.37
1000, 1000 35954.65 19151.19 15770.89 3154.52
2048, 128 7910.14 3383.16 3333.68 649
2048, 2048 22775.24 14007.29 10685.85 2056.58
5000, 500 8781.47 4223.68 3646.09 724.44
20000, 2000 4000.66 2299.21 1757.45 337.51
4 128, 128 73050.19 42551.59 41068.23 6921.87 10324.28
128, 2048 79709.05 52291.78 41164.73 7996.93 10911.86
128, 4096 58311.22 39513.73 27912.48 5736.09 7666.51
500, 2000 68890.48 43818.99 34489.34 6914.68 8456.21
1000, 1000 54576.81 33580.9 27784.74 5251.49 7122.84
2048, 128 11253.32 5467.62 5234.98 827.62 1237.62
2048, 2048 39595.5 24980.93 19432.08 3935.32 5222.98
5000, 500 13558.82 7084.94 6401.56 1092.88 1500.55
20000, 2000 6903.81 4236.84 3303.83 682.48 829.59
8 128, 128 79508.79 53212.55 50849.55 6740.84 17043.54
128, 2048 96005.39 68608.45 61607.7 10393.3 20277.88
128, 4096 69563.18 54827.78 48280.37 8472.35 15282.89
500, 2000 83495.43 58706.39 52583.65 8660.71 17184.24
1000, 1000 64076.75 44705.48 40631.71 5947.72 12851.44
2048, 128 13704.73 7554.38 6988.18 811.96 2165.52
2048, 2048 48948.43 36193.64 30983.35 5136.98 9809.76
5000, 500 17490 10271.8 9210.11 1153.76 2761.28
20000, 2000 10454.76 6835.53 5602.43 918.95 1592.53
Mixtral 8x22B 8 128, 128 22948.57 21876.08 6381.95
128, 2048 49366.99 32415.81 25150.03 6685.99
128, 4096 25753.14 18387.4 4789.13
500, 2000 27429.6 21421.86 5648.46
1000, 1000 19712.35 16573.24 4549.46
2048, 128 2899.84 2794.97 761.56
2048, 2048 15798.59 12244.93 3521.98
5000, 500 4031.79 3645.27 959.14
20000, 2000 2815.76 2227.63 575.02

TP stands for Tensor Parallelism

Reproducing Benchmarked Results

[!NOTE] The only models supported in this workflow are those listed in the table above.

The following tables are references for commands that are used as part of the benchmarking process. For a more detailed description of this benchmarking workflow, see the benchmarking suite documentation.

Commands

For systems other than GH200

Stage Description Command
Dataset Create a synthetic dataset python benchmarks/cpp/prepare_dataset.py --tokenizer=$model_name --stdout token-norm-dist --num-requests=$num_requests --input-mean=$isl --output-mean=$osl --input-stdev=0 --output-stdev=0 > $dataset_file
Build Build a TensorRT-LLM engine trtllm-bench --model $model_name build --tp_size $tp_size --pp_size $pp_size --quantization FP8 --dataset $dataset_file
Run Run a benchmark with a dataset trtllm-bench --model $model_name throughput --dataset $dataset_file --engine_dir $engine_dir

For GH200 systems only

For release v0.17, on GH200 systems, the recommendation is to use the legacy flow based on gptManagerBenchmark to measure performance.

Stage Description Command
Dataset Create a synthetic dataset for engine building python benchmarks/cpp/prepare_dataset.py --tokenizer=$model_name --stdout token-norm-dist --num-requests=$num_requests --input-mean=$isl --output-mean=$osl --input-stdev=0 --output-stdev=0 > $dataset_file
Build Build a TensorRT-LLM engine trtllm-bench --model $model_name build --tp_size $tp_size --quantization FP8 --dataset $dataset_file
Dataset Create a synthetic dataset for benchmarking in json format python benchmarks/cpp/prepare_dataset.py --output=$dataset_file_json --tokenizer=$model_name token-norm-dist --num-requests=$num_requests --input-mean=$isl --output-mean=$osl --input-stdev=0 --output-stdev=0
Run Run a benchmark with a dataset in json format /app/tensorrt_llm/benchmarks/cpp/gptManagerBenchmark --engine_dir $engine_dir --type IFB --api executor --dataset $dataset_file_json --eos_id -1 --log_iteration_data --scheduler_policy guaranteed_no_evict --kv_cache_free_gpu_mem_fraction 0.95 --output_csv result.csv --request_rate -1.0 --enable_chunked_context --warm_up 0

Variables

Name Description
$isl Benchmark input sequence length.
$osl Benchmark output sequence length.
$tp_size Tensor parallel mapping degree to run the benchmark with
$pp_size Pipeline parallel mapping degree to run the benchmark with
$engine_dir Location to store built engine file (can be deleted after running benchmarks).
$model_name HuggingFace model name eg. meta-llama/Llama-2-7b-hf or use the path to a local weights directory
$dataset_file Location of the dataset file generated by prepare_dataset.py
$num_requests The number of requests to generate for dataset generation
$seq_len A sequence length of ISL + OSL

Preparing a Dataset

In order to prepare a dataset, you can use the provided script. To generate a synthetic dataset, run the following command:

python benchmarks/cpp/prepare_dataset.py --tokenizer=$model_name --stdout token-norm-dist --num-requests=$num_requests --input-mean=$isl --output-mean=$osl --input-stdev=0 --output-stdev=0 > $dataset_file

The command will generate a text file located at the path specified $dataset_file where all requests are of the same input/output sequence length combinations. The script works by using the tokenizer to retrieve the vocabulary size and randomly sample token IDs from it to create entirely random sequences. In the command above, all requests will be uniform because the standard deviations for both input and output sequences are set to 0.

For each input and output sequence length combination, the table below details the $num_requests that were used. For shorter input and output lengths, a larger number of messages were used to guarantee that the system hit a steady state because requests enter and exit the system at a much faster rate. For longer input/output sequence lengths, requests remain in the system longer and therefore require less requests to achieve steady state.

Input Length Output Length $seq_len $num_requests
128 128 256 30000
128 2048 2176 3000
128 4096 4224 1500
2048 128 2176 3000
2048 2048 4096 1500
5000 500 5500 1500
1000 1000 2000 3000
500 2000 2500 3000
20000 2000 22000 1000

Engine Building

All engines are built using the trtllm-bench build subcommand. The basic command for FP8 quantized engines is as follows:

trtllm-bench --model $model_name build --tp_size $tp_size --pp_size $pp_size --quantization FP8 --dataset $dataset_file

When providing --dataset in the build subcommand, trtllm-bench build uses high-level statistics of the dataset (average ISL/OSL, max sequence length) and tuning heuristics to optimize engine build settings.

Alternatively, if you would like to build the engine with specific settings, you can do so by specifying the values for max_batch_size and max_num_tokens:

trtllm-bench --model $model_name build --tp_size $tp_size --pp_size $pp_size --quantization FP8 --max_seq_len $seq_len --max_batch_size $max_bs --max_num_tokens $max_token

If you would like to build an FP16 engine without any quantization, simply remove the --quantization FP8 option. If using pre-quantized weights (e.g. nvidia/Llama-3.1-70B-Instruct-FP8 from HuggingFace), please set the --quantization argument to the model dtype to ensure the KV Cache is set to the appropriate dtype.

[!NOTE] If you specify FP8 quantization, the KV cache will automatically be set to FP8 as well!

The trtllm-bench build subcommand will output the path where the engine is located upon a successful build. For example,

===========================================================
ENGINE SAVED: /tmp/meta-llama/Llama-2-7b-hf/tp_1_pp_1
===========================================================

Running the Benchmark

For non GH200 systems

To run the benchmark with the generated data set, simply use the trtllm-bench throughput subcommand. The benchmarker will run an offline maximum throughput scenario such that all requests are queued in rapid succession. You simply need to provide the patch to the engine from the build phase and a generated dataset.

trtllm-bench --model $model_name throughput --dataset $dataset_file --engine_dir $engine_dir

In majority of cases, we also use a higher KV cache percentage by setting --kv_cache_free_gpu_mem_fraction 0.95 in the benchmark command. This allows us to obtain better performance than the default setting of 0.90. We fall back to 0.90 if we hit an out of memory issue.

The results will be printed to the terminal upon benchmark completion. For example,

===========================================================
= ENGINE DETAILS
===========================================================
Model:                  meta-llama/Llama-2-7b-hf
Engine Directory:       /tmp/meta-llama/Llama-2-7b-hf/tp_1_pp_1
TensorRT-LLM Version:   0.12.0
Dtype:                  float16
KV Cache Dtype:         FP8
Quantization:           FP8
Max Input Length:       2048
Max Sequence Length:    4098

===========================================================
= WORLD + RUNTIME INFORMATION
===========================================================
TP Size:                1
PP Size:                1
Max Runtime Batch Size: 4096
Max Runtime Tokens:     8192
Scheduling Policy:      Guaranteed No Evict
KV Memory Percentage:   99.0%
Issue Rate (req/sec):   3.680275266452667e+18
===========================================================
= STATISTICS
===========================================================
Number of requests:             3000
Average Input Length (tokens):  128.0
Average Output Length (tokens): 128.0
Token Throughput (tokens/sec):  23405.927228471104
Request Throughput (req/sec):   182.8588064724305
Total Latency (seconds):        16.406100739
===========================================================

[!WARNING] In some cases, the benchmarker may not print anything at all. This behavior usually means that the benchmark has hit an out of memory issue. Try reducing the KV cache percentage using the --kv_cache_free_gpu_mem_fraction option to lower the percentage of used memory.

Online Serving Measurements

The TensorRT-LLM backend is used to measure the performance of TensorRT-LLM for online serving.

The below table shows the throughput and latency under a serving scenario.

All data in the table below was generated using version 0.14.0, with 500 requests and BF16 precision.
Model GPU TP Input Length Output Length QPS Tput(req/s) Mean TTFT(ms) Mean ITL(ms) Total Token Tput (tok/s) Output Tput (tok/s)
LLaMA 3.1 70B H100 80GB HBM3 4 467 256 2 2 62 21 1406 498
4 4 68 24 2750 973
8 7 92 32 5256 1860
16 12 175 66 8941 3164
32 16 1229 86 11537 4083
INF 16 9123 85 11593 4103
467 16 2 2 53 18 844 28
4 4 58 20 1908 63
8 8 71 24 3795 126
16 16 109 38 7492 248
32 28 1197 482 13655 452
INF 28 9126 548 13719 454
202 214 2 2 48 20 780 401
4 4 51 22 1499 771
8 7 57 25 2702 1390
16 11 74 32 4364 2245
32 14 116 42 5837 3003
INF 16 4482 50 6725 3459
LLaMA 3.1 8B 1 467 256 2 2 23 8 1423 504
4 4 24 9 2624 929
8 8 26 9 5535 1959
16 15 30 11 10636 3765
32 26 50 19 19138 6774
INF 37 3335 39 26614 9420
467 16 2 2 19 7 956 32
4 4 20 7 1910 63
8 8 22 7 3808 126
16 16 24 8 7567 251
32 31 29 10 14894 493
INF 79 3280 193 38319 1269
202 214 2 2 19 7 809 416
4 4 20 8 1586 816
8 7 21 9 3047 1568
16 13 23 10 5597 2879
32 23 27 11 9381 4825
INF 39 1657 21 16117 8291
LLaMA 3.1 70B H200 131GB HBM3 4 467 256 2 2 58 18 1411 499
4 4 63 20 2770 980
8 7 84 27 5328 1886
16 13 165 60 9224 3264
32 16 1279 83 11800 4176
INF 16 9222 83 11826 4185
467 16 2 2 50 15 956 32
4 4 55 16 1909 63
8 8 67 20 3799 126
16 16 103 33 7499 248
32 28 1259 485 13586 450
INF 29 9074 546 13792 457
202 214 2 2 43 17 793 408
4 4 46 18 1524 784
8 7 51 21 2796 1438
16 11 67 28 4639 2386
32 15 112 39 6288 3235
INF 17 4480 48 7230 3719
LLaMA 3.1 8B H200 131GB HBM3 1 467 256 2 2 21 6 1425 504
4 4 23 7 2828 1001
8 8 24 7 5567 1971
16 15 27 9 10761 3809
32 27 44 16 19848 7025
INF 40 3237 36 28596 10121
467 16 2 2 18 5 956 32
4 4 19 6 1910 63
8 8 20 6 3810 126
16 16 22 7 7567 250
32 31 27 9 14927 494
INF 81 3227 190 39007 1291
202 214 2 2 17 6 812 418
4 4 18 6 1597 822
8 7 19 7 3088 1589
16 14 20 8 5771 2969
32 24 24 9 9931 5109
INF 43 1665 19 17861 9189

TP stands for Tensor Parallelism

TTFT stands for Time To First Token

ITL stands for Inter Token Latency

For GH200 systems only

For release v0.17, on GH200 systems, the recommendation is to use gptManagerBenchmark to measure performance. Throughput measurements are reported based on the below commands.

 /app/tensorrt_llm/benchmarks/cpp/gptManagerBenchmark  --engine_dir $engine_dir --type IFB --dataset $dataset_file_json --eos_id -1 --scheduler_policy guaranteed_no_evict --kv_cache_free_gpu_mem_fraction 0.95 --output_csv result.csv --request_rate -1.0 --enable_chunked_context --warm_up 0

The command will run the gptManagerBenchmark binary that will report the throughput and other metrics as part of its output that can be compared with the table in the Throughput Measurements of this README.