Skip to content

support more compression algorithms #1174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jan 5, 2023
Merged

support more compression algorithms #1174

merged 4 commits into from
Jan 5, 2023

Conversation

zhicwu
Copy link
Contributor

@zhicwu zhicwu commented Jan 4, 2023

Added support for more compression algorithms(see table shown below), and enabled client options compress_alogrithm, decompress_alogrithm, and decompress_level.

Algorithm Compress Client Request Decompress Server Response
brotli N Y
bz2 N Y
deflate N Y
gzip N Y
lz4 Y Y
snappy Y N (see ClickHouse/ClickHouse#44885)
xz N Y
zstd Y Y

Closes #1101

@zhicwu zhicwu marked this pull request as draft January 4, 2023 13:08
@zhicwu zhicwu marked this pull request as ready for review January 5, 2023 10:09
@zhicwu zhicwu merged commit 0317788 into ClickHouse:develop Jan 5, 2023
@zhicwu
Copy link
Contributor Author

zhicwu commented Jan 5, 2023

50 concurrent users, each repeats query select number, toDateTime(number), toString(number) from numbers(100000) for 100 times. Both HTTP_CLIENT and APACHE_HTTP_CLIENT could not complete the test without any error.

Option Avg(ms) Median(ms) 90% Line 95% Line 99% Line Min(ms) Max(ms) Error% Throughput(op/s) Received KB/s
compress_algorithm=brotli 390 387 521 566 819 45 1,399 0 124.03 384,309.76
compress_algorithm=bz2 1,040 1,088 1,443 1,490 1,598 125 2,047 0 46.78 144,959.86
compress_algorithm=deflate 402 404 561 604 685 53 924 0 120.58 373,605.40
compress_algorithm=gzip 393 397 534 572 634 53 722 0 123.82 383,653.08
compress_algorithm=lz4 453 459 627 677 769 48 1,111 0 106.82 330,982.44
compress_algorithm=xz 1,222 1,221 1,398 1,439 1,524 109 1,702 0 40.49 125,468.07
compress_algorithm=zstd 396 401 542 588 677 43 899 0 121.93 377,787.14
compress_algorithm=none 460 468 638 688 793 43 1,006 0 105.15 325,790.06
(deafult) 455 462 634 681 764 57 1,022 0 106.17 328,951.40
(default 0.3.2-patch11) 481 491 640 689 771 60 942 0 100.87 312,552.60

Benchmarks(both request and response were compressed using same algorithm, which is lz4 by default)

1 thread

Benchmark                                     (client)  (connection)  (statement)   (type)   Mode  Cnt   Score   Error  Units
Query.selectInt8     ClickhouseHttpJdbc(0.3.2-patch11)         reuse     prepared  default  thrpt   20   97.955 ± 6.514  ops/s
Query.selectInt8                    ClickhouseHttpJdbc         reuse     prepared  default  thrpt   20  112.686 ± 9.231  ops/s
Query.selectInt8              ClickhouseHttpClientJdbc         reuse     prepared  default  thrpt   20   96.415 ± 7.300  ops/s
Query.selectInt8        ClickhouseApacheHttpClientJdbc         reuse     prepared  default  thrpt   20  112.639 ± 8.882  ops/s
Query.selectInt8              ClickhouseHttpJdbc(zstd)         reuse     prepared  default  thrpt   20  108.366 ± 5.153  ops/s
Query.selectInt8        ClickhouseHttpClientJdbc(zstd)         reuse     prepared  default  thrpt   20  101.005 ± 4.174  ops/s
Query.selectInt8  ClickhouseApacheHttpClientJdbc(zstd)         reuse     prepared  default  thrpt   20  104.685 ± 5.008  ops/s

4 threads

Benchmark                                     (client)  (connection)  (statement)   (type)   Mode  Cnt    Score    Error  Units
Query.selectInt8     ClickhouseHttpJdbc(0.3.2-patch11)         reuse     prepared  default  thrpt   20  309.405 ±  8.515  ops/s
Query.selectInt8                    ClickhouseHttpJdbc         reuse     prepared  default  thrpt   20  335.104 ± 21.070  ops/s
Query.selectInt8              ClickhouseHttpClientJdbc         reuse     prepared  default  thrpt   20  312.930 ± 20.013  ops/s
Query.selectInt8        ClickhouseApacheHttpClientJdbc         reuse     prepared  default  thrpt   20  340.629 ± 10.316  ops/s
Query.selectInt8              ClickhouseHttpJdbc(zstd)         reuse     prepared  default  thrpt   20  338.217 ±  7.272  ops/s
Query.selectInt8        ClickhouseHttpClientJdbc(zstd)         reuse     prepared  default  thrpt   20  320.003 ± 16.361  ops/s
Query.selectInt8  ClickhouseApacheHttpClientJdbc(zstd)         reuse     prepared  default  thrpt   20  344.978 ± 11.624  ops/s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant