Skip to content

Commit fa427d8

Browse files
soyeric128soyeric128
authored
updated (#1084)
1 parent 59a00cd commit fa427d8

File tree

2 files changed

+149
-97
lines changed

2 files changed

+149
-97
lines changed

‎docs/en/guides/54-query/06-query-hash.md

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
---
2+
title: Query Hash
3+
---
4+
5+
A Query Hash is an identifier used to represent a unique SQL query. It converts the structure and content of the query into a fixed-length value, so even if the query text has slight differences, the hash will be the same as long as the logical structure is identical. This helps in identifying similar queries and frequently executed queries.
6+
7+
## Query Hash Types
8+
9+
Databend supports for two types of query hashes:
10+
11+
- `query_hash`: The query_hash ensures that repeated queries, even with variations in white space or comments, share the same hash. For example, the following queries share the same hash:
12+
13+
```sql
14+
SELECT * FROM t1 WHERE name = 'jim'
15+
SELECT * FROM t1 WHERE name = 'jim'
16+
```
17+
18+
- `query_parameterized_hash`: The query_parameterized_hash normalizes queries by handling literals involved in comparison predicates (e.g., `=`, `!=`, `>=`, `<=`), enabling the identification of structurally similar queries regardless of the specific values used. For example, the following queries share the same hash:
19+
20+
```sql
21+
SELECT * FROM t1 WHERE name = 'data'
22+
SELECT * FROM t1 WHERE name = 'bend'
23+
```
24+
25+
## Retrieving Hash Values
26+
27+
Databend stores the hash values of historical queries in the columns named `query_hash` and `query_parameterized_hash` in system table [system.query_log](/sql/sql-reference/system-tables/system-query-log). To retrieve the hash values of a query, you can pull them from the system table using a SELECT statement. For example:
28+
29+
```sql
30+
SELECT * FROM books;
31+
32+
┌───────────────────────────────────────────────────────────────┐
33+
│ id │ title │ genre │
34+
├──────────────────┼─────────────────────────┼──────────────────┤
35+
│ 1 │ To Kill a Mockingbird │ Fiction │
36+
│ 2 │ A Brief History of Time │ Science │
37+
└───────────────────────────────────────────────────────────────┘
38+
39+
SELECT query_text, query_hash, query_parameterized_hash
40+
FROM system.query_log
41+
WHERE query_text = 'SELECT * FROM books';
42+
43+
┌───────────────────────────────────────────────────────────────────────────────────────────┐
44+
│ query_text │ query_hash │ query_parameterized_hash │
45+
├─────────────────────┼──────────────────────────────────┼──────────────────────────────────┤
46+
│ SELECT * FROM books │ 7e612be4897104109449c74d3970c9e7 │ 7e612be4897104109449c74d3970c9e7 │
47+
│ SELECT * FROM books │ 7e612be4897104109449c74d3970c9e7 │ 7e612be4897104109449c74d3970c9e7 │
48+
└───────────────────────────────────────────────────────────────────────────────────────────┘
49+
```
50+
51+
## Examples
52+
53+
Suppose we have a table containing the following rows:
54+
55+
```sql
56+
SELECT * FROM books;
57+
58+
┌───────────────────────────────────────────────────────────────┐
59+
│ id │ title │ genre │
60+
├──────────────────┼─────────────────────────┼──────────────────┤
61+
│ 1 │ To Kill a Mockingbird │ Fiction │
62+
│ 2 │ A Brief History of Time │ Science │
63+
└───────────────────────────────────────────────────────────────┘
64+
```
65+
66+
The following queries would share the same hash values:
67+
68+
```sql
69+
SELECT * FROM books WHERE id = 1;
70+
SELECT * FROM books WHERE id = 1;
71+
```
72+
73+
To check them out:
74+
75+
```sql
76+
SELECT query_text, query_hash, query_parameterized_hash
77+
FROM system.query_log
78+
WHERE query_text = 'SELECT * FROM books WHERE id = 1'
79+
OR query_text = 'SELECT * FROM books WHERE id = 1';
80+
81+
┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
82+
│ query_text │ query_hash │ query_parameterized_hash │
83+
├──────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────┤
84+
│ SELECT * FROM books WHERE id = 1 │ ae040c4b3a9388c75e10be76ba407b17 │ b68f516c17d3c15b2c070e4af528464c │
85+
│ SELECT * FROM books WHERE id = 1 │ ae040c4b3a9388c75e10be76ba407b17 │ b68f516c17d3c15b2c070e4af528464c │
86+
│ SELECT * FROM books WHERE id = 1 │ ae040c4b3a9388c75e10be76ba407b17 │ b68f516c17d3c15b2c070e4af528464c │
87+
│ SELECT * FROM books WHERE id = 1 │ ae040c4b3a9388c75e10be76ba407b17 │ b68f516c17d3c15b2c070e4af528464c │
88+
└────────────────────────────────────────────────────────────────────────────────────────────────────────┘
89+
```
90+
91+
The following queries share the same `query_parameterized_hash` value:
92+
93+
```sql
94+
SELECT * FROM books WHERE id = 1;
95+
SELECT * FROM books WHERE id = 2;
96+
97+
SELECT query_text, query_hash, query_parameterized_hash
98+
FROM system.query_log
99+
WHERE query_text = 'SELECT * FROM books WHERE id = 1'
100+
OR query_text = 'SELECT * FROM books WHERE id = 2';
101+
102+
┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
103+
│ query_text │ query_hash │ query_parameterized_hash │
104+
├──────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────┤
105+
│ SELECT * FROM books WHERE id = 1 │ ae040c4b3a9388c75e10be76ba407b17 │ b68f516c17d3c15b2c070e4af528464c │
106+
│ SELECT * FROM books WHERE id = 1 │ ae040c4b3a9388c75e10be76ba407b17 │ b68f516c17d3c15b2c070e4af528464c │
107+
│ SELECT * FROM books WHERE id = 1 │ ae040c4b3a9388c75e10be76ba407b17 │ b68f516c17d3c15b2c070e4af528464c │
108+
│ SELECT * FROM books WHERE id = 1 │ ae040c4b3a9388c75e10be76ba407b17 │ b68f516c17d3c15b2c070e4af528464c │
109+
│ SELECT * FROM books WHERE id = 2 │ 26f135b4936d6a21922074861e5180a4 │ b68f516c17d3c15b2c070e4af528464c │
110+
│ SELECT * FROM books WHERE id = 2 │ 26f135b4936d6a21922074861e5180a4 │ b68f516c17d3c15b2c070e4af528464c │
111+
└────────────────────────────────────────────────────────────────────────────────────────────────────────┘
112+
```

‎docs/en/sql-reference/00-sql-reference/20-system-tables/system-query-log.md

Lines changed: 37 additions & 97 deletions
Original file line numberDiff line numberDiff line change
@@ -5,75 +5,44 @@ title: system.query_log
55
A read-only in-memory table stores all the query logs.
66

77

8-
## Columns
8+
## Examples
99

1010
```sql
11-
CREATE TABLE `query_log` (
12-
`log_type` TINYINT,
13-
`handler_type` VARCHAR,
14-
`tenant_id` VARCHAR,
15-
`cluster_id` VARCHAR,
16-
`sql_user` VARCHAR,
17-
`sql_user_quota` VARCHAR,
18-
`sql_user_privileges` VARCHAR,
19-
`query_id` VARCHAR,
20-
`query_kind` VARCHAR,
21-
`query_text` VARCHAR,
22-
`event_date` DATE,
23-
`event_time` TIMESTAMP,
24-
`current_database` VARCHAR,
25-
`databases` VARCHAR,
26-
`tables` VARCHAR,
27-
`columns` VARCHAR,
28-
`projections` VARCHAR,
29-
`written_rows` BIGINT UNSIGNED,
30-
`written_bytes` BIGINT UNSIGNED,
31-
`written_io_bytes` BIGINT UNSIGNED,
32-
`written_io_bytes_cost_ms` BIGINT UNSIGNED,
33-
`scan_rows` BIGINT UNSIGNED,
34-
`scan_bytes` BIGINT UNSIGNED,
35-
`scan_io_bytes` BIGINT UNSIGNED,
36-
`scan_io_bytes_cost_ms` BIGINT UNSIGNED,
37-
`scan_partitions` BIGINT UNSIGNED,
38-
`total_partitions` BIGINT UNSIGNED,
39-
`result_rows` BIGINT UNSIGNED,
40-
`result_bytes` BIGINT UNSIGNED,
41-
`cpu_usage` INT UNSIGNED,
42-
`memory_usage` BIGINT UNSIGNED,
43-
`client_info` VARCHAR,
44-
`client_address` VARCHAR,
45-
`exception_code` INT,
46-
`exception_text` VARCHAR,
47-
`stack_trace` VARCHAR,
48-
`server_version` VARCHAR,
49-
`session_settings` VARCHAR,
50-
`extra` VARCHAR
51-
)
52-
```
53-
54-
## Examples
11+
SELECT * FROM system.query_log LIMIT 1;
5512

56-
```
57-
*************************** 4. row ***************************
13+
-[ RECORD 1 ]-----------------------------------
5814
log_type: 1
59-
handler_type: MySQL
60-
tenant_id: admin
61-
cluster_id:
15+
log_type_name: Start
16+
handler_type: HTTPQuery
17+
tenant_id: default
18+
cluster_id: default
19+
node_id: zv3rra2vO7f9WUgqIu8WH
6220
sql_user: root
6321
sql_user_quota: UserQuota<cpu:0,mem:0,store:0>
64-
sql_user_privileges: GRANT ALL ON *.*, ROLES: []
65-
query_id: eda2a82b-3667-4ffb-b436-953785178c39
66-
query_kind: Query
67-
query_text: select avg(number) from numbers(1000000)
68-
event_date: 2022-09-08
69-
event_time: 2022-09-08 03:32:39.517
22+
sql_user_privileges: GRANT ALL ON *.*, ROLES: ["account_admin"]
23+
query_id: 7e03cd7a-36fa-463d-afe4-041da4092c45
24+
query_kind: Other
25+
query_text: SHOW TABLES
26+
query_hash: 9c36cac1372650b703400c60dd29042c
27+
query_parameterized_hash: 9c36cac1372650b703400c60dd29042c
28+
event_date: 2024-07-30
29+
event_time: 2024-07-30 04:27:29.296596
30+
query_start_time: 2024-07-30 04:27:29.252170
31+
query_duration_ms: 0
32+
query_queued_duration_ms: 0
7033
current_database: default
7134
databases:
7235
tables:
7336
columns:
7437
projections:
7538
written_rows: 0
7639
written_bytes: 0
40+
join_spilled_rows: 0
41+
join_spilled_bytes: 0
42+
agg_spilled_rows: 0
43+
agg_spilled_bytes: 0
44+
group_by_spilled_rows: 0
45+
group_by_spilled_bytes: 0
7746
written_io_bytes: 0
7847
written_io_bytes_cost_ms: 0
7948
scan_rows: 0
@@ -82,50 +51,21 @@ written_io_bytes_cost_ms: 0
8251
scan_io_bytes_cost_ms: 0
8352
scan_partitions: 0
8453
total_partitions: 0
85-
86-
, skip_header=0, sql_dialect=PostgreSQL, storage_read_buffer_size=1048576, timezone=UTC, unquoted_ident_case_sensitive=0, scope: SESSION
87-
extra:
88-
89-
90-
*************************** 5. row ***************************
91-
log_type: 2
92-
handler_type: MySQL
93-
tenant_id: admin
94-
cluster_id:
95-
sql_user: root
96-
sql_user_quota: UserQuota<cpu:0,mem:0,store:0>
97-
sql_user_privileges: GRANT ALL ON *.*, ROLES: []
98-
query_id: eda2a82b-3667-4ffb-b436-953785178c39
99-
query_kind: Query
100-
query_text: select avg(number) from numbers(1000000)
101-
event_date: 2022-09-08
102-
event_time: 2022-09-08 03:32:39.519
103-
current_database: default
104-
databases:
105-
tables:
106-
columns:
107-
projections:
108-
written_rows: 0
109-
written_bytes: 0
110-
written_io_bytes: 0
111-
written_io_bytes_cost_ms: 0
112-
scan_rows: 1000000
113-
scan_bytes: 8000000
114-
scan_io_bytes: 0
115-
scan_io_bytes_cost_ms: 0
116-
scan_partitions: 0
117-
total_partitions: 0
118-
result_rows: 1
119-
result_bytes: 9
120-
cpu_usage: 24
54+
result_rows: 0
55+
result_bytes: 0
56+
cpu_usage: 8
12157
memory_usage: 0
58+
bytes_from_remote_disk: 0
59+
bytes_from_local_disk: 0
60+
bytes_from_memory: 0
12261
client_info:
123-
client_address: 127.0.0.1:53304
62+
client_address: 192.168.65.1
63+
user_agent: bendsql/0.19.2-1e338e1
12464
exception_code: 0
12565
exception_text:
12666
stack_trace:
127-
server_version:
128-
session_settings: compression=None, empty_as_default=1, field_delimiter=,, flight_client_timeout=60, group_by_two_level_threshold=10000, max_block_size=10000, max_execute_time_in_seconds=0, max_threads=24, quoted_ident_case_sensitive=1, record_delimiter=
129-
, skip_header=0, sql_dialect=PostgreSQL, storage_read_buffer_size=1048576, timezone=UTC, unquoted_ident_case_sensitive=0, scope: SESSION
67+
server_version: v1.2.583-nightly-03c09a571d(rust-1.81.0-nightly-2024-07-25T22:12:15.571684726Z)
68+
session_settings: acquire_lock_timeout=30, aggregate_spilling_bytes_threshold_per_proc=0, aggregate_spilling_memory_ratio=60, auto_compaction_imperfect_blocks_threshold=25, collation=utf8, compact_max_block_selection=10000, cost_factor_aggregate_per_row=5, cost_factor_hash_table_per_row=10, cost_factor_network_per_row=50, create_query_flight_client_with_current_rt=1, data_retention_time_in_days=1, ddl_column_type_nullable=1, disable_join_reorder=0, disable_merge_into_join_reorder=0, disable_variant_check=0, efficiently_memory_group_by=0, enable_aggregating_index_scan=1, enable_auto_fix_missing_bloom_index=0, enable_bloom_runtime_filter=1, enable_cbo=1, enable_clickhouse_handler=0, enable_compact_after_write=1, enable_distributed_compact=0, enable_distributed_copy_into=1, enable_distributed_merge_into=1, enable_distributed_recluster=0, enable_distributed_replace_into=0, enable_dphyp=1, enable_dst_hour_fix=0, enable_experimental_aggregate_hashtable=1, enable_experimental_merge_into=1, enable_experimental_queries_executor=0, enable_experimental_rbac_check=1, enable_geo_create_table=0, enable_hive_parquet_predict_pushdown=1, enable_loser_tree_merge_sort=0, enable_merge_into_row_fetch=1, enable_new_copy_for_text_formats=1, enable_parquet_page_index=1, enable_parquet_prewhere=0, enable_parquet_rowgroup_pruning=1, enable_query_result_cache=0, enable_refresh_aggregating_index_after_write=1, enable_refresh_virtual_column_after_write=1, enable_replace_into_partitioning=1, enable_strict_datetime_parser=1, enable_table_lock=1, enforce_broadcast_join=0, external_server_connect_timeout_secs=10, external_server_request_batch_rows=65536, external_server_request_timeout_secs=180, flight_client_timeout=60, format_null_as_str=1, geometry_output_format=GeoJSON, group_by_shuffle_mode=before_merge, group_by_two_level_threshold=20000, hide_options_in_show_create_table=1, hive_parquet_chunk_size=16384, http_handler_result_timeout_secs=60, idle_transaction_timeout_secs=14400, inlist_to_join_threshold=1024, input_read_buffer_size=4194304, join_spilling_buffer_threshold_per_proc_mb=1024, join_spilling_bytes_threshold_per_proc=0, join_spilling_memory_ratio=60, join_spilling_partition_bits=4, lazy_read_threshold=1000, load_file_metadata_expire_hours=24, max_block_size=65536, max_cte_recursive_depth=1000, max_execute_time_in_seconds=0, max_inlist_to_or=3, max_memory_usage=6577507532, max_result_rows=0, max_set_operator_count=18446744073709551615, max_storage_io_requests=48, max_threads=8, max_vacuum_temp_files_after_query=18446744073709551615, numeric_cast_option=rounding, parquet_fast_read_bytes=16777216, parquet_max_block_size=8192, parse_datetime_ignore_remainder=1, prefer_broadcast_join=1, purge_duplicated_files_in_copy=0, query_flight_compression=LZ4, query_result_cache_allow_inconsistent=0, query_result_cache_max_bytes=1048576, query_result_cache_min_execute_secs=1, query_result_cache_ttl_secs=300, quoted_ident_case_sensitive=1, recluster_block_size=2104802410, recluster_timeout_secs=43200, replace_into_bloom_pruning_max_column_number=4, replace_into_shuffle_strategy=0, sandbox_tenant=, script_max_steps=10000, sort_spilling_batch_bytes=8388608, sort_spilling_bytes_threshold_per_proc=0, sort_spilling_memory_ratio=60, sql_dialect=PostgreSQL, statement_queued_timeout_in_seconds=0, storage_fetch_part_num=2, storage_io_max_page_bytes_for_read=524288, storage_io_min_bytes_for_seek=48, storage_read_buffer_size=1048576, table_lock_expire_secs=20, timezone=UTC, unquoted_ident_case_sensitive=0, use_parquet2=0, scope: SESSION
13069
extra:
131-
```
70+
has_profile: false
71+
```

0 commit comments

Comments
 (0)