Skip to content

Commit e441021

Browse files
dqhl76dqhl76
authored
chore: add system history docs (#2343)
* chore: add system history docs * chore: add config setting for [log.history] * chore: polish details
1 parent 6590a66 commit e441021

File tree

7 files changed

+266
-0
lines changed

7 files changed

+266
-0
lines changed

‎docs/en/guides/10-deploy/04-references/02-node-config/02-query-config.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,27 @@ The following is a list of the parameters available within the [log.tracing] sec
101101
| on | Controls whether tracing is enabled. Default value is 'false' for disabled. Set to 'true' to enable tracing. |
102102
| otlp_endpoint | Specifies the OpenTelemetry Protocol (OTLP) endpoint for tracing. Defaults to `http://127.0.0.1:4317`, but you can replace it with the desired OTLP endpoint. |
103103

104+
### [log.history] Section
105+
106+
The following is a list of the parameters available within the [log.history] section:
107+
108+
109+
| Parameter | Description |
110+
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
111+
| on | Enables or disables the history logging feature. Defaults to false. Set to true to enable history tables. |
112+
| interval | Specifies the interval (in seconds) at which the history log is flushed. Defaults to 2. |
113+
| stage_name | Specifies the name of the staging area that temporarily holds log data before it is finally copied into the table. Defaults to a unique value to avoid conflicts.|
114+
| level | Sets the log level (DEBUG, TRACE, INFO, WARN, or ERROR) for history logging. Defaults to WARN. |
115+
| retention_interval| The interval (in hours) at which the retention process is triggered to check if need to clean up old data. Defaults to 24. |
116+
| tables | Specifies which history tables to enable and their retention policies. This is an array of objects, each with table_name (the name of the history table) and retention (the retention period in hours for that table). |
117+
118+
`tables` are an array of objects, each object has two parameters:
119+
| Parameter | Description |
120+
| --------- | ----------- |
121+
| table_name | The name of the history table. (currently support: `log_history`, `profile_history`, `query_history`, `login_history`) |
122+
| retention | The retention period in hours for that table. |
123+
Note: `log_history` table will be enable in default
124+
104125
## [meta] Section
105126

106127
The following is a list of the parameters available within the [meta] section:

‎docs/en/sql-reference/00-sql-reference/31-system-history-tables/_category_.json

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"label": "System History Tables"
3+
}

‎docs/en/sql-reference/00-sql-reference/31-system-history-tables/index.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
title: System History Tables
3+
---
4+
5+
# System History Tables
6+
7+
System history tables are similar to system tables, but they store data persistently.
8+
9+
These tables are organized in the `system_history` schema and can be queried using standard SQL. They provide a robust foundation for operational insights, compliance reporting, and troubleshooting by enabling you to analyze historical events and trends within your Databend environment.
10+
11+
## Available System Tables
12+
13+
| Table | Description |
14+
|-----------------------------------------------------|-----------------------------------------------------------------|
15+
| [system_history.log_history](log-history.md) | Stores raw log entries from various system components. |
16+
| [system_history.query_history](query-history.md) | Stores structured details of query execution. |
17+
| [system_history.profile_history](profile-history.md)| Stores detailed query execution profiles and statistics. |
18+
| [system_history.login_history](login-history.md) | Records information about user login events. |

‎docs/en/sql-reference/00-sql-reference/31-system-history-tables/log-history.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
---
2+
title: system_history.log_history
3+
---
4+
5+
Stores raw log entries ingested from various nodes. This table acts as the primary source for subsequent log transformations.
6+
7+
All the other log tables are derived from this table, the difference is that other log tables will do some transformations to make the data more structured.
8+
9+
```sql
10+
DESCRIBE system_history.log_history;
11+
12+
╭──────────────────────────────────────────────────────╮
13+
│ Field │ Type │ Null │ Default │ Extra │
14+
│ String │ String │ String │ String │ String │
15+
├──────────────┼───────────┼────────┼─────────┼────────┤
16+
│ timestamp │ TIMESTAMP │ YES │ NULL │ │
17+
│ path │ VARCHAR │ YES │ NULL │ │
18+
│ target │ VARCHAR │ YES │ NULL │ │
19+
│ log_level │ VARCHAR │ YES │ NULL │ │
20+
│ cluster_id │ VARCHAR │ YES │ NULL │ │
21+
│ node_id │ VARCHAR │ YES │ NULL │ │
22+
│ warehouse_id │ VARCHAR │ YES │ NULL │ │
23+
│ query_id │ VARCHAR │ YES │ NULL │ │
24+
│ message │ VARCHAR │ YES │ NULL │ │
25+
│ fields │ VARIANT │ YES │ NULL │ │
26+
│ batch_number │ BIGINT │ YES │ NULL │ │
27+
╰──────────────────────────────────────────────────────╯
28+
```
29+
30+
```sql
31+
SELECT * FROM system_history.log_history LIMIT 1;
32+
33+
*************************** 1. row ***************************
34+
timestamp: 2025-06-03 01:29:49.335455
35+
path: databend_common_meta_client::channel_manager: channel_manager.rs:86
36+
target: databend_common_meta_client::channel_manager
37+
log_level: INFO
38+
cluster_id: test_cluster
39+
node_id: CkdmtwYXLRMhJIvVzl6i11
40+
warehouse_id: NULL
41+
query_id: NULL
42+
message: MetaChannelManager done building RealClient to 127.0.0.1:9191, start handshake
43+
fields: {}
44+
batch_number: 41
45+
```

‎docs/en/sql-reference/00-sql-reference/31-system-history-tables/login-history.md

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
---
2+
title: system_history.login_history
3+
---
4+
5+
Records all login attempts in the system, including successful and failed login attempts. This table is useful for auditing user access and troubleshooting authentication issues.
6+
7+
Login successful example:
8+
```sql
9+
SELECT * FROM system_history.login_history LIMIT 1;
10+
11+
*************************** 1. row ***************************
12+
event_time: 2025-06-03 06:04:57.353108
13+
handler: HTTP
14+
event_type: LoginSuccess
15+
connection_uri: /query
16+
auth_type: Password
17+
user_name: root
18+
client_ip: 127.0.0.1
19+
user_agent: bendsql/0.26.2-unknown
20+
session_id: 9a3ba9d8-44d9-49ca-9446-501deaca15c9
21+
node_id: 765ChL6Ra949Ioeb5LrTs
22+
error_message:
23+
```
24+
25+
Login failed example:
26+
```sql
27+
SELECT * FROM system_history.login_history LIMIT 1;
28+
29+
*************************** 1. row ***************************
30+
event_time: 2025-06-03 06:07:32.512021
31+
handler: MySQL
32+
event_type: LoginFailed
33+
connection_uri:
34+
auth_type: Password
35+
user_name: root1
36+
client_ip: 127.0.0.1:62050
37+
user_agent:
38+
session_id: 4fb87258-865a-402c-8680-e3be1e01b4e6
39+
node_id: 765ChL6Ra949Ioeb5LrTs
40+
error_message: UnknownUser. Code: 2201, Text = User 'root1'@'%' does not exist..
41+
```

‎docs/en/sql-reference/00-sql-reference/31-system-history-tables/profile-history.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
---
2+
title: system_history.profile_history
3+
---
4+
Stores detailed execution profiles for SQL queries in Databend. Each entry provides performance metrics and execution statistics, allowing users to analyze and optimize query performance.
5+
6+
The `profiles` field contains a JSON object with detailed information.
7+
8+
```sql
9+
DESCRIBE system_history.profile_history;
10+
11+
╭─────────────────────────────────────────────────────────╮
12+
│ Field │ Type │ Null │ Default │ Extra │
13+
│ String │ String │ String │ String │ String │
14+
├─────────────────┼───────────┼────────┼─────────┼────────┤
15+
│ timestamp │ TIMESTAMP │ YES │ NULL │ │
16+
│ query_id │ VARCHAR │ YES │ NULL │ │
17+
│ profiles │ VARIANT │ YES │ NULL │ │
18+
│ statistics_desc │ VARIANT │ YES │ NULL │ │
19+
╰─────────────────────────────────────────────────────────╯
20+
```
21+
22+
The `profiles` field can be used to extract specific information. For example, to get the `OutputRows` value for every physical plan, the following query can be used:
23+
```sql
24+
SELECT jq('[.[] | {id, output_rows: .statistics[4]}]', profiles ) AS result FROM system_history.profile_history LIMIT 1;
25+
26+
*************************** 1. row ***************************
27+
result: [{"id":0,"output_rows":1},{"id":3,"output_rows":8},{"id":1,"output_rows":1},{"id":2,"output_rows":1}]
28+
```

‎docs/en/sql-reference/00-sql-reference/31-system-history-tables/query-history.md

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
---
2+
title: system_history.query_history
3+
---
4+
5+
Records detailed logs of all SQL query executions in Databend. For each query, two entries are generated: one when the query starts and another when it finishes. This table is valuable for monitoring query activity, auditing user actions, and analyzing performance metrics.
6+
7+
```sql
8+
SELECT * FROM system_history.query_history LIMIT 2;
9+
10+
*************************** 1. row ***************************
11+
log_type: 1
12+
log_type_name: Start
13+
handler_type: HTTPQuery
14+
tenant_id: test_tenant
15+
cluster_id: test_cluster
16+
node_id: CkdmtwYXLRMhJIvVzl6i11
17+
sql_user: root
18+
sql_user_quota: NULL
19+
sql_user_privileges: NULL
20+
query_id: 55758078-e85d-4fac-95fc-0203ba0d68b2
21+
query_kind: Query
22+
query_text: SELECT * FROM system_history.log_history LIMIT 1
23+
query_hash: e67623933f7ca8c66be47f5645f02367
24+
query_parameterized_hash: 100446ab318d76cf2bfd2aaecff07305
25+
event_date: 2025-06-03
26+
event_time: 2025-06-03 01:32:49.864759
27+
query_start_time: 2025-06-03 01:32:49.843950
28+
query_duration_ms: 0
29+
query_queued_duration_ms: 0
30+
current_database: default
31+
written_rows: 0
32+
written_bytes: 0
33+
join_spilled_rows: 0
34+
join_spilled_bytes: 0
35+
agg_spilled_rows: 0
36+
agg_spilled_bytes: 0
37+
group_by_spilled_rows: 0
38+
group_by_spilled_bytes: 0
39+
written_io_bytes: 0
40+
written_io_bytes_cost_ms: 0
41+
scan_rows: 0
42+
scan_bytes: 0
43+
scan_io_bytes: 0
44+
scan_io_bytes_cost_ms: 0
45+
scan_partitions: 0
46+
total_partitions: 0
47+
result_rows: 0
48+
result_bytes: 0
49+
bytes_from_remote_disk: 0
50+
bytes_from_local_disk: 0
51+
bytes_from_memory: 0
52+
client_address: 127.0.0.1
53+
user_agent: bendsql/0.26.2-unknown
54+
exception_code: 0
55+
exception_text:
56+
server_version: v1.2.747-nightly-1ed0ae8571(rust-1.88.0-nightly-2025-06-03T01:16:19.413296000Z)
57+
query_tag:
58+
has_profile: NULL
59+
peek_memory_usage: {}
60+
*************************** 2. row ***************************
61+
log_type: 2
62+
log_type_name: Finish
63+
handler_type: HTTPQuery
64+
tenant_id: test_tenant
65+
cluster_id: test_cluster
66+
node_id: CkdmtwYXLRMhJIvVzl6i11
67+
sql_user: root
68+
sql_user_quota: NULL
69+
sql_user_privileges: NULL
70+
query_id: 55758078-e85d-4fac-95fc-0203ba0d68b2
71+
query_kind: Query
72+
query_text: SELECT * FROM system_history.log_history LIMIT 1
73+
query_hash: e67623933f7ca8c66be47f5645f02367
74+
query_parameterized_hash: 100446ab318d76cf2bfd2aaecff07305
75+
event_date: 2025-06-03
76+
event_time: 2025-06-03 01:32:49.878900
77+
query_start_time: 2025-06-03 01:32:49.843950
78+
query_duration_ms: 34
79+
query_queued_duration_ms: 0
80+
current_database: default
81+
written_rows: 0
82+
written_bytes: 0
83+
join_spilled_rows: 0
84+
join_spilled_bytes: 0
85+
agg_spilled_rows: 0
86+
agg_spilled_bytes: 0
87+
group_by_spilled_rows: 0
88+
group_by_spilled_bytes: 0
89+
written_io_bytes: 0
90+
written_io_bytes_cost_ms: 0
91+
scan_rows: 38
92+
scan_bytes: 30922
93+
scan_io_bytes: 0
94+
scan_io_bytes_cost_ms: 0
95+
scan_partitions: 0
96+
total_partitions: 0
97+
result_rows: 1
98+
result_bytes: 1156
99+
bytes_from_remote_disk: 7580
100+
bytes_from_local_disk: 0
101+
bytes_from_memory: 0
102+
client_address: 127.0.0.1
103+
user_agent: bendsql/0.26.2-unknown
104+
exception_code: 0
105+
exception_text:
106+
server_version: v1.2.747-nightly-1ed0ae8571(rust-1.88.0-nightly-2025-06-03T01:16:19.413296000Z)
107+
query_tag:
108+
has_profile: NULL
109+
peek_memory_usage: {"CkdmtwYXLRMhJIvVzl6i11":571809}
110+
```

0 commit comments

Comments
 (0)