1
- # How Databend Optimizer Works
1
+ ---
2
+ title : How Databend Optimizer Works
3
+ ---
2
4
3
5
## Core Concepts
4
6
@@ -18,12 +20,14 @@ Databend's query optimizer is built on several key abstractions that work togeth
18
20
Databend collects and uses these statistics to guide optimization decisions:
19
21
20
22
** Table Statistics:**
23
+
21
24
- ` num_rows ` : Number of rows in the table
22
25
- ` data_size ` : Size of the table data in bytes
23
26
- ` number_of_blocks ` : Number of storage blocks
24
27
- ` number_of_segments ` : Number of segments
25
28
26
29
** Column Statistics:**
30
+
27
31
- ` min ` : Minimum value in the column
28
32
- ` max ` : Maximum value in the column
29
33
- ` null_count ` : Number of null values
@@ -133,12 +137,14 @@ Databend's query optimizer passes through four distinct phases to transform SQL
133
137
** 1. Subquery Decorrelation (SubqueryDecorrelatorOptimizer)**
134
138
135
139
** SQL Example:**
140
+
136
141
``` sql
137
142
SELECT * FROM customers c
138
143
WHERE c .total_orders > (SELECT AVG (total_orders) FROM customers WHERE region = c .region )
139
144
```
140
145
141
146
** Before:**
147
+
142
148
```
143
149
Filter (c.total_orders > Subquery)
144
150
└─ Scan (customers as c)
@@ -149,6 +155,7 @@ Filter (c.total_orders > Subquery)
149
155
```
150
156
151
157
** After:**
158
+
152
159
```
153
160
# Correlated subquery transformed into join operation
154
161
Join (c.region = r.region)
@@ -165,18 +172,21 @@ Filter (c.total_orders > r.avg_total)
165
172
** 2. Statistics-based Aggregate Optimization (RuleStatsAggregateOptimizer)**
166
173
167
174
** SQL Example:**
175
+
168
176
``` sql
169
177
SELECT MIN (price) FROM products
170
178
```
171
179
172
180
** Before:**
181
+
173
182
```
174
183
Aggregate (MIN(price))
175
184
└─ EvalScalar
176
185
└─ Scan (products)
177
186
```
178
187
179
188
** After:**
189
+
180
190
```
181
191
# MIN aggregate replaced with pre-computed value from statistics
182
192
EvalScalar (price_min)
@@ -188,18 +198,21 @@ EvalScalar (price_min)
188
198
** 3. Statistics Collection (CollectStatisticsOptimizer)**
189
199
190
200
** SQL Example:**
201
+
191
202
``` sql
192
203
SELECT * FROM orders WHERE region = ' Asia'
193
204
```
194
205
195
206
** Before:**
207
+
196
208
```
197
209
Filter (region = 'Asia')
198
210
└─ Scan (orders)
199
211
[No statistics]
200
212
```
201
213
202
214
** After:**
215
+
203
216
```
204
217
Filter (region = 'Asia')
205
218
└─ Scan (orders)
@@ -216,11 +229,13 @@ Filter (region = 'Asia')
216
229
** 4. Aggregate Normalization (RuleNormalizeAggregateOptimizer)**
217
230
218
231
** SQL Example:**
232
+
219
233
``` sql
220
234
SELECT COUNT (id), COUNT (* ), COUNT (DISTINCT region) FROM orders GROUP BY region
221
235
```
222
236
223
237
** Before:**
238
+
224
239
```
225
240
Aggregate (
226
241
GROUP BY [region],
@@ -232,6 +247,7 @@ Aggregate (
232
247
```
233
248
234
249
** After:**
250
+
235
251
```
236
252
# Optimized aggregates
237
253
EvalScalar (COUNT(*) AS count_id, COUNT(*) AS count_star)
@@ -244,18 +260,21 @@ EvalScalar (COUNT(*) AS count_id, COUNT(*) AS count_star)
244
260
```
245
261
246
262
** What it does:** Optimizes aggregate functions by:
247
- 1 . Rewriting COUNT(non-nullable) to COUNT(* )
248
- 2 . Reusing a single COUNT(* ) for multiple count expressions
263
+
264
+ 1 . Rewriting COUNT(non-nullable) to COUNT(\* )
265
+ 2 . Reusing a single COUNT(\* ) for multiple count expressions
249
266
3 . Eliminating DISTINCT when counting columns that are already in GROUP BY
250
267
251
268
** 5. Filter Pull-up (PullUpFilterOptimizer)**
252
269
253
270
** SQL Example:**
271
+
254
272
``` sql
255
273
SELECT * FROM orders o JOIN customers c ON o .customer_id = c .id WHERE o .region = ' Asia' AND c .status = ' active'
256
274
```
257
275
258
276
** Before:**
277
+
259
278
```
260
279
Filter (c.status = 'active')
261
280
└─ Filter (o.region = 'Asia')
@@ -265,6 +284,7 @@ Filter (c.status = 'active')
265
284
```
266
285
267
286
** After:**
287
+
268
288
```
269
289
# Filters pulled up to the top
270
290
Filter (o.region = 'Asia' AND c.status = 'active' AND o.customer_id = c.id)
@@ -284,17 +304,20 @@ Filter (o.region = 'Asia' AND c.status = 'active' AND o.customer_id = c.id)
284
304
#### Filter Pushdown Rules
285
305
286
306
** SQL Example:**
307
+
287
308
``` sql
288
309
SELECT * FROM orders WHERE region = ' Asia'
289
310
```
290
311
291
312
** Before:**
313
+
292
314
```
293
315
Filter (region = 'Asia')
294
316
└─ Scan (orders)
295
317
```
296
318
297
319
** After (PushDownFilterScan rule):**
320
+
298
321
```
299
322
# Filter pushed down to scan layer
300
323
Scan (orders, pushdown_predicates=[region = 'Asia'])
@@ -305,18 +328,21 @@ Scan (orders, pushdown_predicates=[region = 'Asia'])
305
328
#### Limit Pushdown Rules
306
329
307
330
** SQL Example:**
331
+
308
332
``` sql
309
333
SELECT * FROM orders ORDER BY order_date LIMIT 10
310
334
```
311
335
312
336
** Before:**
337
+
313
338
```
314
339
Limit (10)
315
340
└─ Sort (order_date)
316
341
└─ Scan (orders)
317
342
```
318
343
319
344
** After (PushDownLimitSort rule):**
345
+
320
346
```
321
347
# Limit pushed through sort
322
348
Sort (order_date)
@@ -329,17 +355,20 @@ Sort (order_date)
329
355
#### Elimination Rules
330
356
331
357
** SQL Example:**
358
+
332
359
``` sql
333
360
SELECT * FROM orders WHERE 1 = 1
334
361
```
335
362
336
363
** Before:**
364
+
337
365
```
338
366
Filter (1=1)
339
367
└─ Scan (orders)
340
368
```
341
369
342
370
** After (EliminateFilter rule):**
371
+
343
372
```
344
373
# Redundant filter removed
345
374
Scan (orders)
@@ -350,11 +379,13 @@ Scan (orders)
350
379
** 7. Aggregate Splitting (RecursiveRuleOptimizer - SplitAggregate)**
351
380
352
381
** SQL Example:**
382
+
353
383
``` sql
354
384
SELECT region, SUM (amount) FROM orders GROUP BY region
355
385
```
356
386
357
387
** Before:**
388
+
358
389
```
359
390
# Single-phase aggregation (mode: Initial)
360
391
Aggregate (
@@ -366,6 +397,7 @@ Aggregate (
366
397
```
367
398
368
399
** After:**
400
+
369
401
```
370
402
# Two-phase aggregation
371
403
Aggregate (
@@ -388,11 +420,13 @@ Aggregate (
388
420
** 8. Join Order Optimization (DPhpyOptimizer)**
389
421
390
422
** SQL Example:**
423
+
391
424
``` sql
392
425
SELECT * FROM orders o JOIN customers c ON o .customer_id = c .id JOIN products p ON o .product_id = p .id WHERE c .region = ' Asia'
393
426
```
394
427
395
428
** Before (original order):**
429
+
396
430
```
397
431
Join
398
432
├─ Join
402
436
```
403
437
404
438
** After (optimized order):**
439
+
405
440
```
406
441
# Optimized join order based on cost estimation
407
442
Join
@@ -424,18 +459,21 @@ This optimizer is particularly important for queries involving multiple joins, w
424
459
** 9. Single Join to Inner Join Conversion (SingleToInnerOptimizer)**
425
460
426
461
** SQL Example:**
462
+
427
463
``` sql
428
464
SELECT o.* FROM orders o LEFT SINGLE JOIN customers c ON o .customer_id = c .id
429
465
```
430
466
431
467
** Before:**
468
+
432
469
```
433
470
LeftSingleJoin (o.customer_id = c.id)
434
471
├─ Scan (orders as o)
435
472
└─ Scan (customers as c)
436
473
```
437
474
438
475
** After:**
476
+
439
477
```
440
478
# Single join converted to inner join
441
479
InnerJoin (o.customer_id = c.id)
@@ -448,11 +486,13 @@ InnerJoin (o.customer_id = c.id)
448
486
** 10. Join Condition Deduplication (DeduplicateJoinConditionOptimizer)**
449
487
450
488
** SQL Example:**
489
+
451
490
``` sql
452
491
SELECT * FROM t1, t2, t3 WHERE t1 .id = t2 .id AND t2 .id = t3 .id AND t3 .id = t1 .id
453
492
```
454
493
455
494
** Before:**
495
+
456
496
```
457
497
Join (t2.id = t3.id AND t3.id = t1.id)
458
498
├─ Scan (t3)
@@ -462,6 +502,7 @@ Join (t2.id = t3.id AND t3.id = t1.id)
462
502
```
463
503
464
504
** After:**
505
+
465
506
```
466
507
# Removed transitive join condition
467
508
Join (t2.id = t3.id)
@@ -483,18 +524,21 @@ This optimization reduces the number of join conditions that need to be evaluate
483
524
** 11. Join Commutation (CommuteJoin Rule)**
484
525
485
526
** SQL Example:**
527
+
486
528
``` sql
487
529
SELECT * FROM orders o JOIN customers c ON o .customer_id = c .id
488
530
```
489
531
490
532
** Before (orders is larger than customers):**
533
+
491
534
```
492
535
Join (o.customer_id = c.id)
493
536
├─ Scan (orders as o) # Larger table (10M rows)
494
537
└─ Scan (customers as c) # Smaller table (100K rows)
495
538
```
496
539
497
540
** After (CommuteJoin rule applied):**
541
+
498
542
```
499
543
# Join order swapped to put smaller table on left
500
544
Join (c.id = o.customer_id)
@@ -515,6 +559,7 @@ Since Databend typically uses the right side as the build side in hash joins, th
515
559
** 12. Cost-Based Implementation Selection (CascadesOptimizer)**
516
560
517
561
** SQL Example:**
562
+
518
563
``` sql
519
564
SELECT customer_name, SUM (total_price) as total_spend
520
565
FROM customers JOIN orders ON customers .id = orders .customer_id
@@ -606,6 +651,7 @@ Costs are calculated recursively - a plan's total cost includes all its operatio
606
651
Databend's query optimizer employs a sophisticated, multi-stage pipeline to transform user SQL queries into highly efficient physical execution plans. It leverages core concepts like SExpr for plan representation, a rich set of transformation rules, detailed statistics, and a cost model to explore and evaluate various plan alternatives.
607
652
608
653
The process involves:
654
+
609
655
1 . ** Preparation:** Decorrelating subqueries and gathering necessary statistics.
610
656
2 . ** Logical Optimization:** Applying rule-based transformations (like filter pushdown, aggregate normalization) to refine the logical plan structure.
611
657
3 . ** Join Optimization:** Strategically determining the best join order and methods using techniques like dynamic programming.
0 commit comments