fix: drop a deep BinaryExpr chain iteratively to avoid a stack overflow#23198
fix: drop a deep BinaryExpr chain iteratively to avoid a stack overflow#23198mdashti wants to merge 1 commit into
BinaryExpr chain iteratively to avoid a stack overflow#23198Conversation
100b14c to
f7ed788
Compare
test_stack_overflow against a deep-recursion stack overflowtest_stack_overflow cases on a large explicit stack
f7ed788 to
a46cc36
Compare
test_stack_overflow cases on a large explicit stacka46cc36 to
ff9369c
Compare
test_stack_overflow cases on a large stack
adriangb
left a comment
There was a problem hiding this comment.
i'm curious if this is a test-only concern or also a real production issue. should we implement an iterative Drop?
| std::thread::Builder::new() | ||
| .stack_size(64 * 1024 * 1024) |
There was a problem hiding this comment.
wdyt of this reply: "doesn't this defeat the purpose of the test? can we keep the smaller stack size for the planning (which is what we are trying to ensure is iterative) but move the value into a large stack thread to run drop()?"
There was a problem hiding this comment.
Thanks. You're right. It's a real production issue, so I went with the iterative Drop. The BinaryExpr children hold a BoxedExpr whose Drop tears a deep chain down off the call stack (39e458c5c), so the original test_stack_overflow cases pass with no stack trick. impl Drop for Expr wasn't viable (E0509 from the by-value destructuring in map_children), so the newtype wraps just the recursive edges. Repurposed the PR for it.
ff9369c to
5a710be
Compare
A long `OR`/`AND` chain builds an `Expr` as deep as the chain, so its recursive `Drop` overflows the stack. The `BinaryExpr` children hold a smart pointer whose `Drop` tears the chain down iteratively.
5a710be to
39e458c
Compare
test_stack_overflow cases on a large stackBinaryExpr chain iteratively to avoid a stack overflow
| std::thread::Builder::new() | ||
| .stack_size(64 * 1024 * 1024) |
There was a problem hiding this comment.
Thanks. You're right. It's a real production issue, so I went with the iterative Drop. The BinaryExpr children hold a BoxedExpr whose Drop tears a deep chain down off the call stack (39e458c5c), so the original test_stack_overflow cases pass with no stack trick. impl Drop for Expr wasn't viable (E0509 from the by-value destructuring in map_children), so the newtype wraps just the recursive edges. Repurposed the PR for it.
Which issue does this close?
None filed.
Rationale for this change
This PR makes a deep
BinaryExprchain drop iteratively instead of recursively.A left-associative chain like
a OR b OR ...parses with shallow recursion (the left side accumulates in a loop), sorecursion_limitnever bounds its length.sql_expr_to_logical_exprbuilds it iteratively (the stack machine from #1444), so planning is fine. But the resultingExpris as deep as the chain, and its recursiveDropthrough the boxed children overflows the stack. The tree-walk passes are#[recursive]-protected, so theDropis the one recursion the protection can't reach.impl Drop for Exprisn't viable: it forbids moving fields out ofExpr, whichmap_childrenand the rewriters do by value (E0509).What changes are included in this PR?
Wrap the recursive
BinaryExpr.left/rightedges in aBoxedExprnewtype whoseDropdetaches each node's children into a heap work-list before the node drops.ExprandBinaryExprkeep noDrop, so the by-value destructuring elsewhere is untouched, andBinaryExpr::newkeeps itsBox<Expr>signature. The unbounded-depth edges are only these chains; other nesting is capped byrecursion_limit.The rest of the diff is mechanical: a few consumers move
BinaryExproperands by value, so each*leftbecomes*left.into_inner().Are these changes tested?
Yes.
deep_binary_expr_chain_drops_without_overflowingbuilds a 200000-deepORchain and drops it on a normal stack. The existingtest_stack_overflowcases cover the SQL planning path, and this fix is what lets them pass without a large-stack workaround.Are there any user-facing changes?
No.