Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add dynamic parameter expression #780

Merged
merged 19 commits into from
Feb 14, 2025
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions proto/substrait/algebra.proto
Original file line number Diff line number Diff line change
Expand Up @@ -957,6 +957,7 @@ message Expression {
Cast cast = 11;
Subquery subquery = 12;
Nested nested = 13;
DynamicParameter dynamic_parameter = 14;

// deprecated: enum literals are only sensible in the context of
// function arguments, for which FunctionArgument should now be
Expand Down Expand Up @@ -1611,6 +1612,16 @@ message Expression {
}
}

// Expression that represents a dynamic parameter.
// Dynamic parameters are identified by a surrogate key within a plan.
message DynamicParameter {
// The type of the dynamic parameter.
Type type = 1;

// The surrogate key used within a plan to reference a specific parameter binding.
uint32 parameter_anchor = 2;
jcamachor marked this conversation as resolved.
Show resolved Hide resolved
}

// The description of a field to sort on (including the direction of sorting and null semantics)
message SortField {
Expression expr = 1;
Expand Down
5 changes: 5 additions & 0 deletions proto/substrait/plan.proto
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,11 @@ message Plan {
// unused. In many cases, a consumer may be able to work with a plan even if
// one or more message types defined here are unknown.
repeated string expected_type_urls = 5;

// An optional list of bindings for dynamic parameters used in this plan.
// The index of each entry in the list corresponds to the parameter_anchor
// of a DynamicParameter expression, mapping it to its runtime value.
repeated Expression.Literal parameter_bindings = 7;
jcamachor marked this conversation as resolved.
Show resolved Hide resolved
}

// This message type can be used to deserialize only the version of a Substrait
Expand Down
1 change: 1 addition & 0 deletions site/docs/expressions/_config
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ arrange:
- table_functions.md
- user_defined_functions.md
- embedded_functions.md
- dynamic_parameters.md
12 changes: 12 additions & 0 deletions site/docs/expressions/dynamic_parameters.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Dynamic Parameter Expression

The dynamic parameter expression represents a placeholder within an expression whose value is determined at runtime.
This is particularly useful for parameterized queries where certain values are not known until execution.
Additionally, using dynamic parameters can enable other use cases, such as sharing execution plans without embedding sensitive information.

A dynamic parameter expression includes the following properties:

| Property | Description | Required |
|--------------------|-------------------------------------------------------------------------------|----------|
| `type` | Specifies the expected data type of the dynamic parameter. | Yes |
| `parameter_anchor` | A surrogate key used within a plan to reference a specific parameter binding. | Yes |
8 changes: 8 additions & 0 deletions site/docs/relations/basics.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,14 @@ A guarantee that data output from this operation is provided with a sort order.
| Clustered | Ensures that all equal values are coalesced (but no ordering between values is defined). E.g. for values 1,2,3,1,2,3, output could be any of the following: 1,1,2,2,3,3 or 1,1,3,3,2,2 or 2,2,1,1,3,3 or 2,2,3,3,1,1 or 3,3,1,1,2,2 or 3,3,2,2,1,1. | N/A, may appear anywhere but will be coalesced. |


## Dynamic Parameters

Dynamic parameters act as placeholders within expressions, with their values determined at runtime.
This is particularly useful for *parameterized queries*, *query reuse*, and *execution plan sharing*, where specific values are not known ahead of time or need to be replaced dynamically during execution.

### Dynamic Parameter Bindings
In a plan's relational tree, dynamic parameters are referenced in expressions without predefined values.
Their values are supplied via parameter bindings, which can either be included in the plan's optional bindings field or supplied separately at execution time.



Expand Down
Loading