Skip to content

perf: move query params to POST body (fix #454) - #455

Open
cjrh wants to merge 5 commits into
ClickHouse:mainfrom
cjrh:perf-move-query-params-post-body
Open

perf: move query params to POST body (fix #454)#455
cjrh wants to merge 5 commits into
ClickHouse:mainfrom
cjrh:perf-move-query-params-post-body

Conversation

@cjrh

@cjrh cjrh commented Aug 7, 2026

Copy link
Copy Markdown

Summary

Fixes #454.

Server-side query parameters supplied with [Query::param()] are now sent as multipart/form-data fields in the POST request body instead of URI query parameters. This avoids URI-length limits when queries bind many parameters or large values.

Candidate CHANGELOG entry

 * Server-side query parameters supplied with [`Query::param()`] are now sent as              
     `multipart/form-data` fields in the POST request body instead of URI query                 
     parameters. This avoids URI-length limits when queries bind many parameters                
     or large values.                                                                           
     * The `Query::param()` API and parameter serialization are unchanged; callers              
       should see the same ClickHouse query semantics.                                          
     * Settings named `param_*` follow the same behavior.                                       
     * For parameterized queries, clickhouse-rs now controls HTTP body-framing                  
       headers: it sets the multipart `Content-Type` and removes caller-provided                
       `Content-Length` and `Transfer-Encoding`. Applications that set, sign, or                
       otherwise depend on those headers must update their request configuration.     

Note the point about the HTTP headers. For users setting custom headers, this is a breaking change.

Implementation Notes

Query::param() continues to serialize values with the existing parameter serializer and records them as param_<name> settings. While building a query request, clickhouse-rs now separates those param_* settings from ordinary ClickHouse settings:

  • Ordinary settings and roles remain URI query parameters.
  • When parameters are present, the request becomes a multipart/form-data POST body containing a query field followed by one field per param_<name>.
  • Parameter fields are sorted for deterministic request construction. The multipart boundary is chosen to avoid collisions with the query or parameter values, and parameter names are validated before being used as form-field names.
  • clickhouse-rs sets the multipart Content-Type and removes caller-supplied Content-Length and Transfer-Encoding, allowing the HTTP body implementation to frame the request correctly.
  • Parameter-free queries retain the previous raw-SQL POST body and explicit Content-Length behavior.

Unit tests cover multipart field construction, boundary collisions, parameter validation, header handling, compression, and preservation of ordinary settings. Integration coverage includes parameterized queries with large values that would otherwise approach URI-length limits.

The branch also fixes the opt-in uuid RowBinary adapter for Option<Vec<Uuid>>. Its optional wrapper now delegates Some values through the crate's UUID-vector serializer/deserializer, preserving ClickHouse's binary UUID representation. Regression tests cover both nullable states.

On the UUID RowBinary change

I use clickhouse-rs via diesel-clickhouse, and I use UUIDs a lot which is how I found this.

This is a targeted fix for structs that map a Rust Option<Vec<uuid::Uuid>> to a ClickHouse Nullable(Array(UUID)) column using #[serde(with = "clickhouse::serde::uuid_vec::option")].

For RowBinary, a non-null Nullable(...) value starts with a 0 byte; NULL is represented by 1. A present Array(UUID) then contains its length, followed by each UUID encoded as two little-endian u64 values.

Previously, serializing Some(vec) delegated directly to the UUID-vector adapter. That adapter correctly encoded the array and UUID elements, but bypassed Serde's outer Option handling, so it omitted the required leading 0 non-null marker. The generated bytes were therefore not valid Nullable(Array(UUID)) RowBinary.

The old deserializer had the inverse problem: it handled the outer Option but delegated a present vector to Serde's generic Vec<Uuid> implementation, rather than the crate's UUID-vector adapter. It consequently could not deserialize valid non-null Nullable(Array(UUID)) bytes.

The fix preserves the outer Option layer and delegates only its present inner value to the UUID-vector adapter. The regression test verifies exact bytes and round-trips both Some(vec![...]) and None.

None was already encoded correctly. This change is behind the opt-in uuid feature and is independent of the multipart query-parameter change. I do need it for diesel-clickhouse though.

Checklist

Delete items not relevant to your PR:

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided so that we can include it in CHANGELOG later

@CLAassistant

CLAassistant commented Aug 7, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@cjrh
cjrh force-pushed the perf-move-query-params-post-body branch from d4cb50d to 620d8bd Compare August 7, 2026 23:49
@mshustov
mshustov requested a lite review from Copilot August 14, 2026 15:54

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR changes how ClickHouse server-side query parameters (param_*, including those created via Query::param()) are transported: instead of being sent as URI query parameters, they are now sent as multipart/form-data fields in the POST request body to avoid URI length limits. It also fixes RowBinary serde for Option<Vec<Uuid>> behind the uuid feature, and strengthens CI/test coverage around these behaviors.

Changes:

  • Send param_* settings as multipart form fields in the POST body (while keeping ordinary settings/roles in the URI query string), and control framing headers for multipart requests.
  • Add unit/integration tests for multipart request construction, header handling, and large parameter payloads.
  • Fix uuid_vec::option RowBinary serde so Option<Vec<Uuid>> correctly preserves the Nullable marker and uses the UUID adapter on deserialization (plus regression tests).

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/it/query.rs Adds integration coverage for ordinary settings + params, and for large parameter payloads that would previously risk URI-length issues.
src/test/handlers.rs Adds a test-only handler to record full HTTP requests for transport-level assertions.
src/serde.rs Fixes uuid_vec::option serde to preserve Nullable encoding and use the crate’s UUID-vector adapter.
src/rowbinary/tests.rs Adds regression tests for RowBinary round-tripping of Nullable(Array(UUID)) via Option<Vec<Uuid>>.
src/request_body.rs Makes RequestBody::full accept Into<Bytes> to support multipart bodies efficiently.
src/query.rs Implements multipart request-body construction for param_*, validates names, selects a non-colliding boundary, and adds transport/unit tests.
src/lib.rs Updates Client::with_header docs to reflect new header framing behavior.
docs/index.mdx Removes outdated documentation claiming server-side parameter binding is unsupported.
.github/workflows/ci.yml Expands CI coverage (clippy with uuid feature) and tests against an additional minimum ClickHouse version.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/query.rs Outdated
Comment thread src/lib.rs Outdated
@cjrh
cjrh force-pushed the perf-move-query-params-post-body branch from 620d8bd to 08a5fff Compare September 4, 2026 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move query URI param_* bind values to the POST body

3 participants