Keep caller spelling of header names and match names case-insensitively (#1378)

* Keep caller spelling of header names and match names case-insensitively

HTTP.jl 2.0 through 2.7 canonicalized every header name on insertion
(`providerId` became `Providerid`) because lookups compared stored names
with `==` against a canonicalized key. HTTP.jl 1.x kept the caller's
spelling and only canonicalized when `canonicalize_headers=true`, which
was off by default. Servers that treat header names as case-sensitive
data, such as Azure Service Bus custom properties, received the rewritten
names, and 2.x offered no opt-out: `canonicalize_headers=false` only
logged a "has no effect" warning.

Fix the cause instead of the spelling: every `Headers` lookup and update
(`header`, `headers`, `haskey`, `hasheader`, `setheader`, `appendheader`,
`removeheader`, `headercontains`, `header_keys`, `merge!`) now matches
names with an ASCII case fold, so stored names no longer need to be
canonical and insertion keeps them as given. HTTP/1 sends them as
spelled; HTTP/2 still lowercases. The HTTP/1 parser already
canonicalized received names; the HTTP/2 client and server decoders now
do so explicitly, so received messages look the same as before. The few
exact-name checks outside the helpers (default Accept-Encoding, HTTP/2
Host-vs-:authority, request display) fold case too.

`canonicalize_headers=true` rewrites request header names into
`canonical_header_key` form again, as in 1.x, and the keyword no longer
warns.

Fixes #1377.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep case-insensitive header enumeration linear

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Author: quinnj

CHANGELOG.md

@@ -52,6 +52,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   `max_line_bytes` may not exceed `max_header_bytes`. ([#1362])
 
 ### Changed
+- `HTTP.Headers` keeps each header name as the caller spelled it and matches
+  names case-insensitively, as HTTP.jl 1.x did. HTTP.jl 2.0 through 2.7
+  rewrote every name into `Content-Type` form on insertion, so a request header
+  such as `providerId` went out as `Providerid`. That breaks servers that treat
+  header names as case-sensitive data, such as Azure Service Bus custom
+  message properties. HTTP/1 now sends names as spelled, HTTP/2 still sends
+  them in lowercase, and headers received from the network still use
+  `Content-Type` form. `canonicalize_headers=true` works again: it sends
+  request header names in `Content-Type` form, and no longer logs a
+  "has no effect" warning. ([#1377])
 - Raised the default HTTP/1 per-line limit (request/status lines and single
   header lines) from 8 KiB to 64 KiB, matching Python's `http.client`. Real
   origins send single header lines longer than 8 KiB — a 9,695-byte
@@ -969,3 +979,4 @@ See changes for 0.9.15: this release is equivalent to 0.9.15 with [#752] reverte
 [#1361]: https://github.com/JuliaWeb/HTTP.jl/issues/1361
 [#1362]: https://github.com/JuliaWeb/HTTP.jl/issues/1362
 [#1371]: https://github.com/JuliaWeb/HTTP.jl/issues/1371
+[#1377]: https://github.com/JuliaWeb/HTTP.jl/issues/1377

docs/src/guides/client.md

@@ -549,9 +549,10 @@ as temporary compatibility, not as the preferred API:
 - `retry_delays` and `retry_check` should become `retry_if`, `retries`, and
   `retry_bucket`
 - `sslconfig` and `socket_type_tls` should move to transport/TLS configuration
-- `canonicalize_headers`, `detect_content_type`,
-  `observelayers`, `logerrors`, and `logtag` are accepted for compatibility
-  where possible
+- `detect_content_type`, `observelayers`, `logerrors`, and `logtag` are
+  accepted for compatibility where possible
+- `canonicalize_headers=true` sends request header names in `Content-Type`
+  form, as in 1.x; by default names are sent as spelled
 
 See the [migration guide](migration-1x.md) for before/after examples.
 

docs/src/guides/migration-1x.md

@@ -168,7 +168,12 @@ req = HTTP.Request(
 ## Headers
 
 `HTTP.Headers` is the canonical mutable header container. It preserves pair
-order and canonicalizes header keys on insertion.
+order and each header name as you spell it, as 1.x did. Lookups such as
+`HTTP.header(headers, "Content-Type")` match names in any case. Unlike 1.x,
+headers received from the network use `Content-Type` form. To send every
+request header name in that form, pass `canonicalize_headers=true`, as in 1.x.
+HTTP/2, which 2.0 uses for HTTPS when the server supports it, always sends
+names in lowercase; pass `protocol=:h1` to a server that needs exact spelling.
 
 Before:
 
@@ -676,9 +681,11 @@ abruptly:
 - `retry_delays` and `retry_check`: accepted, but use `retry_if`,
   `retries`, and `retry_bucket`
 - `sslconfig` and `socket_type_tls`: accepted, but configure the transport
-- `canonicalize_headers`, `detect_content_type`,
-  `observelayers`, `logerrors`, and `logtag`: accepted for compatibility, but
-  not the preferred 2.0 observation/configuration surface
+- `detect_content_type`, `observelayers`, `logerrors`, and `logtag`:
+  accepted for compatibility, but not the preferred 2.0
+  observation/configuration surface
+- `canonicalize_headers`: supported with its 1.x meaning. `true` sends request
+  header names in `Content-Type` form. The default sends them as spelled.
 
 Treat these as temporary migration aids. New code should use the documented
 2.0 API names.

src/http2_client.jl

@@ -1651,7 +1651,7 @@ function _decode_response_headers(headers::Vector{HeaderField})::Tuple{Int,Heade
         if _is_h2_connection_specific_header(name) || name == "transfer-encoding" || name == "te"
             throw(ProtocolError("forbidden HTTP/2 response header $(repr(name))"))
         end
-        appendheader(out, name, normalized)
+        appendheader(out, canonical_header_key(name), normalized)
     end
     status === nothing && throw(ProtocolError("missing HTTP/2 :status pseudo-header"))
     return status::Int, out
@@ -1667,7 +1667,7 @@ function _decode_h2_trailer_headers(headers::Vector{HeaderField})::Headers
         _valid_trailer_header_name(name) || throw(ProtocolError("invalid HTTP/2 trailer header $(repr(name))"))
         normalized = _normalize_strict_header_field_value(value)
         normalized === nothing && throw(ProtocolError("invalid HTTP/2 trailer field value for $(repr(name))"))
-        appendheader(out, name, normalized)
+        appendheader(out, canonical_header_key(name), normalized)
     end
     return out
 end

src/http2_server.jl

@@ -982,7 +982,7 @@ function _validate_h2_request_headers!(headers::Vector{HeaderField})::Tuple{Stri
             end
             continue
         end
-        appendheader(out_headers, name, normalized)
+        appendheader(out_headers, canonical_header_key(name), normalized)
     end
     method === nothing && throw(ProtocolError("missing HTTP/2 :method pseudo-header"))
     # RFC 9113 8.3.1: "If the :authority pseudo-header field is present, the
@@ -998,7 +998,7 @@ function _validate_h2_request_headers!(headers::Vector{HeaderField})::Tuple{Stri
     # directly rather than calling the `headers(::Headers, key)` accessor.)
     if authority !== nothing
         for (entry_name, entry_value) in out_headers.entries
-            entry_name == "Host" || continue
+            _ascii_equal_fold(entry_name, "Host") || continue
             entry_value == authority || throw(ProtocolError("HTTP/2 Host header does not match :authority"))
         end
     end

src/http_client.jl

@@ -224,7 +224,6 @@ end
 function _handle_client_compat_kwargs(;
     copyheaders=nothing,
     pool=nothing,
-    canonicalize_headers=nothing,
     detect_content_type=nothing,
     observelayers=nothing,
     retry_delays=nothing,
@@ -235,7 +234,6 @@ function _handle_client_compat_kwargs(;
     logtag=nothing,
 )::Nothing
     pool === nothing || _warn_ignored_client_compat_kw("pool")
-    canonicalize_headers === nothing || _warn_ignored_client_compat_kw("canonicalize_headers")
     detect_content_type === nothing || _warn_ignored_client_compat_kw("detect_content_type")
     observelayers === nothing || _warn_ignored_client_compat_kw("observelayers")
     retry_delays === nothing || _warn_ignored_client_compat_kw("retry_delays")
@@ -1855,9 +1853,20 @@ function _normalize_headers_input(headers_input, copyheaders::Bool=true)::Header
     throw(ArgumentError("unsupported headers input type $(typeof(headers_input))"))
 end
 
+# `canonicalize_headers=true`: rewrite caller-spelled names in place, e.g.
+# `x-request-id` becomes `X-Request-Id`.
+function _canonicalize_header_names!(headers::Headers)::Nothing
+    entries = headers.entries
+    @inbounds for i in eachindex(entries)
+        key, value = entries[i]
+        entries[i] = canonical_header_key(key) => value
+    end
+    return nothing
+end
+
 function _apply_default_accept_encoding!(headers::Headers, decompress::Union{Nothing,Bool})::Nothing
     decompress === false && return nothing
-    any(h -> h[1] == "Accept-Encoding", headers.entries) && return nothing
+    haskey(headers, "Accept-Encoding") && return nothing
     setheader(headers, "Accept-Encoding", "gzip, deflate")
     return nothing
 end
@@ -2152,7 +2161,6 @@ function request(
         _handle_client_compat_kwargs(
             copyheaders=copyheaders,
             pool=pool,
-            canonicalize_headers=canonicalize_headers,
             detect_content_type=detect_content_type,
             observelayers=observelayers,
             retry_delays=retry_delays,
@@ -2188,6 +2196,7 @@ function request(
         request_url = parsed.url
         req_headers = _normalize_headers_input(headers, copyheaders)
         _apply_client_default_headers!(req_headers, client)
+        canonicalize_headers === true && _canonicalize_header_names!(req_headers)
         normalized_cookies = _normalize_cookies_input(cookies)
         sink = _resolve_response_sink(response_stream)
         sse_callback === nothing || sink === nothing || throw(ArgumentError("sse_callback cannot be combined with response_stream"))
@@ -2323,6 +2332,9 @@ Keyword arguments:
   mutate it until the call completes; final contents are unspecified. Other
   header inputs become a new collection either way. Retry attempts still
   receive isolated headers.
+- `canonicalize_headers`: header names are sent as the caller spelled them
+  (HTTP/2 sends them in lowercase). `true` rewrites every request header name
+  into [`canonical_header_key`](@ref) form first, as in HTTP.jl 1.x
 - `decompress`: `nothing`/`true` auto-decompress gzip and deflate responses, `false` leaves wire bytes untouched
 - `max_decompressed_size`: cap, in bytes, on an auto-decompressed response body; reading past it throws `DecompressionLimitError`, guarding against decompression bombs. Defaults to 64 MiB; `0` disables the limit
 - `sse_callback`: callback receiving `(event)` or `(stream, event)` for
@@ -2354,7 +2366,7 @@ Keyword arguments:
 HTTP.jl 2.0 accepts several HTTP.jl 1.x keywords as migration shims:
 `readtimeout` maps to `read_idle_timeout`; `pool`, `retry_delays`,
 `retry_check`, `sslconfig`, `socket_type_tls`,
-`canonicalize_headers`, `detect_content_type`, `logerrors`, `logtag`, and
+`detect_content_type`, `logerrors`, `logtag`, and
 `observelayers` are accepted so older call sites fail less abruptly. Prefer the
 2.0 forms listed above for new code: `client` / `transport` for pooling,
 `retry_if` / `retry_bucket` for retries, Reseau `Transport` TLS configuration

src/http_core.jl

@@ -638,22 +638,30 @@ end
 """
     Headers
 
-Ordered, case-canonicalized collection of header pairs.
+Ordered collection of header pairs.
 
 `Headers` deliberately behaves like `Vector{Pair{String, String}}` so code
 written against the long-standing pair-vector header representation can reuse
-the same helper functions. Keys are canonicalized on insertion, but pair order
-is preserved. Constructing from a vector, dictionary, or tuple goes through
-`mkheaders`: the source is left unchanged, keys are canonicalized, and adjacent
-duplicate keys are joined the way `appendheader` joins them.
+the same helper functions. Pair order is preserved.
+
+Header names are case-insensitive (RFC 9110 §5.1). Every lookup and update
+(`header`, `setheader`, `removeheader`, `haskey`, ...) matches a name in any
+case, but a name is stored exactly as it was given. HTTP/1 sends it in that
+spelling; HTTP/2 sends every name in lowercase. Headers read from the network
+are stored in [`canonical_header_key`](@ref) form (`Content-Type`), so a
+response looks the same over HTTP/1 and HTTP/2.
+
+Constructing from a vector, dictionary, or tuple goes through `mkheaders`: the
+source is left unchanged, and adjacent duplicate keys are joined the way
+`appendheader` joins them.
 """
 mutable struct Headers <: AbstractVector{Pair{String,String}}
     entries::Vector{Pair{String,String}}
 
     """Create and return an empty `Headers` collection."""
     Headers() = new(Pair{String,String}[])
 
-    """Copy pair storage without re-canonicalizing keys in an existing `Headers`."""
+    """Copy the pair storage of an existing `Headers`."""
     Headers(headers::Headers) = new(copy(headers.entries))
 end
 
@@ -677,7 +685,7 @@ Headers(items::Tuple) = mkheaders(items)
 """
     Headers(items...; kwargs...) -> Headers
 
-Construct a canonicalized Headers from items and/or kwargs
+Construct a Headers from items and/or kwargs.
 """
 Headers(items::Union{Pair,Tuple}...; kwargs...) = mkheaders(items...; kwargs...)
 
@@ -702,7 +710,7 @@ Base.iterate(headers::Headers, state...) = iterate(headers.entries, state...)
 Base.getindex(headers::Headers, i::Int) = headers.entries[i]
 
 @inline function _header_pair(key, value)::Pair{String,String}
-    return canonical_header_key(String(key)) => String(value)
+    return String(key) => String(value)
 end
 
 function Base.setindex!(headers::Headers, item, i::Int)
@@ -716,7 +724,7 @@ end
 
 function Base.merge!(headers::Headers, items)
     for (key, value) in Headers(items)
-        if key == "Set-Cookie"
+        if _ascii_equal_fold(key, "Set-Cookie")
             push!(headers.entries, key => value)
         else
             setheader(headers, key => value)
@@ -796,23 +804,26 @@ mkheaders(items::Tuple{Vararg{Union{Pair,Tuple}}}; kwargs...) =
 
 mkheaders(items::Union{Pair,Tuple}...; kwargs...) = mkheaders(Base.Iterators.flatten((items, kwargs)))
 
-"""Return a newly allocated `Vector{String}` of header keys in insertion order."""
+"""
+Return a newly allocated `Vector{String}` of header keys in insertion order.
+Names that differ only in case are listed once, in their first spelling.
+"""
 function header_keys(headers::Headers)::Vector{String}
     out = String[]
     seen = Set{String}()
     for (key, _) in headers
-        key in seen && continue
-        push!(seen, key)
+        folded = _ascii_lowercase_string(key)
+        folded in seen && continue
+        push!(seen, folded)
         push!(out, key)
     end
     return out
 end
 
 """Return the first value for `key`, or `default` if the header is absent."""
 function header(headers::Headers, key::AbstractString, default="")
-    canon = canonical_header_key(key)
     for (name, value) in headers
-        name == canon && return value
+        _ascii_equal_fold(name, key) && return value
     end
     return default
 end
@@ -824,10 +835,9 @@ Return a freshly allocated vector containing all values for `key` in stored
 order. Returns `String[]` when the header is absent.
 """
 function headers(headers::Headers, key::AbstractString)::Vector{String}
-    canon = canonical_header_key(key)
     out = String[]
     for (name, value) in headers
-        name == canon && push!(out, value)
+        _ascii_equal_fold(name, key) && push!(out, value)
     end
     return out
 end
@@ -840,10 +850,10 @@ end
 """
     headers[key] -> String
 
-Dict-style indexing on `Headers`. Returns the canonical first value for
-`key`, throwing `KeyError(key)` if the header is absent or has empty value.
-Use [`HTTP.header`](@ref) when you want a string default instead of an
-exception.
+Dict-style indexing on `Headers`. Returns the first value for `key`, matched
+case-insensitively, throwing `KeyError(key)` if the header is absent or has
+empty value. Use [`HTTP.header`](@ref) when you want a string default instead
+of an exception.
 """
 function Base.getindex(headers::Headers, key::AbstractString)::String
     v = header(headers, key)
@@ -867,9 +877,8 @@ Dict-style `haskey` on `Headers`. Returns `true` when `key` is present
 (case-insensitive), regardless of whether its value is empty.
 """
 function Base.haskey(headers::Headers, key::AbstractString)::Bool
-    canon = canonical_header_key(key)
     for (name, _) in headers
-        name == canon && return true
+        _ascii_equal_fold(name, key) && return true
     end
     return false
 end
@@ -884,9 +893,8 @@ Return `true` when any stored header value for `key` matches `value`
 case-insensitively.
 """
 function hasheader(headers::Headers, key::AbstractString, value::AbstractString)::Bool
-    canon = canonical_header_key(key)
     for (name, current) in headers
-        name == canon || continue
+        _ascii_equal_fold(name, key) || continue
         _ascii_equal_fold(current, value) && return true
     end
     return false
@@ -897,7 +905,8 @@ end
     setheader(headers, key, value) -> Headers
 
 Replace all stored values for `key` with `value`, preserving the first matching
-position if the key already exists and appending it otherwise. Returns the
+position if the key already exists and appending it otherwise. Names match in
+any case; the kept entry takes the spelling of `key`. Returns the
 mutated `headers`. [`setheader!`](@ref) is the same function under the
 conventional mutating-name spelling.
 """
@@ -909,7 +918,7 @@ function setheader(headers::Headers, header::Pair)
     write_idx = 1
     @inbounds for read_idx in eachindex(entries)
         entry = entries[read_idx]
-        if first(entry) == key
+        if _ascii_equal_fold(first(entry), key)
             if first_idx == 0
                 first_idx = write_idx
                 entries[write_idx] = item
@@ -940,18 +949,18 @@ end
 
 Append a header value to `headers`.
 
-If the previous stored header has the same name and the key is not
-`Set-Cookie`, the value is merged into the previous entry with a comma
-(no whitespace), as permitted by RFC 9110 §5.3 and required by common
-request-signing canonicalizations.
+If the previous stored header has the same name (in any case) and the key is
+not `Set-Cookie`, the value is merged into the previous entry, which keeps its
+spelling, with a comma (no whitespace), as permitted by RFC 9110 §5.3 and
+required by common request-signing canonicalizations.
 Otherwise a new pair is appended. [`appendheader!`](@ref) is the same
 function under the conventional mutating-name spelling.
 """
 function appendheader(headers::Headers, header::Pair)
     item = _header_pair(header.first, header.second)
     if !isempty(headers.entries)
         last_header = headers.entries[end]
-        if first(item) != "Set-Cookie" && first(last_header) == first(item)
+        if !_ascii_equal_fold(first(item), "Set-Cookie") && _ascii_equal_fold(first(last_header), first(item))
             headers.entries[end] = first(last_header) => string(last(last_header), ",", last(item))
             return headers
         end
@@ -972,12 +981,11 @@ Remove every stored header for `key` and return the mutated `headers`.
 mutating-name spelling.
 """
 function removeheader(headers::Headers, key::AbstractString)
-    canon = canonical_header_key(key)
     entries = headers.entries
     write_idx = 1
     @inbounds for read_idx in eachindex(entries)
         entry = entries[read_idx]
-        if first(entry) == canon
+        if _ascii_equal_fold(first(entry), key)
             continue
         end
         if write_idx != read_idx
@@ -1139,9 +1147,8 @@ tokens rather than one opaque string.
 """
 function headercontains(headers::Headers, key::AbstractString, token::AbstractString)::Bool
     needle = token isa String ? (token::String) : String(token)
-    canon = canonical_header_key(key)
     for (name, value) in headers
-        name == canon || continue
+        _ascii_equal_fold(name, key) || continue
         _header_value_contains_token(value, needle) && return true
     end
     return false