Checksum mismatch for registries served with `Content-Encoding: gzip` (e.g. GitHub Pages)

[mgeisler]

We use a custom registry at Proton and found that we get a checksum mismatch when crate_universe fetches .crate files from it. The registry is hosted on GitHub Pages, which applies Content-Encoding: gzip based on the client's Accept-Encoding header.

I was able to reproduce this in a self-contained repository, please see https://github.com/mgeisler/gh-pages-registry-demo.

Environment

Bazel	9.1.0
rules_rust	0.70.0
OS	macOS 26 (arm64)

Repository structure

Path	Purpose
greet/	Library crate published to the registry
hello/	Binary crate that depends on greet via the registry

The registry is live at https://mgeisler.github.io/gh-pages-registry-demo/.

Reproducing the bug

Building with Cargo works:

cd hello && cargo run

Building with Bazel fails (Cargo.lock is already committed, no repinning needed):

cd hello && bazel run :main

Bazel output:

WARNING: Download from https://mgeisler.github.io/gh-pages-registry-demo/crates/greet-0.1.0.crate failed:
  class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException
  Checksum was   dd22f693a0dc4bcd8d59c30b12ffe47ee355586964f211ba0878892ba46dfd3a
             but wanted 93afd84f9dc5cd2b9dacc42f1dec536595ed068214a844c4e0a8a7032a7397f0
ERROR: An error occurred during the fetch of repository 'rules_rust++crate+crates__greet-0.1.0':
  Error in download_and_extract: java.io.IOException: Error downloading
  [https://mgeisler.github.io/gh-pages-registry-demo/crates/greet-0.1.0.crate]:
  Checksum was   dd22f693a0dc4bcd8d59c30b12ffe47ee355586964f211ba0878892ba46dfd3a
             but wanted 93afd84f9dc5cd2b9dacc42f1dec536595ed068214a844c4e0a8a7032a7397f0
ERROR: Build did NOT complete successfully

Verifying the root cause with Curl

Without Accept-Encoding — what Cargo does; digest matches Cargo.lock:

curl -L "https://mgeisler.github.io/gh-pages-registry-demo/crates/greet-0.1.0.crate" \
  | sha256sum
# → 93afd84f9dc5cd2b9dacc42f1dec536595ed068214a844c4e0a8a7032a7397f0  ✓

With Accept-Encoding: gzip — what Bazel's HTTP client sends:

curl -L -H "Accept-Encoding: gzip" \
  "https://mgeisler.github.io/gh-pages-registry-demo/crates/greet-0.1.0.crate" \
  | sha256sum
# → dd22f693a0dc4bcd8d59c30b12ffe47ee355586964f211ba0878892ba46dfd3a  ✗

The response headers confirm that Fastly re-compresses the already-gzipped .crate:

content-type:     application/octet-stream
content-encoding: gzip
vary:             Accept-Encoding

We can verify this by hand:

curl -L -H "Accept-Encoding: gzip" \
    "https://mgeisler.github.io/gh-pages-registry-demo/crates/greet-0.1.0.crate" \
    | gunzip | sha256sum
# → 93afd84f9dc5cd2b9dacc42f1dec536595ed068214a844c4e0a8a7032a7397f0  ✓

Bazel hashes the gzip-encoded wire bytes: the .crate (already a .tar.gz) is gzipped a second time by Fastly, and Bazel checksums that doubly-compressed payload without decompressing it first.

Root cause

Bazel's downloader does not strip Content-Encoding: gzip before computing the SHA-256 checksum, contrary to how every other HTTP client behaves. The issue affects any Cargo registry whose CDN honours Accept-Encoding: gzip on binary responses — GitHub Pages (fronted by Fastly) is a reliable example.

Possible fixes

Suggested by my AI, I would of course love to hear other ideas:

1. Bazel-side (preferred): compute SHA-256 after decompressing Content-Encoding: gzip, consistent with other HTTP client libraries.
2. rules_rust: do not emit Accept-Encoding: gzip when issuing crate download requests (e.g. via a --header downloader flag).
3. rules_rust: expose a per-URL or per-registry option to suppress specific request headers in the generated http_archive calls.

[UebelAndre]

Thanks for the report! It seems http_archive doesn't expose any control over headers. This is probably a feature request that should be sent to https://github.com/bazelbuild/bazel. But for a fix here, a custom repository rule could be implemented that matches http_archive sets custom headers on repository_ctx.download_and_extract::headers (bazelbuild/bazel#17829) to match Cargo's behavior. I'm not sure if this would be acceptable in all cases. Some exploration is probably needed but I'd be happy to review a design/findings/PR

[mgeisler]

Hi @UebelAndre, thanks for the prompt reply! Would you like me to file the issue in bazelbuild/bazel as well?

I would like to contribute a fix, but I'm not sure I'll find the time right now. Luckily, we have worked around the problem for now by deploying a small proxy on our machines. From a bit of testing, it seems to work fine:

#!/usr/bin/env python3
"""Proxy that strips Accept-Encoding."""

import errno
import hashlib
import http.server
import os
import sys
import urllib.error
import urllib.request
from datetime import datetime, timezone

UPSTREAM = "https://company-registry.example.net"
LOG_FILE = os.path.join(os.environ.get("TMPDIR", "/tmp"),
                        "proton-registry-proxy.log")
port = 7755  # set from argv in __main__


def log(msg: str) -> None:
    ts = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
    with open(LOG_FILE, "a") as f:
        f.write(f"{ts} {msg}\n")


class Handler(http.server.BaseHTTPRequestHandler):

    def log_message(self, *_):
        pass

    def do_GET(self):
        req = urllib.request.Request(f"{UPSTREAM}{self.path}")
        req.add_unredirected_header("Accept-Encoding", "identity")
        try:
            with urllib.request.urlopen(req) as r:
                data = r.read()
                sha = hashlib.sha256(data).hexdigest() if self.path.endswith(
                    ".crate") else None
                extra = f" sha256={sha}" if sha else ""
                log(f"{r.status} {self.path} ({len(data)} bytes){extra}")
                self.send_response(r.status)
                for k, v in r.headers.items():
                    if k.lower() not in ("transfer-encoding", "connection",
                                         "content-length"):
                        self.send_header(k, v)
                self.send_header("Content-Length", str(len(data)))
                self.end_headers()
                self.wfile.write(data)
        except urllib.error.HTTPError as e:
            log(f"{e.code} {self.path} ERROR")
            self.send_error(e.code)


if __name__ == "__main__":
    port = int(sys.argv[1]) if len(sys.argv) > 1 else port
    log(f"starting on 127.0.0.1:{port}")
    try:
        server = http.server.HTTPServer(("127.0.0.1", port), Handler)
    except OSError as e:
        if e.errno == errno.EADDRINUSE:
            sys.exit(0)  # another instance is already running
        raise
    server.serve_forever()

We then have a downloader.cfg with

rewrite company-registry\.example\.net/(.*) http://127.0.0.1:7755/$1

which we register in .bazelrc with:

common --downloader_config=tools/downloader.cfg

Finally, we start the proxy automatically with a Bazel/Bazelisk wrapper located in tools/bazel:

#!/usr/bin/env bash

PROXY_PORT=7755
WORKSPACE="$(cd "$(dirname "$0")/.." && pwd)"
PROXY_SCRIPT="$WORKSPACE/tools/proton-registry-proxy.py"

# Use Python for the port check — nc/netcat is not always installed.
_port_open() {
    python3 - "$1" 2>/dev/null <<'EOF'
import socket, sys
s = socket.socket()
s.settimeout(0.1)
if s.connect_ex(("127.0.0.1", int(sys.argv[1]))) != 0:
    sys.exit(1)
EOF
}

# The proxy exits silently if already running (EADDRINUSE), so start it
# unconditionally and wait for the port to be ready.
python3 "$PROXY_SCRIPT" "$PROXY_PORT" &
disown $!
for _ in $(seq 40); do
    _port_open "$PROXY_PORT" && break
    sleep 0.05
done

# For shutdown, stop the proxy after Bazel exits so the two stay in sync.
if [[ "$1" == "shutdown" ]]; then
    "$BAZEL_REAL" "$@"
    EXIT=$?
    pkill -f "proton-registry-proxy.py" 2>/dev/null || true
    exit $EXIT
fi

exec "$BAZEL_REAL" "$@"

I hope this can be helpful to people until the underlying problem is fixed.