Vault 2.x billing fails with "lz4: bad magic number"

[danilobuerger]

Describe the bug

The monthly "SSH duration-adjusted certificate count" (and the sibling SSH OTP count) is written to storage as plain, uncompressed JSON, but read back through a code path that treats the first stored byte as a compression canary. The LZ4 canary is the ASCII character '4' (0x34). Whenever the stored counter's decimal JSON representation begins with the digit 4 (e.g. 4, 4.1096, 42, 400, 4096), the reader misclassifies the plain number as LZ4-compressed data and fails:

error decoding current SSH duration adjusted cert count: failed to decompress JSON: lz4: bad magic number

To Reproduce

podman run --rm -p 8200:8200 docker.io/hashicorp/vault:2.0.3 server -dev -dev-tls -dev-root-token-id root -dev-listen-address 0.0.0.0:8200

and

export VAULT_ADDR=https://127.0.0.1:8200 VAULT_SKIP_VERIFY=true VAULT_TOKEN=root
vault secrets enable -path=ssh -max-lease-ttl=10000h ssh
vault write ssh/config/ca generate_signing_key=true
vault write ssh/roles/repro key_type=ca allow_user_certificates=true allowed_users='*' ttl=3000h max_ttl=4000h
ssh-keygen -t ed25519 -N '' -f ./id
vault write ssh/sign/repro public_key=@./id.pub ttl=3000h valid_principals=ubuntu
sleep 70
vault read sys/billing/overview

outputs:

Error reading sys/billing/overview: Error making API request.

URL: GET https://127.0.0.1:8200/v1/sys/billing/overview
Code: 500. Errors:

* 1 error occurred:
        * error building billing data for month 2026-06: error retrieving SSH duration-adjuested cert counts for current month: error decoding current SSH duration adjusted cert count: failed to decompress JSON: lz4: bad magic number

Expected behavior

Not fail with an error.

Environment:

• Vault Server Version (retrieve with vault status): verified in 2.0.0 and 2.0.3
• Vault CLI Version (retrieve with vault version): n/a
• Server Operating System/Architecture: n/a

Vault server configuration file(s):

Additional context

[dosubot]

Hi @danilobuerger! I'm Dosu and I'm helping the HashiCorp, an IBM Company team.

This is a valid bug. I was able to confirm the root cause by tracing through the code:

Write path — storeSSHDurationAdjustedCertCountLocked() encodes the float64 cert count with jsonutil.EncodeJSON(), which produces plain uncompressed JSON (e.g. 4.1096\n) and writes it directly to storage with no canary byte prefix [1].

Read path — getStoredSSHDurationAdjustedCertCountLocked() calls StorageEntry.DecodeJSON(), which in turn calls compressutil.Decompress(). That function inspects the first byte of the stored data against known compression canary bytes [2]. Since CompressionCanaryLZ4 is defined as byte '4' (0x34) [3], any stored numeric value whose JSON representation begins with the digit 4 is misclassified as LZ4-compressed data and fails decompression.

The same class of false-positive would theoretically apply to values starting with G (gzip canary), L (LZW canary), or S (Snappy canary) — but since the stored value is a number, only the '4' case is reachable in practice.

The asymmetry is clear: EncodeJSON writes raw JSON without a canary, while DecodeJSON always attempts canary-based decompression first. For top-level JSON objects/arrays (starting with { or [) this is safe, but for bare numeric primitives it's not.

I didn't find any existing PR or issue tracking this fix. Tagging @heatherezell to flag this for product and engineering review.

Written by @dosu, approved by heatherezell

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[danilobuerger]

@dosu the PR is at #32015

[dosubot]

Thanks for the PR link, @danilobuerger! The fix looks solid — switching from jsonutil.EncodeJSON/DecodeJSON to strconv-based storage (aligning with how PKI and OIDC counters already work) cleanly avoids the canary byte collision, and the strings.TrimSpace handling for legacy values is a nice touch for backward compatibility.

One note: please make sure the CLA has been signed — the maintainers will need that before the PR can be merged. If you haven't already, the CLA bot should prompt you on the PR.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[heatherezell]

Is this for Vault Enterprise? If so, I recommend you open a support ticket. Reference the issue and PR numbers. Thanks!

[danilobuerger]

@heatherezell no its not

[heatherezell]

@heatherezell no its not

Got it! Thanks for the clarification. I'll get this to engineering. :)