Preserve LLVM archive alternatives and add feature changelog (#878)

Fix the remaining xz/zstd mismatch in the single-release distribution
helper and add a history-backed changelog.

# AG;DR

## Fix

#870 changed the Python updater to retain all archive formats and made
the toolchain selector exclude zstd until Bazel can extract LLVM's
large-window archives. The shell helper still had #836's old filter,
which discarded xz whenever the same archive was available as zstd.

- Retain xz, zstd, and gzip archives and their individual checksums in
`extra_distributions.sh`.
- Leave extraction compatibility decisions in the toolchain selector;
this does not enable zstd selection.
- Correct the README's stale zstd-preference claim, updater scope, and
dependencies.

## Changelog

Add `CHANGELOG.md` with release headings and bullets, newest first.
Historical feature introductions are checked against the first
containing release tags; current changes are grouped under the version
heading `1.11.0`. Distinguish earlier functionality from later
enhancements, including Linux versus macOS ThinLTO, initial versus
expanded dynamic-libstdc++ support, and MSan introduction versus
correctness fixes.

## Regression coverage

- Execute the real shell helper against offline release metadata,
allowing only the metadata request and rejecting unexpected archive
downloads.
- Cover stable and prerelease versions, both LLVM naming schemes,
xz/zstd/gzip alternatives and individual formats, exact checksum
preservation, unrelated-asset filtering, and no-match errors.
- Extend the Python updater test to preserve all three formats and
checksums.
- Run both script suites in a Linux/macOS CI job and include it in
`required_checks_done.needs`.

## Verification

- Demonstrated the new shell regression fails before the fix in all four
stable/prerelease and naming-scheme combinations, specifically because
xz is missing.
- `python3 -m unittest discover -s utils -p '*_test.py' -v`: all 8 tests
pass on macOS.
- `USE_BAZEL_VERSION=9.0.0 bazel test //... --test_output=errors
--nocache_test_results`: all 21 tests executed and passed on the rebased
head.
- `bash -n` for both shell entry points.
- Repository lint/push hooks, `git diff --check`, and YAML validation
that the required-check gate includes every declared job and has no
unknown dependencies.

Fresh CI: [Tests run
36820892858](https://github.com/bazel-contrib/toolchains_llvm/actions/runs/36820892858).
Protected squash auto-merge is enabled; this is the final PR in the #873
→ #875 → #876 → #877 → #878 sequence. The changelog heading remains `##
1.11.0`.

Author: helly25

.github/workflows/tests.yml

@@ -23,6 +23,16 @@ jobs:
         uses: actions/checkout@v7
       - name: Trunk Check
         uses: trunk-io/trunk-action@v2.0.0
+  distribution_scripts_test:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [macos-latest, ubuntu-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v7
+      - name: Test distribution scripts without network downloads
+        run: python3 -m unittest discover -s utils -p '*_test.py' -v
   test:
     strategy:
       fail-fast: false
@@ -336,6 +346,7 @@ jobs:
     name: required_checks_done
     needs:
       - lint
+      - distribution_scripts_test
       - test
       - lsan_macos_test
       - cpp_modules_test

CHANGELOG.md

@@ -0,0 +1,180 @@
+# Changelog
+
+Versions below refer to **toolchains_llvm**, not LLVM or Bazel. Historical
+entries record notable user-facing feature introductions, checked against
+the first release tag containing their implementation. They are not an
+exhaustive list of fixes, dependency updates, or newly catalogued LLVM archives.
+See the [GitHub releases](https://github.com/bazel-contrib/toolchains_llvm/releases)
+for release artifacts and the [README](README.md) for configuration and requirements.
+
+## 1.11.0
+
+### Added
+
+- Generic, checksum-pinned downloadable linkers selected by name through
+  `linker`, with `linker_version` / `linker_versions` supporting exact versions
+  and `first` / `latest` requirements. The built-in catalogue initially contains
+  mold; other linker names can be supplied through inline or JSONC catalogues.
+- Linker catalogue controls, including custom entries, disabling the built-in
+  catalogue, and the `linker_features` capability list.
+- Optional mold version inference from an explicitly injected mold Bazel
+  module. Depending on that module is not required to use a catalogued mold
+  release with an explicit version or requirement.
+- Feature-based toolchain overrides with signed predicates such as
+  `features = ["thin_lto", "-msan"]`, optional target filters, default fallback,
+  ambiguity checks, and configurable merge/reset behavior. Matching uses
+  configuration-level `--features` / `--host_features`, not rule-local features.
+- Linker-managed macOS ThinLTO through configuration-level `--features=thin_lto`,
+  including independent `--host_features` handling for build tools. Linux's
+  existing distributed ThinLTO path is unchanged.
+- Python-based distribution updater that refreshes both the LLVM catalogue and
+  the mold catalogue, retaining the BCR mold version and the three newest stable
+  releases. `utils/update_distributions.sh` remains the shell entry point.
+
+### Fixed
+
+- Retain xz, zstd, and gzip archive metadata, but automatically select only xz
+  or gzip until Bazel can extract LLVM's large-window zstd archives. Both
+  distribution helper scripts now retain all formats instead of dropping xz
+  when an equivalent zstd asset exists.
+- Warn when LLVM 23 on macOS uses its bundled linker with the known SDK/TAPI
+  incompatibility, suggesting the native `auto` linker override.
+- Include `libLTO.dylib` in Apple linker sandbox inputs and omit redundant `-lm`
+  on macOS.
+- Apply MSan instrumentation to C compilation as well as C++, and fail loudly
+  when MSan is requested on an unsupported target, including macOS.
+- Correct the README's archive-selection policy and distribution updater
+  dependencies; exercise both distribution scripts in Linux/macOS CI.
+
+## [1.10.0](https://github.com/bazel-contrib/toolchains_llvm/tree/v1.10.0)
+
+- Configurable linker selection: empty/default for the linker bundled with
+  LLVM, `auto` for the execution platform's native linker, or an explicit
+  absolute path.
+
+## [1.9.1](https://github.com/bazel-contrib/toolchains_llvm/tree/v1.9.1)
+
+- Experimental C++ named modules, including `clang-scan-deps` integration.
+
+## [1.9.0](https://github.com/bazel-contrib/toolchains_llvm/tree/v1.9.0)
+
+- Explicit LLVM prerelease selection and prerelease-aware requirements.
+- External JSONC distribution files and disabling the built-in LLVM catalogue.
+- Direct zstd archive recognition.
+- LeakSanitizer integration.
+
+## [1.8.0](https://github.com/bazel-contrib/toolchains_llvm/tree/v1.8.0)
+
+- JSONC distribution buckets and the named distribution helper scripts.
+- Per-target toolchain roots through `target_toolchain_roots`.
+- `extra_linker_files` and per-target extra-file extension tags.
+- Expanded dynamic-libstdc++/Yocto sysroot layouts and SDK libc++ on Darwin.
+- MSan with an instrumented libc++ overlay and the current sanitizer feature
+  integration.
+- Bare-metal RISC-V 64-bit targets.
+
+## [1.7.0](https://github.com/bazel-contrib/toolchains_llvm/tree/v1.7.0)
+
+- Custom rules-based C++ features via `extra_known_features` and
+  `extra_enabled_features`.
+- RISC-V 64-bit Linux targets.
+
+## [1.6.0](https://github.com/bazel-contrib/toolchains_llvm/tree/v1.6.0)
+
+- `first` / `latest` LLVM version requirements and environment-derived version
+  requirements.
+- Additive `extra_*` flag attributes.
+
+## [1.5.0](https://github.com/bazel-contrib/toolchains_llvm/tree/v1.5.0)
+
+- `extra_llvm_distributions`, including full URL/path entries.
+- Header parsing and fastbuild compiler flags.
+- Bare-metal x86-64 and RISC-V 32-bit targets.
+
+## [1.4.0](https://github.com/bazel-contrib/toolchains_llvm/tree/v1.4.0)
+
+- More convenience tool aliases.
+- WebAssembly WASI Preview 1 and Linux ARMv7 targets.
+
+## [1.3.0](https://github.com/bazel-contrib/toolchains_llvm/tree/v1.3.0)
+
+- Initial `dynamic-stdc++` support.
+- C-only compiler flags.
+- Additional exported Clang tools/libraries.
+- WebAssembly `wasm32-unknown-unknown` and `wasm64-unknown-unknown` targets.
+
+## [1.2.0](https://github.com/bazel-contrib/toolchains_llvm/tree/v1.2.0)
+
+- Custom platform constraints and optional libunwind linking.
+
+## [1.1.0](https://github.com/bazel-contrib/toolchains_llvm/tree/v1.1.0)
+
+- `llvm-profdata` support for coverage data merging.
+
+## [1.0.0](https://github.com/bazel-contrib/toolchains_llvm/tree/1.0.0)
+
+- Strict header dependencies through `layering_check`.
+- Additional compiler sandbox inputs through `extra_compiler_files`.
+- Explicit execution-platform configuration distinct from the configuring host.
+
+## [0.9](https://github.com/bazel-contrib/toolchains_llvm/tree/0.9)
+
+- Bzlmod integration.
+- Per-host LLVM versions through `llvm_versions`.
+
+## [0.8.1](https://github.com/bazel-contrib/toolchains_llvm/tree/0.8.1)
+
+- Apple Silicon host support using LLVM 15 distributions.
+
+## [0.8](https://github.com/bazel-contrib/toolchains_llvm/tree/0.8)
+
+- `target_settings` for toolchain selection.
+- An `llvm-symbolizer` target.
+
+## [0.7](https://github.com/bazel-contrib/toolchains_llvm/tree/0.7)
+
+- Per-platform download URLs/checksums and mirror configuration.
+- C++ standard and standard-library selection, and flag overrides.
+- Convenience `clang-format`, `llvm-cov`, and `omp` targets.
+
+## [0.6](https://github.com/bazel-contrib/toolchains_llvm/tree/0.6)
+
+- Bring Your Own LLVM through `toolchain_roots` and a separate LLVM distribution
+  repository.
+- Target-aware cross-compilation and AArch64 support.
+- Authenticated downloads.
+- Inheritance of Bazel's Unix toolchain configuration, including Linux ThinLTO.
+
+## [0.4.4](https://github.com/bazel-contrib/toolchains_llvm/tree/0.4.4)
+
+- Toolchain registration support.
+
+## [0.4](https://github.com/bazel-contrib/toolchains_llvm/tree/0.4)
+
+- Custom sysroots.
+
+## [0.2](https://github.com/bazel-contrib/toolchains_llvm/tree/0.2)
+
+- Automatic OS/distribution detection and explicit distribution selection.
+
+## [0.1](https://github.com/bazel-contrib/toolchains_llvm/tree/0.1)
+
+- Initial WORKSPACE-based LLVM toolchain.
+- Explicit LLVM version selection.
+- Absolute-path/sandbox configuration.
+
+## Interpreting availability
+
+- Introduction does not imply support for every Bazel version, LLVM release,
+  platform, or later-added option. Consult the current README for requirements.
+- Dynamic libstdc++ first appeared in 1.3.0; its expanded sysroot/Yocto support
+  arrived in 1.8.0.
+- MSan's instrumented-libc++ integration dates to 1.8.0. The C-instrumentation
+  and unsupported-target fixes above do not make MSan a new 1.11.0 feature.
+- Linux ThinLTO is inherited from Bazel/rules_cc, so its behavior also depends
+  on those versions. The 1.11.0 addition is the macOS integration.
+- Native and explicit-path linker selection shipped in 1.10.0. Downloadable
+  catalogued linkers are a separate addition in 1.11.0.
+- Zstd recognition arrived in 1.9.0; it does not guarantee that Bazel can
+  extract every zstd archive. The 1.11.0 compatibility fix deliberately
+  excludes zstd from automatic selection while retaining its metadata.

README.md

@@ -3,7 +3,7 @@
 ## Quickstart
 
 See notes on the [release](https://github.com/bazel-contrib/toolchains_llvm/releases)
-for how to get started.
+for how to get started, and the [changelog](CHANGELOG.md) for feature availability.
 
 NOTE: For releases prior to 0.10.1, please also see [these notes](REPO_RENAME.md).
 
@@ -866,17 +866,21 @@ Entry keys are either a tarball **basename** (URL derived via `base_url`)
 or a **full URL/path** (used verbatim, bypassing `base_url`). Comments are
 stripped before parsing, and trailing commas are tolerated.
 
-LLVM archives compressed as `.tar.zst`, `.tar.xz`, or `.tar.gz` are
-supported. When equivalent zstd and xz archives are available, automatic
-selection and the helper scripts prefer the much faster-to-extract zstd form.
+The distribution data and both helper scripts retain `.tar.zst`, `.tar.xz`,
+and `.tar.gz` archives with their checksums. Automatic selection currently
+uses only xz or gzip: released Bazel versions cannot unpack LLVM 23's
+large-window zstd archives. Retaining zstd metadata does not enable its use;
+archive compatibility is decided by the toolchain selector, not the scripts.
 
 ### Two helper scripts
 
-- **`utils/update_distributions.sh`** — refreshes
-  [`toolchain/distributions/github.jsonc`](toolchain/distributions/github.jsonc)
-  by paging through the GitHub releases API for `llvm/llvm-project` and
-  rewriting the file in place. Use this when contributing a new LLVM release
-  to the bundled list. The script also regenerates the test golden file so
+- **`utils/update_distributions.sh`** — invokes `utils/update_distributions.py`
+  to refresh [`github.jsonc`](toolchain/distributions/github.jsonc) from LLVM
+  releases and [`linkers.jsonc`](toolchain/distributions/linkers.jsonc) from
+  mold releases. The linker catalogue includes the mold version in the Bazel
+  Central Registry and the three newest stable mold releases. Use this when
+  contributing new releases to the bundled lists.
+  The script also regenerates the distribution test golden files so
   the diff stays self-contained. No tarballs are downloaded — checksums come
   from GitHub's release-asset `.digest` field, with existing values
   preserved for older assets that predate that field. Set `GITHUB_TOKEN` to
@@ -890,10 +894,13 @@ selection and the helper scripts prefer the much faster-to-extract zstd form.
   tarballs and computing SHA-256 locally for older assets that don't have a
   `.digest`. See `-h` for details.
 
-Both scripts run on Linux, macOS, and on Windows under Git Bash / MSYS2 /
-WSL. They require `bash`, `curl`, `jq`, and `awk`;
-`utils/extra_distributions.sh` additionally needs `sha256sum` (Linux, Git
-Bash) or `shasum` (macOS).
+The shell entry points run on Linux, macOS, and on Windows under Git Bash /
+MSYS2 / WSL. The updater requires Python 3.9 or newer and `bazel` to regenerate
+goldens; its shell wrapper additionally requires `bash`. It uses `gh` for
+authentication when available and `GITHUB_TOKEN` is unset, and optionally uses
+Trunk or the repository's Prettier launcher for formatting.
+`utils/extra_distributions.sh` requires `bash`, `curl`, `jq`, `awk`, and either
+`sha256sum` (Linux, Git Bash) or `shasum` (macOS).
 
 ## Prior Art
 

utils/BUILD.bazel

@@ -1,5 +1,11 @@
 load("@rules_python//python:defs.bzl", "py_test")
 
+py_test(
+    name = "extra_distributions_test",
+    srcs = ["extra_distributions_test.py"],
+    data = ["extra_distributions.sh"],
+)
+
 py_test(
     name = "update_distributions_test",
     srcs = [

utils/extra_distributions.sh

@@ -92,18 +92,13 @@ curl "${curl_args[@]}" \
   >"${release_json}"
 
 # Asset table: name <TAB> digest (may be empty) <TAB> download URL.
+# Keep every compression format, just like update_distributions.py. Archive
+# compatibility is decided by the toolchain selector, not this data collector.
 assets_tsv="${tmp_dir}/assets.tsv"
 jq -r --arg version "${llvm_version}" '
-  .assets as $assets
-  | .assets[]
+  .assets[]
   | select(.name | test("^(clang[+]llvm|LLVM)-.*tar[.](zst|xz|gz)$"))
   | select(.name | contains("-" + $version + "-"))
-  | .name as $name
-  | select(
-      ($name | endswith(".tar.xz") | not) or
-      (($name | sub("[.]tar[.]xz$"; ".tar.zst")) as $zstd_name |
-       ($assets | any(.name == $zstd_name) | not))
-    )
   | [.name, ((.digest // "") | sub("^sha256:"; "")), .browser_download_url] | @tsv
 ' "${release_json}" >"${assets_tsv}"
 

utils/extra_distributions_test.py

@@ -0,0 +1,84 @@
+"""Exercise the real extra-distributions helper without network downloads."""
+
+import ast
+import json
+import os
+from pathlib import Path
+import subprocess
+import tempfile
+import unittest
+
+
+class ExtraDistributionsTest(unittest.TestCase):
+    def run_helper(self, version, assets):
+        with tempfile.TemporaryDirectory(
+            prefix="llvm-extra-distributions-", dir=os.environ.get("TEST_TMPDIR")
+        ) as directory:
+            root = Path(directory)
+            metadata = root / "metadata.json"
+            metadata.write_text(json.dumps({"assets": assets}))
+            curl = root / "curl"
+            # Only the release metadata request is allowed. Any attempt to
+            # download an archive fails instead of reaching the network.
+            curl.write_text("""#!/bin/sh
+for arg in "$@"; do url="$arg"; done
+if [ "$url" != "$LLVM_TEST_RELEASE_URL" ]; then
+  echo "Unexpected download: $url" >&2
+  exit 1
+fi
+cat "$LLVM_TEST_RELEASE_JSON"
+""")
+            curl.chmod(0o755)
+            env = dict(os.environ)
+            env.pop("GITHUB_TOKEN", None)
+            env.update({
+                "PATH": str(root) + os.pathsep + os.environ.get("PATH", os.defpath),
+                "LLVM_TEST_RELEASE_URL": "https://api.github.com/repos/llvm/llvm-project/releases/tags/llvmorg-" + version,
+                "LLVM_TEST_RELEASE_JSON": str(metadata),
+            })
+            return subprocess.run(
+                ["bash", str(Path(__file__).with_name("extra_distributions.sh")),
+                 "-t", str(root / "output"), "-v", version],
+                env=env, capture_output=True, text=True, check=False, timeout=30,
+            )
+
+    def assert_archives(self, version, prefix, suffixes):
+        names = [prefix + ".tar." + suffix for suffix in suffixes]
+        expected = {name: format(index, "064x") for index, name in enumerate(names, start=1)}
+        assets = [
+            {"name": name, "digest": "sha256:" + digest,
+             "browser_download_url": "https://example.invalid/" + name}
+            for name, digest in expected.items()
+        ]
+        # Neither source archives nor binaries for another version belong in
+        # the output, even when GitHub attaches them to the requested release.
+        assets.extend({"name": name} for name in [
+            "llvm-project-" + version + ".src.tar.xz",
+            "LLVM-99.0.0-Linux-X64.tar.xz",
+        ])
+        result = self.run_helper(version, assets)
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertEqual(ast.literal_eval("{\n" + result.stdout + "\n}"), expected)
+
+    def test_keeps_all_compressions_and_checksums(self):
+        for version in ["23.1.2", "23.1.0-rc3"]:
+            for prefix in [
+                "LLVM-" + version + "-Linux-X64",
+                "clang+llvm-" + version + "-x86_64-linux-gnu",
+            ]:
+                with self.subTest(version=version, prefix=prefix):
+                    self.assert_archives(version, prefix, ["xz", "zst", "gz"])
+
+    def test_keeps_single_compression(self):
+        for suffix in ["xz", "zst", "gz"]:
+            with self.subTest(suffix=suffix):
+                self.assert_archives("23.1.2", "LLVM-23.1.2-Linux-X64", [suffix])
+
+    def test_no_matching_archives_fails(self):
+        result = self.run_helper("23.1.2", [])
+        self.assertNotEqual(result.returncode, 0)
+        self.assertIn("no matching tarball assets found", result.stderr)
+
+
+if __name__ == "__main__":
+    unittest.main()