Tool staging#1519
Draft
novas0x2a wants to merge 8 commits into
Draft
Conversation
The toolchain layer was still using one path for two different jobs: where the foreign build should invoke a tool and which files should be staged for it under EXT_BUILD_DEPS. Split those apart so toolchains can describe both explicitly. This lets prebuilt CMake keep its execroot path while still giving staged tools like Meson a distinct staged invoke path when they need one.
The current staging path still treats tools as path strings with guessed runfiles sidecars. That keeps EXT_BUILD_DEPS/bin working, but it is not how Bazel expects executable tools to be launched and it leaks filename heuristics into the framework. Keep the staged bin entry as the compatibility surface and drop the adjacent manifest copies. The framework now stages the explicit launcher entry, handles absolute and already-staged paths correctly, and stops carrying manifest state through the native tool shim. Keep the improved Meson wrapper for all built Meson targets. Meson still needs a wrapper today because child Python re-execs do not inherit the import roots that rules_python sets up for the parent launcher. Add focused smoke coverage for the staged launcher contract and wire the Meson examples into a small tool-staging suite so macOS and Windows CI can rerun this change without waiting on full //... coverage.
The first Buildkite Windows failures showed two shared-layer regressions. Built Meson still needs the explicit runfiles and repo-mapping manifests next to the staged bin entry on Windows. The launcher itself was staged, but the adjacent manifest files were gone, so the Python launcher fell back to a path that did not exist. Prebuilt CMake also cannot be invoked from the flattened staged bin entry. That loses the install-tree-relative layout CMake uses to find CMAKE_ROOT. Keep the PATH-facing bin entry, but run prebuilt CMake from its execroot install tree instead. Also treat staged_path as the literal path inside EXT_BUILD_DEPS instead of running it back through resolve_command.
The current Windows rerun exposed a second staged Meson launcher bug. Restoring the runfiles and repo-mapping manifests was not enough. With `.bazelrc.common` forcing `--nobuild_python_zip` on Windows, the `rules_python` launcher also expects its sibling bootstrap file next to the staged `.exe`. Stage launcher-adjacent support files explicitly from the tool's own direct outputs in the executable directory. That keeps the Windows launcher contract intact without going back to guessed sidecars or staging the whole target closure.
The next Windows rerun exposed a broader staging bug in the shared layer. `tool_entries` still included preinstalled host tools, so Windows builds were trying to stage names like `cmake.exe` and `m4` out of the execroot. At the same time, the Windows helper was still copying `*-config` files even when the caller explicitly said `replace_in_files = False`, which turned PATH-facing `pkg-config` tool entries into writable copies for no reason. Only stage bin entries for toolchains that actually have a Bazel target behind them, and only copy `*-config`-style files when replacement is enabled. That keeps `EXT_BUILD_DEPS/bin` as the compatibility surface without treating host tools or executables as editable payloads.
The Windows examples rerun narrowed the remaining failure to Meson path metadata. We were telling the build script to invoke `bin/meson_tool*.exe`, but the Bazel `py_binary` output for this tool is the bare launcher name plus sibling bootstrap files. That left the staged path pointing at a file that was never created. Use the real staged launcher basename again. The launcher support files are already staged explicitly, so Meson still gets the adjacent files it needs on Windows without forcing an `.exe` suffix that does not match the artifact.
The next Windows examples rerun showed that the staged Meson launcher was now found and executed, but it still could not find its interpreter. The stage-1 Python launcher derives its runfiles tree as `<argv0>.exe.runfiles` on Windows even when the staged launcher file itself is on the bare basename. We were already staging the launcher, manifest, repo mapping, and sibling bootstrap files, but not the runfiles-dir alias that the launcher actually probes first. Carry the launcher's sibling runfiles dir through the tool metadata and stage it under the Windows alias next to relocated launchers. That keeps the fix in the shared staging layer instead of teaching Meson or individual examples about another Windows-only path quirk.
The previous Windows follow-up staged the `.exe.runfiles` alias, but it used the wrong primitive. `symlink_to_dir` treats its target as a parent directory. That is fine when we want to drop a source directory under a parent, but it is wrong for the Python launcher alias case. There the target path itself needs to behave like the runfiles directory. Use `symlink_contents_to_dir` for the launcher runfiles alias so the staged alias contains the expected runfiles layout at its own path. That matches what the Windows stage-1 launcher probes for stage-2 bootstrap and interpreter resolution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.