Skip to content

Commit 34e82cd

Browse files
philscrickeylev
andauthored
feat: provide access to arbitrary interpreters (bazel-contrib#2507)
There are some use cases that folks want to cover here. They are discussed in [this Slack thread][1]. The high-level summary is: 1. Users want to run the exact same interpreter that Bazel is running to minimize environmental issues. 2. It is useful to pass a target label to third-party tools like mypy so that they can use the correct interpreter. This patch adds to @rickeylev's work from bazel-contrib#2359 by adding docs and a few integration tests. [1]: https://bazelbuild.slack.com/archives/CA306CEV6/p1730095371089259 --------- Co-authored-by: Richard Levasseur <[email protected]>
1 parent 0a3704d commit 34e82cd

13 files changed

+430
-25
lines changed
+41
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
:::{default-domain} bzl
2+
:::
3+
:::{bzl:currentfile} //python/bin:BUILD.bazel
4+
:::
5+
6+
# //python/bin
7+
8+
:::{bzl:target} python
9+
10+
A target to directly run a Python interpreter.
11+
12+
By default, it uses the Python version that toolchain resolution matches
13+
(typically the one marked `is_default=True` in `MODULE.bazel`).
14+
15+
This runs a Python interpreter in a similar manner as when running `python3`
16+
on the command line. It can be invoked using `bazel run`. Remember that in
17+
order to pass flags onto the program `--` must be specified to separate
18+
Bazel flags from the program flags.
19+
20+
An example that will run Python 3.12 and have it print the version
21+
22+
```
23+
bazel run @rules_python//python/bin:python \
24+
`--@rule_python//python/config_settings:python_verion=3.12 \
25+
-- \
26+
--version
27+
```
28+
29+
::::{seealso}
30+
The {flag}`--python_src` flag for using the intepreter a binary/test uses.
31+
::::
32+
33+
::::{versionadded} VERSION_NEXT_FEATURE
34+
::::
35+
:::
36+
37+
:::{bzl:flag} python_src
38+
39+
The target (one providing `PyRuntimeInfo`) whose python interpreter to use for
40+
{obj}`:python`.
41+
:::

docs/toolchains.md

+43-2
Original file line numberDiff line numberDiff line change
@@ -396,7 +396,7 @@ provide `Python.h`.
396396

397397
This is typically implemented using {obj}`py_cc_toolchain()`, which provides
398398
{obj}`ToolchainInfo` with the field `py_cc_toolchain` set, which is a
399-
{obj}`PyCcToolchainInfo` provider instance.
399+
{obj}`PyCcToolchainInfo` provider instance.
400400

401401
This toolchain type is intended to hold only _target configuration_ values
402402
relating to the C/C++ information for the Python runtime. As such, when defining
@@ -556,4 +556,45 @@ of available toolchains.
556556
Currently the following flags are used to influence toolchain selection:
557557
* {obj}`--@rules_python//python/config_settings:py_linux_libc` for selecting the Linux libc variant.
558558
* {obj}`--@rules_python//python/config_settings:py_freethreaded` for selecting
559-
the freethreaded experimental Python builds available from `3.13.0` onwards.
559+
the freethreaded experimental Python builds available from `3.13.0` onwards.
560+
561+
## Running the underlying interpreter
562+
563+
To run the interpreter that Bazel will use, you can use the
564+
`@rules_python//python/bin:python` target. This is a binary target with
565+
the executable pointing at the `python3` binary plus its relevent runfiles.
566+
567+
```console
568+
$ bazel run @rules_python//python/bin:python
569+
Python 3.11.1 (main, Jan 16 2023, 22:41:20) [Clang 15.0.7 ] on linux
570+
Type "help", "copyright", "credits" or "license" for more information.
571+
>>>
572+
$ bazel run @rules_python//python/bin:python --@rules_python//python/config_settings:python_version=3.12
573+
Python 3.12.0 (main, Oct 3 2023, 01:27:23) [Clang 17.0.1 ] on linux
574+
Type "help", "copyright", "credits" or "license" for more information.
575+
>>>
576+
```
577+
578+
You can also access a specific binary's interpreter this way by using the
579+
`@rules_python//python/bin:python_src` target. In the example below, it is
580+
assumed that the `@rules_python//tools/publish:twine` binary is fixed at Python
581+
3.11.
582+
583+
```console
584+
$ bazel run @rules_python//python/bin:python --@rules_python//python/bin:interpreter_src=@rules_python//tools/publish:twine
585+
Python 3.11.1 (main, Jan 16 2023, 22:41:20) [Clang 15.0.7 ] on linux
586+
Type "help", "copyright", "credits" or "license" for more information.
587+
>>>
588+
$ bazel run @rules_python//python/bin:python --@rules_python//python/bin:interpreter_src=@rules_python//tools/publish:twine --@rules_python//python/config_settings:python_version=3.12
589+
Python 3.11.1 (main, Jan 16 2023, 22:41:20) [Clang 15.0.7 ] on linux
590+
Type "help", "copyright", "credits" or "license" for more information.
591+
>>>
592+
```
593+
Despite setting the Python version explicitly to 3.12 in the example above, the
594+
interpreter comes from the `@rules_python//tools/publish:twine` binary. That is
595+
a fixed version.
596+
597+
:::{note}
598+
The `python` target does not provide access to any modules from `py_*`
599+
targets on its own. Please file a feature request if this is desired.
600+
:::

python/BUILD.bazel

+1
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ filegroup(
3535
name = "distribution",
3636
srcs = glob(["**"]) + [
3737
"//python/api:distribution",
38+
"//python/bin:distribution",
3839
"//python/cc:distribution",
3940
"//python/config_settings:distribution",
4041
"//python/constraints:distribution",

python/bin/BUILD.bazel

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
load("//python/private:interpreter.bzl", _interpreter_binary = "interpreter_binary")
2+
3+
filegroup(
4+
name = "distribution",
5+
srcs = glob(["**"]),
6+
visibility = ["//:__subpackages__"],
7+
)
8+
9+
_interpreter_binary(
10+
name = "python",
11+
binary = ":python_src",
12+
target_compatible_with = select({
13+
"@platforms//os:windows": ["@platforms//:incompatible"],
14+
"//conditions:default": [],
15+
}),
16+
visibility = ["//visibility:public"],
17+
)
18+
19+
# The user can modify this flag to source different interpreters for the
20+
# `python` target above.
21+
label_flag(
22+
name = "python_src",
23+
build_setting_default = "//python:none",
24+
)

python/private/common.bzl

+17
Original file line numberDiff line numberDiff line change
@@ -543,3 +543,20 @@ def target_platform_has_any_constraint(ctx, constraints):
543543
if ctx.target_platform_has_constraint(constraint_value):
544544
return True
545545
return False
546+
547+
def runfiles_root_path(ctx, short_path):
548+
"""Compute a runfiles-root relative path from `File.short_path`
549+
550+
Args:
551+
ctx: current target ctx
552+
short_path: str, a main-repo relative path from `File.short_path`
553+
554+
Returns:
555+
{type}`str`, a runflies-root relative path
556+
"""
557+
558+
# The ../ comes from short_path is for files in other repos.
559+
if short_path.startswith("../"):
560+
return short_path[3:]
561+
else:
562+
return "{}/{}".format(ctx.workspace_name, short_path)

python/private/interpreter.bzl

+82
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# Copyright 2025 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Implementation of the rules to access the underlying Python interpreter."""
16+
17+
load("@bazel_skylib//lib:paths.bzl", "paths")
18+
load("//python:py_runtime_info.bzl", "PyRuntimeInfo")
19+
load(":common.bzl", "runfiles_root_path")
20+
load(":sentinel.bzl", "SentinelInfo")
21+
load(":toolchain_types.bzl", "TARGET_TOOLCHAIN_TYPE")
22+
23+
def _interpreter_binary_impl(ctx):
24+
if SentinelInfo in ctx.attr.binary:
25+
toolchain = ctx.toolchains[TARGET_TOOLCHAIN_TYPE]
26+
runtime = toolchain.py3_runtime
27+
else:
28+
runtime = ctx.attr.binary[PyRuntimeInfo]
29+
30+
# NOTE: We name the output filename after the underlying file name
31+
# because of things like pyenv: they use $0 to determine what to
32+
# re-exec. If it's not a recognized name, then they fail.
33+
if runtime.interpreter:
34+
# In order for this to work both locally and remotely, we create a
35+
# shell script here that re-exec's into the real interpreter. Ideally,
36+
# we'd just use a symlink, but that breaks under certain conditions. If
37+
# we use a ctx.actions.symlink(target=...) then it fails under remote
38+
# execution. If we use ctx.actions.symlink(target_path=...) then it
39+
# behaves differently inside the runfiles tree and outside the runfiles
40+
# tree.
41+
#
42+
# This currently does not work on Windows. Need to find a way to enable
43+
# that.
44+
executable = ctx.actions.declare_file(runtime.interpreter.basename)
45+
ctx.actions.expand_template(
46+
template = ctx.file._template,
47+
output = executable,
48+
substitutions = {
49+
"%target_file%": runfiles_root_path(ctx, runtime.interpreter.short_path),
50+
},
51+
is_executable = True,
52+
)
53+
else:
54+
executable = ctx.actions.declare_symlink(paths.basename(runtime.interpreter_path))
55+
ctx.actions.symlink(output = executable, target_path = runtime.interpreter_path)
56+
57+
return [
58+
DefaultInfo(
59+
executable = executable,
60+
runfiles = ctx.runfiles([executable], transitive_files = runtime.files).merge_all([
61+
ctx.attr._bash_runfiles[DefaultInfo].default_runfiles,
62+
]),
63+
),
64+
]
65+
66+
interpreter_binary = rule(
67+
implementation = _interpreter_binary_impl,
68+
toolchains = [TARGET_TOOLCHAIN_TYPE],
69+
executable = True,
70+
attrs = {
71+
"binary": attr.label(
72+
mandatory = True,
73+
),
74+
"_bash_runfiles": attr.label(
75+
default = "@bazel_tools//tools/bash/runfiles",
76+
),
77+
"_template": attr.label(
78+
default = "//python/private:interpreter_tmpl.sh",
79+
allow_single_file = True,
80+
),
81+
},
82+
)

python/private/interpreter_tmpl.sh

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/bin/bash
2+
3+
# --- begin runfiles.bash initialization v3 ---
4+
# Copy-pasted from the Bazel Bash runfiles library v3.
5+
set -uo pipefail; set +e; f=bazel_tools/tools/bash/runfiles/runfiles.bash
6+
# shellcheck disable=SC1090
7+
source "${RUNFILES_DIR:-/dev/null}/$f" 2>/dev/null || \
8+
source "$(grep -sm1 "^$f " "${RUNFILES_MANIFEST_FILE:-/dev/null}" | cut -f2- -d' ')" 2>/dev/null || \
9+
source "$0.runfiles/$f" 2>/dev/null || \
10+
source "$(grep -sm1 "^$f " "$0.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
11+
source "$(grep -sm1 "^$f " "$0.exe.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
12+
{ echo>&2 "ERROR: cannot find $f"; exit 1; }; f=; set -e
13+
# --- end runfiles.bash initialization v3 ---
14+
15+
set +e # allow us to check for errors more easily
16+
readonly TARGET_FILE="%target_file%"
17+
MAIN_BIN=$(rlocation "$TARGET_FILE")
18+
19+
if [[ -z "$MAIN_BIN" || ! -e "$MAIN_BIN" ]]; then
20+
echo "ERROR: interpreter executable not found: $MAIN_BIN (from $TARGET_FILE)"
21+
exit 1
22+
fi
23+
exec "${MAIN_BIN}" "$@"

python/private/py_executable.bzl

+6-22
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ load(
4848
"filter_to_py_srcs",
4949
"get_imports",
5050
"is_bool",
51+
"runfiles_root_path",
5152
"target_platform_has_any_constraint",
5253
"union_attrs",
5354
)
@@ -447,7 +448,7 @@ def _create_executable(
447448
)
448449

449450
def _create_zip_main(ctx, *, stage2_bootstrap, runtime_details, venv):
450-
python_binary = _runfiles_root_path(ctx, venv.interpreter.short_path)
451+
python_binary = runfiles_root_path(ctx, venv.interpreter.short_path)
451452
python_binary_actual = venv.interpreter_actual_path
452453

453454
# The location of this file doesn't really matter. It's added to
@@ -522,7 +523,7 @@ def _create_venv(ctx, output_prefix, imports, runtime_details):
522523

523524
if not venvs_use_declare_symlink_enabled:
524525
if runtime.interpreter:
525-
interpreter_actual_path = _runfiles_root_path(ctx, runtime.interpreter.short_path)
526+
interpreter_actual_path = runfiles_root_path(ctx, runtime.interpreter.short_path)
526527
else:
527528
interpreter_actual_path = runtime.interpreter_path
528529

@@ -543,11 +544,11 @@ def _create_venv(ctx, output_prefix, imports, runtime_details):
543544
# may choose to write what symlink() points to instead.
544545
interpreter = ctx.actions.declare_symlink("{}/bin/{}".format(venv, py_exe_basename))
545546

546-
interpreter_actual_path = _runfiles_root_path(ctx, runtime.interpreter.short_path)
547+
interpreter_actual_path = runfiles_root_path(ctx, runtime.interpreter.short_path)
547548
rel_path = relative_path(
548549
# dirname is necessary because a relative symlink is relative to
549550
# the directory the symlink resides within.
550-
from_ = paths.dirname(_runfiles_root_path(ctx, interpreter.short_path)),
551+
from_ = paths.dirname(runfiles_root_path(ctx, interpreter.short_path)),
551552
to = interpreter_actual_path,
552553
)
553554

@@ -646,23 +647,6 @@ def _create_stage2_bootstrap(
646647
)
647648
return output
648649

649-
def _runfiles_root_path(ctx, short_path):
650-
"""Compute a runfiles-root relative path from `File.short_path`
651-
652-
Args:
653-
ctx: current target ctx
654-
short_path: str, a main-repo relative path from `File.short_path`
655-
656-
Returns:
657-
{type}`str`, a runflies-root relative path
658-
"""
659-
660-
# The ../ comes from short_path is for files in other repos.
661-
if short_path.startswith("../"):
662-
return short_path[3:]
663-
else:
664-
return "{}/{}".format(ctx.workspace_name, short_path)
665-
666650
def _create_stage1_bootstrap(
667651
ctx,
668652
*,
@@ -676,7 +660,7 @@ def _create_stage1_bootstrap(
676660
runtime = runtime_details.effective_runtime
677661

678662
if venv:
679-
python_binary_path = _runfiles_root_path(ctx, venv.interpreter.short_path)
663+
python_binary_path = runfiles_root_path(ctx, venv.interpreter.short_path)
680664
else:
681665
python_binary_path = runtime_details.executable_interpreter_path
682666

python/private/site_init_template.py

+3-1
Original file line numberDiff line numberDiff line change
@@ -163,7 +163,9 @@ def _maybe_add_path(path):
163163
if cov_tool:
164164
_print_verbose_coverage(f"Using toolchain coverage_tool {cov_tool}")
165165
elif cov_tool := os.environ.get("PYTHON_COVERAGE"):
166-
_print_verbose_coverage(f"Using env var coverage: PYTHON_COVERAGE={cov_tool}")
166+
_print_verbose_coverage(
167+
f"Using env var coverage: PYTHON_COVERAGE={cov_tool}"
168+
)
167169

168170
if cov_tool:
169171
if os.path.isabs(cov_tool):

tests/interpreter/BUILD.bazel

+52
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Copyright 2024 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
load(":interpreter_tests.bzl", "PYTHON_VERSIONS_TO_TEST", "py_reconfig_interpreter_tests")
16+
17+
# For this test the interpreter is sourced from the current configuration. That
18+
# means both the interpreter and the test itself are expected to run under the
19+
# same Python version.
20+
py_reconfig_interpreter_tests(
21+
name = "interpreter_version_test",
22+
srcs = ["interpreter_test.py"],
23+
data = [
24+
"//python/bin:python",
25+
],
26+
env = {
27+
"PYTHON_BIN": "$(rootpath //python/bin:python)",
28+
},
29+
main = "interpreter_test.py",
30+
python_versions = PYTHON_VERSIONS_TO_TEST,
31+
)
32+
33+
# For this test the interpreter is sourced from a binary pinned at a specific
34+
# Python version. That means the interpreter and the test itself can run
35+
# different Python versions.
36+
py_reconfig_interpreter_tests(
37+
name = "python_src_test",
38+
srcs = ["interpreter_test.py"],
39+
data = [
40+
"//python/bin:python",
41+
],
42+
env = {
43+
# Since we're grabbing the interpreter from a binary with a fixed
44+
# version, we expect to always see that version. It doesn't matter what
45+
# Python version the test itself is running with.
46+
"EXPECTED_INTERPRETER_VERSION": "3.11",
47+
"PYTHON_BIN": "$(rootpath //python/bin:python)",
48+
},
49+
main = "interpreter_test.py",
50+
python_src = "//tools/publish:twine",
51+
python_versions = PYTHON_VERSIONS_TO_TEST,
52+
)

0 commit comments

Comments
 (0)