feat(venv): Create a relocatable venv shim (#546)

Related to #522 and associated issues.

This PR introduces a new venv tool, what for lack of better words I'm
calling the "shim".

While standard interpreters do support the use of relative paths to the
interpreter in `pyvenv.cfg` -- for instance `home = ./bin/python` is
legal -- we still need a `bin/python` which is portable. Existing
relocatable virtual environment solutions (uv, conda) still ultimately
create both an absolute path reference in `pyvenv.cfg` and usually in
the `bin/python` symlink to a specific interpreter on the filesystem. UV
does this statically as is standard, Conda will also do this statically
with an explicit update process as part of the conda unpack.

We can't create relocatable symlinks, and we also can't dynamically
correct static symlinks without continuing to have #339 as a problem.

What we can do is dynamically identify a Python interpreter and hoodwink
it with regards to its own path, causing it to load a virtualenv which
is itself entirely relocatable.

As with many other tools, the Python interpreter introspects its
`argv[0]` to figure out its own nominal path and identify the home, see
[2], [3] for the gory details. This means that when the interpreter is
invoked via a symlink under normal execution, the "path" of the
interpreter per `argv[0]` is the path of the symlink with respect to
which `pyvenv.cfg` can be identified and the virtualenv activated
automatically. The Darwin platform provides the `_NSGetExecutablePath`
libc call which is an alternative mechanism for determining the "path"
by which the interpreter is invoked.

This PR introduces a shim tool which can be emplaced as the target of
`bin/python`, which will consult the `pyvenv.cfg` of a conventional
virtualenv to find the requested interpreter version, and will attempt
to delegate to an identified interpreter in such a way as to make the
interpreter believe that its path is that of the shim tool.

On a conventional unix this is as simple as lying about the value of
`argv[0]`, on Darwin the `PYTHONEXECUTABLE` environment flag must be set
to make Python disbelieve the value of `_NSGetExecutablePath` which is
harder to hoodwink.

Using this tool, a relocatable virtualenv can be structured as follows

```
./pyvenv.cfg                        # conventional
./bin/python -> ./aspect_venv_shim  # customized "interpreter"
./bin/python3 -> ./python           # conventional
./bin/python3.{N} -> ./python        # conventional
./bin/aspect_venv_shim
./lib/python3.${N}/site-packages/...                 # conventional; standard contents
```

We should be able to create one of these venvs by updating our `uv`
dependency, setting the `relocatable=True` flag when creating
virtualenvs, and attempting to specify the custom interpreter path of
`./bin/aspect_venv_shim` so that the "python" symlink and its siblings
will enter this pipeline.

Using this as our "interpreter" will allow us to pull venv creation
forwards from runtime to a normal build action, and allow us to create
conda-pack like structures which require no unpack post-processing
directly from that venv.

The downside of this approach is that as with other `exec` based Python
launchers such as Pex or Bazel's `--run_under` it is likely to interfere
with debugging tools that want to instrument a Python process, as the
bootloader is not an interpreter and cannot be analyzed as such.

### Changes are visible to end-users: yes

- Searched for relevant documentation and updated as needed: yes
- Breaking change (forces users to change their own code or config): no
- Suggested release notes appear below: no

### Test plan

- [x] Built for MacOS; made a relocatable venv, customized the
`pyvenv.cfg` to be `home = ./bin/python`, updated the `bin/python` link
to be `bin/aspect_venv_shim`, confirmed that invoking the venv's python
links did cause the venv to load.
- [x] Built for Linux; made a relocatable venv, customized the
`pyvenv.cfg` to be `home = ./bin/python`, updated the `bin/python` link
to be `bin/aspect_venv_shim`, confirmed that invoking the venv's python
links did cause the venv to load.

### Notes
[1]
https://discuss.python.org/t/interpreter-independent-isolated-virtual-environments/5378/53
[2] https://github.com/python/cpython/blob/main/Modules/getpath.py#L293
[3] https://github.com/python/cpython/blob/main/Modules/getpath.c#L774

Author: arrdem

Cargo.Bazel.lock

@@ -1,5 +1,5 @@
 {
-  "checksum": "81a92b86099c7c9b831ce831b33ad24c552a0c97ef3d8dc37836db3d08529ec4",
+  "checksum": "7c8cd2683de76ab9fae29a74881389aefee00f8d83f53bd006aba44b4dd3731d",
   "crates": {
     "addr2line 0.24.2": {
       "name": "addr2line",
@@ -23131,6 +23131,33 @@
       "license_ids": [],
       "license_file": null
     },
+    "venv_shim 0.1.0": {
+      "name": "venv_shim",
+      "version": "0.1.0",
+      "package_url": "https://github.com/aspect-build/rules_py",
+      "repository": null,
+      "targets": [],
+      "library_target_name": null,
+      "common_attrs": {
+        "compile_data_glob": [
+          "**"
+        ],
+        "deps": {
+          "common": [
+            {
+              "id": "miette 7.2.0",
+              "target": "miette"
+            }
+          ],
+          "selects": {}
+        },
+        "edition": "2021",
+        "version": "0.1.0"
+      },
+      "license": "Apache 2",
+      "license_ids": [],
+      "license_file": null
+    },
     "version_check 0.9.5": {
       "name": "version_check",
       "version": "0.9.5",
@@ -26950,7 +26977,8 @@
   "workspace_members": {
     "py 0.1.0": "py/tools/py",
     "unpack_bin 0.1.0": "py/tools/unpack_bin",
-    "venv_bin 0.1.0": "py/tools/venv_bin"
+    "venv_bin 0.1.0": "py/tools/venv_bin",
+    "venv_shim 0.1.0": "py/tools/venv_shim"
   },
   "conditions": {
     "aarch64-apple-darwin": [

Cargo.lock

@@ -4,6 +4,7 @@ members = [
     "py/tools/py",
     "py/tools/venv_bin",
     "py/tools/unpack_bin",
+    "py/tools/venv_shim",
 ]
 
 [workspace.package]

Cargo.toml

@@ -80,8 +80,9 @@ crate.from_cargo(
     manifests = [
         "//:Cargo.toml",
         "//py/tools/py:Cargo.toml",
-        "//py/tools/venv_bin:Cargo.toml",
         "//py/tools/unpack_bin:Cargo.toml",
+        "//py/tools/venv_bin:Cargo.toml",
+        "//py/tools/venv_shim:Cargo.toml",
     ],
 )
 use_repo(crate, "crate_index")

MODULE.bazel

@@ -281,8 +281,9 @@ crates_repository(
     manifests = [
         "//:Cargo.toml",
         "//py/tools/py:Cargo.toml",
-        "//py/tools/venv_bin:Cargo.toml",
         "//py/tools/unpack_bin:Cargo.toml",
+        "//py/tools/venv_bin:Cargo.toml",
+        "//py/tools/venv_shim:Cargo.toml",
     ],
 )
 

WORKSPACE

@@ -36,8 +36,9 @@ crates_repository(
     manifests = [
         "@aspect_rules_py//:Cargo.toml",
         "@aspect_rules_py//py/tools/py:Cargo.toml",
-        "@aspect_rules_py//py/tools/venv_bin:Cargo.toml",
         "@aspect_rules_py//py/tools/unpack_bin:Cargo.toml",
+        "@aspect_rules_py//py/tools/venv_bin:Cargo.toml",
+        "@aspect_rules_py//py/tools/venv_shim:Cargo.toml",
     ],
 )
 

e2e/smoke/WORKSPACE.bazel

@@ -0,0 +1,19 @@
+load("//tools/release:defs.bzl", "rust_binary")
+
+rust_binary(
+    name = "shim",
+    srcs = [
+        "src/main.rs",
+    ],
+    deps = [
+        "@crate_index//:miette",
+    ],
+)
+
+alias(
+    name = "venv_shim",
+    actual = ":shim",
+    visibility = [
+        "//visibility:public",
+    ],
+)

py/tools/venv_shim/BUILD.bazel

@@ -0,0 +1,22 @@
+[package]
+name = "venv_shim"
+version.workspace = true
+categories.workspace = true
+homepage.workspace = true
+repository.workspace = true
+license.workspace = true
+edition.workspace = true
+readme.workspace = true
+rust-version.workspace = true
+
+[features]
+debug = []
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[[bin]]
+name = "python_shim"
+path = "src/main.rs"
+
+[dependencies]
+miette = { workspace = true }

py/tools/venv_shim/Cargo.toml

@@ -0,0 +1,168 @@
+use miette::miette;
+use std::env;
+use std::fs;
+use std::io::{self, BufRead};
+use std::os::unix::process::CommandExt;
+use std::path::{Path, PathBuf};
+use std::process::Command;
+
+fn find_pyvenv_cfg(start_path: &Path) -> Option<PathBuf> {
+    let parent = start_path.parent()?.parent()?;
+    let cfg_path = parent.join("pyvenv.cfg");
+    if cfg_path.exists() && cfg_path.is_file() {
+        Some(cfg_path)
+    } else {
+        None
+    }
+}
+
+fn extract_pyvenv_version_info(cfg_path: &Path) -> Result<Option<String>, io::Error> {
+    let file = fs::File::open(cfg_path)?;
+    let reader = io::BufReader::new(file);
+    for line in reader.lines() {
+        let line = line?;
+        if let Some((key, value)) = line.split_once("=") {
+            let key = key.trim();
+            let value = value.trim();
+            if key == "version_info" {
+                return Ok(Some(value.to_string()));
+            }
+        }
+    }
+    Ok(None)
+}
+
+fn parse_version_info(version_str: &str) -> Option<String> {
+    // To avoid pulling in the regex crate, we're gonna do this by hand.
+    let parts: Vec<_> = version_str.split(".").collect();
+    match parts[..] {
+        [major, minor, ..] => Some(format!("{}.{}", major, minor)),
+        _ => None,
+    }
+}
+
+fn compare_versions(version_from_cfg: &str, executable_path: &Path) -> bool {
+    if let Some(file_name) = executable_path.file_name().and_then(|n| n.to_str()) {
+        return file_name.ends_with(&format!("python{}", version_from_cfg));
+    } else {
+        false
+    }
+}
+
+fn find_python_executables(version_from_cfg: &str, exclude_dir: &Path) -> Option<Vec<PathBuf>> {
+    let python_prefix = format!("python{}", version_from_cfg);
+    let path_env = env::var_os("PATH")?;
+
+    let binaries: Vec<_> = env::split_paths(&path_env)
+        .filter_map(|path_dir| {
+            let potential_executable = path_dir.join(&python_prefix);
+            if potential_executable.exists() && potential_executable.is_file() {
+                Some(potential_executable)
+            } else {
+                None
+            }
+        })
+        .filter(|potential_executable| potential_executable.parent() != Some(exclude_dir))
+        .filter(|potential_executable| compare_versions(version_from_cfg, &potential_executable))
+        .collect();
+
+    if binaries.len() > 0 {
+        Some(binaries)
+    } else {
+        None
+    }
+}
+
+fn main() -> miette::Result<()> {
+    let current_exe = env::current_exe().unwrap();
+    let args: Vec<_> = env::args().collect();
+
+    #[cfg(feature = "debug")]
+    println!("[aspect] Current executable path: {:?}", current_exe);
+
+    let pyvenv_cfg_path = match find_pyvenv_cfg(&current_exe) {
+        Some(path) => {
+            #[cfg(feature = "debug")]
+            eprintln!("[aspect] Found pyvenv.cfg at: {:?}", path);
+            path
+        }
+        None => {
+            return Err(miette!("pyvenv.cfg not found one directory level up."));
+        }
+    };
+
+    let version_info_result = extract_pyvenv_version_info(&pyvenv_cfg_path).unwrap();
+    let version_info = match version_info_result {
+        Some(v) => {
+            #[cfg(feature = "debug")]
+            eprintln!("[aspect] version_info from pyvenv.cfg: {}", v);
+            v
+        }
+        None => {
+            return Err(miette!("version_info key not found in pyvenv.cfg."));
+        }
+    };
+
+    let target_python_version = match parse_version_info(&version_info) {
+        Some(v) => {
+            #[cfg(feature = "debug")]
+            eprintln!("[aspect] Parsed target Python version (major.minor): {}", v);
+            v
+        }
+        None => {
+            return Err(miette!("Could not parse version_info as x.y."));
+        }
+    };
+
+    let exclude_dir = current_exe.parent().unwrap();
+    if let Some(python_executables) = find_python_executables(&target_python_version, exclude_dir) {
+        #[cfg(feature = "debug")]
+        {
+            eprintln!(
+                "[aspect] Found potential Python interpreters in PATH with matching version:"
+            );
+            for exe in &python_executables {
+                println!("[aspect] - {:?}", exe);
+            }
+        }
+
+        let interpreter_path = &python_executables[0];
+        let exe_path = current_exe.to_string_lossy().into_owned();
+        let exec_args = &args[1..];
+
+        #[cfg(feature = "debug")]
+        eprintln!(
+            "[aspect] Attempting to execute: {:?} with argv[0] as {:?} and args as {:?}",
+            interpreter_path, exe_path, exec_args,
+        );
+
+        let mut cmd = Command::new(interpreter_path);
+        cmd.args(exec_args);
+
+        // Lie about the value of argv0 to hoodwink the interpreter as to its
+        // location on Linux-based platforms.
+        if cfg!(target_os = "linux") {
+            cmd.arg0(&exe_path);
+        }
+
+        // On MacOS however, there are facilities for asking the C runtime/OS
+        // what the real name of the interpreter executable is, and that value
+        // is preferred while argv[0] is ignored. So we need to use a different
+        // mechanism to lie to the target interpreter about its own path.
+        //
+        // https://github.com/python/cpython/blob/68e72cf3a80362d0a2d57ff0c9f02553c378e537/Modules/getpath.c#L778
+        // https://docs.python.org/3/using/cmdline.html#envvar-PYTHONEXECUTABLE
+        if cfg!(target_os = "macos") {
+            cmd.env("PYTHONEXECUTABLE", exe_path);
+        }
+
+        let _ = cmd.exec();
+
+        return Ok(());
+    } else {
+        return Err(miette!(
+            "No suitable Python interpreter found in PATH matching version '{}'.",
+            &version_info,
+        ));
+    }
+}