Error: KeyError: '__builtins__' with globals when importing from a callback on Python >= 3.10

[nertpinx]

Bug Description

When executing with GIL and running a python code with some globals set, which calls back into rust, then trying to import a module in this lower stack fails with KeyError: 'builtins'.

Steps to Reproduce

use pyo3::types::{PyDictMethods, PyTracebackMethods};

#[pyo3::pyclass]
#[derive(Clone)]
struct Cls {
    code: String,
}

impl Cls {
    fn reproducer(&mut self, py: pyo3::Python) -> pyo3::PyResult<()> {
        let variables = pyo3::types::PyDict::new(py);
        variables.set_item("cls", pyo3::Py::new(py, self.clone())?)?;

        let code = std::ffi::CString::new(self.code.as_str())?;
        let output = py.run(code.as_c_str(), None, Some(&variables));

        if let Err(ref err) = output {
            if let Some(tb) = err.traceback(py) {
                eprint!("{}", tb.format()?);
            }
        }

        output?;

        println!("Running with globals now");

        let output = py.run(code.as_c_str(), Some(&variables), None);

        if let Err(ref err) = output {
            if let Some(tb) = err.traceback(py) {
                eprint!("{}", tb.format()?);
            }
        }

        output?;

        Ok(())
    }
}

#[pyo3::pymethods]
impl Cls {
    fn func(&mut self, py: pyo3::Python) -> pyo3::PyResult<()> {
        /* Importing anything here, even an empty name "" gives the same error */
        py.import("math")?;

        println!("func exec");

        Ok(())
    }

}

fn main() -> eyre::Result<()> {
    let mut cls = Cls {
        code: r#"
cls.func()
"#.to_string(),
    };
    pyo3::prepare_freethreaded_python();

    pyo3::Python::with_gil(|py| {
        /* Importing here works */
        py.import("math")?;

        println!("Imported");

        cls.reproducer(py)
    })?;

    Ok(())
}

[package]
name = "pyo3-test"
version = "0.1.0"
edition = "2021"

[dependencies]
eyre = "0.6"
pyo3 = "0.23"

This will print:

Imported
func exec
Running with globals now
Traceback (most recent call last):
  File "<string>", line 2, in <module>
Error: KeyError: '__builtins__'

Location:
    src/main.rs:62:5

Backtrace

The traceback is not that interesting or helpful:

Traceback (most recent call last):
  File "<string>", line 2, in <module>
Error: KeyError: '__builtins__'

Your operating system and version

Gentoo Linux

Your Python version (python --version)

Python 3.13.2

Your Rust version (rustc --version)

rustc 1.82.0 (f6e511eec 2024-10-15)

Your PyO3 version

0.23.4

How did you install python? Did you use a virtualenv?

emerge python, no virtualenv

Additional Info

• This is similar to Python 3.{8,9}: NameError: __build_class__ not found when globals are provided to py.run() #3370 with the exception that this is two levels down the stack (rust->python->rust->py.import()).
• The issue goes away with the abi3-py39 feature due to the cfg feature gate in https://github.com/PyO3/pyo3/blob/main/src/marker.rs#L652
• It also did not happen with pyo3 0.22.3
• It might be some cpython issue, just like in the Python 3.{8,9}: NameError: __build_class__ not found when globals are provided to py.run() #3370 issue, but I hope it can be fixed with just a change in the feature gate in run_code() as It would be nice to have this fixed before the 0.24 release in 0.24 Release #4905 but I understand that the rabbit hole might ho deeper than I think, so fingers crossed 🤞 it does not.

[ngoldbaum]

The issue goes away with the abi3-py39 feature due to the cfg feature gate in https://github.com/PyO3/pyo3/blob/main/src/marker.rs#L652

The #[cfg(not(Py_3_10))] is new in 0.23, I added it because I ran into some test failures related to this code on the free-threaded build. That said, I can also reproduce the KeyError you've reported here on the free-threaded build so we definitely need a more fine-grained fix. I suspect that this is indeed a CPython bug and I'd encourage you to report it upstream (although they may ask you for a reproducer that uses just the C API...).

Maybe we can hold a critical section on the __builtins__ dict if it hasn't been filled? I'll take a look if that fixes it and avoids errors from races trying to add __builtins__

[nertpinx]

Thanks for looking into it. I filed an upstream issue with the reproducer, but also noticed that PyImport_Import() has specifically a different documentation: https://docs.python.org/3/c-api/import.html#c.PyImport_Import

This is a higher-level interface that calls the current “import hook function” (with an explicit level of 0, meaning absolute import). It invokes the import() function from the builtins of the current globals. This means that the import is done using whatever import hooks are installed in the current environment.

[ngoldbaum]

I opened #4921