crash with sub-interpreters and threading

[ycdtosa]

Hi Everyone

I am trying to get a simple prototype where objects are created in the main thread, each of them owning a sub-interpreter. These objects will be used from different threads calling some embed python script, but the object keeps the subinterpreter alive until it goes out of scope, but I am not understanding the subinterpreters or when are some objects destroyed. probably once the Main interpreter is gone, and it is crashing on me.

The next sample does work and is based pybind11 example about subinterpreters.

#include <iostream>
#include <pybind11/embed.h>
#include <pybind11/gil.h>
#include <pybind11/subinterpreter.h>

namespace py = pybind11;

PYBIND11_EMBEDDED_MODULE(printer, m, py::multiple_interpreters::per_interpreter_gil()) {
    m.def("which", [](const std::string& when) {
        std::cout << when << "; Current Interpreter is "
                << py::subinterpreter::current().id()
                << std::endl;
    });
}

// forgive me for this, I don't want to the main interpreter unless someone needs it.
// Here it will be created only if required, and it will outlive main, and be destroyed afterwards
bool CreateMainInterpreter()
{
  py::scoped_interpreter *MainInterpreter = new py::scoped_interpreter;
  return true;
}

int main() {
    bool Initialize{ CreateMainInterpreter() };

    try  {
      // in the main thread, and the main interpreter
      py::module_::import("printer").attr("which")("First init"); 

      { // <--- I would want a different thread for this block.
        py::subinterpreter sub = py::subinterpreter::create(); //(1) create, but the gil goes back to the main interpreter
        py::subinterpreter_scoped_activate guard(sub);         //(2) activate this subinterpreter
        py::module_::import("printer").attr("which")("Created sub"); //(3) call code from subinterpreter, the gil is captured
      }; // (4)
    
    }  catch (py::error_already_set &e) {
      std::cerr << "EXCEPTION " << e.what() << std::endl;
      return 1;
    }

    return 0;
}

I am assuming that at (4) gil is released (back to main/previous interpreter)
the also subinterpreter is deactivated when scope(sub) goes out of scope
and then the subinterpreter is destroyed when sub goes out of scope

The output is the expected one...

First init; Current Interpreter is 0
Created sub; Current Interpreter is 1

Now, if I wrap the task in a thread, the code will segmentation fault.

      {  // wrap previous code in a thread
          std::thread Task{ []() {
              // (5) nothing preventing the task from starting like it was doing before
              py::subinterpreter sub = py::subinterpreter::create(); //(1) create, but the gil goes back to the main interpreter
              py::subinterpreter_scoped_activate guard(sub);         //(2) activate this subinterpreter
              py::module_::import("printer").attr("which")("Created sub"); //(3) call code from subinterpreter, the gil is captured
          });

          // (6) Nothing required for the Task to release the subprocess gil, etc and die, et all.
          Task.join();
      } 

sadlly I get...

First init; Current Interpreter is 0
Segmentation fault (core dumped)

#0  0x0000775852a77884 in PyErr_Fetch () from /lib/x86_64-linux-gnu/libpython3.13.so.1.0
#1  0x0000603954412ac3 in pybind11::error_scope::error_scope (this=0x775851cfecc8) at _deps/pybind11-src/include/pybind11/detail/common.h:1166
#2  0x000060395443edbd in pybind11::subinterpreter::create (cfg=...) at _deps/pybind11-src/include/pybind11/subinterpreter.h:81
#3  0x000060395443eb49 in pybind11::subinterpreter::create () at _deps/pybind11-src/include/pybind11/subinterpreter.h:138
#4  0x000060395441278c in main::$_0::operator() (this=0x60398e5421c8) at /home/ycdtosa/python_subinterpreters/src/main.cpp:32
#5  0x0000603954412765 in std::__invoke_impl<void, main::$_0> (__f=...) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61
#6  0x0000603954412725 in std::__invoke<main::$_0> (__fn=...) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:96
#7  0x00006039544126fd in std::thread::_Invoker<std::tuple<main::$_0> >::_M_invoke<0ul> (this=0x60398e5421c8) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_thread.h:301
#8  0x00006039544126d5 in std::thread::_Invoker<std::tuple<main::$_0> >::operator() (this=0x60398e5421c8) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_thread.h:308
#9  0x0000603954412649 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main::$_0> > >::_M_run (this=0x60398e5421c0) at /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_thread.h:253
#10 0x00007758524e1224 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x000077585229cb7b in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:448
#12 0x000077585231a7b8 in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Any ideas on what assumptions I got wrong?

Thanks

[ycdtosa]

I worked it out.
It seems the main process/thread needs to let the GIL go.

int main() {
    bool Initialize{ CreateMainInterpreter() };

    try
    {
      py::module_::import("printer").attr("which")("First init");

      //py::gil_scoped_release release;  // <-- this would do the trick, but anywhere before the join should work.

      std::thread Task( []() {
        py::gil_scoped_acquire scope;
        py::subinterpreter sub = py::subinterpreter::create();
        py::subinterpreter_scoped_activate guard(sub);
        py::module_::import("printer").attr("which")("Created sub");
      } );

      py::gil_scoped_release release;
      Task.join();
    }

    catch (py::error_already_set &e)
    {
      std::cerr << "EXCEPTION " << e.what() << std::endl;
      throw std::runtime_error( e.what() );
    }

    return 0;
}