How to safely convert std::runtime_error that contains non UTF8 words to Python RuntimeError after pybind v2.7

[TianciXiao]

I implement a pybind method, which will call another lib to do some things. But this lib may raise exceptions contains non UTF8 words like '0xff'. So when I catch that exception and try to raise my own std::runtime_error. This pybind method will raise 'UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position xxx: invalid start byte' when I call MyFunc from Python.

.def("MyFunc", []() {
  try {
    call_other_func();
  } catch (const std::exception &err) {
    raise std::runtime_error("Call MyFunc failed, err: " + std::string(err.what()));
  }
})

This situation only occurs with pybind after v2.7. I guess it may be relative to the change about py::str https://pybind11.readthedocs.io/en/stable/upgrade.html#v2-7

So is there a safe way to convert std::runtime_error that contains non UTF8 words to Python RuntimeError?

[ycdtosa]

[TianciXiao]

I guess you replied to the wrong person. But congratulations!

[ycdtosa]

Ouch. Thanks. I was reading your issue because I ran into the same issue. Lets see if I get lucky in this one too.

…

On Fri, Nov 28, 2025, 02:34 Tianci Xiao ***@***.***> wrote: I guess you replied to the wrong person. But congratulations! — Reply to this email directly, view it on GitHub <#5900 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFEHGLM462GKVOZZD4SOQT366RB7AVCNFSM6AAAAACMRGYVE2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKMJQGEZDONQ> . You are receiving this because you commented.Message ID: ***@***.***>

[TianciXiao]

Here is my solution for now. But I think this should be done in pybind when it tries to convert std::runtime_error to Python RuntimeError.

.def("MyFunc", []() {
  try {
    call_other_func();
  } catch (const std::exception &err) {
    std::string error_msg = "Call MyFunc failed, err: " + std::string(err.what());
    // Decode the error message to UTF-8 and replace non UTF-8 characters with backslash.
    py::handle utf8_str_handle = PyUnicode_DecodeUTF8(error_msg.data(), error_msg.size(), "backslashreplace");
    if (!utf8_str_handle) {
      throw py::error_already_set();
    }
    py::str utf8_str = py::reinterpret_steal<py::str>(utf8_str_handle);
    raise std::runtime_error(utf8_str);
  }
})

[ycdtosa]

yeap. It looks very verbose.