[BUG]: scoped_interpreter destructor does not respond

[bogdan-lab]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

Problem description

I am trying to make a singleton wrapper for pybind11::scoped_interpreter. The idea is, that class will create separate thread for python, then create there scoped_interpreter and do all python calculation in that thread. The problem is that in this case scoped_interpreter destructor freezes. It actually feels like some deadlock, however I do not have evidence of that. It freezes on the line

detail::get_local_internals().registered_types_cpp.clear();

Interestingly, it freezes only in case when I try to join the python thread (interpreter is the local variable of that thread) in singleton class destructor. If I create separate method for joining python thread and call it somewhere before leaving the main(), interpreter destructor works OK.

Windows Server 2019 Standart
Visual Studio C++ 2019

Reproducible example code

class WorkerExample {
 public:
  static WorkerExample& Access() {
    static WorkerExample we;
    return we;
  }

  void Stop() {
    do_work_ = false;
    if (python_thread_.joinable()) python_thread_.join();
  }

  ~WorkerExample() { Stop(); }

 private:
  WorkerExample() {
    python_thread_ = std::thread([this]() {
      pybind11::scoped_interpreter guard{};

      while (do_work_) {
        // place holder for calculations, but bug can be reproduced without them
      }
    });
  }

  std::thread python_thread_;
  std::atomic<bool> do_work_ = true;
};

int main() {
  auto& we = WorkerExample::Access();
  // uncomment line below in order to avoid freezing in
  // scoped_interpreter destructor
  // we.Stop();

  return 0;
}

[bogdan-lab]

I found out that inside detail::get_local_internals() static variable is created.
Thus when I try to destroy py::scope_interpreter inside my singleton destructor I create static variable inside another static variable destructor.
Such code does not work well with requirements for the order of static variable destruction.
The solution is to create all static variables you may need in the "parent static" constructor.
So I placed get_local_internals() call in the WorkerExample constructor like this:

WorkerExample() {
  python_thread_ = std::thread([this]() {
    pybind11::scoped_interpreter guard{};            
    pybind11::detail::get_local_internals();

    while (do_work_) {
       // place holder for calculations, but bug can be reproduced without them
     }
   });
}

Now my code freezes in the constructor inside the call of get_local_internals()....

[bogdan-lab]

By putting pybind11::detail::get_local_internals() call inside the WorkerExample constructor I tried to guarantee that the static variable which is created in get_local_internals() will be created before the WorkerExample and therefore it will be safe to access it in the WorkerExample destructor.

However I placed the call in the thread and therefore guarantee which I pursued was not achieved.
The freezing goes away if I wait in the constructor for those local_internals to be created.
The code of the constructor is the following

WorkerExample() {

  std::promise<void> preparation_promise;

  python_thread_ = std::thread([this]() {
    pybind11::scoped_interpreter guard{};            
    pybind11::detail::get_local_internals();

    preparation_promise.set_value();
    while (do_work_) {
       // place holder for calculations, but bug can be reproduced without them
     }
   });
   
   preparation_promise.get_future().wait();
}

Now, when I wait until the local_internals are created I do not have any freezes inside the WorkerExample destructor.
Talking about freezing in WorkerExample constructor, which I mentioned above, I think I misinterpreted that.
I saw freezing in the thread, but I did not check how did execution reached there. Maybe it happened when WorkerExample destructor was calling python_thread_.join().

Anyway I still feel that my solution is not completely correct.
The rule says, that static variables will be destroyed in the order reversed to their creation order and programmer should guarantee that such order will be correct.
Now, when I see that this fix works, I start to think that this "order of creation/destruction" is in terms of time and not in terms of "code" (like line, which creates an instance is beneath or above the other line).
And I doubt that it is the way it works....

[bogdan-lab]

The problem was solved in the PR above.