RuntimeError: can't start new thread

[caohuongls]

I'm encountering an issue with my Raspberry Pi Zero 2W. The speakers connected via the 2 HAT won’t activate voice until I restart the service. This happens when I haven't used it for a few hours.

``
Sep 22 03:19:56 3dprint run[611]: Traceback (most recent call last):
Sep 22 03:19:56 3dprint run[611]: File "/root/wyoming-openwakeword/.venv/lib/python3.9/site-packages/wyoming/server.py", line 41, in run
Sep 22 03:19:56 3dprint run[611]: if not (await self.handle_event(event)):
Sep 22 03:19:56 3dprint run[611]: File "/root/wyoming-openwakeword/wyoming_openwakeword/handler.py", line 73, in handle_event
Sep 22 03:19:56 3dprint run[611]: ensure_loaded(
Sep 22 03:19:56 3dprint run[611]: File "/root/wyoming-openwakeword/wyoming_openwakeword/handler.py", line 256, in ensure_loaded
Sep 22 03:19:56 3dprint run[611]: state.ww_threads[model_key].start()
Sep 22 03:19:56 3dprint run[611]: File "/usr/lib/python3.9/threading.py", line 874, in start
Sep 22 03:19:56 3dprint run[611]: _start_new_thread(self._bootstrap, ())
Sep 22 03:19:56 3dprint run[611]: RuntimeError: can't start new thread

[vilhalmer]

I am seeing this also on two Zero 2W units. It looks to me like a leak, and I suspect it would eventually happen on any system but because this is 32-bit it runs out of resources much faster.

[vilhalmer]

These aren't very scientific, but some observations.

On a system which is stuck:

$ sudo lsof | grep openwakeword | wc -l
54468

$ sudo lsof | grep openwakeword | awk '{ print $3; }' | sort | uniq | wc -l  # Number of threads with open files
226

On a freshly started instance:

$ sudo lsof | grep openwakeword | wc -l
218
$ sudo lsof | grep openwakeword | awk '{ print $3; }' | sort | uniq | wc -l
9

Most of those lines are a handle to the model:

python3     586 32599 python3               root  209r      REG      179,2   206380     383319 /home/vil/wyoming-openwakeword/wyoming_openwakeword/models/ok_nabu_v0.1.tflite

It doesn't seem to be a file descriptor limit though, soft limit is set to the default of 1024 and it's nowhere near that:

$ sudo ls /proc/586/fd | wc -l
233

[rockstar2020]

Did you guys figure out what causes this error? It's so frustrating!
Temporarily I've added try/catch to the handler.py file so that it forces the code to exit which I'm hoping it'll force the service to restart when this error happens.
Hopefully somebody can come up with a smarter solution rather than a band-aid.