Unload model after being not used for some time

[hgruber]

Unlike to #66 it's probably also a good idea to unload the model after e.g. 300s of not being used.
For ollama server this is the standard. My graphics card uses 40W more of power with any model being loaded.

[fedirz]

I think I'll have this feature implemented by EOW. For now, you can use the DELETE /api/ps/{model_name:path} route to manually offload the model.

[samos123]

I would like this behind a flag so I can keep the model loaded for improved latency.

[Foddy]

Has anyone already found a solution for the problem, similar to how Ollama does it? Unfortunately, my GPU also constantly uses around 45W-50W in idle mode, which becomes quite expensive to operate given the local electricity prices (especially if the model is not used for an extended period) :S

[fedirz]

Has anyone already found a solution for the problem, similar to how Ollama does it? Unfortunately, my GPU also constantly uses around 45W-50W in idle mode, which becomes quite expensive to operate given the local electricity prices (especially if the model is not used for an extended period) :S

FYI, I'm currently working on this

[fedirz]

The feature has been implemented in #92 and is available on master. Will wait a couple of days before make a new release with the changes. Any feedback would be appreciated!

[hgruber]

Thanks, that was a quick one! Model unloading works, but some GPU Memory is still used by the process which keeps the GPU in P0 state and continues to consume more power than necessary:

|=========================================+======================+======================|
|   0  Tesla P40                      On  | 00000000:01:00.0 Off |                    0 |
| N/A   40C    P0              51W / 250W |    152MiB / 23040MiB |      0%	Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage	|
|=======================================================================================|
|    0   N/A  N/A   2441259	 C   ...ter-whisper-server/.venv/bin/python	 150MiB |
+---------------------------------------------------------------------------------------+

[fedirz]

@hgruber I'm aware of this, but unfortunately, this is an upstream issue which doesn't seem resolved yet. See (this)[https://github.com/SYSTRAN/faster-whisper/issues/992]. If you find a workaround, please LMK and I'll implement it.