Enhancement: Lower Latency by Optimizing Server Infrastructure

[Ankur2606]

We aim to lower the latency of our AI voice assistant system to provide a more responsive user experience. Currently, the system has room for improvement in terms of request processing time.

Steps for Improvement:

1. Switch to Gunicorn servers:

   • Replace the current Flask development server with Gunicorn.
   • Utilize multiple workers for handling concurrent requests.
   • Fine-tune the worker threads and timeout settings to balance speed and reliability.

2. Optimize I/O operations:

   • Ensure non-blocking and asynchronous I/O for network and database operations to reduce bottlenecks.

3. Caching Strategy:

   • Implement an effective caching mechanism (Redis/Memcached) to store frequent model inferences and common user requests.

Action Items:

• Integrate Gunicorn in the project.
• Refactor the backend to make I/O operations asynchronous where possible.
• Add a caching layer for frequent requests.
• Utilize Up-to-date latest models potentially providing low latency and accurate response