AI Communication

A real-time voice conversation app for practicing spoken English with an AI partner. Learners talk naturally with a friendly AI agent over LiveKit, then receive bilingual (English–Vietnamese) feedback on their speaking.

Features

• Real-time voice chat with an AI conversation partner (speech in, speech out)
• Casual, friend-like dialogue: short replies, follow-up questions, occasional new phrases
• Voice activity detection for natural turn-taking
• Post-conversation AI feedback: overall impression, specific feedback, pronunciation respelling suggestions, and tips
• Bilingual feedback (English + Vietnamese)

Tech Stack

Component	Technology
Real-time audio	LiveKit (Agents, Server SDK, React components)
Speech-to-Text	Deepgram nova-2
LLM (conversation)	OpenAI gpt-4o-mini
Text-to-Speech	OpenAI TTS (alloy voice)
VAD	Silero
Feedback LLM	Groq – llama-3.3-70b-versatile
Token / feedback API	Flask, Flask-CORS
Frontend	React 18, Vite

Architecture

┌──────────────┐   token    ┌────────────────┐
│  React App   │ ─────────▶ │  Flask server  │  /getToken, /agentToken, /aiFeedback
│  (Vite)      │ ◀───────── │  (server.py)   │ ──▶ Groq (feedback)
└──────┬───────┘            └────────────────┘
       │ WebRTC audio
       ▼
┌──────────────┐            ┌────────────────┐
│ LiveKit room │ ◀────────▶ │  AI Agent      │  Deepgram STT → GPT-4o-mini → OpenAI TTS
└──────────────┘            │  (agent.py)    │
                            └────────────────┘

Project Structure

├── backend/
│   ├── agent.py          # LiveKit voice agent (STT / LLM / TTS / VAD)
│   ├── server.py         # Flask API: LiveKit tokens + AI feedback
│   ├── function.py       # Groq feedback generation
│   ├── prompts.py        # Welcome message & system prompts
│   └── requirements.txt
├── frontend/
│   ├── src/
│   │   ├── App.jsx
│   │   ├── FeedbackAi.jsx                     # Feedback view
│   │   └── components/
│   │       ├── LiveKitModal.jsx               # Room connection
│   │       └── SimpleVoiceAssistant.jsx       # Voice UI
│   ├── vite.config.js
│   └── package.json
└── test_api.py           # Test script for /aiFeedback

Getting Started

Requirements

• Python 3.10+
• Node.js 18+
• A LiveKit server (LiveKit Cloud or self-hosted)
• API keys: OpenAI, Deepgram, Groq

1. Backend

cd backend
pip install -r requirements.txt

Create backend/.env:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_PUBLIC_URL=wss://your-project.livekit.cloud   # optional
LIVEKIT_API_KEY=your_livekit_key
LIVEKIT_API_SECRET=your_livekit_secret
OPENAI_API_KEY=your_openai_key
DEEPGRAM_API_KEY=your_deepgram_key
GROQ_API_KEY=your_groq_key
LLM_TEMPERATURE=1.0                                   # optional

Run the API server (port 8080):

python server.py

Run the voice agent in another terminal:

python agent.py dev

2. Frontend

cd frontend
npm install
npm run dev

Create frontend/.env if needed:

VITE_LIVEKIT_URL=wss://your-project.livekit.cloud

Note: The frontend currently points to the deployed backend (https://aicom.aespwithai.com) in src/components/LiveKitModal.jsx and vite.config.js. For local development, switch both to http://127.0.0.1:8080.

API

Method	Endpoint	Description
GET	/getToken?name=&room=	LiveKit access token for the user (auto-generates a room if omitted)
GET	/agentToken?room=	LiveKit access token for the AI agent
POST	/aiFeedback/	Generate speaking feedback from a conversation transcript

Example feedback request:

{
  "text": "Agent: Hi there! Which topic do you want to talk about? User: I want to talk about Vietnam's history...",
  "bilingual": true
}

Response:

{
  "feedback": "SECTION 1 - ENGLISH ... SECTION 2 - VIETNAMESE ..."
}

Test it with:

python test_api.py

Author

Nguyen Quang Hung – @hung1962003