Voice Agent — LiveKit + Groq + Twilio

A conversational AI voice agent that books appointments, supports live monitoring, and performs warm transfers to human agents via Twilio.

Demo

📹 Watch the full demo on Loom

Architecture

Caller (Browser) ──► LiveKit Room ◄── Python Agent (LiveKit Agents SDK)
                                                │
                                    ┌───────────┴───────────┐
                                 Groq API              Twilio API
                              (LLM + STT + TTS)     (warm transfer)
                                                │
                                        call_state.json
                                                │
                              Monitor Dashboard (polls every 1s)

Tech Stack

Layer	Technology
Voice Agent	Python, LiveKit Agents SDK v1.6
LLM	Groq — Llama 3.3 70B
STT	Groq — Whisper Large v3 Turbo
TTS	Cartesia
Frontend	Next.js 14, Tailwind CSS, LiveKit JS SDK
Telephony	Twilio (outbound call for warm transfer)
Storage	JSON files (appointments + call state)

Features

• Voice conversation — natural back-and-forth with AI receptionist "Alex"
• Appointment booking — collects name, reason, date/time, phone; checks availability; confirms and stores booking
• Live monitor dashboard — real-time transcript, agent state, and appointment details via polling (no auth required, no mic access)
• Warm transfer — triggered by keywords (billing, complaint, talk to a human); Twilio calls a real phone number, speaks a summary, connects on accept
• Post-call summary — Groq LLM summarizes the full conversation when call ends

Prerequisites

• Node.js 18+
• Python 3.11–3.13 (3.14 may have dependency issues)
• ffmpeg — brew install ffmpeg

Setup

1. Clone & configure environment

git clone https://github.com/Keyur279/voice-agent.git
cd voice-agent
cp .env.example .env
# Fill in all values in .env

2. Backend

cd backend
python3 -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate
pip install -r requirements.txt
pip install livekit-plugins-cartesia

3. Frontend

cd frontend
npm install

Environment Variables

Create a .env file in the root with:

# LiveKit Cloud — cloud.livekit.io
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=APIxxxxxxx
LIVEKIT_API_SECRET=your_api_secret

# Groq — console.groq.com (free)
GROQ_API_KEY=gsk_xxxxxxx

# Cartesia — play.cartesia.ai (free tier)
CARTESIA_API_KEY=sk_car_xxxxxxx

# Twilio — twilio.com (free trial)
TWILIO_ACCOUNT_SID=ACxxxxxxx
TWILIO_AUTH_TOKEN=your_auth_token
TWILIO_PHONE_NUMBER=+1xxxxxxxxxx   # your purchased Twilio number
TRANSFER_TARGET_NUMBER=+91xxxxxxxxxx  # phone to ring on warm transfer

Running (3 terminals required)

Terminal 1 — Backend API server

cd backend
source venv/bin/activate
python server.py

Runs on http://localhost:8000

Terminal 2 — LiveKit Agent worker

cd backend
source venv/bin/activate
python agent.py dev

Terminal 3 — Frontend

cd frontend
npm run dev

Runs on http://localhost:3000

Testing the Full Flow

Test 1 — Voice call & appointment booking

1. Open http://localhost:3000
2. Click Start Call, allow microphone
3. Wait for Alex to greet you (~2 seconds)
4. Say: "Hi, I'd like to book an appointment"
5. Answer questions: name → reason → date/time → phone number
6. Alex confirms and reads back the booking

Verify booking: http://localhost:8000/api/appointments

Test 2 — Live monitor dashboard

1. Open http://localhost:3000/monitor in a separate tab
2. Start a call in the first tab
3. Watch transcript and agent state update live in the monitor tab (no login, no mic needed)

Test 3 — Warm transfer

During a call, say any of:

• "I have a billing complaint"
• "I want to talk to a human"
• "Can I speak to a real person"
• "I want to speak to a manager"

Alex says "please hold" → your phone (TRANSFER_TARGET_NUMBER) rings from the Twilio number → call speaks a summary and asks to press 1 (accept) or 2 (decline).

Test 4 — Post-call summary

Click Disconnect after any call. The summary appears automatically in the monitor dashboard.

Flow Explanations

Appointment Booking

1. Caller clicks "Start Call" → gets a LiveKit token → joins the room
2. Python agent connects to the same room and greets the caller
3. Groq Whisper transcribes speech → Llama 3.3 generates responses
4. Agent collects: name, reason, preferred date/time, contact number
5. check_availability tool confirms the slot is free
6. book_appointment tool stores the booking in appointments.json
7. Agent reads the confirmation ID back to the caller

Live Monitoring

1. Watcher opens /monitor — no login, no microphone access
2. Page polls /api/call-state every second
3. Agent writes transcript, state, and appointment events to call_state.json
4. Dashboard updates in real time with transcript, agent state, and booking details

Warm Transfer

1. Caller mentions billing, complaint, or asks for a human
2. Intent detected via keyword matching in the transcript
3. Groq generates a short call summary
4. Agent says "please hold" → Twilio dials TRANSFER_TARGET_NUMBER
5. Twilio plays the summary and asks human agent to press 1 (accept) or 2 (decline)
6. If accepted: agent tells caller they're connected and exits
7. If declined: agent returns to caller — "our team isn't available right now"

Post-call Summary

• Generated by Groq LLM (Llama 3.3) from the full conversation transcript
• Written to call_state.json when the call ends
• Displayed automatically in the monitor dashboard

Project Structure

voice-agent/
├── .env.example
├── .gitignore
├── README.md
├── backend/
│   ├── requirements.txt    # Python dependencies
│   ├── agent.py            # LiveKit voice agent — conversation, booking, transfer
│   ├── tools.py            # check_availability + book_appointment tool calls
│   ├── transfer.py         # Twilio warm transfer logic
│   ├── prompts.py          # System prompt + summary prompt
│   └── server.py           # FastAPI — token API, call state, appointments
└── frontend/
    ├── app/
    │   ├── page.tsx            # Caller UI — Start Call, visualizer, disconnect
    │   └── monitor/page.tsx    # Monitor dashboard — transcript, state, summary
    ├── components/
    │   ├── AgentStatus.tsx     # Listening/Thinking/Speaking indicator
    │   ├── TranscriptFeed.tsx  # Scrolling transcript bubbles
    │   └── TakeoverButton.tsx  # (available for future watcher takeover)
    └── lib/livekit.ts          # Token fetch + event parsing helpers