LiveKit Tutorial: Build an AI Voice Agent From Scratch (2026)

Prerequisites and Project Setup

You'll need a LiveKit Cloud account (the free Build tier is enough for this tutorial), API keys for a speech-to-text provider (Deepgram), an LLM provider (OpenAI), and a TTS provider (Deepgram Aura-2, Cartesia, or ElevenLabs). Node.js 18+ or Python 3.9+ both work — this tutorial uses Python for the agent worker since it's the more common choice in production LiveKit deployments today.

Step 1: Define the Agent

A LiveKit agent is a worker process that joins a room and runs a voice pipeline. The core building block is an AgentSession, which wires together VAD (to detect when someone is speaking), STT, the LLM, and TTS into one coordinated loop. This is the piece that handles interruption logic for you — when the user starts talking while the agent is speaking, the session automatically stops TTS output and reprocesses the new input, which is the single hardest thing to get right if you build this from scratch.

Step 2: Give the Agent Tools

A voice agent that can only talk isn't useful for business automation — it needs to take actions. LiveKit Agents lets you register Python functions as tools the LLM can call mid-conversation, the same function-calling pattern used in text-based LLM apps. For a restaurant or receptionist use case, this is where check_availability(), book_appointment(), or transfer_call() live.

The critical engineering discipline here: write tight, unambiguous docstrings for each tool. The LLM decides when to call a tool based on the description you give it — vague descriptions cause it to either call the wrong tool or hallucinate parameters instead of asking a clarifying question.

Step 3: Handle Turn Detection Properly

The default voice-activity-detection approach (Silero VAD) works on silence gaps, but it struggles with natural pauses mid-sentence ('I want to book a table for... let me check... four people'). For production quality, pair it with a turn-detection-aware STT model like Deepgram Flux, which is trained specifically to distinguish a mid-thought pause from an actual end of turn. This single change is usually the biggest perceived-quality improvement you can make to a voice agent.

Step 4: Connect a Real Phone Number

LiveKit has native SIP support, which means you can route an inbound phone number (via Twilio, Telnyx, or LiveKit's own phone number product) directly into a room your agent joins — the same agent code that worked in a browser demo now answers real phone calls with no architecture changes. This is the step that turns a tutorial project into an actual product.

Step 5: Deploy to Production

For production, deploy your agent worker to LiveKit Cloud's managed agent hosting rather than running it as a long-lived process you manage yourself — this gets you auto-scaling, observability (transcripts, trace spans, logs per session), and avoids cold-start issues that plague self-managed deployments. LiveKit Cloud's Build tier is free for prototyping; production traffic needs at minimum the Ship tier ($50/month) to avoid hard usage caps.

Latency Budget: What to Actually Tune

Cost of Running This in Production

On LiveKit Cloud, agent session minutes are billed at $0.01/min on top of the plan's included allotment, WebRTC minutes at roughly $0.0004-0.0005/min, and SIP/telephony minutes at $0.003-0.004/min. Add Deepgram STT (~$0.0048-0.0077/min), an LLM (roughly $0.005-0.02/min depending on model and conversation length), and TTS (roughly $0.01-0.07/min depending on provider). A realistic all-in cost for a GPT-4o-mini + Deepgram + Cartesia stack lands around $0.04-0.08 per minute at moderate volume — before any LiveKit plan minimums.

Implementation Checklist

• Prototype in a browser room before wiring up telephony — isolate voice pipeline bugs from SIP/telephony bugs
• Write explicit, narrow tool docstrings — this is the #1 lever for reducing hallucinated tool calls
• Swap in a turn-detection-aware STT model before going to production, not after
• Load test with simulated concurrent calls at your expected peak, not average volume
• Set up LiveKit Cloud's observability dashboard before launch, not after your first incident
• Confirm your LiveKit plan tier supports the compliance requirements you need (HIPAA is Scale-tier and above)

Common Mistakes

• Building the interruption/turn-taking logic from scratch instead of using the Agents framework's built-in handling
• Using a non-streaming TTS model, which adds 1-2+ seconds of dead air before the agent starts speaking
• Deploying directly to Twilio without going through LiveKit's SIP layer, losing the framework's session management
• Skipping the Ship-tier upgrade and hitting hard usage caps mid-launch on the free Build tier
• Not testing what happens when a tool call fails (e.g., calendar API times out) — the agent needs a graceful fallback line, not silence

FAQs