How I built a sub-500ms latency voice agent from scratch

Nick Tikhonov's February 2026 write-up builds a phone voice agent from first principles and ships the code as shuo, an MIT-licensed repository of about 600 lines of Python. The post treats turn-taking as the hard problem: deciding when the caller has finished and when the agent should stop talking. The first pass uses Silero VAD and a pre-recorded reply to get the loop running, then shows where silence-based detection breaks down on real speech. The second pass switches to Deepgram Flux, which handles streaming transcription and end-of-turn detection in one model, and builds the full pipeline: Flux into the LLM, the LLM into ElevenLabs text-to-speech over pre-warmed WebSockets, and audio back to Twilio as μ-law over a WebSocket, all served by FastAPI. Later sections cover running locally, deploying on Railway in an EU region, and model selection. The latency figures come from this setup: about 1.6 seconds end to end when orchestrated from Turkey, about 690 ms server-side once hosted in the EU, and about 400 ms average after moving the LLM from OpenAI gpt-4o-mini to Groq's llama-3.3-70b. Keeping TTS sockets warm saved roughly 300 ms. The author estimates a Vapi setup with the same models at about 840 ms. The repo's README walks through ngrok setup, an .env of API keys and a pytest suite.

Topic

Agentic

Level

Intermediate

Format

Tutorial

Cost

Free

Duration

~12 min read, plus setup time to run the open-source shuo repo (Twilio, Deepgram, OpenAI and ElevenLabs keys required)

Provider

Nick Tikhonov

Hands-on

Yes — code/exercises

Certificate

None

• ✓Why turn-taking is the hardest part of building a real-time voice agent
• ✓How Silero VAD silence detection works and where it misfires on real callers
• ✓How Deepgram Flux combines streaming transcription with end-of-turn detection in one model
• ✓How to stream Twilio μ-law phone audio over WebSockets into a FastAPI server
• ✓How to chain LLM output into ElevenLabs text-to-speech so first audio arrives sooner
• ✓How pre-warmed TTS connections and EU hosting cut hundreds of milliseconds of latency
• ✓How switching from gpt-4o-mini to Groq llama-3.3-70b changed time to first token

• •Reports measured latency at each stage, from about 1.6 s locally to about 400 ms hosted
• •The companion repo shuo is MIT licensed, about 600 lines of Python, and had 672 GitHub stars in October 2026
• •The Show HN post reached 570 points and 153 comments in March 2026
• •HN commenters disputed the comparison with Vapi, which also handles tool calls, webhooks and logging on each turn
• •The author said on HN that LiveKit or a similar framework is likely the better production choice and that the from-scratch build was for understanding each component

This page was written from 3 sources, 2 on domains other than ntik.me.

1. 1.ntik.me — voice agentvendor
2. 2.github.com — shuo
3. 3.news.ycombinator.com — item