Voice agents
by OpenAI
Speech-to-speech or chained pipeline — the voice-agent decision, made properly.
Overview
This is OpenAI's official guide to building voice agents, and its central act is a decision rather than a walkthrough. It contrasts a speech-to-speech architecture, where a realtime model such as gpt-realtime-2.1 consumes and emits live audio in a single multimodal pass for the lowest latency and the most natural turn-taking, against a chained pipeline that runs speech-to-text, then a text model such as gpt-5.6, then text-to-speech — trading latency for explicit control over the intermediate transcript and letting an existing text agent be reused unchanged. The sections run: choose the right architecture, recommended starting points, build a speech-to-speech voice agent, build a chained voice workflow, the shared agent building blocks, and next steps. Code is SDK-first: TypeScript RealtimeAgent and RealtimeSession for browser assistants over WebRTC, and the Python Agents SDK VoicePipeline wrapping an existing Agent with function tools and streamed audio events, with WebSocket transport for server-side use. The final section is the one that matters most to engineers already shipping agents — voice changes the transport, not the architecture, so tool calling, streaming and state management, handoffs to specialist agents, guardrails and human review, MCP-backed integrations and observability all carry over. It links onward to the realtime API reference, session lifecycle, WebRTC connection details and realtime prompting.
At a Glance
- Topic
- Agentic
- Level
- Intermediate
- Format
- Guide
- Cost
- Free
- Duration
- ~30-45 min read, plus runnable TypeScript and Python SDK examples
- Provider
- OpenAI
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Choose deliberately between speech-to-speech and chained STT-LLM-TTS voice architectures
- ✓Build a browser voice assistant using RealtimeAgent and RealtimeSession over WebRTC
- ✓Wrap an existing Python text agent in a VoicePipeline with function tools
- ✓Stream audio events and manage session state during a live conversation
- ✓Apply guardrails and human review to a spoken, low-latency interaction
- ✓Hand off between specialist agents mid-conversation via tool-call-driven transfers
- ✓Pick WebRTC for browser clients and WebSocket for server-side audio transport
Highlights
- •Leads with the architecture decision instead of one blessed path — that choice is the expensive one to reverse later
- •Current rather than stale: it names gpt-realtime-2.1 and gpt-5.6, not the long-deprecated realtime preview models
- •Makes the reuse argument explicit — tools, handoffs, guardrails, MCP and observability transfer from text agents untouched
- •Backed by openai/openai-realtime-agents, a 6.9k-star MIT-licensed Next.js reference app demonstrating chat-supervisor and sequential-handoff patterns
- •Covers both the TypeScript and Python SDK paths with runnable snippets rather than pseudocode
Who It’s For
Best For
- ✓Agent engineers adding voice to an existing text agent
- ✓Developers building phone, kiosk or in-browser support agents
- ✓Teams choosing between realtime speech-to-speech and a chained voice pipeline
Prerequisites
- •Familiarity with the OpenAI Agents SDK or a comparable agent framework
- •Working TypeScript or Python, plus an OpenAI API key
- •Basic understanding of WebRTC or WebSocket audio transport helps
FAQ
What is Voice agents?
OpenAI's official guide to building voice agents, organized around the architectural choice that is expensive to reverse: a single realtime speech-to-speech model versus a chained speech-to-text, reasoning, text-to-speech pipeline. Shows both paths in code with the TypeScript RealtimeAgent and the Python VoicePipeline, and explains which agent building blocks — tools, handoffs, guardrails, MCP — carry over unchanged from text agents.
Is Voice agents free?
Voice agents is free to access.
What level is Voice agents for?
Voice agents is aimed at a intermediate audience. Recommended background: Familiarity with the OpenAI Agents SDK or a comparable agent framework, Working TypeScript or Python, plus an OpenAI API key, Basic understanding of WebRTC or WebSocket audio transport helps.
How long does Voice agents take?
Expect roughly ~30-45 min read, plus runnable TypeScript and Python SDK examples. Most learners work through it at their own pace.
What will I learn from Voice agents?
You'll learn: Choose deliberately between speech-to-speech and chained STT-LLM-TTS voice architectures; Build a browser voice assistant using RealtimeAgent and RealtimeSession over WebRTC; Wrap an existing Python text agent in a VoicePipeline with function tools; Stream audio events and manage session state during a live conversation; Apply guardrails and human review to a spoken, low-latency interaction; Hand off between specialist agents mid-conversation via tool-call-driven transfers; Pick WebRTC for browser clients and WebSocket for server-side audio transport.
Topics
Sources
This page was written from 2 sources, 1 on domains other than developers.openai.com.