OpenAI speech and voice API integration

OpenAI offers language, speech and image models through an API your software calls over the internet. You send text or audio, receive text, audio or structured data, and pay by usage.

We have used OpenAI models in voice agent experiments.

• The feature needs speech and language from one account. OpenAI covers transcription, text generation and synthetic voice, which keeps an early voice or call-notes prototype to one provider and one bill.
• Output can be checked by code or a person. Pulling a name, date and budget from an enquiry into set fields works well, because a validator or a reviewer catches bad results.
• Your contracts allow text to go to an outside provider.

• The data can’t leave your machines. Start with a local open model (Ollama) or compare setups (hosted against self-hosted).
• Answers must come only from your records, and there is no retrieval layer yet. The model alone fills gaps with guesses. Build retrieval first (RAG), with the model citing passages and refusing when nothing matches.

There are two common ways to build a voice feature on hosted models. A realtime speech-to-speech model takes the caller’s audio and answers in audio within one session. With fewer hand-offs, replies can start sooner, tone carries through, and interruptions are handled inside the session.

A pipeline transcribes the speech, sends the text to a language model, then turns the reply into speech. Each step is a separate choice, so the transcription, model and voice can come from different providers. The text in the middle can also be logged, checked against rules or filtered before anything is said aloud. The cost is an extra hop at each step, which adds delay, and tone in the caller’s voice is lost at transcription.

Choose realtime when a natural back-and-forth matters most. Choose a pipeline when you need to control or record exactly what the agent says, or want to swap one part later. Either way, audio needs a real-time connection: WebRTC, a WebSocket or a phone (SIP) link. LiveKit is one way to run that layer (LiveKit). Both approaches are billed differently from text requests, so compare costs on your own call lengths.

For text tasks such as turning an enquiry into fields, OpenAI’s structured outputs make the reply match a JSON schema you supply. That removes parsing failures but not wrong values: a date can be well-formed and still wrong. Validate business rules in code, and send doubtful records to a person.

• A model update changes answers. Keep a fixed set of real cases and rerun it before any switch.
• Waiting on a live call. Each step in a voice flow adds delay, so test the whole flow by phone before agreeing what an agent should handle.

• AI voice agents
• AI automation