Quick answer: To build an AI receptionist you wire nine parts in order: a phone number or SIP trunk, a telephony bridge, a speech pipeline (STT, LLM and TTS, or one speech-to-speech model), a prompt and knowledge base, booking tools, warm transfer, voicemail handling, recording consent and AI disclosure, then tests and monitoring. Pipecat says its quickstart takes under 5 minutes; the phone line, transfer, voicemail and compliance parts are the real build.
This is the build order; the engineer-day estimate is in how long it takes to build a voice AI agent in-house. Each step below names the Pipecat, LiveKit Agents, Twilio or OpenAI documentation that covers it, read on 14 October 2026, and the last section says when not to build at all.
How do I build an AI receptionist agent myself?
If your starting point is “I want to build an AI receptionist for my business”, work through these nine steps in order. Each depends on the one before: a booking tool cannot be tested without a voice pipeline, and a warm transfer cannot work without an outbound leg to dial the human.
- Get a phone number or SIP trunk that can receive calls and place an outbound leg.
- Choose a telephony bridge: raw audio, text, SIP into a framework, or SIP straight to a model.
- Choose the speech architecture: a cascade or a speech-to-speech model.
- Write the prompt and knowledge base for a listener, not a reader.
- Add booking as tool calls.
- Build the warm transfer, including the branch where nobody answers.
- Handle voicemail both ways: take messages inbound, detect machines outbound.
- Settle recording consent and AI disclosure for every jurisdiction you take calls from.
- Test, then monitor.
What to do on day one. Before you open a console, write down the five call types your receptionist must finish without a human (for example: new booking, reschedule, opening hours, “speak to someone”, message for a named person). Every later step is tested against that list. Then run one framework quickstart, to learn whether you can live with its language and SDK before you buy a number.
The finish state. You are done when a stranger can ring at 2 am, hear the AI disclosure in the greeting, book a slot that appears in your calendar exactly once, ask for a person and either reach one with context or leave a message, and when that call appears in your monitoring with a reviewable transcript.
The receptionist build bill of materials
Each component, its documented options, and the failure a caller meets if it is missing.
| # | Component | Documented options (read 14 October 2026) | What breaks if it is missing |
|---|---|---|---|
| 1 | Phone number or SIP trunk | LiveKit Phone Numbers (buy a number, create a dispatch rule); a third-party SIP trunk (Twilio, Telnyx, Plivo and others in LiveKit’s provider guides); a Twilio number driving TwiML | Nobody can call it. Without an outbound route as well, warm transfers and callbacks fail even on an inbound-only receptionist; a cold SIP REFER forward instead needs call transfers enabled on your provider trunk. |
| 2 | Telephony bridge | Twilio Media Streams (bidirectional, <Connect><Stream>); Twilio ConversationRelay; LiveKit SIP; Pipecat over Twilio, Telnyx, Plivo or Exotel WebSockets, or Daily PSTN/SIP; OpenAI Realtime API SIP |
Audio never reaches your code, or arrives in a format your speech models were not tuned for (8 kHz μ-law on Twilio Media Streams). |
| 3 | Speech pipeline | Cascade: separate STT, LLM and TTS services. Speech-to-speech: OpenAI Realtime or GPT-Live, via Pipecat or direct | Either the agent cannot hear or speak, or it sounds natural but follows complex booking instructions less reliably than a cascade (Pipecat’s guidance, step 3). |
| 4 | System prompt and knowledge base | Instructions in the session config; retrieval (LiveKit’s Build Agents section covers RAG); TTS text normalisation per Twilio’s ConversationRelay best practices | Wrong opening hours, invented prices, “$20.50” read aloud as symbols, answers to questions a receptionist should refuse. |
| 5 | Booking tools | Function calling in Pipecat or LiveKit Agents; tools delegated from a speech-to-speech session | “I’ve booked you in” with nothing in the calendar, or a double booking when the caller interrupts mid-write. |
| 6 | Transfer to a human | LiveKit cold transfer (SIP REFER) and WarmTransferTask (beta, Python); ConversationRelay end message with handoffData; OpenAI Realtime refer endpoint |
The caller who says “let me speak to someone” is trapped, or is dropped into a ringing phone with no context. |
| 7 | Voicemail and machine handling | A take-a-message tool for inbound; Twilio AMD, LiveKit AMD or Pipecat VoicemailDetector for outbound legs | Messages are lost, or the agent talks over a voicemail greeting on a callback or a transfer leg. |
| 8 | Recording consent and AI disclosure | Greeting line (ConversationRelay welcomeGreeting, or the agent’s first turn); provider or in-pipeline recording (Pipecat AudioBufferProcessor) |
An unlawful recording in a jurisdiction that needs all-party consent; an outbound callback made without the consent the FCC requires in the US. |
| 9 | Testing and monitoring | Pipecat Evals (scripted and simulated callers); LiveKit test framework (pytest and Vitest), simulations, OpenTelemetry export; Pipecat metrics (TTFB, processing time, usage) | A prompt edit silently stops the booking tool being called, and you find out from a customer. |
A receptionist is not one model; it is nine components, and callers only ever experience the weakest one.
Step 1: Which phone number or SIP trunk do I need?
You need a number that rings into your stack and an outbound route out of it, and the second half is the one people skip. LiveKit’s warm-transfer guide states that “in order for your agent to transfer calls to another number, you need outbound trunk configuration”, and that receiving calls additionally needs an inbound trunk and a dispatch rule. So on LiveKit a receptionist that never makes a sales call still needs outbound trunking the moment it does an agent-assisted warm transfer. A cold transfer is different: it goes out as a SIP REFER through your provider trunk, as below.
The simplest documented LiveKit path is a LiveKit Phone Number plus a dispatch rule; LiveKit says that with its own numbers “no inbound trunk configuration or verification is required”. For cold transfers by SIP REFER, LiveKit says your provider trunk must allow call transfers; on Twilio that means enabling call transfer and PSTN transfers on the trunk. For Australian builders, Twilio’s Media Streams overview lists the Australia (AU1) Region as supported, alongside Ireland (IE1).
Step 2: Which telephony bridge connects the call to my agent?
The bridge decides what your code receives, and so how much of the speech stack you own. There are four documented shapes.
| Bridge | What your code receives | Who runs speech recognition and synthesis | Transfer route, as documented |
|---|---|---|---|
| Twilio Media Streams, bidirectional | Raw audio over a WebSocket: base64 audio/x-mulaw, sample rate 8000, one channel |
You | Not part of the stream itself; Pipecat lists “no advanced call center features like transfers or reconnects” as the limitation of its WebSocket connections, so plan transfers in your call-control layer |
| Twilio ConversationRelay | Transcribed caller speech as structured messages; you send text back | Twilio, with selectable providers (Deepgram, Google, Amazon, ElevenLabs model cards are published) | Send an end message with handoffData; Twilio returns control of the call and posts that data to your <Connect> action URL |
| SIP into a framework (LiveKit SIP; Pipecat with Daily SIP or PSTN) | The caller as a SIP participant in a room; your agent joins the room | You, through the framework’s plugins | SIP REFER (cold), or an agent-assisted warm transfer |
| SIP straight to the model (OpenAI Realtime API) | A realtime.call.incoming webhook; you accept or reject, then attach a WebSocket to monitor |
The model, speech-to-speech | refer endpoint; OpenAI says “the downstream system handles the rest of the call flow for the caller” |
On Twilio, “a Call can have only one bidirectional Stream active at a time”, and unidirectional forks share a four-track limit with SIPREC, real-time transcription and answering machine detection. On the direct-to-model path, OpenAI’s telephony page has you create a project webhook first, point your trunk at sip.api.openai.com (or sip-eu.api.openai.com for European data residency) with your project ID over TLS, and allow outbound TLS on port 5061 plus bidirectional SRTP over UDP to four published /28 ranges. The page also has a GPT-Live tab; follow the one for your API.
Pipecat’s production guidance adds the constraint that decides hosting: “a caller expects to hear something within a second or two of being connected”, which it says can often “rule out cold-start-heavy patterns”. Choose the bridge after reading step 6, because it decides whether a warm transfer is a setting or a project.
Step 3: Should I use STT, LLM and TTS, or a speech-to-speech model?
Pipecat’s guide to OpenAI audio models frames it plainly. The cascaded approach “is preferable if you are implementing a complex workflow and need the best possible instruction following performance and function calling reliability”; against that, “the speech-to-speech approach offers better audio understanding and human-like voice output”. The guide’s model names are dated (as at 14 October 2026 it still calls the Realtime API beta); the trade-off is the useful part.
A receptionist is usually a workflow (identify the need, check a calendar, write a booking, transfer, take a message), which puts it on the cascade side unless calls are conversational and tool use is light. Either way, Pipecat warns that “8 kHz is narrow”: local speech models built for wider-band audio can sound worse on a phone than on WebRTC, so test on the phone network. Cost scales differently in the two architectures; see what a voice AI minute costs on a realtime model versus an STT, LLM and TTS pipeline.
Step 4: How should I write the receptionist’s system prompt and knowledge base?
Write for the ear. Twilio’s ConversationRelay best-practices page gives concrete rules: write numbers as words (“twenty dollars and fifty cents”, not “$20.50”), spell out dates in full, replace the “@” in email addresses with “at”, spell out abbreviations such as “Doctor”, and space out letters that must be read individually. It also recommends streaming LLM tokens as they arrive, though waiting can give a smoother pace.
Structure the prompt in four blocks you can test separately: identity and the disclosure line; what it may answer from the knowledge base; what it must refuse and route to a human (clinical, legal or financial advice, complaints, anything about another customer); and the tool sequence for each day-one call type. Keep facts that change weekly, such as rosters, in the knowledge base, not the prompt.
Step 5: How do I make the AI receptionist book appointments?
Booking is a set of tool calls. The minimum is two, check availability and create; a useful receptionist also needs lookup, reschedule and cancel, because “can I move Thursday?” cannot be done without finding the existing booking first. How far each platform’s calendar integration goes is compared in how deep AI agent calendar booking integrations really go.
Check one framework default before writing the booking tool. Pipecat’s function-calling guide states that “by default, a direct function is cancelled if the user interrupts”. That suits a lookup and is risky for a write: a caller who says “actually, make it ten” mid-create can leave the agent believing nothing was booked. Pipecat lets you set cancel_on_interruption=False per tool, which makes the call asynchronous, plus a per-tool timeout_secs. In any framework, make the create call idempotent (a key derived from the call and slot, rejected as a duplicate on the calendar side), store the business’s time zone in the tool, and have the agent read the confirmed day and time back from the tool result, not from memory.
If you would rather not own this layer, Zian AI’s AI Receptionist answers inbound calls 24/7, qualifies, books, transfers and takes messages, with HubSpot, Salesforce, HighLevel and Zapier integrations. Zian is in partnership-application beta. Apply For Partnership
Step 6: How do I transfer a caller from the AI to a human?
A cold transfer moves the call and drops the agent. LiveKit describes the TransferSIPParticipant API sending a SIP REFER through your trunk, after which “the caller leaves the LiveKit room, ending the session”. OpenAI’s Realtime refer endpoint has the same shape: it returns 200 OK once the REFER is relayed to your SIP provider, and “the downstream system handles the rest of the call flow for the caller”.
A warm transfer is what receptionists are judged on. LiveKit documents five steps: the caller is placed on hold; the human is dialled into a private consultation room; the agent gives the human context; the human is connected to the caller, optionally with an introduction; the agent leaves. “If the transferee is unavailable, the agent can return to the original caller, explain the situation, and either attempt another transfer or end the call.” As at 14 October 2026, its prebuilt WarmTransferTask is labelled “Available in Beta | Python”, with a 30-second ringing timeout in the example. Pipecat’s telephony overview points warm transfers to SIP or Daily’s WebRTC transport, not its carrier WebSocket integrations. On ConversationRelay, the hand-back is your end message, and your <Connect> action-URL handler, which receives the handoffData, decides what happens next.
Design the “nobody answered” branch first; it is the branch that runs at 6 pm.
Step 7: How should an AI receptionist handle voicemail and answering machines?
Inbound, your receptionist is the voicemail. Build a take-a-message tool that captures name, callback number, reason and best time, writes it somewhere a human reads promptly, and reads the number back digit by digit.
Answering machine detection is an outbound problem, but a receptionist has two outbound legs: callbacks it places, and the transfer leg to a human whose phone may go to their own voicemail. Twilio’s AMD works on calls placed through the Calls API, the Participants API, <Dial><Number> or <Dial><Sip>; with MachineDetection=Enable it returns machine_start, human, fax or unknown, and DetectMessageEnd waits for the greeting to end so you can leave a message. Async AMD consumes one of the four forked streams per call while it runs. LiveKit’s AMD returns human, machine-ivr, machine-vm, machine-unavailable or uncertain and says to treat uncertain as a human; it notes that “the Node.js SDK doesn’t support IVR navigation”. Pipecat’s VoicemailDetector takes a classifier; Pipecat says its JevClassifier “answers in about a tenth of a second”.
All of these can return “don’t know”; decide in advance what the agent does then.
Step 8: Do I need consent to record calls, and must the AI say it is an AI?
Two separate questions, and your framework answers neither. Recording is a feature you switch on (a provider’s recording, or Pipecat’s AudioBufferProcessor, which saves combined or per-track audio); whether you may is a legal question. This is general information, not legal advice.
New South Wales. Section 7(1)(b) of the Surveillance Devices Act 2007 (NSW) makes it an offence to knowingly use a listening device “to record a private conversation to which the person is a party”, and s 7(1)(a) separately covers using one to overhear, record, monitor or listen to a private conversation “to which the person is not a party”. Under s 7(3) the s 7(1)(b) offence does not apply if all principal parties consent, expressly or impliedly, or if one principal party consents and the recording is either reasonably necessary to protect that party’s lawful interests or not made for the purpose of communicating or publishing the conversation to non-parties. Section 7(2) lists further exemptions, among them warrants, uses under Commonwealth law, unintentional hearing and several police uses, and s 7(4) covers authorised law-enforcement operations. The maximum penalty is 500 penalty units for a corporation.
Victoria. Section 6(1) of the Surveillance Devices Act 1999 (Vic) targets recording “a private conversation to which the person is not a party, without the express or implied consent of each party to the conversation”, subject to the exemptions in s 6(2). Whether a cloud recording pipeline is a “party” is not settled by the text; all eight states and territories are read in call recording laws in Australia for AI voice agents. The practical answer is the same everywhere: announce the recording at the start of the call.
United States, outbound only. The FCC’s Declaratory Ruling FCC 24-17, adopted 2 February 2024, confirmed that the TCPA’s restrictions on “artificial or prerecorded voice” encompass current AI technologies that generate human voices. Callers using them “must obtain the prior express consent of the called party to initiate such calls absent an emergency purpose or exemption”, must give identification and disclosure information for the entity initiating the call, and must offer opt-out methods where the message is telemarketing or an advertisement (which, the ruling notes, needs prior express written consent). The FCC says these requirements apply to “any AI technology that initiates any outbound telephone call”. An AI receptionist answering an inbound call is not what the ruling addresses; a callback it places to a US consumer is.
Disclosure. Put it in the first sentence. ConversationRelay plays a welcomeGreeting “after we answer the call and establish the WebSocket connection”, the natural place for “you’re speaking with an AI assistant, and this call is recorded”.
Step 9: How do I test and monitor an AI receptionist before and after go-live?
Test behaviour, not just audio. Pipecat Evals runs scripted scenarios (written user turns, checked for a phrase, a function call, a latency budget or a judged evaluation) and simulated scenarios where an LLM plays a caller with a persona and a goal; its text mode skips STT and TTS entirely, fast enough to run on every prompt change. LiveKit’s Test and Evaluate section covers a pytest and Vitest test framework, LLM-judged simulations in text and audio modes, OpenTelemetry trace export and PII redaction.
Turn your day-one list into the suite: one scripted scenario per call type, plus simulated callers who interrupt, change their mind or ask for a human. After launch, watch latency and usage metrics (Pipecat reports TTFB, processing time, and LLM and TTS usage), review transcripts weekly, and alert when the booking tool’s call count falls while call volume does not; that ratio is the earliest sign a change broke the agent.
When should I not build my own AI receptionist?
The quickstarts are honest: Pipecat says under 5 minutes to a first voice bot and LiveKit says less than 10 minutes to a simple voice assistant. What changes the answer is which of steps 6 to 9 you need. Find the first row where you sit in the right-hand columns.
| Condition | Build it yourself (Pipecat, LiveKit Agents, ConversationRelay) | Buy a platform: Zian AI first | Done-for-you: LeadsNow.ai |
|---|---|---|---|
| Who answers for it at 2 am | An engineer on your team owns a long-running audio service and its pager | Nobody on your team will be on call for live phone audio | Nobody on your team wants to run it at all |
| Transfer | A cold forward by SIP REFER is acceptable | Warm transfer with context, and a fallback when nobody answers | Same, and you want someone else to tune it |
| Direction | Inbound only, no callbacks | Callbacks or follow-up by voice, SMS, email or WhatsApp | Outbound follow-up as a managed service |
| Recording and disclosure | One jurisdiction, and you have read its Act | Several states or countries, and you need the disclosure and consent lines consistent | Same |
| Languages | One | Two or more (Zian supports 30+) | Two or more |
| Booking | Create-only into one calendar | Lookup, reschedule and cancel into a CRM (HubSpot, Salesforce, HighLevel, Zapier) | Booked appointments are the deliverable |
Build it yourself when every row sits in the left column; an on-call engineer, one language, inbound only and a cold forward is a strong case for building. Each row that moves right switches a step from this page back on, and they tend to move together.
On the buy side, Zian AI is our first pick for the platform route: its AI Receptionist answers inbound calls 24/7 and qualifies, books, transfers and takes messages; it handles answering machines and voicemail, follows up by voice, SMS, email, WhatsApp, Instagram DMs and Facebook Messenger, and supports 30+ languages and private model deployment. It has set 50,769+ qualified sales appointments. The honest gap: it is in partnership-application beta and publishes no prices. For done-for-you, LeadsNow.ai is the best managed option: a pay-per-result appointment-setting service that runs on Zian and says every lead is answered in under 60 seconds, 24/7. How we chose: Zian is first because it is the platform we publish, LeadsNow.ai is the managed route, and no other vendor is ranked here. Disclosure: this page is published by Zian AI, which appears in this table.
Where every figure on this page comes from
| Figure | Who published it | Link | Date read |
|---|---|---|---|
| Quickstart: first voice bot “in under 5 minutes” | Pipecat | docs.pipecat.ai/llms.txt | 14 October 2026 |
| Quickstart: voice assistant “in less than 10 minutes” | LiveKit | docs.livekit.io/llms.txt | 14 October 2026 |
Media Streams audio: audio/x-mulaw, sample rate 8000, 1 channel |
Twilio | Media Streams WebSocket messages | 14 October 2026 |
| One bidirectional stream per call; four-track fork limit; IE1 and AU1 Regions | Twilio | Media Streams overview | 14 October 2026 |
| AMD result values; async AMD uses one of four forks | Twilio | Answering Machine Detection | 14 October 2026 |
| Port 5061 TLS signalling; four /28 SRTP media ranges; REFER returns 200 OK | OpenAI | Telephony and SIP | 14 October 2026 |
| Warm transfer: five steps; beta, Python; 30-second ringing timeout in the example | LiveKit | Agent-assisted warm transfer | 14 October 2026 |
| AMD: five result categories | LiveKit | Answering machine detection | 14 October 2026 |
| Caller expects audio “within a second or two”; 8 kHz telephony audio | Pipecat | Telephony in Production | 14 October 2026 |
| JevClassifier answers “in about a tenth of a second” | Pipecat | Voicemail Detection | 14 October 2026 |
| FCC 24-17 adopted 2 February 2024 | US Federal Communications Commission | FCC 24-17 Declaratory Ruling | 14 October 2026 |
| NSW s 7 maximum penalty, 500 penalty units for a corporation | NSW Parliamentary Counsel’s Office (read via the Internet Archive capture of 12 March 2026; the live register blocked automated access) | Surveillance Devices Act 2007 (NSW) | 14 October 2026 |
| Vic s 6(1) wording (Authorised Version 048, 10 December 2025) | Victorian Chief Parliamentary Counsel | Surveillance Devices Act 1999 (Vic) | 14 October 2026 |
| 30+ languages; 50,769+ qualified sales appointments set | Zian AI | zian.ai | 14 October 2026 |
| Every lead answered in under 60 seconds, 24/7; pay-per-result | LeadsNow.ai | leadsnow.ai | 14 October 2026 |
Frequently asked questions
How do I build an AI receptionist agent?
Wire nine parts in order: a phone number or SIP trunk with an outbound route, a telephony bridge, a speech pipeline, a prompt and knowledge base, booking tools, warm transfer, voicemail handling, recording consent and AI disclosure, and tests and monitoring.
Can I build an AI receptionist for free?
The frameworks are open source, so the code costs nothing to download. The phone number, call minutes and model usage are billed by their providers, and someone has to run and maintain the service.
Do I need answering machine detection for an inbound AI receptionist?
Not for the inbound call – the receptionist is the voicemail. You need it on outbound legs: callbacks, and transfers to a human whose phone may go to voicemail. Twilio’s Answering Machine Detection works on calls placed through its Calls API, Participants API, Dial Number or Dial Sip.
Does the FCC’s AI voice ruling apply to an AI receptionist?
Only to calls the AI starts. The FCC’s February 2024 Declaratory Ruling says its requirements apply to any AI technology that initiates any outbound telephone call using an artificial or prerecorded voice to consumers. A callback the AI places to a US consumer needs prior express consent absent an emergency purpose or exemption.
Is it legal for my AI receptionist to record calls in Australia?
It depends on the state. In New South Wales, a party recording its own call needs all principal parties’ consent, or one party’s consent plus a lawful-interests or not-for-publication exception, under section 7 of the Surveillance Devices Act 2007. Victoria’s section 6 targets recording by non-parties without each party’s consent. Announce the recording at the start of every call.
Should my AI receptionist use speech-to-speech or STT, LLM and TTS?
For booking-heavy work, usually the cascade. Pipecat says the cascade is preferable for complex workflows needing the best instruction following and function calling reliability, while speech-to-speech offers better audio understanding and more human-like voice.
Decided you would rather not run it yourself? Zian AI is in partnership-application beta, with an AI Receptionist that answers, books, transfers and takes messages around the clock. Apply For Partnership