The ElevenLabs Voice Agent Playbook: From Demo to Production Calls

By the end of this playbook you will have:

• A scope sheet for one voice agent: the job, the callers, the finish line, and the hard "never" rules
• A system prompt structured for voice, with tool rules and an escalation rule
• Webhook tools for the lookups and actions the agent needs, with authentication and plain-language responses
• A phone number connected through the native Twilio integration or your own SIP trunk
• A human transfer that carries a summary to the person picking up
• Guardrails, privacy, and retention settings that match the data you handle
• An automated test suite (simulation, next reply, and tool call tests) that runs before every change
• Post-call webhooks that write a structured record into your CRM, verified with HMAC signatures
• A versioned rollout plan so prompt and tool changes reach callers gradually

Octacer note: We build and fix voice agents on ElevenLabs, Retell, and custom telephony stacks. Most failures we are asked to rescue are not voice problems. They are tool problems: a webhook that cannot return an answer, a booking the agent confirms without a real appointment behind it, or a transfer nobody tested. This playbook front-loads those.

1. An ElevenLabs account with access to ElevenLabs Agents (the agents platform in the ElevenLabs dashboard).
2. A phone path: a Twilio account and number, or a SIP trunk from your carrier or PBX (Telnyx, Plivo, Bandwidth, Sinch, and others have documented setups).
3. An endpoint the agent can call for lookups and actions. This can be an n8n, Make, or Zapier webhook for a first version, or your own API.
4. A CRM or database where call outcomes should land (HubSpot, GoHighLevel, EspoCRM, or similar).
5. One person who owns the agent's behavior and reviews calls weekly.

Pick a single job with a clear finish line. Good first jobs:

• Answer after-hours calls, capture the caller's need, and book a callback slot
• Confirm, reschedule, or cancel an existing appointment
• Qualify an inbound lead with three to five questions and route hot leads to a person

Write a one-page scope sheet:

The "never" list matters more than the "always" list. It becomes your prompt rules, your guardrails, and your test cases.

ElevenLabs' own prompting guide treats the system prompt as system design, not a personality blurb. A structure that works well on calls:

1. Role and goal: one or two sentences. "You answer calls for [Company]. Your goal is to book a consultation or hand the caller to a person."
2. Voice style: short sentences, one question at a time, read numbers and times naturally, confirm spelling of names and emails.
3. Tool rules: which tool to call, when, and what to do with the result. "Always call check_availability before offering times. Only offer times it returns."
4. Escalation rule: exactly when to call the transfer tool.
5. Never rules: copied from your scope sheet.
6. Closing: how to confirm and end the call.

Keep facts that change (opening hours, prices, staff names) out of the prompt. Put them in the knowledge base or behind a tool so you can update them without touching the agent's behavior.

Use dynamic variables for per-call context such as the caller's name or account tier, so one agent can serve many callers without separate copies.

Webhook tools let the agent call your APIs mid-conversation. The model fills in query, body, and path parameters from the conversation and the parameter descriptions you write, so the descriptions are part of your prompt.

Rules we follow for every tool:

• One tool, one job. check_availability, book_slot, find_contact, create_callback. Avoid a generic "do_anything" endpoint.
• Return a sentence the agent can read back. For example {"status":"booked","spoken_summary":"Booked for Tuesday 4 November at 3 pm UK time.","event_id":"..."}. A bare 200 with no body teaches the agent to improvise.
• Return explicit failures. {"status":"no_slots","spoken_summary":"There are no openings this week."} is much safer than an error the agent has to interpret.
• Answer synchronously. If your endpoint queues work and replies "accepted", the agent cannot know whether the booking happened. We wrote about this exact failure in The Webhook That Couldn't Answer Back and the related Zapier and GHL webhook KB article.
• Authenticate. Use the tool's headers or auth connections so your endpoint can reject calls that do not come from your agent.
• Keep it fast. Every second a tool takes is a second of silence on the phone. ElevenLabs lets you add tool call sounds during execution, but the real fix is a fast endpoint.

If you prefer, ElevenLabs also supports MCP servers as tool sources and has security guidance for third-party servers. Treat MCP tools with the same least-privilege rules as webhooks.

Check: the agent must never say "booked" unless the booking tool returned a real event ID. Our KB article on voice agents that confirm bookings without an appointment covers the fix in detail.

Use the knowledge base for stable reference material: services, service areas, policies, FAQs. For larger document sets, enable RAG so only relevant chunks are retrieved per turn.

Practical rules:

• One topic per document, with clear headings. Messy PDFs produce messy answers.
• Anything that changes daily (availability, order status, balances) belongs behind a tool, not in the knowledge base.
• Add a prompt rule: if the answer is not in the knowledge base or a tool result, say so and offer a callback.

You have two main paths.

Native Twilio integration. Import your Twilio number with its Account SID and Auth Token, and ElevenLabs configures the voice webhooks for you (Twilio native integration). This is the fastest route, and some transfer options (warm transfer messages and blind transfer) depend on it.

SIP trunking. If you already run a PBX or a carrier SIP trunk, connect it with SIP trunking. ElevenLabs supports digest authentication or IP-based access control, TLS for signaling, and media encryption options. Calls to the ElevenLabs SIP endpoint must include an identifier in the SIP URI (for example a number in E.164 format).

Whichever you choose:

• Use a dedicated number for the agent while you test, then move your main number over.
• Decide what happens when the agent is unavailable: forward to voicemail or a person, not a dead line.
• Set a maximum conversation duration. The default is 600 seconds and the allowed range is 60 to 7,200 seconds. A booking call rarely needs more than a few minutes.

The transfer to number system tool hands the call to a phone number or SIP URI. It supports three transfer types:

• Conference transfer (default): dials the destination, joins them, and drops the agent. With native Twilio, a warm message can brief the person picking up.
• Blind transfer: sends the call straight through and keeps the original caller ID. Native Twilio only.
• SIP REFER: for SIP calls where your trunk allows REFER. No warm message.

Set it up so the human never starts cold:

1. Write the transfer conditions into the prompt, matching your scope sheet.
2. Fill the operator message with a one-line summary: who is calling, what they want, and what the agent already tried.
3. Decide what happens if nobody answers: take a message, create a callback task in the CRM, and tell the caller when to expect it.
4. Test it during and outside office hours.

The end call and voicemail detection tools are worth enabling for outbound calls, so the agent does not talk to an answering machine as if it were a person.

Guardrails. ElevenLabs Guardrails work at three levels: system prompt hardening (with a Focus guardrail), user input validation (a Manipulation guardrail that detects prompt injection and can end the conversation), and agent response validation (Content and Custom guardrails that check each reply before it is spoken). For your most critical "never" rules, put them in both the prompt and a custom guardrail.

Privacy and retention. Decide what you store before the first real call:

• Retention controls how long transcripts and recordings are kept.
• Audio saving controls whether recordings are kept at all.
• Conversation history redaction removes sensitive entities from stored history.
• Zero Retention Mode stops recordings and PII-bearing transcripts from being stored after the call. You then rely on post-call webhooks to keep your own record, and ElevenLabs notes it limits their ability to debug calls.

If you handle health data, read ElevenLabs' HIPAA page and confirm the contract terms with them before you take real patient calls.

Authentication for web and app agents. If the same agent also runs in a website widget or app, use signed URLs generated on your server (they expire after 15 minutes) or a domain allowlist. Never put your ElevenLabs API key in client-side code. Our client-side AI key playbook covers how to clean that up if it already happened.

ElevenLabs Agent Testing offers three test types:

Start with ten to twenty tests built from your scope sheet:

• Happy path: caller books a slot, and the booking tool is called with the right date and time zone
• No slots available: the agent offers a callback instead of inventing a time
• Tool failure: the agent apologizes, does not claim success, and transfers or takes a message
• Caller asks for a person: the transfer tool is called
• Off-topic or manipulation attempts: the agent stays in scope
• Every "never" rule: one test each

When a real call goes wrong, use create test from this conversation so the same failure cannot come back silently.

Post-call webhooks send the transcript, analysis results, and metadata after each call. There are also audio webhooks and call initiation failure webhooks.

Set it up properly:

1. Verify the HMAC signature in the ElevenLabs-Signature header with the shared secret, using the SDK helpers. Reject anything that fails.
2. Return 200 quickly, then process. ElevenLabs auto-disables a webhook after repeated consecutive failures.
3. Use data collection to extract structured fields (name, email, intent, booked yes or no) from each call. Data collection supports string, boolean, integer, and number fields, and the field description is the extraction instruction.
4. Define success evaluation criteria such as "a booking was made" or "the caller was transferred", so you can measure outcomes instead of reading every transcript.
5. Write one CRM activity per call with the summary, outcome, and a link back to the conversation, and create a follow-up task when the outcome needs one.

If you run the receiver in n8n, add an approval or review step for anything that changes customer records beyond logging. Our n8n human-in-the-loop playbook shows the pattern.

Agent versioning gives you immutable versions, branches, and traffic splitting. Note that once versioning is enabled on an agent it cannot be turned off, so decide early.

A safe change process:

1. Make prompt or tool changes on a branch.
2. Run the full test suite against the branch.
3. Send a small share of traffic to the branch and compare outcomes.
4. Merge to main when the numbers hold, or roll back by moving traffic.

For structured comparisons, ElevenLabs also has Experiments for A/B tests on live traffic. Set up alerts for error-rate spikes so a broken tool does not run unnoticed overnight.

• [ ] Scope sheet signed off, including the "never" list
• [ ] Every tool returns a spoken summary and an explicit failure status
• [ ] Booking or write tools return a real ID, and the prompt forbids confirming without it
• [ ] Tool endpoints authenticate requests from the agent
• [ ] Transfer tested in and out of hours, with a no-answer fallback
• [ ] Max call duration set
• [ ] Guardrails on, with custom rules for the most critical "never" items
• [ ] Retention, audio saving, and redaction set to match your data policy
• [ ] API key only on the server; signed URLs or allowlist for web use
• [ ] Test suite passing, including failure paths
• [ ] Post-call webhook verifies HMAC and writes to the CRM
• [ ] Versioning on, alerts on, and a named owner reviewing calls weekly

• Letting the model state facts it should look up. Times, prices, and statuses come from tools.
• Tools that only say "OK". The agent fills the gap with something plausible and wrong.
• Testing only the happy path. Most bad calls happen when a tool fails or the caller goes off script.
• Editing the live agent directly. Use a branch and the test suite.
• No owner. Voice agents drift as your business changes. Someone needs to read transcripts every week.

How long does it take to build an ElevenLabs voice agent?

A demo can be running the same day. A production agent with tools, transfer, privacy settings, tests, and CRM sync usually takes one to two days of focused work for a single job, plus a week of monitored live traffic.

Do I need Twilio to use ElevenLabs voice agents on the phone?

No. Twilio has a native integration, but you can also connect your own SIP trunk or PBX, and ElevenLabs documents setups for several carriers. Some transfer features, such as warm transfer messages and blind transfer, currently depend on the native Twilio integration.

Can an ElevenLabs agent book appointments in my calendar?

Yes, through webhook tools to your calendar or booking system, or through built-in integrations such as Cal.com, Calendly, and Google Calendar. The important part is that the agent only confirms a time the tool actually returned.

How do I stop the agent from making things up?

Keep changing facts behind tools, tell the agent to say it does not know when a fact is missing, add guardrails for critical rules, and test failure paths. Our hallucination containment playbook goes deeper.

How do I get call data into HubSpot or GoHighLevel?

Enable post-call webhooks, verify the HMAC signature, extract structured fields with data collection, and write one activity per call to the CRM. ElevenLabs also lists a native HubSpot integration.

Is an ElevenLabs voice agent secure enough for customer calls?

It can be, if you apply the same controls as any AI agent: least-privilege tools, authenticated endpoints, guardrails, retention settings, and logging. See The AI Security Playbook for the full checklist.

What should I measure after launch?

Containment (calls finished without a human), successful bookings or outcomes, transfers and why they happened, tool error rates, and average call length. Review a sample of transcripts every week.

We design, build, and repair voice agents on ElevenLabs, Retell, and custom telephony, and connect them to HubSpot, GoHighLevel, EspoCRM, calendars, and n8n or Make workflows. Typical work includes:

• A voice agent build for one job, with tools, transfer, tests, and CRM sync
• Rescue work on agents that confirm bookings that do not exist, go silent on slow tools, or transfer badly. See System Rescue
• A fixed-price Automation Sprint for the webhook and CRM side
• Engineering Capacity if you want a dedicated engineer on your voice stack

Book a call and bring your call flow. We will help you scope the first job and the "never" list.

• Fix wrong-time bookings: Wrong-Time Booking Agent Triage
• Secure your agents: The AI Security Playbook
• Add approvals to write actions: n8n Human-in-the-Loop Tool Gates
• See our AI build work: AI Apps and Integrations

• ElevenLabs Docs, ElevenAgents overview: https://elevenlabs.io/docs/eleven-agents/overview
• ElevenLabs Docs, Prompting guide: https://elevenlabs.io/docs/eleven-agents/best-practices/prompting-guide
• ElevenLabs Docs, Webhook tools: https://elevenlabs.io/docs/eleven-agents/customization/tools/webhook-tools
• ElevenLabs Docs, Tool call sounds: https://elevenlabs.io/docs/eleven-agents/customization/tools/tool-configuration/tool-call-sounds
• ElevenLabs Docs, Model Context Protocol: https://elevenlabs.io/docs/eleven-agents/customization/tools/mcp
• ElevenLabs Docs, MCP integration security: https://elevenlabs.io/docs/eleven-agents/customization/tools/mcp/security
• ElevenLabs Docs, Knowledge base: https://elevenlabs.io/docs/eleven-agents/customization/knowledge-base
• ElevenLabs Docs, Retrieval-Augmented Generation: https://elevenlabs.io/docs/eleven-agents/customization/knowledge-base/rag
• ElevenLabs Docs, Dynamic variables: https://elevenlabs.io/docs/eleven-agents/customization/personalization/dynamic-variables
• ElevenLabs Docs, Twilio native integration: https://elevenlabs.io/docs/eleven-agents/phone-numbers/twilio-integration/native-integration
• ElevenLabs Docs, SIP trunking: https://elevenlabs.io/docs/eleven-agents/phone-numbers/sip-trunking
• ElevenLabs Docs, Conversation flow: https://elevenlabs.io/docs/eleven-agents/customization/conversation-flow
• ElevenLabs Docs, Transfer to number: https://elevenlabs.io/docs/eleven-agents/customization/tools/system-tools/transfer-to-number
• ElevenLabs Docs, End call: https://elevenlabs.io/docs/eleven-agents/customization/tools/system-tools/end-call
• ElevenLabs Docs, Voicemail detection: https://elevenlabs.io/docs/eleven-agents/customization/tools/system-tools/voicemail-detection
• ElevenLabs Docs, Guardrails: https://elevenlabs.io/docs/eleven-agents/best-practices/guardrails
• ElevenLabs Docs, Retention: https://elevenlabs.io/docs/eleven-agents/customization/privacy/retention
• ElevenLabs Docs, Audio saving: https://elevenlabs.io/docs/eleven-agents/customization/privacy/audio-saving
• ElevenLabs Docs, Conversation history redaction: https://elevenlabs.io/docs/eleven-agents/customization/privacy/conversation-history-redaction
• ElevenLabs Docs, Zero Retention Mode (per-agent): https://elevenlabs.io/docs/eleven-agents/customization/privacy/zrm
• ElevenLabs Docs, HIPAA: https://elevenlabs.io/docs/eleven-agents/legal/hipaa
• ElevenLabs Docs, Agent authentication: https://elevenlabs.io/docs/eleven-agents/customization/authentication
• ElevenLabs Docs, Agent Testing: https://elevenlabs.io/docs/eleven-agents/customization/agent-testing
• ElevenLabs Docs, Post-call webhooks: https://elevenlabs.io/docs/eleven-agents/workflows/post-call-webhooks
• ElevenLabs Docs, Data collection: https://elevenlabs.io/docs/eleven-agents/customization/agent-analysis/data-collection
• ElevenLabs Docs, Agent versioning: https://elevenlabs.io/docs/eleven-agents/operate/versioning
• ElevenLabs Docs, Experiments: https://elevenlabs.io/docs/eleven-agents/operate/experiments
• ElevenLabs Docs, Alerts: https://elevenlabs.io/docs/eleven-agents/operate/alerts