Skip to main content
Status: In Development · Playground demo coming soon. The recipe below is complete and runnable today; only the hosted playground showcase is pending.
A phone-grade voice concierge built on Pipecat with ElevenLabs TTS and Deepgram STT. Synap is wired in as a frame processor so memory injection happens automatically before each LLM call, and the turn is ingested afterward, all within voice latency budgets.
Python-only recipe. Pipecat is a Python-native framework and does not have a TypeScript port. If you need TypeScript voice, build on LiveKit Agents instead. See Patterns → Voice Agent on LiveKit for the pattern (Python-only there too) or wrap your own STT/LLM/TTS on the JS side.

What you’ll build

A voice agent that:
  • Answers a phone call or live mic session
  • Recalls caller history mid-call from prior calls: preferences, prior issues, on-going situations
  • Stays inside voice latency budgets: Synap’s fast mode is built for the latency-critical retrieval path
  • Records and ingests every turn for next time
  • Handles natural-feeling phone interactions: interruptions, short replies, repeat handling
Est. build time: 60-90 minutes (most of it is STT/TTS provider setup).

When to use this recipe

Build this if:
  • You’re building a phone agent (inbound IVR replacement, outbound calling, kiosk voice UI)
  • Caller continuity across calls is the value: “I know who you are without you stating your account number”
  • You need sub-second total round-trip latency
  • You can carry Python on the call-handling side

Architecture at a glance

Synap sits on the LLM frame in Pipecat’s pipeline. The retrieval is on the critical path (must be fast). The ingestion is fire-and-forget (must not block the next utterance).

Stack

Prerequisites

  • A Synap API key. See Authentication
  • Deepgram API key
  • ElevenLabs API key + chosen voice ID
  • OpenAI API key
  • A way to ingest audio (Twilio call → Pipecat WebRTC bridge is common)
  • Python 3.11+

Install

Configure

Build it

1. Identity & scoping

Voice has a stable identifier: the caller’s phone number (from CNAM / SIP From / Twilio webhook).
  • customer_id = a stable UUID for your business
  • user_id = a stable UUID derived from the caller phone (E.164; hash first if your privacy posture requires)
  • conversation_id = the same per-caller UUID, rolling: the relationship is the conversation; this isn’t per-call
Synap ids must be valid UUIDs, so don’t pass the raw phone number. Derive a deterministic UUID from it with uuid.uuid5(...) (shown below): the same phone always maps to the same id, which is exactly the rolling continuity you want.

2. The Pipecat pipeline

The maximem-synap-pipecat package exposes SynapContextHook and SynapMemoryHook as Pipecat frame processors. Drop them in around the LLM.

3. Tools (optional)

Phone agents lean on tools too: transfer to a human, look up an order, send an SMS follow-up. Pipecat’s OpenAILLMService supports function tools; wire them like you would in any agent.

4. Latency budget

Voice has a tight window of conversational comfort end-to-end. Where the time goes, fastest to slowest: The expensive part is the LLM. Synap stays well inside the budget. If you see drift:
  1. Move the SDK to the same region as your call-handling box.
  2. Lower max_results to 3.
  3. Cache the last context fetch for 10s: voice turns are tight in time.
  4. Switch the LLM to gpt-4o-mini for short replies.

Run & verify

First call
Two weeks later
The agent picked up the prior issue without you wiring anything case-specific. Phone calls feel continuous because they are.

Customize / extend

  • Outbound voice campaigns → flip the pipeline; have your telephony provider place outbound calls into the same Pipecat pipeline.
  • LiveKit instead of Pipecat → see Patterns → Voice Agent on LiveKit. Same memory model, different transport.
  • Voice journaling / personal companion → set max_results higher and use a richer system prompt; see AI Companion for the persona shape.
  • Coach by voice → adapt AI Coach over Pipecat. Voice tracking of workouts is natural (“I just ran 5k”).
  • Tier-2 escalation by voice → on transfer_to_agent, post a memory-grounded summary into your queue so the human agent has full context the moment they pick up.

Troubleshooting

Replies feel slow
  • Profile each stage. LLM first-token is usually the bottleneck. Reduce max_results, switch model, or pre-emit a filler (“let me check…”) if your TTS supports interruption-friendly start.
Concierge re-asks for the caller’s name every call
  • Caller-name memory isn’t being ingested. After the first call, capture name explicitly with a tool or system rule: “if the caller introduces themselves, store via Synap before responding.”
Context fetch times out
  • Run Synap in the same region. Check network. If degraded, fall back to no-context gracefully; the call should proceed without memory rather than fail.
Caller speaks over the agent and confuses the pipeline
  • Pipecat handles VAD-driven interruption; tune SileroVADAnalyzer sensitivity for your audio path.
Phone number not stable (caller ID withheld)
  • Fall back to a session-only ID + ask for an account number / OTP in-flow. Don’t write to long-term memory until identity is confirmed.