Voice agent at a glance
Pipeline runs left-to-right inside the framework layer. Policy stacks on top, infrastructure stacks below. Four cross-cutting concerns run through all three.
Agent-specific · what to say, what's allowed
Framework · the real-time turn loop
Infra · where it runs and lands
Agent
What you own · decision & content
How it sounds voice persona
Persona + promptPure differentiation. Versioned.
What it can do capabilities
ToolsYour product surface. Framework provides the hook.
KB / RAG contentWhat & how to retrieve.
Memory rulesWhat's worth remembering.
How it behaves conversation policy
Flow / business policySlots, escalations, commits.
Confirmation gatesRead-back slots, regex affirm.
Quality & safety enforced rules
Guardrail rulesRefusal style, jailbreak, HIPAA/PCI.
Eval signalsWhat "good" means in your domain.
↓ ↓ ↓
Framework
The real-time turn loop — mechanism only, policy plugs in from above
Callerin
▶
AEC + VADecho · voice
▶
STTinterims
▶
Dialogturn + spec
▶
LLM+ tools
▶
TTSstreamed
▶
TransportWebRTC/SIP
▶
Callerout
Barge-in · transport-level flush, <20 ms cancel
Endpointing · adaptive per slot kind
Speculation · fire LLM/tools on stable interim
Confirmation hook · regex fast-path, LLM fallback
Safety hook · pre-LLM input + post-LLM output filters, ~10 ms · rules from agent
The framework should be the most swappable layer; the hooks above run under the cert-gate budget without stalling audio. Business logic in here = your moat coupled to a replaceable component.
↓ ↓ ↓
Infra
Substrate · model-agnostic, ops-owned
Compute where inference runs
Model servingGPU pool, autoscaling, warm starts, colocation.
Filter modelsPII scrub, classifier inference, refusal cascade.
State where data lives
Vector storeRAG index. Hot cache for speculation.
Memory storeRedis / KV, caller-keyed recall.
Recording + storageRetention, encryption, residency, PII.
Operations how you run it
ObservabilityTraces, audio+transcript, cost-per-call.
Eval harnessBatch sim calls, CI gating, regression dash.
Edge / transport the network boundary
Transport hostsWebRTC SFU · PSTN bridge · TURN.
Server-side AECWebRTC AEC3 on PSTN (no client canceller).
Cross-cutting · run through all three layers
These do not live in any one bucket; each one needs an owner on each layer. Treat them as standing concerns, not features to bolt on at the end.
Latency budgeting
Evaluation & simulation
Observability & tracing
Guardrails & safety