Where the decisions live

Nearly every voice-agent component is split. The framework owns the mechanism (how a turn is detected, how streams are wired). Infra owns the substrate (where it runs, scales, lands). The agent owns the policy (what to say, which model, what's allowed). Most boundary mistakes come from putting policy in the framework.

1Framework

Universal across all voice agents and latency-critical. Owns the verb — the real-time turn loop.

2Infrastructure

Running, scaling, storing, observing. Model-agnostic, ops-owned. Owns the where.

3Agent-specific

Where you differentiate — domain logic, knowledge, behavior, policy. Owns the what/which.

● primary owner ◐ partial / shared Framework Infrastructure Agent-specific
Component
Frame
Infra
Agent
Where the boundary sits

Cross-cutting concerns

Four concerns don't live in any single bucket — they audit all three. Forced into one column they get under-resourced, which is exactly how voice agents ship sounding bad.
Concern
Framework
Infrastructure
Agent-specific
Latency budgeting
Overlap or stack: endpointing window, speculative LLM start, sentence-level TTS streaming.
Sets the floor: model colocation, GPU proximity, jitter buffer, transport choice, warm pools.
Work per turn: prompt size, serial tool/RAG hops, output verbosity, model size.
Evaluation & simulation
Replay/injection hooks: recorded audio + synthetic turns through the real loop deterministically.
Harness at scale: batch simulated calls, transcript + audio store, CI gating.
What "good" means: adversarial caller personas, scoring rubrics, golden conversations.
Observability & tracing
Per-turn events: endpoint fired, barge-in, STT-final, LLM TTFT, TTS first-byte, tool spans.
Collect, store, visualize: trace backend, aligned audio+transcript, cost-per-call attribution.
What to flag: which failures matter, semantic quality signals, business KPIs.
Guardrails & safety
Inline interception: checks run under the latency budget without stalling audio.
Hosts filter models & PII scrubbing: retention, encryption, residency on recordings.
Owns the rules: what's blocked, refusal style, jailbreak handling, HIPAA/PCI logic.
The dotted (◐) rows in the matrix above are where teams most often get the boundary wrong. Retrieval, guardrails, tools, model selection all feel like framework concerns; the policy belongs to the agent. The framework offers the hook, never the rule.