CCAR-P Domain 4: Production Telemetry, Observability & Cost Attribution
Tracking key Service Level Indicators (SLIs), distributed tracing for multi-agent chains, token accounting by departmental cost centers, and cache hit efficiency.
1. The Four Golden Signals of LLM Observability
1. Latency Metrics
Track p50, p95, and p99 Time-to-First-Token (TTFT) and total generation duration. Alert on p99 degradation.
2. Token Accounting
Split tracking into input_tokens, output_tokens, cache_creation_input_tokens, and cache_read_input_tokens.
3. Error & Throttling Rates
Monitor HTTP 429 rate limit percentages and HTTP 529 overload spikes to trigger circuit breaker fallbacks.
4. Semantic Quality & Drift
Sample production completions for automated LLM-as-a-judge scoring to detect semantic drift and hallucination rates over time.
2. Departmental Cost Attribution & Showback
In multi-tenant enterprise architectures, an API Gateway intercepts every outgoing Claude request, attaches custom metadata tags (e.g. cost_center: "wealth_management", app_id: "portfolio_advisor"), and records token metrics to corporate data warehouses for precise internal chargeback reporting.
3. Distributed Tracing for Multi-Agent Systems
Using OpenTelemetry, assign a root TraceID to every user transaction. Each sub-agent call (Planner → Researcher → Critic) creates a nested Span tracking individual token counts, tool execution latency, and error states.