Everything you need to measure AI quality

Correctness, relevance, faithfulness, and grounding quality rolled into one pillar. Combines dataset-based scoring from LangSmith, LLM-as-judge evaluation for subjective criteria, and RAG-specific metrics like Faithfulness and Context Recall.

• Dataset-based scoring with LangSmith, Promptfoo, or custom runner
• LLM-as-judge for non-deterministic criteria with confidence scores
• RAG metrics: Faithfulness, Context Recall, Context Precision
• Piecewise linear normalization to 0–1 scale per metric

Track failures, timeouts, rate limits, tool-call success rates, and error budgets derived from your OpenTelemetry traces. Set SLOs and get alerted before your users notice.

• Automatic error rate and timeout detection from OTel spans
• Tool-call success rate tracking for agentic workflows
• SLO-based error budget monitoring
• Rate limit and fallback event detection

Latency distributions, token usage, cost per request, and cache hit rates. Every API call is converted to dollars using provider pricing so you can optimize spend without guessing.

• P50/P95/P99 latency tracking across all pipeline stages
• Per-request cost calculation using live provider pricing
• Token usage analysis with cached vs. uncached breakdown
• Cache hit rate monitoring and optimization signals

Policy compliance, guardrail pass rates, and unsafe content detection. Safety is special — critical violations cap your entire Quality Index regardless of how well other pillars score.

• Guardrail pass/fail rates across all requests
• Content safety scoring and PII detection
• Policy compliance checks with configurable rules
• Safety gate: violations cap the overall Quality Index