Documentation

Deploy in 5 minutes. See your first Quality Index in 30. Everything you need to get started.

Quick StartAPI ReferenceConnectors

Quick Start

Get from zero to a live Quality Index in four steps. The platform runs entirely on your machine via Docker Compose.

1

Deploy the platform

Clone the repo and run Docker Compose. The platform starts on port 3000.

terminal
git clone https://github.com/qualityindex/qualityindex.git
cd qualityindex
docker compose up -d
2

Connect OpenTelemetry

Update your application's OTel Collector config to point at the platform's OTLP endpoint.

terminal
# Add to your otel-collector-config.yaml
exporters:
  otlp/qualityindex:
    endpoint: "localhost:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      exporters: [otlp/qualityindex]
3

Create your first Eval Card

Open Eval Studio and define what quality means for your AI application. Start with a persona template or build from scratch.

terminal
# Example Eval Card (YAML config)
name: "Product Search Relevance"
task: "product-search"
pillar: task-quality
rubric:
  - metric: relevance-score
    threshold: 0.8
    weight: 0.6
  - metric: faithfulness
    threshold: 0.9
    weight: 0.4
gate: false  # monitor, don't block
4

View your Quality Index

Open the dashboard. Your Quality Index is computed from available signals immediately — runtime health metrics from OTel plus eval scores as they arrive.

terminal
# Open your browser
open http://localhost:3000/dashboard

# Your Quality Index updates in real-time
# as telemetry and eval results flow in.

API Reference

The REST API provides programmatic access to your Quality Index, Eval Cards, connectors, resource inventory, regressions, releases, and webhooks.

Authentication

All API requests require a Bearer token in the Authorization header. Create API keys in the dashboard under Settings → API Keys. Keys are scoped to a single workspace.

request header
Authorization: Bearer qi_live_a1b2c3d4e5f6...

Unauthenticated requests return 401 Unauthorized. Requests outside key scope return 403 Forbidden.

Base URL: https://api.qualityindex.ai/api/v1

Quality Index

Read and compare your Quality Index scores.

GET/api/v1/indexCurrent composite score and all 4 pillar scores with weights
GET/api/v1/index/historyTime-series data; query params: from, to, interval (hourly|daily|weekly)
GET/api/v1/index/compareCompare two releases side by side; query params: base_release, head_release
example response
// GET /api/v1/index → 200 OK
{
  "quality_index": 87,
  "updated_at": "2026-03-07T14:32:00Z",
  "pillars": {
    "task_quality": { "score": 92, "weight": 0.40 },
    "reliability":  { "score": 88, "weight": 0.25 },
    "efficiency":   { "score": 81, "weight": 0.20 },
    "safety":       { "score": 84, "weight": 0.15 }
  }
}

Eval Cards

Create, manage, and trigger evaluations.

GET/api/v1/eval-cardsList all cards with status, pillar, and gate flag
POST/api/v1/eval-cardsCreate a card (name, pillar, rubric[], data_binding, gate)
GET/api/v1/eval-cards/:idSingle card with latest scores
PATCH/api/v1/eval-cards/:idUpdate a card
DELETE/api/v1/eval-cards/:idDelete a card
POST/api/v1/eval-cards/:id/runTrigger an eval run manually
example response
// POST /api/v1/eval-cards → 201 Created
{
  "id": "ec_2xKj9mNpQr",
  "name": "Product Search Relevance",
  "pillar": "task_quality",
  "status": "active",
  "gate": false,
  "rubric": [
    { "metric": "relevance-score", "threshold": 0.8, "weight": 0.6 },
    { "metric": "faithfulness",    "threshold": 0.9, "weight": 0.4 }
  ],
  "data_binding": "traces.product-search",
  "latest_score": null,
  "created_at": "2026-03-07T10:15:00Z"
}

Connectors

Monitor and trigger data source syncs.

GET/api/v1/connectorsList connectors with status and last sync time
POST/api/v1/connectors/:type/syncTrigger sync (type: otel, langsmith, github)
GET/api/v1/connectors/:type/healthDetailed health check for a connector
example response
// GET /api/v1/connectors → 200 OK
{
  "connectors": [
    {
      "type": "otel",
      "status": "healthy",
      "last_sync": "2026-03-07T14:28:00Z",
      "spans_ingested_24h": 148320
    },
    {
      "type": "langsmith",
      "status": "healthy",
      "last_sync": "2026-03-07T14:25:00Z",
      "runs_synced_24h": 4210
    },
    {
      "type": "github",
      "status": "degraded",
      "last_sync": "2026-03-07T12:00:00Z",
      "error": "Rate limit exceeded, retry at 13:00 UTC"
    }
  ]
}

Resources

AI resource inventory discovered from traces.

GET/api/v1/resourcesList models, prompts, retrievers, and guardrails
GET/api/v1/resources/:id/tracesRecent traces for a specific resource
example response
// GET /api/v1/resources → 200 OK
{
  "resources": [
    {
      "id": "res_gpt4o",
      "type": "model",
      "name": "gpt-4o",
      "provider": "openai",
      "first_seen": "2026-02-18T09:00:00Z",
      "trace_count_24h": 12840,
      "avg_latency_ms": 620
    },
    {
      "id": "res_product_rag",
      "type": "retriever",
      "name": "product-catalog-rag",
      "provider": "pinecone",
      "first_seen": "2026-02-20T11:30:00Z",
      "trace_count_24h": 8420,
      "avg_latency_ms": 185
    }
  ]
}

Regressions

Surface and investigate quality regressions.

GET/api/v1/regressionsTop regressions ranked by impact score
GET/api/v1/regressions/:idDetail with root cause analysis and linked PR
example response
// GET /api/v1/regressions → 200 OK
{
  "regressions": [
    {
      "id": "reg_8kLmN3",
      "title": "Relevance drop in product-search",
      "impact_score": 9.2,
      "pillar": "task_quality",
      "metric": "relevance-score",
      "baseline": 0.91,
      "current": 0.74,
      "detected_at": "2026-03-06T22:15:00Z",
      "root_cause": "Prompt template v2.4 removed category constraint",
      "linked_pr": "https://github.com/acme/app/pull/481"
    }
  ]
}

Releases

Quality Index snapshots per release.

GET/api/v1/releasesList releases with Quality Index score
GET/api/v1/releases/:tagSingle release detail
example response
// GET /api/v1/releases → 200 OK
{
  "releases": [
    {
      "tag": "v2.4.1",
      "quality_index": 87,
      "deployed_at": "2026-03-07T08:00:00Z",
      "commit_sha": "a1b2c3d",
      "regressions_detected": 0
    },
    {
      "tag": "v2.4.0",
      "quality_index": 79,
      "deployed_at": "2026-03-05T16:30:00Z",
      "commit_sha": "e4f5g6h",
      "regressions_detected": 2
    }
  ]
}

Webhooks

Subscribe to platform events in real time.

POST/api/v1/webhooksRegister a URL for events (regression.detected, index.changed, eval.completed)
GET/api/v1/webhooksList registered webhooks
DELETE/api/v1/webhooks/:idRemove a webhook
example response
// POST /api/v1/webhooks → 201 Created
{
  "id": "wh_Qr7xTp",
  "url": "https://acme.com/hooks/qualityindex",
  "events": ["regression.detected", "index.changed"],
  "secret": "whsec_a1b2c3...truncated",
  "created_at": "2026-03-07T10:00:00Z"
}

Ready to get started?

Deploy the platform locally and see your first Quality Index in under 30 minutes.

Explore FeaturesView Pricing