AI observability: knowing your LLM feature degraded before your users tell you
Traditional monitoring tells you an AI feature responded successfully. It cannot tell you the answers got worse last Tuesday, which is the failure that actually happens.
By Quality AboveAll · · 8 min read
- Trace every request end to end: retrieval, prompt, model version, output, latency and cost.
- Sample and score real production outputs continuously, because quality degrades without erroring.
- Alert on refusal rate, error rate, latency and spend anomalies, not just on availability.
Why standard monitoring is insufficient
An AI feature can be fully available, returning 200s within latency budget, while giving materially worse answers than last week, because a model was updated, a document set changed, or a prompt edit had an unintended effect.
That is a silent failure mode, and users usually respond by quietly using the feature less rather than reporting it. By the time adoption metrics show the decline, you have lost months and cannot say when it started.
Trace the whole request
Record what was retrieved and with what scores, the assembled prompt, the model and version, the raw output, any guardrail decisions, latency per stage and token counts. Without this, debugging a bad answer is guesswork.
Standard tooling handles this well. OpenTelemetry traces work perfectly for AI pipelines, and using the same observability stack as the rest of your system beats a separate AI-specific tool that nobody has dashboards for.
The first question about a bad answer is always what the model was given. If you cannot answer that, everything after it is speculation.
Sample and score continuously
Take a sample of real production outputs on a schedule and score them, by model-graded evaluation against a rubric, human review, or both. This is the only way to detect gradual quality decline, and it should feed the same metrics pipeline as everything else.
Track the score as a time series with alerting on decline. Combine it with your golden-set results from CI, described in LLM evaluation, so you have both a controlled measure and a live one.
The operational signals
Refusal rate is an unusually informative metric: a sudden rise means inputs changed, retrieval broke, or the model became more conservative, and it often moves before anything else does. Error and timeout rates per provider matter for the same reason.
Cost per request is a quality signal as much as a financial one. A jump usually means prompts grew or retrieval is returning more than intended, and it is frequently the first visible symptom of a configuration regression. Optimisation is covered in LLM cost optimization.
Detect drift in inputs and behaviour
Watch the distribution of incoming requests. Users find uses you did not design for, and a feature performing well on its intended distribution can perform badly on what it is actually receiving, without anything technically failing.
Log the questions that produced no confident answer and review them regularly. That list is simultaneously a quality signal, a content gap report, and a roadmap of what users actually want, which makes it one of the highest-value artefacts the system produces.
Frequently asked questions
Do we need a specialised LLM observability tool?
Not necessarily. Standard tracing and metrics tooling covers most needs if you instrument the AI-specific fields. Specialised tools add convenience for prompt and evaluation management.
How often should we sample outputs for quality?
Continuously at a low rate, with a larger review after any change to prompts, models or retrieval. Daily scoring of a modest sample catches most regressions quickly.
What is the most useful single alert?
A sustained rise in refusal or no-answer rate. It moves early and usually indicates retrieval or input problems before quality scores register the decline.
Running AI in production without knowing whether quality is holding? A free 30-minute consultation covers what to instrument and what to alert on.
