AI Observability

AI observability — often called LLM observability — is the practice and tooling that give teams visibility into how AI applications behave in production: every prompt, retrieval step, tool call, token and unit of cost in a request, captured as a trace that can be inspected when output quality is in doubt^[1].

Background

Traditional application monitoring (Datadog, New Relic, CloudWatch) tracks uptime, latency and error rates. A generative AI application can return HTTP 200 with a fluent, confident and entirely wrong answer, so those metrics say nothing about whether it works. Failures are semantic: a retrieval step silently returning irrelevant context after a prompt update two weeks earlier, a malformed tool argument, a drift in cost per conversation^[1][2].

Langfuse describes tracing as the most important observability tool for LLM applications because it captures the full context of each request — prompts, responses, tool calls and their relationships — with each trace recording timing, inputs, outputs and metadata for every operation^[1]. Because agent systems chain many model calls per user action, the bug is rarely in the final call: it is inside a retrieval span several steps back.

Key capabilities

• Tracing — hierarchical spans for each LLM call, retrieval step and tool execution, often instrumented via OpenTelemetry-compatible conventions.
• Evaluation — offline testing against golden datasets, online sampling of production traffic, and LLM-as-a-judge for graded quality signals.
• Cost and token accounting — per-feature and per-endpoint spend, so a prompt change that doubles token use is visible immediately.
• Prompt and version management — comparing runs across prompt versions, with regression alerts on deployment.
• Guardrail and PII monitoring — detecting blocked outputs and sensitive data flowing through prompts.

Landscape

Four platforms dominate the LLMOps observability conversation: LangSmith, the LangChain-native tracing platform; Langfuse, an open-source (MIT-licensed) option that can be self-hosted and emphasises cost visibility; Arize Phoenix, strong on RAG evaluation and enterprise monitoring; and W&B Weave, oriented towards experimentation and prompt research, with Helicone offering proxy-based instrumentation^[2][3]. Selection usually follows the team's framework (LangChain or LangGraph teams default to LangSmith) and its deployment preference (data-sensitive teams default to self-hosted Langfuse).

>See Also

• Langfuse
• LangSmith
• MLOps
• LLM Evaluation
• AI Guardrails