LLM Observability Platforms
A home for traces and evals
Langfuse, Arize Phoenix, LangSmith, Helicone and Datadog provide tracing UIs, eval pipelines, prompt registries and dataset management out of the box. Pick by cost, self-hosting and eval depth — most teams start with one and standardize contracts around it.
What you will learn
- Compare LLM observability platforms
- Select by workload, budget and hosting
- Keep contracts portable across tools
What is this?
LLM observability platforms package the whole stack: SDKs that emit traces with gen-AI attributes, a UI to inspect prompts/outputs, eval runners (golden sets, judges), prompt versioning and cost dashboards. They're the fastest route from zero to visible, though they add a dependency and a data-flow.
Why do you need it?
- Rolling your own trace UI + eval infra is weeks of work.
- Completed platforms already speak the gen_ai attribute standard.
- They switch cost: self-hosted vs SaaS changes data-residency and ops burden.
Mental model
Platforms
Buying a fitted kitchen
You could frame, wire, plumb and tile it yourself; buying fitted units gets the same kitchen in days with fewer custom mistakes.
How it works
- An SDK (or gateway exporter) sends OTel-format traces to the platform.
- The platform ingests spans, renders timelines, shows prompt/response and cost.
- Eval runs attach scores to traces; dashboards aggregate by version.
- Portable contracts (OTLP, standard attributes) let you swap platforms.
Event → platform → answer.
In production
- A startup runs Langfuse Cloud with golden evals and prompt versioning.
- A strict bank self-hosts Langfuse on-prem to keep prompts inside the perimeter.
- A big org standardizes on Datadog for AI since the rest of the stacks already live there.
Best practices
- Emit standard semconv so your contracts outlive any specific tool.
- Redact at the SDK/source before anything ships to the platform.
- Start with one platform, keep data-export enabled, and re-evaluate yearly.
- Cap sample rates per traffic class to control ingest cost.
Common mistakes
- Onboarding three platforms at once — same traces, triple cost.
- No redaction: full prompts and chatter flow to a vendor.
- Picking on feature-marketing without testing self-hosting or scale limits.
Debugging tips
- Traces missing in the UI → exporter/OTLP endpoint or sample rate misconfig.
- Cost numbers differ from billing → platform pricing table vs actual; verify usage fields.
- Eval scores not on traces → the eval run needs the same trace ids.
Interview checkpoint
Hands-on lab
- Build a minimal span batch
- Serialize as JSON with trace ids
- Push to a collector endpoint over HTTP
Wire an OTLP-Style Exporter
30 min · advanced
Quick quiz
A rule of thumb when choosing a platform?
Platforms always require sending data to a cloud you don't control.