OpenTelemetry (OTel) has become the de facto open standard for collecting telemetry data from modern distributed applications. Instead of relying on vendor-specific SDKs, OpenTelemetry provides a common framework for generating **Traces**, **Metrics**, and **Logs**, allowing organizations to export observability data to a wide variety of backends such as Jaeger, Grafana Tempo, Prometheus, Elastic, Datadog, Splunk, Dynatrace, Honeycomb, AWS X-Ray, Azure Monitor, and many others.
For traditional applications, OpenTelemetry helps developers understand request flows across multiple microservices, identify performance bottlenecks, detect failures, and correlate metrics with logs and traces. As AI applications have evolved into distributed, multi-agent systems, OpenTelemetry has naturally extended to become one of the strongest foundations for **AI Observability**.
Unlike conventional applications, AI workloads involve several additional dimensions that require observability:
* Agent orchestration
* Multiple LLM invocations
* RAG retrieval pipelines
* MCP tool execution
* Prompt engineering
* Token consumption
* AI cost
* Model selection
* User conversations
* AI quality metrics
OpenTelemetry allows all of these to be attached as **trace attributes**, **events**, and **child spans**, giving developers complete end-to-end visibility into an AI request.
---
# What we built
Across the two blog articles, we progressively evolved a simple FastAPI application into a production-inspired AI system instrumented with OpenTelemetry.
We covered:
## Part 1
* Installing OpenTelemetry SDK
* Configuring the OpenTelemetry Collector
* Running Jaeger using Docker Compose
* Creating spans
* Exporting traces
* Viewing traces in Jaeger
* Instrumenting a simple AI endpoint
This established the foundation for distributed tracing.
---
## Part 2
We then enhanced the same application to instrument advanced AI workflows.
### Q2 — Multi-Agent Reasoning Chains
We traced
* Supervisor agent
* Research agent
* Retriever agent
* Tool agent
* Validation agent
* Summarizer agent
while recording
* Agent handoffs
* Workflow execution
* Reasoning events
* Token usage
* Cost
* Execution latency
This allows engineers to understand exactly how an agentic workflow executed.
---
### Q3 — Prompt Explosion Detection
Instead of only measuring token usage, we monitored
* Original prompt size
* Expanded prompt size
* Prompt amplification ratio
* Additional tokens introduced
* Source responsible for prompt growth
This helps identify unnecessary prompt expansion before it causes excessive cost and latency.
---
### Q4 — AI Cost Attribution
We demonstrated cost tracking at multiple levels.
* Per span
* Per conversation
* Per user
* Per tenant
* Total request
This makes it possible to answer questions like
* Which tenant spends the most?
* Which conversation exceeded budget?
* Which agent is most expensive?
---
### Q5 — RAG Retrieval Quality
Rather than treating retrieval as a black box, we monitored
* Retrieved documents
* Retrieved chunks
* Similarity score
* Retrieval latency
* Context utilization
* Retrieval quality
This provides visibility into whether poor LLM responses are caused by retrieval rather than the model itself.
---
### Q6 — MCP Tool Usage
We instrumented every MCP invocation.
For each tool execution we captured
* MCP Server
* Tool Name
* Transport
* Latency
* Retry count
* Status
* Response size
* Request ID
This allows developers to identify unreliable external dependencies in an agentic workflow.
---
# Important AI Observability Principles
Throughout the examples we also introduced several production best practices.
### Attribute useful metadata
Rather than storing only latency, record
* model
* provider
* tokens
* cost
* conversation ID
* tenant
* user
* workflow
---
### Use events for reasoning
Instead of creating unnecessary spans, capture
* reasoning decisions
* handoffs
* retries
* validation
* planning
as events inside spans.
---
### Avoid high-cardinality attributes
Avoid storing
* Full prompts
* Complete documents
* Entire conversations
inside spans.
Instead prefer
* Prompt hash
* Prompt size
* Token count
* Conversation ID
to reduce storage cost.
---
### Aggregate intelligently
Record detailed information at the span level while also aggregating key metrics at the overall trace or conversation level.
Examples include
* Total tokens
* Total cost
* Total latency
* Total tool calls
* Number of agent hops
This provides both fine-grained diagnostics and high-level operational insights.
---
# Why OpenTelemetry is an Excellent Foundation for AI
OpenTelemetry is not an AI observability product—it is an observability framework. That distinction is important because it means you can instrument your AI applications once and send the telemetry to virtually any backend or AI observability platform. As the ecosystem evolves, your instrumentation remains stable while your choice of backend can change.
It also integrates naturally with modern AI frameworks such as:
* LangChain
* LangGraph
* LlamaIndex
* AutoGen
* CrewAI
* Semantic Kernel
* OpenAI Agents SDK
* Amazon Bedrock Agents
This makes it an ideal foundation for enterprise AI systems.
---
# What's Next
OpenTelemetry provides the raw telemetry, but many AI-specific platforms build on top of it to offer higher-level capabilities such as prompt management, evaluations, hallucination analysis, experiment tracking, model comparisons, and dataset management.
The natural next step is to explore how OpenTelemetry integrates with tools such as **Langfuse**, **LangSmith**, **OpenLIT**, **Arize Phoenix**, **MLflow**, **Helicone**, and **Traceloop**, combining standard observability with AI-native analytics for a complete view of modern AI applications.
**One key takeaway:** treat OpenTelemetry as the **observability backbone** of your AI platform. Instrument once, enrich traces with AI-specific metadata, and build increasingly sophisticated monitoring—from simple request tracing to comprehensive visibility into multi-agent reasoning, RAG quality, costs, governance, and production reliability.
