How to Trace AI Agents with OpenTelemetry

When an autonomous AI agent fails in production, standard logging outputs a disjointed wall of text. You see that an error occurred, but you cannot determine:

• Which planning node generated the malformed query?
• Did the failure stem from a stale memory retrieval 5 turns ago or an unexpected tool response?
• Which sub-agent in a multi-agent hierarchy consumed 80% of the total latency?

Solving this requires Distributed Tracing tailored to graph-based agent state machines.

This guide explores how to model and trace multi-agent workflows using OpenTelemetry (OTel), propagate context across asynchronous boundaries, and instrument leading frameworks like LangGraph and CrewAI.

The Anatomy of an Agent Distributed Trace

In classic microservices, a trace follows an HTTP request through service A, service B, and a database. In an AI agent system, a trace follows the lifecycle of an intent through state transformations, LLM reasoning, and tool calls.

Trace: Run 01J8K9P4 (Total: 4.82s, $0.084, Risk: 0.04)
│
├──  span: agent.orchestrator [root] ────────────────────────── (4.82s)
│   ├── span: node.planner ──────────────────────── (0.85s)
│   │   └── span: llm.openai.gpt-4o ─────────────── (0.82s)
│   │
│   ├── span: agent.researcher [sub-agent] ───────── (2.40s)
│   │   ├── span: node.search_web ────────────────── (1.10s)
│   │   │   └── span: tool.tavily_search ─────────── (1.05s)
│   │   └── span: node.summarize ─────────────────── (1.25s)
│   │       └── span: llm.anthropic.claude-3-7 ───── (1.20s)
│   │
│   └── span: node.synthesizer ───────────────────── (1.45s)
│       ├── span: tool.postgres_write ────────────── (0.12s)
│       └── span: llm.openai.gpt-4o ──────────────── (1.30s)

Key Span Types in Agent Telemetry:

1. Workflow / Graph Span: Encompasses the entire user request from inception to final output.
2. Node / Step Span: Represents a single state transition (e.g., "router", "researcher", "critic").
3. LLM Invocation Span: The raw generation call, recording model name, token counts, temperature, and latency.
4. Tool Execution Span: The external execution boundary (database, REST API, sandbox code runner).

Context Propagation Across Multi-Agent Systems

When an orchestrator agent delegates a task to a sub-agent over a message broker (Kafka, RabbitMQ) or an HTTP API, the trace context must travel with the payload.

OpenTelemetry uses the standard W3C TraceContext format (traceparent header):

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              │  └─────────────┬────────────────┘ └───────┬────────┘ └─ flags
           version          trace_id                   parent_id

Injecting & Extracting Context in Python:

from opentelemetry import trace
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

tracer = trace.get_tracer("splyntra.agent.propagator")
propagator = TraceContextTextMapPropagator()

# --- 1. SENDER: Orchestrator delegating to Sub-Agent ---
def delegate_to_subagent(task_payload: dict) -> dict:
    with tracer.start_as_current_span("orchestrator.delegate") as span:
        carrier = {}
        # Inject current span context into carrier dictionary
        propagator.inject(carrier)
        
        task_payload["_telemetry_carrier"] = carrier
        return send_to_queue(task_payload)

# --- 2. RECEIVER: Sub-Agent executing task ---
def process_subagent_task(task_payload: dict):
    carrier = task_payload.get("_telemetry_carrier", {})
    # Extract parent context
    extracted_context = propagator.extract(carrier=carrier)
    
    # Start span parented by orchestrator span across network boundary
    with tracer.start_as_current_span("subagent.execute", context=extracted_context) as span:
        span.set_attribute("gen_ai.agent.name", "worker_subagent")
        # Do work...

Tracing LangGraph Workflows

LangGraph models agent workflows as cyclical state graphs. To capture node transitions seamlessly, attach an OpenTelemetry tracer to node execution callbacks:

from typing import TypedDict, Annotated
from langgraph.graph import StateGraph, END
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("splyntra.langgraph.tracer")

class AgentState(TypedDict):
    messages: list
    current_node: str

def traced_node(node_name: str):
    """Decorator to automatically wrap LangGraph nodes with OTel spans."""
    def decorator(func):
        def wrapper(state: AgentState):
            with tracer.start_as_current_span(
                f"langgraph.node.{node_name}",
                attributes={
                    "langgraph.node.name": node_name,
                    "langgraph.state.message_count": len(state.get("messages", [])),
                }
            ) as span:
                try:
                    result = func(state)
                    span.set_status(Status(StatusCode.OK))
                    return result
                except Exception as e:
                    span.record_exception(e)
                    span.set_status(Status(StatusCode.ERROR, str(e)))
                    raise e
        return wrapper
    return decorator

# Define graph with traced nodes
@traced_node("planner")
def plan_step(state: AgentState):
    return {"current_node": "planner", "messages": state["messages"] + ["plan created"]}

@traced_node("executor")
def execute_step(state: AgentState):
    return {"current_node": "executor", "messages": state["messages"] + ["task executed"]}

workflow = StateGraph(AgentState)
workflow.add_node("planner", plan_step)
workflow.add_node("executor", execute_step)
workflow.set_entry_point("planner")
workflow.add_edge("planner", "executor")
workflow.add_edge("executor", END)

app = workflow.compile()

Tracing CrewAI Hierarchical Teams

CrewAI structures agents into hierarchical teams with a manager and specialized workers. Here is how to trace Crew executions:

from crewai import Agent, Task, Crew, Process
from opentelemetry import trace

tracer = trace.get_tracer("splyntra.crewai.tracer")

def run_traced_crew(topic: str):
    with tracer.start_as_current_span(
        "crewai.crew_execution",
        attributes={
            "crewai.topic": topic,
            "crewai.process_type": "hierarchical",
        }
    ) as crew_span:
        researcher = Agent(
            role='Senior Research Analyst',
            goal=f'Uncover cutting-edge developments in {topic}',
            backstory="You are an expert researcher.",
            verbose=False
        )
        
        task1 = Task(
            description=f"Analyze top 5 trends in {topic}",
            expected_output="Bullet list of 5 key trends",
            agent=researcher
        )
        
        crew = Crew(
            agents=[researcher],
            tasks=[task1],
            process=Process.sequential
        )
        
        with tracer.start_as_current_span("crewai.kickoff") as kickoff_span:
            result = crew.kickoff()
            kickoff_span.set_attribute("crewai.output_length", len(str(result)))
            
        crew_span.set_attribute("gen_ai.agent.status", "completed")
        return result

Debugging Common Agent Failure Modes with Traces

Next Steps & Related Technical Guides

• How to Monitor AI Agents with OpenTelemetry — Foundation guide to OTel GenAI setup.
• AI Agent Observability: What You Need to Monitor in Production — Core observability architecture.
• How to Track LLM Costs in AI Agents — Tracking per-span costs in recursive graphs.
• AI Agent Security: A Practical Guide — Securing tool boundaries across distributed traces.