Tracing in Agent Observability: What It Is and Why It Matters

When an agent misbehaves, the first question is always the same: what did it actually do? Logs almost never answer that question. Traces do, and once you have readable ones, most agent debugging drops from hours to minutes.

Why logs fall apart for agents#

A normal request handler does one thing. An agent runs a loop: call the model, get a tool request, run the tool, feed the result back, call the model again, maybe spawn a subagent, and repeat until it decides it is done. One user message can turn into thirty operations, and the shape of those operations changes run to run because the model is nondeterministic.

Log lines flatten all of that into an interleaved stream. If two runs execute at the same time, their lines mix together. If a tool call fails three levels deep inside a subagent, the error shows up as one line with no context about which run, which iteration, or which parent decision caused it. You end up grepping for request IDs and reconstructing the story by hand.

The story you are trying to reconstruct is a tree: this run called the model, which called this tool, which called this database. A trace is that tree, recorded for you.

The anatomy: traces, spans, and steps#

Three terms cover almost everything.

A trace is the complete record of one run, end to end. For an agent, that usually means one user message in and one final answer out, or one execution of a scheduled job. Every trace gets a trace_id, and everything that happened during the run carries it.

A span is one unit of work inside the trace. A span has a name, a start time, an end time, a status (ok or error), a set of key-value attributes, and a pointer to its parent span. That parent pointer is what turns a bag of spans into a tree: the root span is the run itself, and every step nests underneath.

A step is not a formal term, but you will hear it constantly. LangSmith calls spans "runs", some tools say "steps", agent frameworks talk about "iterations". They all map onto the same idea: a node in the tree. If you understand spans, you understand all of them.

Here is what a support agent run looks like as a trace:

agent.run                        8.4s   ok
├── llm.plan                     1.2s   ok
├── tool.search_orders           0.9s   ok
├── llm.decide                   1.1s   ok
├── retriever.policy_docs        0.4s   ok
│   └── db.vector_query          0.3s   ok
├── subagent.refunds             3.9s   ok
│   ├── llm.plan                 0.8s   ok
│   ├── tool.refund_policy       0.5s   ok
│   └── llm.answer               2.4s   ok
└── llm.final_answer             0.9s   ok

Every question you would normally grep for is now visible at a glance: what ran, in what order, nested under what, and how long each piece took. The subagent is just a subtree. A parallel fan-out shows up as siblings with overlapping timestamps.

What belongs on a span#

Timings and structure come free. The real debugging power is in attributes, the key-value pairs you attach to each span. A span that says "an LLM call happened" is barely useful. A span that says which model, with what input, returning what output, at what token cost, is a debugging session in a box.

A rough guide to what to record per span kind:

If you use an instrumentation library (LangChain, the OpenAI SDK instrumentors, and friends all have them), most of this is captured automatically. You add manual spans for the parts that are yours: custom tools, business logic, post-processing. With OpenTelemetry-style APIs that looks like this:

import json
from opentelemetry import trace
 
tracer = trace.get_tracer("support-agent")
 
def run_tool(name: str, args: dict):
    with tracer.start_as_current_span(f"tool.{name}") as span:
        span.set_attribute("tool.name", name)
        span.set_attribute("tool.args", json.dumps(args)[:2000])
        try:
            result = TOOLS[name](**args)
            span.set_attribute("tool.result_chars", len(str(result)))
            return result
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(trace.Status(trace.StatusCode.ERROR))
            raise

Because the span is opened with start_as_current_span, it automatically nests under whatever span was active when the tool ran. You get the tree without managing it yourself.

A worked example: walking the tree#

Here is the kind of bug traces are built for. A user asks the support agent whether order 4832 can be refunded. The agent confidently answers that no refund is possible. That is wrong, and nothing errored, so the logs show a clean run.

Walking the trace:

1. The root span is ok. Total time 6.1s, nothing unusual.
2. llm.plan looks sensible: the model decided to look up the order first. Good.
3. tool.search_orders is ok, but its attributes show rows_returned: 0. Suspicious.
4. The same span's tool.args shows {"order_id": "#4832"}. The tool expects an integer. The model passed the ID with a # prefix, the lookup silently matched nothing, and the tool returned an empty result instead of an error.
5. llm.final_answer shows the model doing its best with an empty context: it assumed the order did not exist and answered accordingly.

Root cause in five clicks: a silent type mismatch between the model's tool call and the tool's contract. The fix is input normalization in the tool plus a validation error instead of an empty result. Without the trace, this is an afternoon of adding print statements and trying to reproduce a nondeterministic run. With it, the broken step is sitting right there with its inputs attached.

Latency debugging works the same way. If p95 jumped, open a slow trace and find the long pole. In the tree above, subagent.refunds costs 3.9s of the 8.4s total, and 2.4s of that is one llm.answer call. That is where the optimization goes (shorter context, a faster model for that step, or caching the policy lookup, which I covered in caching agent tool calls).

Traces are the raw material for everything else#

Debugging is the first payoff, but not the biggest one. Once every run is a structured tree, you can build on top of it:

Metrics. Aggregate spans to get error rate per tool, p95 latency per span kind, token cost per run. This is exactly the "what to measure" layer I wrote about in observability for LLM apps, and traces are where those numbers come from.

Evaluation. Sample real traces into datasets and score them with judges or assertions. A trace gives the evaluator the full context of what the agent did, not just the final answer. I walked through that workflow in evaluating agents with LangSmith.

Alerting. Alert on span-level signals (a tool's error rate, a subagent's latency) instead of only on process-level symptoms.

Wrapping up#

A trace is the tree of everything one agent run did; spans are the nodes; attributes make the nodes worth reading. Instrument once, and you get debugging, metrics, evals, and alerts from the same data.

The natural next question is what standard to build this on so you are not locked into one vendor's SDK. That standard is OpenTelemetry, and it is the subject of the next post: what OpenTelemetry is and why it matters for agent observability.