observingagents

Why I Stopped Guessing Prompts and Started Tracing Every Span

Changing a single sentence in a system prompt felt like blindfolded surgery—until I borrowed distributed tracing from microservices and saw every step in one timeline.

Dany Paredes
Dany ParedesFirst-person field note · Inspired by AI Agent Observability

Early on, whenever an agent gave a weird answer, my workflow was embarrassingly predictable: open the prompt file, tweak two adjectives, re-run the script, and hope for the best.

Sometimes it worked. Most of the time, fixing one edge case quietly broke three others. I wasn’t engineering—I was guessing in the dark.

One request, seven invisible hops

When a user asks a seemingly simple question like “Summarize my last invoice,” your application doesn’t make one call. Behind the scenes, mine was doing this:

  • Root Span: Receiving the user’s message.
  • Span 1 (Retrieval): Querying a vector store for context.
  • Span 2 (Tool Call): Calling fetchInvoice(id) against PostgreSQL.
  • Span 3 (LLM Call): Sending the prompt, retrieved chunks, and invoice JSON to the model.

Without a trace connecting those spans, I had console.log fragments scattered across three files and zero idea which step introduced the bad data.

Adopting OpenTelemetry instead of another custom logger

My first instinct was to write a quick JSON logger. I’m glad I threw that away.

Using OpenTelemetry (OTel) with GenAI semantic conventions meant every span automatically carried the fields I actually needed when debugging at midnight:

  • gen_ai.system and gen_ai.request.model
  • gen_ai.usage.input_tokens and gen_ai.usage.output_tokens
  • Exact parent-child timing between retrieval, tool execution, and generation

The first time I opened a waterfall timeline of a slow agent run, the bottleneck wasn’t the LLM at all—it was a redundant database lookup inside a tool call eating 1.4 seconds on every turn.

You can’t optimize what you can’t see, and you can’t trust a prompt you only understand by guessing.