Why I Stopped Guessing Prompts and Started Tracing Every Span
Changing a single sentence in a system prompt felt like blindfolded surgery—until I borrowed distributed tracing from microservices and saw every step in one timeline.
Early on, whenever an agent gave a weird answer, my workflow was embarrassingly predictable: open the prompt file, tweak two adjectives, re-run the script, and hope for the best.
Sometimes it worked. Most of the time, fixing one edge case quietly broke three others. I wasn’t engineering—I was guessing in the dark.
One request, seven invisible hops
When a user asks a seemingly simple question like “Summarize my last invoice,” your application doesn’t make one call. Behind the scenes, mine was doing this:
- Root Span: Receiving the user’s message.
- Span 1 (Retrieval): Querying a vector store for context.
- Span 2 (Tool Call): Calling
fetchInvoice(id)against PostgreSQL. - Span 3 (LLM Call): Sending the prompt, retrieved chunks, and invoice JSON to the model.
Without a trace connecting those spans, I had console.log fragments scattered across three files and zero idea which step introduced the bad data.
Adopting OpenTelemetry instead of another custom logger
My first instinct was to write a quick JSON logger. I’m glad I threw that away.
Using OpenTelemetry (OTel) with GenAI semantic conventions meant every span automatically carried the fields I actually needed when debugging at midnight:
gen_ai.systemandgen_ai.request.modelgen_ai.usage.input_tokensandgen_ai.usage.output_tokens- Exact parent-child timing between retrieval, tool execution, and generation
The first time I opened a waterfall timeline of a slow agent run, the bottleneck wasn’t the LLM at all—it was a redundant database lookup inside a tool call eating 1.4 seconds on every turn.
You can’t optimize what you can’t see, and you can’t trust a prompt you only understand by guessing.