observingagents

The Night My Agent Returned 200 OK and Booked the Wrong Flight

Every dashboard was green. Zero exceptions in Sentry, 180ms latency, and a happy 200 OK. Yet the agent confidently confirmed a flight to Texas instead of Paris.

Dany Paredes
Dany ParedesFirst-person field note · Inspired by AI Agent Observability

For ten years, my mental model of production monitoring was simple: if the HTTP status is 200 OK, latency is low, and Sentry is quiet, the system is healthy.

That mental model broke on a Tuesday night.

I had wired up an assistant that could look up travel itineraries and trigger a booking endpoint. A test user asked: “Book a flight to Paris for next Friday.” My backend received the request, called the model, executed the tool, and returned 200 OK in under two seconds.

Then I read the actual reply the user saw:

“Done! I booked your flight to Texas for next Friday.”

No stack trace. No 500 error. No timeout. From the perspective of traditional APM tools, it was one of the cleanest requests of the evening.

The gap between uptime and truth

That night taught me the difference between infrastructure health and semantic correctness. Traditional observability answers “Did the server crash?” Agent observability has to answer “Did the agent actually do what the user meant?”

When I finally dug into the raw inputs and tool arguments, the culprit wasn’t a bug in my REST controller. A stale context chunk from a previous test session had leaked into the system prompt, nudging the tool call toward an airport code in Texas while the model’s final summary cheerfully confirmed the action.

What I changed the very next morning

I stopped treating LLM calls like black-box HTTP requests and started recording three things on every single run:

  1. The exact prompt assembly — system instructions, retrieved context chunks, and user message together.
  2. Every tool call’s arguments and raw output — not just whether the function returned, but what it received.
  3. A lightweight semantic check — comparing the user’s intent against the tool parameters before celebrating a 200 OK.

Once I could see the full chain of thought and tool arguments side by side, that “mysterious” hallucination took thirty seconds to diagnose.