Why exact-match testing breaks down for agents
Traditional software tests typically assert that a given input produces one specific, exact output. That approach mostly falls apart for agents. The same correct underlying behavior, say, successfully looking up an order and canceling it, can produce a final response that's worded differently, structured differently, or arrives after a different sequence of tool calls each time you run it, even when nothing is actually wrong. An exact-match assertion on the agent's text output will fail constantly on perfectly correct runs, training you to either ignore your tests or loosen them so much they stop catching real problems.
The fix is to shift what you're checking from exact text to outcomes and behavior. Instead of asserting the final response equals a specific string, you assert things like: did the agent call the correct tool at all, did it reach the correct final state (the order actually got canceled, the record actually got updated), and did it avoid taking an unsafe or incorrect action along the way. These are checks on what the agent did and where it ended up, not on the precise phrasing it used to describe doing it, and they remain meaningful even as wording varies run to run.