How to Evaluate AI Agents

Why it's hard

Agents take multi-step, non-deterministic paths. The same task can succeed via different routes, so you can't just diff against a fixed answer.

What to measure

• Task success — did it achieve the goal? (the headline metric)
• Efficiency — steps, tokens, cost, latency.
• Trajectory — did it use the right tools correctly?
• Safety — did it avoid forbidden actions?

Approach

Curate representative tasks with checkable success criteria, run them repeatedly (agents are stochastic), and track success rate over versions. Trace every run so failures are diagnosable, not mysterious.