How to evaluate AI agents comes down to three stages run in order: an offline eval set of 50 to 200 real cases taken from your own history, a written rubric a reviewer can apply to one case in about 30 seconds, and a live measurement behind a named approver once the first two pass. Skip the first two and all you have is a demo, which is a sample of size one chosen by the person selling it.
Picture the operations lead of a travel company in Colombo 4, with nine people on a support desk. She is our invention, but the meeting is not. A vendor has just run twelve queries through an agent live on the call and every answer came back correct. She cannot name what is wrong with what she watched, so she signs, and in week three finds the agent confidently applying a refund rule nobody ever wrote down.
Every US dollar figure below converts at LKR 328.75, the Central Bank of Sri Lanka’s indicative USD/LKR spot rate for 10 September 2026.
Why “it looked good in the demo” is not an evaluation
Published agent benchmarks show how far a good first impression sits from reliability. In the 2024 paper introducing tau-bench, which simulates retail and airline conversations with real tool calls and written policies, Yao and colleagues report that “even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks”, and that those agents “are quite inconsistent (pass^8 <25% in retail)”. Pass^8 is their measure of getting the same task right in all eight independent attempts.
Read them together and the demo problem is obvious: an agent can be right once and wrong on the same task next run, so twelve queries chosen by the vendor say nothing about a shift of two hundred.
Evaluation is therefore not one judgement but three, in order: does it handle work shaped like ours, do we agree what a good answer looks like, and does it hold up on live work with a person in front of it. That order follows the argument in The SDLC Didn’t Get Faster. It Got Restructured. When producing output gets cheap, checking it becomes the constraint.
Stage one: an eval set from your own history
An eval set is a frozen collection of real cases with the correct outcome written down for each, made before the agent sees any of them. It is the cheapest artefact here, and the one everyone skips.
How many. Between 50 and 200 cases for one workflow, with at least 10 in every category you intend to automate. The floor matters more than the total: a set of 200 holding one refund exception tells you nothing about refund exceptions, and one is what unwatched sampling gives you.
How to sample. Take a fixed window, usually the last quarter, list the cases by category, then sample at random inside each rather than picking the ones you remember. Add the awkward inputs deliberately: the truncated message, the customer asking three things at once, the case where two rules point opposite ways. A set of tidy work tests the half of the job that was never the problem.
How to write the expected outcome. Not “a good reply”. The exact action and fields: which record is read, which template is used, what is escalated, and what happens when the right move is to do nothing. If that is hard to write you have found the real defect, and the fix is the spec, not the model. Our five part shape is in how to write a spec for AI agents.
How to anonymise. Replace names, identity numbers, card fragments and phone numbers with stable tokens, each consistent across cases so a repeat customer is still recognisable. With Sri Lanka’s Data Protection Authority now in place, the safe default is that an eval set never leaves your systems identifiable and the mapping stays with you, not the vendor. This is not legal advice.
Stage two: a rubric a reviewer can apply in 30 seconds
Anthropic’s documentation on defining success criteria insists criteria be specific and measurable: “accurate sentiment classification” rather than “good performance”, and “Less than 0.1% of outputs out of 10,000 trials flagged for toxicity by the content filter” rather than “Safe outputs”. The same discipline fits on a card at the scale of a support desk.
Score every case against five checks, each a plain yes or no.
- Grounded. Every fact traces to a record the agent actually read. No invented order number, date or balance.
- Rule followed. It applied the written rule, exception clause included, not a reasonable-sounding version of it.
- Scope. It did what was asked and nothing more. No extra email, no second action, no helpful edit to a record.
- Form. The approved template, the right language, the right amounts in the right currency.
- Stopped correctly. Where the case was out of bounds it refused, escalated and said why, instead of answering confidently.
A case passes only if all five pass. Partial credit sounds fairer and is worse, because any score that has to be argued about ends up decided in favour of whoever wants the project to proceed.
Have two reviewers score the same 20 cases independently before scoring the rest. If they disagree on more than two or three, stop. That is proof the rule was never written down, and no evaluation survives that.
Tools help, but an AI agent evaluation still needs a human verdict
Every major model platform ships eval tooling that will store, run and score a set. Use it. What it cannot do is decide the verdict. In a systematic study of position bias in LLM-as-a-judge, published in June 2024 and revised through 2025, Shi and colleagues scored 15 LLM judges across MTBench and DevBench, covering 22 tasks and over 150,000 evaluation instances, and proposed repetition stability, position consistency and preference fairness as measures. They found position bias is not random and varies substantially by judge and by task. A judge whose verdict shifts with the order it sees two answers in is a useful filter and a poor referee, so let a model triage and have a person score what matters.
Stage three: live, behind an approver, measured weekly
A frozen set cannot contain next month’s customer. Once a category clears the rubric it goes live in draft only: the agent proposes, a named person approves or corrects with a reason, nothing leaves the building unreviewed.
The number is approval rate, the share of drafts that reviewer approves untouched, read weekly against a baseline recorded before the agent was switched on. An edited draft is not approved. How it is produced, and why we cannot polish it, is in what an AI agent’s approval rate means. It catches what a frozen set cannot: drift in your inputs, and the reviewer who has started clicking approve on everything.
The three stages side by side
| Stage | Input | Who scores | Metric | Pass threshold | What happens on fail |
|---|---|---|---|---|---|
| Stage 0: baseline | One week of the work as done today | The approver, ten seconds an item | Share sent untouched today | None: later stages are read against it | No baseline, no evaluation. Stop |
| Stage 1: eval set | 50 to 200 anonymised cases from last quarter, each with a written expected outcome | Nobody yet: the set is built and frozen | Cases per category | At least 10 per category you intend to automate | The category leaves scope until the cases exist |
| Stage 2: rubric | Output on the frozen set, run twice | Two of your reviewers, 30 seconds a case | Share passing all five checks, by category | Above the stage 0 baseline, stable across both runs | Rewrite the rule, not the prompt, and rescore |
| Stage 3: live | Real work, drafted, nothing sent unreviewed | The named approver, weekly | Approval rate: drafts approved untouched | Improvement on the day one baseline by day 30 | The hire is not confirmed, the balance not invoiced |
A worked example: a support queue
Back to the travel desk in Colombo 4, figures invented. It handles about 600 messages a month. The supervisor logs a week of replies as sent or changed, then they export the last quarter, sort it into five categories and pull 24 from each for a set of 120.
Stage 2 is where the useful thing happens, and it is not a score. The failures cluster. Confirmations and schedule notices pass all five checks in both runs. Refund exceptions fail on rule followed in both runs, because the desk applies a courtesy waiver that lives in the supervisor’s head and appears in no document. That is not a verdict on the agent. It is a map of which parts of this desk were ever written down, made by her own staff.
Confirmations and schedule notices go live behind her. Refunds stay with the humans until somebody writes the waiver down.
What we will not claim
specshop.dev is an AI consultancy and AI agency in Colombo, Sri Lanka, led by Janaka Ediriweera, Principal AI and Product Management Consultant. We publish no accuracy percentage for our own agents, and you should discount anyone who does: the vendor picks the test set, the prompt and the run that gets reported.
What we sell is the method above and the terms around it. A workflow mapping sprint is US$2,300, about LKR 756,125, credited in full against the recruitment fee if you hire within 30 days. Agent salaries start at US$1,225 a month, about LKR 402,700. The recruitment fee is US$9,500, about LKR 3,123,125, one off, covering deployment, tool connections, the first month’s salary and the 30 day probation with weekly scorecards. If approval rate has not improved on its day one baseline by day 30, you do not confirm the hire and the balance is never invoiced.
If your own engineers run the three stages instead, better still, and that is what the two weekends of our AI training programme are for, at US$500 to US$2,000 a seat by track, about LKR 164,375 to LKR 657,500. To bring in people who already work this way, hire AI engineers in Sri Lanka sets out the cost.
What to do this week
Build the eval set. Pick one workflow, export the last quarter, sort it into categories and count them, because the count alone tells you which parts of the job you have never written down.
Then write expected outcomes for 20 of those cases, and have a second person do the same 20 independently. Where the two of you differ is where every agent, and every new hire, will guess.
Questions people actually ask.
How do you test an AI agent before deployment?
Test it in three stages: run it against a frozen eval set of real cases with expected outcomes written before the agent saw them, score each case against a short binary rubric applied by two of your own reviewers, then run it live in draft form behind a named approver who has to approve before anything is sent. Run the offline stage twice rather than once, because the tau-bench authors found that agents scoring under 50% on tasks were also “quite inconsistent (pass^8 <25% in retail)", meaning a single successful run is weak evidence.
How many examples do you need to evaluate an AI agent?
Between 50 and 200 cases for a single workflow, with a floor of at least 10 cases in every category you intend to automate. The per category floor matters more than the headline number, because a 200 case set containing one refund exception can tell you nothing about refund exceptions, which are usually the cases that carry the money and the risk.
What is an eval set?
An eval set is a fixed, anonymised collection of real cases from your own history, each with the correct outcome written down in advance, held unchanged so that different agents, prompts and versions can be scored against exactly the same work. It is the only part of an evaluation that stays yours: whichever vendor you use, the set keeps working, and it makes the next comparison a morning’s work rather than a project.
What metrics matter for AI agents in a business?
Two: the share of eval cases that pass every check in your rubric, measured offline before anything goes live, and the approval rate on real work afterwards, which is the share of drafts a named reviewer approves without editing. Ignore volume metrics entirely, because an agent producing forty drafts a day that all get rewritten has added a review queue rather than removed work.
Can a non-technical team evaluate an AI agent?
Yes, and the scoring stage is better done by the people who own the work than by engineers, because they are the ones who know which exception applies. The technical part is exporting and anonymising the cases, which is a day of work for one engineer, after which a supervisor with a five item rubric can score a case in about 30 seconds without knowing anything about how the agent is built.
Sources
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan, arXiv:2406.12045, June 2024 — the abstract’s findings that “even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks”, that they “are quite inconsistent (pass^8 <25% in retail)”, and the definition of pass^k as a measure of the reliability of agent behaviour over multiple trials. Fetched 11 September 2026.
- Define your success criteria, Claude platform documentation, Anthropic, 2026 — the requirement that success criteria be specific, measurable, achievable and relevant, the replacement of “good performance” with “accurate sentiment classification”, and the safety example contrasting “Safe outputs” with “Less than 0.1% of outputs out of 10,000 trials flagged for toxicity by the content filter”. Fetched 11 September 2026.
- Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge, Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma and Soroush Vosoughi, arXiv:2406.07791, June 2024, revised November 2025 — the study of 15 LLM judges across MTBench and DevBench covering 22 tasks and over 150,000 evaluation instances, the finding that position bias varies substantially by judge and task and is not attributable to random chance, and the three proposed metrics of repetition stability, position consistency and preference fairness. Fetched 11 September 2026.
- Daily Indicative USD/LKR Spot Exchange Rates, Central Bank of Sri Lanka, 2026 — the indicative rate of LKR 328.75 per US$1 on 10 September 2026, used for every conversion in this post.