How to run an AI pilot that decides anything comes down to four rules: one workflow rather than a platform, one named approver rather than a committee, one number measured weekly against a day-one baseline, and a decision on day 30 written before day one. A pilot missing any of the four cannot end, because nothing in it can turn into a yes or a no.
Picture the finance director of a freight forwarder in Colombo 2. She is our invention, but the document on her desk is not unusual: an AI platform, six modules, a three-month rollout, a promise of forty per cent time saved. She does not want to refuse it and she does not want to sign it. She wants to know cheaply whether any of it is true in her office, on her invoices. That is a pilot, and almost nobody runs one properly.
Every US dollar figure below converts at LKR 328.75, the Central Bank of Sri Lanka’s indicative USD/LKR spot rate for 10 September 2026.
Why most AI pilots never end
The evidence is blunt. MIT NANDA’s The GenAI Divide: State of AI in Business 2025, published in July 2025 and built on 52 structured interviews, 153 survey responses from senior leaders and a review of more than 300 public AI deployments, reports that 95% of organisations see no business return from generative AI, and that only 5% of custom enterprise AI tools reach production. A year earlier, Gartner predicted on 29 July 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, blaming poor data quality, inadequate risk controls, escalating costs and unclear business value. Gartner’s Rita Sallam put it plainly: executives are impatient for returns, “yet organizations are struggling to prove and realize value”.
Struggling to prove is the operative phrase, and it is a design fault, not a technology one. Three things go missing.
No baseline. Nobody wrote down what the work cost before the pilot started, so there is nothing to compare against and the argument runs on impressions. Impressions favour whoever is most enthusiastic.
No approver. The output is judged by the room. When quality belongs to everyone it belongs to nobody, and a disagreement about whether a draft was good enough has no way to settle.
No end date. A pilot without a written decision date does not fail. It dissolves into a tool three people use, a licence renewing quietly, and a board question nobody can answer with a number.
The freight forwarder’s proposal adds a fourth fault: six modules cannot be evaluated in thirty days by a team that also has a business to run.
Rule one: one workflow, not a platform
Pick a workflow that runs at least twenty times a week, has an output somebody checks, and belongs to one person. Invoice follow-up. Booking confirmations. The delivery-order paperwork one clerk retypes from PDFs.
The volume threshold matters. A workflow running twice a week gives you eight observations in a month, not enough to tell a trend from a mood. Twenty times a week gives you roughly eighty, which is.
Anthropic’s evaluation documentation argues the same from the model side: design evaluations that mirror your real-world task distribution, and “prioritize volume over quality”. Include the awkward inputs too, among them “ambiguous test cases where even humans would find it hard to reach an assessment consensus”. A pilot that only sees the tidy half of your work has tested nothing you will rely on.
One more reason to resist the platform version: a platform pilot tests a vendor’s product, while a workflow pilot tests whether your business has written down how that work is done, and that answer is yours to keep.
Rule two: a named approver and a day-one baseline
One person, with a job title, who already owns the output. Not a steering group. Their job during the pilot is what it was before: decide whether this goes out.
Then measure the baseline before anything is switched on. For one week, that person logs two counts: how many outputs they sent as drafted, and how many they changed. Record the counts, the week and who produced them. It takes ten seconds an item and it is the most valuable hour of the exercise, because without it every later number is unanchored.
The baseline also protects you against the commonest pilot illusion, improvement that was really just attention: work watched by a project team gets done better.
Rule three: one number, measured weekly
One number. Not a dashboard.
We use approval rate: the share of drafts the named reviewer approves untouched. Approved with an edit is not approved, and a comma counts. It is harsh on purpose, because any softer rule needs adjudicating, and adjudicated rules end up decided in the vendor’s favour. We set out how it is produced in what an AI agent’s approval rate means.
Your number does not have to be ours. It has to pass four tests: your staff produce it, not the vendor; it is countable in under a minute a day; it has a day-one value; and it counts work removed, not work produced. That last test kills the most flattering metric in every pilot report. An agent generating forty drafts a day that all get rewritten has added a review queue on top of the original job.
Read it weekly, and read the direction rather than the level. Week one is meant to be unflattering: the system knows the workflow but not your house style, your escalation habits or the three customers you handle differently for reasons nobody wrote down.
Rule four: a decision on day 30, written before day one
Before anything is installed, write one paragraph and have both sides sign it. The AI pilot success criteria go there, in this form: on 30 October, if approval rate on this workflow has risen from its day-one baseline of X to Y or better, we proceed to Z, and if not, we stop and pay nothing further.
Three properties make it work. It names a date, so the pilot ends with something other than a meeting. It names a threshold, so nobody argues afterwards about what counted. And it prices failure, the part vendors quietly leave out and the part you should insist on.
Be specific, too, about the difference between the pilot and the rollout, an argument we made in Four of the Five Steps Are Gradual. One Is a Cliff. Getting a workflow to draft well is gradual. Letting it act with no person in front of it is a cliff, and thirty days is not evidence for taking that step, only for deciding whether it is worth preparing for.
A pilot plan you can copy
This is the AI pilot project plan we run. Adjust the workflow, keep the shape.
| Week | What happens | Who does it | What is measured | The decision at the end of the week |
|---|---|---|---|---|
| Week 0 | Workflow chosen, its rule written on one page with three examples, decision paragraph signed | You and the vendor | Weekly volume | Proceed only if the rule fits on one page |
| Week 1 | The approver logs each output as sent untouched or changed, no AI involved | The approver, ten seconds an item | The day-one baseline | Both sides sign it. No baseline, no pilot |
| Week 2 | The agent drafts, the approver corrects with a reason, nothing goes out unreviewed | Approver, vendor on corrections | Approval rate | Continue, or stop if the tool cannot reach its data |
| Week 3 | Corrections sorted into house style, missing rule, or judgement | Approver and vendor, one 30-minute review | Approval rate, correction mix | Judgement-heavy categories leave the scope |
| Week 4 | The remaining categories run untouched, measuring the system not the tuning | The approver | Approval rate against the baseline | Day 30: apply the signed paragraph. Proceed, narrow, or stop |
Week three is the step people drop and the one that pays. The mix of corrections shows where the boundary between automatable and not sits in your business, decided by your own reviewer rather than a demo. That map is worth having even if you hire nobody.
What a pilot should cost, and what a vendor who refuses one is telling you
specshop.dev is an AI consultancy and AI agency in Colombo, Sri Lanka, led by Janaka Ediriweera, Principal AI and Product Management Consultant. Here is what we charge, as a reference point for reading someone else’s quote.
A workflow mapping sprint is US$2,300, about LKR 756,125: two video sessions, your three busiest workflows, staff hours against each, a report inside 48 hours. It is credited in full against the recruitment fee if you hire within 30 days. If you go on to hire an AI agent, salaries start at US$1,225 a month, about LKR 402,700, and the senior delivery engineer is US$3,000 a month, about LKR 986,250. The recruitment fee is US$9,500, about LKR 3,123,125, one-off, half to begin, covering deployment, connection to your inbox, WhatsApp and tools, the first month’s salary and the full 30-day probation with weekly scorecards. From day one we measure the share of the agent’s drafts your reviewers approve unedited, and if it has not improved from its day-one baseline by day 30, you do not confirm the hire and the balance of the recruitment fee is never invoiced.
That is a pilot with the decision written before day one and failure priced in advance. The wider arithmetic sits in what AI automation costs in Sri Lanka.
Now hold any proposal against it. Gartner’s 2024 analysis put the cost of transforming a business model with generative AI at US$5 million to US$20 million, so the gap between a pilot and a programme is no rounding error. A vendor who will not scope a single-workflow pilot, or who wants the licence signed before the measurement starts, is telling you the value is not measurable at small scale. That may be honest. It is still your risk.
What to do this week
Choose the workflow. Spend an hour writing its rule on one page with three worked examples, and note every question you cannot answer, because that list is why the last pilot did not end.
Then start counting, before you talk to anybody. One approver, two tallies, five working days. You will own a day-one baseline whether or not you sign anything, and it is the only number that makes the next conversation short. The first thirty minutes with an AI consultant in Colombo costs nothing and includes an honest answer, sometimes “not yet”.
Questions people actually ask.
How long should an AI pilot last?
Thirty days is enough for a workflow that runs at least twenty times a week, because that gives you roughly eighty observations and four weekly readings, which is sufficient to tell a direction from a fluctuation. Anything longer usually means the scope was too broad to decide on, and anything shorter measures the novelty of being watched rather than the work.
What should an AI pilot measure?
One number that your own staff produce, that can be counted in under a minute a day, that has a value recorded before the pilot started, and that describes work removed rather than work produced. We use approval rate, the share of drafts a named reviewer approves untouched, because volume-based metrics flatter a system that generates forty drafts a day which all have to be rewritten.
How much should an AI pilot cost?
A single-workflow pilot should cost a low four-figure sum in US dollars or nothing at all, and it should never require a platform licence signed before the measurement begins. Our workflow mapping sprint is US$2,300, about LKR 756,125, credited in full against the recruitment fee if you hire within 30 days, and the 30-day probation that follows a hire is covered by the US$9,500 recruitment fee, about LKR 3,123,125, of which the balance is never invoiced if the approval rate has not improved.
Who should be involved in an AI pilot?
Three people, at most: the named approver who already owns the output and judges every draft, one sponsor who can sign the decision paragraph and release the data, and one person on the vendor side who fixes things. A steering group is a warning sign, because quality that belongs to a committee cannot be measured and disagreements about it have no way to settle.
What happens after an AI pilot succeeds?
You narrow before you widen: keep the categories where the number moved, formally exclude the judgement-heavy ones, and only then add the second workflow. Resist removing the human review gate at the same time as you expand scope, because expanding what is drafted is gradual while removing the approver is a step change in risk, and the two should never be taken in the same month.
Sources
- The GenAI Divide: State of AI in Business 2025, MIT NANDA (Aditya Challapally, Chris Pease, Ramesh Raskar, Pradyumna Chari), July 2025 — the finding that 95% of organisations are seeing no business return from generative AI and that only 5% of custom enterprise AI tools reach production, and the methodology of 52 structured interviews, 153 survey responses from senior leaders and a review of more than 300 publicly disclosed AI deployments conducted between January and June 2025. Fetched 11 September 2026.
- Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025, Gartner press release, 29 July 2024 — “At least 30% of generative AI (GenAI) projects will be abandoned after proof of concept by the end of 2025, due to poor data quality, inadequate risk controls, escalating costs or unclear business value”; Rita Sallam’s remark that executives are impatient for returns “yet organizations are struggling to prove and realize value”; and deployment costs for transforming business models with generative AI of US$5 million to US$20 million. Fetched 11 September 2026.
- Create strong empirical evaluations, Claude platform documentation, Anthropic, 2026 — the eval design principles “Be task-specific: Design evals that mirror your real-world task distribution”, “Prioritize volume over quality”, and the listed edge cases including irrelevant or nonexistent input data, overly long inputs and “ambiguous test cases where even humans would find it hard to reach an assessment consensus”. Fetched 11 September 2026.
- Daily Indicative USD/LKR Spot Exchange Rates, Central Bank of Sri Lanka, 2026 — the indicative rate of LKR 328.75 per US$1 on 10 September 2026, used for every conversion in this post.