By KYN AI Advisory Team — AI implementation specialists, Singapore
Evaluate AI output the way you would audit a new hire: against a labelled set of real cases, with the error rate counted by type, then with a standing spot check once it is live. Build the test set from 100 or more of your own past cases, label them by hand, run the agent on them, and count the misses that cost money separately from the misses that do not. A single accuracy percentage hides the errors that matter.
Why does an overall accuracy number mislead?
Take an agent that sorts incoming replies into six labels. It gets 94 of 100 test cases right. That sounds fine. Now ask which 6 failed. If they are "out of office" labelled as "not now", nobody is harmed. If they are "unsubscribe" labelled as "interested", you have emailed someone who asked you to stop. The same number covers both.
So the first rule is to separate errors by cost. Before testing, write a short list: which wrong answers are annoying, which are expensive, which are unlawful or irreversible. Count each class on its own. Set a different tolerance for each. For the irreversible class the tolerance is zero misses in the test set, plus a control that does not depend on the model at all.
How do you build a test set?
A test set is a file of real inputs with the correct answer written beside each one, by a person who knows the work.
- Pull real cases. Take 100 to 200 from the last few months: emails, invoices, bank lines, enquiries. Real inputs are messier than invented ones.
- Cover the tail on purpose. Random samples hold few of the rare cases. Add examples of each rare class you care about: a complaint, a legal threat, a foreign-currency receipt, a forwarded thread with three signatures.
- Label by hand. One person labels. A second person labels a sample. Where they disagree, the rule is unclear, and you fix the rule before you test the model.
- Freeze it. Keep a version. Never tune the prompt on the same cases you score on, or the score measures memory, not skill. Hold back a portion that nobody touches.
- Refresh it. Add new failure cases from live use. A test set that stops growing stops matching reality.
A set of 100 gives rough error rates. Treat a one-case difference as noise at that size. Read the pattern of errors, not the decimals.
What do you measure?
Pick measures that match the task type.
| Task | Primary measure | Watch separately | |---|---|---| | Classification (reply labels, ticket types) | Error rate per label | Costly confusions, such as unsubscribe read as interest | | Extraction (invoice fields, receipt totals) | Field-level exact match | Amounts and dates, which carry money | | Matching (bank lines to invoices) | Share of correct top suggestion | Wrong high-confidence matches | | Drafting (emails, summaries) | Reviewer edit rate | Factual errors, invented details | | Retrieval and Q&A | Share of answers supported by a cited source | Answers with no source |
For drafting, we count how often a reviewer changes the draft and what they change. A draft that needs a fix to a fact is a worse failure than one that needs a softer tone.
Two further numbers are cheap and useful:
- Abstention rate. How often the agent says it does not know and hands over. An agent that never abstains is guessing.
- Confidence calibration. Of the answers labelled high confidence, what share were right? If high confidence is nearly always right, you can route those to batch review. If it is wrong one time in five, you cannot.
How do you test for the failures that tests miss?
A labelled set tests the cases you thought of. Add adversarial cases for the ones you did not.
- Hostile input. Replies and web pages containing instructions aimed at the agent. See prompt injection in business AI agents.
- Stale data. A record that is eight months out of date. Does the agent say so, or write confidently?
- Missing fields. An invoice with no date. Does it guess?
- Long and odd inputs. A 40-message thread; a scanned page; text in two languages.
Run these as a separate suite, and require that the agent hands over or flags, not that it answers.
What does a live spot check look like?
A test set describes the past. Live use drifts: customers change how they write, vendors change invoice layouts, a model update shifts behaviour. So the check continues after launch.
- Weekly sample. A reviewer takes a fixed number of outputs, such as 25, drawn at random plus a few of every flagged class. They mark right or wrong. This takes minutes.
- Error log with a type. Each wrong output is tagged with its type. Patterns show in a month.
- Trigger for action. Decide in advance what error rate or what single severe miss pauses the agent and returns work to a person.
- Every action logged. If an output cannot be traced to an input, a model version and an actor, it cannot be audited.
We also use the review queue as data. When a reviewer edits or rejects an agent's output, that edit is a labelled example. It goes into the test set the same week.
What does it cost to evaluate?
Mostly person time. The model cost is small. At KYN's live prices, sorting a reply costs about S$0.01 and a group benefits comparison about S$0.03, so running a 200-case test set through a task like that costs a few dollars. Labelling the set is the real cost: expect a day or two of a knowledgeable person's time for the first version, then minutes a week.
That is a small price against the alternative, which is finding the error rate from a customer.
What are the trade-offs?
- Test sets age. Without refresh, the score holds while real quality falls.
- Small sets are noisy. Hundreds of cases give direction, not precision. Rare costly failures need targeted cases, not volume.
- Human labels disagree. That is a signal about the rule, so settle it before blaming the model.
Where KYN fits
KYN builds agents with their own restricted accounts, every action logged with the actor, and outbound messages, money, filings and deletions held for a named person. We evaluate each agent on a labelled set from your own cases during the build, and keep a weekly spot check in the monthly retainer. See operations and security. The first 14 days are a review of your stack and interviews with your team; a full build ships in under 90 days.
Frequently asked questions
How many test cases do I need?
Start with 100 to 200 real cases, with extra examples of each rare but costly class. It gives a rough error rate by type, which is enough to decide whether to launch with review.
What error rate is acceptable?
It depends on the cost of the error. For low-cost mistakes, an occasional miss with review is workable. For irreversible actions such as sending a message or posting money, the control should not rely on the model being right.
How often should I check an agent after launch?
Weekly at first. A random sample of about 25 outputs, plus any flagged cases, takes minutes and catches drift early. Move to fortnightly only after several clean weeks.
Checked 30 October 2026.
Curious whether Marketing agents fits your business? Talk to KYN on WhatsApp — no forms, just a conversation.
