By KYN AI Advisory Team — AI implementation specialists, Singapore
Measure an AI reply desk with five numbers: label accuracy, costly misses, edit rate, response time and cost per reply. Take them from a fixed test set before launch and from a weekly sample after. Set pass marks in advance. A desk that is fast and cheap but misses unsubscribes is failing, whatever the averages say.
Why measure at all?
Because a reply desk fails quietly. It writes fluent text whether it is right or wrong. Without measures, the first sign of a problem is a customer complaint. With them, you see it in the numbers a week earlier.
Measures also decide how much freedom the desk gets. A label you have tested at high accuracy can run unattended. A label you have not tested stays under review. The numbers set the boundary. For how the desk works end to end, see how an AI reply desk classifies and drafts replies.
What are the five numbers?
| Measure | Definition | Why it matters | |---|---|---| | Label accuracy | Replies where desk label equals the human label, divided by replies tested | Routing depends on it | | Costly misses | Count of unsubscribe, complaint or interested replies mislabelled | Small number, large harm | | Edit rate | Drafts changed by a reviewer before sending, divided by drafts | Shows draft quality and review load | | Response time | Time from reply received to answer sent, split into desk time and human time | Shows where delay sits | | Cost per reply | Model and infrastructure cost for one reply, plus review minutes | Shows if the desk pays |
Report accuracy per label as well as overall. A high overall figure can hide a label that is wrong half the time.
How do you build the test set?
- Pull 100 real replies from the last quarter. Include every complaint and unsubscribe you can find, so rare labels are tested.
- Have two people label each reply on their own.
- Where they disagree, discuss and agree one label. Note the disagreements, which show weak definitions.
- Freeze the set. Do not tune the prompt against it more than you must, or the score stops meaning anything.
- Keep a second, unseen set of 50 for a final check.
Add new hard cases monthly: a reply in mixed languages, a long thread, a message that tries to instruct the agent. The set grows into a regression test.
How do you measure edit rate?
Log every draft and every final message. Compare them.
- Unchanged: sent as drafted.
- Light edit: a word or phrase changed.
- Rewrite: most of the text replaced.
- Rejected: not sent.
Track the shares weekly. Then sort the edits into causes: tone, length, wrong fact, wrong structure, missed escalation. The cause tells you what to fix. A pile of "wrong fact" edits points to the source material. A pile of "tone" edits points to the voice sheet.
Set a gate before widening autonomy. For example: a label may run unattended only when at least nine in ten drafts go out unchanged over four weeks and no costly miss occurred. Choose your own numbers. They are a business decision, not a standard.
How do you measure response time?
Split it. The desk controls one part, your people the other.
- Desk time: received to draft ready.
- Review wait: draft ready to approved.
- Total time: received to sent.
Report the median and the slowest tenth, in working hours and outside them. The median hides the replies that waited all weekend. If review wait is the largest part, the fix is staffing or notification, not a faster model.
How do you measure cost per reply?
Add three items.
- Model calls: classification, drafting, and any checks.
- Infrastructure: hosting, storage, logging, shared across replies.
- Review time: minutes a person spends per reply, multiplied by their hourly cost.
In KYN's live systems for an insurance brokerage, sorting one reply costs about S$0.01 and a group benefits comparison document about S$0.03, depending on the model and the text. Those are model costs only. Review time usually outweighs them, which is why edit rate belongs next to cost. Compare against what the same work costs by hand. See AI sales agent cost per month.
How often should you review?
- Daily: a count of escalations and any costly miss.
- Weekly: a sample of 50 replies compared with a person's label, plus edit rate and response time.
- Monthly: accuracy by label and language, cost per reply, and changes to the test set.
- After any change to the model, prompt or label list: re-run the frozen set before release.
Write results in one table with the date. Trends matter more than a single reading. If accuracy falls two weeks running, find out what changed in your inbox: a new product, a campaign, a new language.
What does a worked example look like?
Take an illustrative desk that handled 200 replies in a week. A person labelled a sample of 50.
| Measure | Result | Pass mark | Verdict | |---|---|---|---| | Label accuracy | 44 of 50 match | 45 of 50 | Fail: 44 | | Costly misses | 1 unsubscribe labelled as question | 0 | Fail | | Edit rate | 31 of 60 drafts unchanged | 42 of 60 unchanged | Fail: 31 | | Median desk time | under 1 minute | Under 2 minutes | Pass | | Median review wait | 5 hours | Under 4 hours | Fail |
All figures are illustrative, and the pass marks are examples. The reading is clear. Speed is fine. Accuracy is close, but the unsubscribe miss is serious, so the team adds a code rule for stop words and re-runs the frozen set. The edit rate points to source material, so the team reviews the 29 edited drafts and groups them. The review wait points to staffing, so the team adds a notification and a second reviewer.
What should the weekly report show?
Keep it to one screen. The reader should see the decision first.
- The five numbers, this week against last week.
- Every costly miss, with a link to the reply and the rule that failed.
- The three most common edit causes.
- Labels eligible to move from review to unattended, and the evidence.
- Changes made this week and their effect.
If a number is missing because the system cannot produce it, say so. Do not fill the gap with an estimate. A measure that is guessed once is quoted forever.
Where KYN fits
KYN builds the measures into the reply desk. Every label, draft, edit and send is logged with the actor, so the five numbers come from the system, not from a spreadsheet kept by hand. The desk runs in your own cloud account under a restricted account, and outbound messages wait for a named person until the measures justify more. See sales systems and security.
In the 14-day review we pull a sample of your past replies, label them with your team and set the baseline for the five numbers. The full build ships in under 90 days. Pricing is a one-off build fee plus a monthly retainer of at least twelve months, quoted after that review.
Frequently asked questions
What accuracy should a reply desk reach?
There is no standard figure. Set it per label by the cost of an error. Unsubscribe and complaint labels need near-complete recall, with a code rule behind the model. A "question" label can tolerate more errors because a reviewer sees it anyway.
How many replies do I need to test?
Start with 100 real replies, enough to see rare labels if you select for them. Add to the set each month. For a label that is rare in your inbox, collect more examples of it deliberately.
Can I measure without labelling replies by hand?
Not for accuracy. Someone must label a sample, because the desk cannot mark its own homework. You can measure edit rate and response time from logs alone.
Checked 30 October 2026.
Curious whether Marketing agents fits your business? Talk to KYN on WhatsApp — no forms, just a conversation.
