By KYN AI Advisory Team — AI implementation specialists, Singapore
To evaluate an AI agent vendor, ask fifteen questions in five groups: what data the agent sees, what it may do alone, what evidence the vendor shows, how it is secured, and what you keep if you leave. A good vendor answers each in specifics and lets you test it on your own data. A weak vendor answers with feature names. Intercom publishes its own guidance on evaluating AI agents, and the questions below extend that thinking to any vendor, including KYN.
Group 1: What data does the agent see?
1. Which of our systems does it read, and which does it write to? A good answer lists each system and the permission level. A weak answer says "it integrates with everything." Test: ask for the permission scopes it requests, and read them.
2. Where is our data stored, and in whose account? A good answer names the cloud provider, region and account owner. Ask whether the data sits in an account in your name. A weak answer is "in our secure cloud."
3. Is our data used to train models? Get the answer in the contract, not in a sales call. Ask for the model provider's data-use terms by name.
Group 2: What may it do alone?
4. Which actions run without a person, and which wait for approval? A good answer is a table. Outbound messages to customers, anything about money, filings and deletions should wait for a named person. See approval gates for high-risk agent actions.
5. Can we change those limits ourselves? A rule you cannot see or change is a risk you cannot manage. Ask to see the setting in the product.
6. What happens when it is unsure? A good agent stops and hands over with the context. A weak one guesses. Test it with ten questions it should not answer and count how many it declines.
Group 3: What evidence do they show?
7. Can we test it on our own cases before signing? Give the vendor 100 real examples with the right outcome marked by your team. Score it. A vendor that will only demo on its own data is selling a demo.
8. How do you measure errors, and what is your current rate? Ask for the method, the sample size and the date. A single percentage with no method is marketing. Ask which error matters most for your case, such as a missed handover, and how it is measured.
9. Which customers run this in production, and for how long? A good answer names industries and durations, and offers a reference call. Beware of "in pilot with Fortune 500 clients" with no detail.
Group 4: How is it secured?
10. How do people and agents sign in? Look for named accounts through a proper identity service, no shared logins, and separate restricted accounts for agents. Our security note shows the pattern we use.
11. How is one customer's data kept from another's? For a multi-tenant product, ask how isolation is enforced. Row-level security in the database is a strong answer. See multi-tenant database security: what to ask your web developer.
12. How does it resist hostile input? A customer message or web page can contain instructions aimed at the agent. Ask how the vendor tests for it. See prompt injection in business AI agents and the AI agent security checklist.
Group 5: What do we keep if we leave?
13. Who owns the code, configuration and prompts? Rights should pass to you on payment, in writing. For a subscription product, ask what you can export. Our guide who owns the data and prompts in a custom agent project goes through this.
14. If we end the contract, does the system keep running? For a hosted product, no. For a system built in your own cloud account, it can. Ask for the exit plan in writing.
15. What does it cost over three years, not three months? Ask for the build or setup fee, the monthly cost, the usage cost, the review time and the cost of changes. Compare totals, not the headline price.
How do you score the answers?
Score each question 0, 1 or 2.
- 0: no answer, or an answer that avoids the question.
- 1: an answer in general terms, with no document or test behind it.
- 2: an answer in specifics, backed by a document, a setting you can see or a test you ran.
A total under 18 out of 30 means you do not yet know enough to buy. Between 18 and 24, run a paid pilot with a small scope. Above 24, you are choosing between informed options. The thresholds are our judgement, not an industry standard. Adjust them to your risk.
Which questions matter most for an SME?
Four carry the most weight for a small company without a security team: 4 (what waits for approval), 7 (test on your cases), 13 (ownership) and 15 (three-year cost). Get those right and the others are fixable. Get one wrong and the rest do not rescue it. Our note on the hidden cost of buying AI tools your business doesn't need shows how the last one goes wrong.
What does a test on your own data look like?
Collect 100 real cases from the last quarter. Remove personal details you do not need to share. For each case, have your most experienced person write the correct outcome. Give the vendor the inputs only. Score the outputs as correct, acceptable, wrong or unsafe. Unsafe means the agent did something that should have waited for a person, or invented a fact. Report each category separately. Ask the vendor to explain every unsafe result and what would change. Their response shows how they handle errors after launch, which matters more than the score on the day.
Where KYN fits
You can ask KYN the same fifteen questions. We build custom systems in a cloud account in your name. Rights in the custom code and configuration pass to you on full payment, and you can revoke KYN's role while everything keeps running. Agents have their own restricted accounts. Every action is logged with the actor. Outbound messages, money, filings and deletions wait for a named person.
The first 14 days are a review of your stack and interviews with your team. A full build ships in under 90 days. Pricing is a one-off build fee and a monthly retainer of at least twelve months, quoted after the review. See case studies and pricing.
Frequently asked questions
How many vendors should an SME compare?
Three. One packaged product, one adjacent option such as a platform feature in a tool you already use, and one custom build. More than three costs more time than it saves.
Should we run a pilot before signing?
Yes, if the vendor allows it. Keep the scope to one task, one team and four weeks, and score it against 100 real cases you marked in advance.
What is the biggest red flag?
A vendor that will not let you test on your own data, or that cannot say who owns the output when the contract ends. Either answer tells you how the relationship will go.
Checked 30 October 2026.
Curious whether Marketing agents fits your business? Talk to KYN on WhatsApp — no forms, just a conversation.
