You have three proposals on your desk, and each one promises "an intelligent AI agent working around the clock". The slides use the same vocabulary, the architecture diagrams look alike, and only the totals differ. The real differences show up later: the first time the agent gives a customer a wrong answer, the first time your price list changes, or the day you decide to move to another provider. This article gives you ten questions to put to any AI automation vendor before you sign, with notes on what a considered answer sounds like and what should make you pause.
The questions work for any vendor, including us. They are most useful when you ask them in the first meeting and ask for the answers in writing, so you can compare proposals side by side.
Why a polished demo tells you very little
AI automation has an awkward property: demos almost always look good. The model answers rehearsed questions fluently, the bot greets you politely, the integration diagram is tidy. The problems appear on real traffic. Customers type with typos, switch between English and Arabic or Russian mid-sentence, ask about things that are not in the knowledge base, send voice notes or a photo of a receipt. So the thing to evaluate is not how the system behaves at its best, but how the vendor thinks about the worst case.
The ten questions below follow that logic: three about the task and the outcome, three about data and errors, two about ownership and support, and two about money and people.
Questions about the task and the outcome
1. Which process would you automate first, and why that one?
A good vendor asks before they propose. They want to know how many enquiries you receive, where they get lost, who answers them today and how long it takes. If a solution arrives before anyone has looked at your numbers, you are being sold a ready-made product, not an automation of your process. A strong first candidate is a repetitive process with a clear metric and visible losses, such as replying to inbound enquiries outside office hours.
2. Can we see a system that is running in production, not a demo?
Ask to see something live: a working bot, an anonymised conversation log, a write-up of a real project. A screen recording of a scripted scenario is weak evidence. If the vendor publishes post-mortems, including ones about their own failures, treat that as a good sign. It means they review their mistakes rather than hide them.
3. How will the pilot work, and how will we decide whether to continue or stop?
A pilot without criteria turns into endless tweaking. Before the start, agree on the scope (which part of the process), the duration, the metric (time to first reply, share of enquiries reaching a salesperson, number of escalations) and the value that counts as success. Agree separately on the stop condition: what would have to happen for you to close the project honestly. We describe what the first weeks of a pilot usually look like in our guide to AI pilot success criteria, and how to calculate the effect in how to measure AI ROI.
Questions about data, errors and control
4. Where will customer data be stored, and who will have access to it?
The answer should be specific: which services are involved, which country the servers are in, whether conversations are sent to a third-party language model provider, and who on the vendor's team can see the database. The rules depend on where you and your customers are. In the UK that means UK GDPR; the UAE and Saudi Arabia each have their own personal data protection laws; Uzbekistan requires personal data of its citizens to be stored on servers inside the country. How these apply to your setup is a question for your lawyer, but a vendor should at least know which regime applies to you and have an answer ready.
Ask about access too. Are you expected to hand over your CRM login, or will the vendor get a separate key with limited permissions that you can revoke in one click?
5. What will the agent never do without human approval?
The right answer is a list. For example: it does not offer discounts, does not confirm refunds, does not send bulk messages, does not move deals past a certain stage. Just as important is how that limit is enforced. "We don't configure it that way" and "the system technically cannot do it" are different things. We learned this ourselves: an outreach automation bypassed its own approval gate and sent 42 messages without sign-off. The write-up and how we made the gate mandatory in code are in the approval gate case.
6. How will we know when the agent gets something wrong?
Language models sometimes answer confidently and incorrectly. That is not a reason to avoid them, but it is a reason to ask: is every conversation logged, who reviews the log and how often, what happens when the agent does not know the answer (does it hand over to a person or improvise), and does anyone get an alert if the CRM integration stops responding? "Our model doesn't make mistakes, we've tuned it well" is a red flag.
Questions about ownership and support
7. Who owns the code, the prompts, the knowledge base and the accounts?
Imagine that a year from now you decide to switch vendors. What stays with you? The healthy setup is that service accounts (the WhatsApp or Telegram bot, API keys, hosting) are registered to your company, the knowledge base can be exported in a usable format, and code and documentation are handed over on request. If everything lives in the vendor's account and "cannot be transferred", you are not buying a system but a subscription to one supplier. That can be acceptable, but it should be a conscious choice.
8. What does support after launch actually include?
An agent ages along with your business: prices, product range, opening hours and delivery terms all change. Find out who updates the knowledge base, whether your team can correct an answer without a developer, how quickly a fix is made if the agent starts answering wrongly, and what counts as urgent. Also ask what happens when the model provider releases a new version. The agent's behaviour can shift, and someone has to check it.
Questions about money and people
9. What makes up the cost, now and six months from now?
Beyond the build, an AI system has running costs: language model usage billed by the volume of text processed, hosting, and paid services such as speech recognition. Ask for these to be listed separately, with an explanation of how they grow as conversation volume grows. A good vendor will show you a cost model with assumptions you can check yourself. For a breakdown of typical budget lines, see our AI automation cost breakdown.
10. Who exactly will work on the project, and what do you need from us?
Find out who your main contact is, who writes the code and who is responsible for scripts and texts. Most importantly, ask how much of your team's time the project will take. Automation without the client's involvement does not work: someone has to explain how sales actually run, collect answers to common questions and review the first conversations. If a vendor says nothing is needed from you, that work has most likely just not been accounted for.
Cheat sheet: considered answer vs red flag
| Question | Considered answer | Red flag |
|---|---|---|
| What to automate first | Asks for your enquiry and process numbers first | Proposes a solution before discussing the process |
| Live system | Shows a production project or a public write-up | Only a video of a scripted scenario |
| Pilot | Metric, duration and stop condition in writing | "Let's launch and see" |
| Data | Names services, server locations, access rights | "It's all secure, don't worry" |
| Human approval | A list of actions and how the limit is enforced technically | "The agent will figure it out" |
| Errors | Conversation log, escalation, failure alerts | "The model doesn't make mistakes" |
| Ownership | Accounts in your name, exportable knowledge base | Everything in the vendor's account, no transfer |
| Support | Who, how fast, and what counts as urgent | "Just get in touch if anything comes up" |
| Cost | Build and running costs shown separately | One total with no breakdown |
| Team | Named roles and an estimate of your time | "You don't need to do anything" |
What to put in the contract
Verbal answers belong in the agreement. A minimum checklist:
- the process and the pilot scope: what the agent does and what it does not do;
- the success metric, pilot duration and stop condition;
- the list of actions that require human approval;
- where data is stored, who has access and how access is revoked;
- whose name the accounts and keys are registered in, and what is handed over on termination;
- the scope of support and how urgent issues are handled;
- running costs: who pays them and how they are reported.
This list does not replace legal advice, but it covers the points people usually remember only after the first incident. The typical problems that arise when these agreements are missing are collected in 7 mistakes when implementing AI. More on launching projects is in the Implementation section.
FAQ
Should we go with the cheapest vendor?
Price only means something alongside the answers above. A low figure may simply leave out support, conversation logging or running costs. Compare proposals on an identical scope of work.
Does the vendor need access to our whole CRM?
Usually not. The agent needs specific actions: create a lead, read a status, add a note. Ask the vendor to describe the minimum permissions and issue a separate key that you can revoke.
Can we skip the pilot and roll out to the whole process at once?
You can, but the risk is higher: any errors will hit your entire customer flow at once. A pilot on one channel or one type of enquiry lets you check answer quality and fix the knowledge base before scaling.
How can we tell whether a vendor understands AI or just plugs in someone else's builder?
Ask what they would do if the model started answering worse after a provider update, and how they test answers before launch. A concrete reply with examples of their checks tells you more than a list of technologies on a slide.
If you would like to walk through these questions using your own process, get in touch. The cost depends on scope and data; the first conversation is there to work it out.