A lot of AI pilots end with the same verdict: "it seems to work". The agent answers messages, the team has got used to it, yet nobody can say whether anything actually improved or whether a full rollout is worth paying for. That happens when a pilot is treated as a smaller version of the real thing rather than as an experiment with a question it must answer. This article walks through the first two to three weeks step by step: which numbers to capture before you start, what to agree in writing up front, and how to reach one of three decisions at the end — scale, fix, or stop.
If you are after a day-by-day plan for building the agent itself, that lives in How to Implement AI in 2 Weeks Without a Technical Team. This piece is about something else: turning those weeks into a test whose result tells you clearly what to do next.
A pilot tests a hypothesis — it is not a demo
A demo shows what a system can do. A pilot answers whether it works on your data, with your customers and your staff. The difference is that a pilot has a hypothesis that can turn out to be wrong. Some well-formed examples:
- "The agent handles routine questions about delivery and payment without a manager stepping in, and complaints about answers do not rise."
- "The assistant extracts data from incoming invoices well enough that the accountant corrects no more than one field in ten."
- "Enquiries that arrive overnight get a first reply within five minutes, and by morning the sales team sees them already qualified."
A weak hypothesis sounds like "let's try AI in customer service" or "let's see what happens". It can be neither confirmed nor disproved, so the final decision ends up being made on gut feel.
A good hypothesis covers one process. If support, sales and document handling all go into the same pilot, three weeks later you will have three half-finished results and no answer.
Week 0: the work before launch
The most underrated part of a pilot happens before the agent sends its first message. Skip it and you will have nothing to compare against at the end.
Take a baseline
Look at the last two to four weeks of the process and count, by hand or from a CRM export: how many enquiries came in, how long replies took, what share were routine, how many went unanswered. The numbers will be imperfect, and that is fine. What matters is that they are counted the same way they will be counted during the pilot. For what belongs on the cost and benefit sides, see How to Measure AI ROI.
Write the success criteria down first
The thresholds that define a successful pilot are written before launch and agreed by both sides — an email thread is enough. Agree them afterwards and both parties will, without meaning to, bend them towards whatever happened.
Name an owner on your side
This is someone who knows the process, looks at the agent's output every day and has the authority to say "that's wrong". Without them, mistakes pile up for a week and then everyone discovers them at once.
Set the boundaries
Decide what the agent does on its own, what it only suggests, and what it must never do: quote delivery dates, promise discounts, respond to formal complaints. It is easier to start narrow and widen the scope later than the other way round.
Week 1: shadow mode
In the first week the agent works on real incoming messages, but nothing leaves without a human. It prepares a draft reply or a summary card; a member of staff reviews it, edits it and sends it themselves. You get two things at once: customers never see the mistakes, and you build honest statistics — what share of drafts went out untouched, what share needed small edits, and what share had to be rewritten.
Log every correction. A four-column sheet is enough: what came in, what the agent proposed, what was actually sent, and the type of error. After a week the log shows where the problem sits — in the knowledge base (the agent did not know the answer), in the instructions (it knew, but phrased it badly), or in the task itself (the question genuinely needs a human).
Also check that the "human in the loop" is enforced in code, not just agreed in a meeting. We learned this ourselves: in our own outreach automation the sender bypassed the approval gate and 42 messages went out unapproved — the incident review is public. The takeaway for a pilot is simple: if the rules say the agent only suggests, make sure it is technically unable to send on its own.
Week 2: limited live launch
If, after week one, most drafts go out unchanged or with minor edits, the agent can be given some independence — but not across the whole flow. Ways to narrow it down:
- by request type: only delivery and opening-hours questions, everything else goes to a person;
- by time: the agent replies on its own overnight and at weekends, and keeps drafting during working hours;
- by channel: WhatsApp or website chat only, while phone and email stay as they are.
The log continues in week two, but the question shifts. Now you look beyond answer quality to customer behaviour: did the conversation lead to an order or booking, did the customer ask for a real person, did they come back with the same question? The process owner keeps reading a sample of conversations — twenty or thirty a day is a reasonable minimum.
Week 3: measure and decide
A third week is not always needed. If volume is low and two weeks produced few enquiries, take it, so conclusions are not drawn from a few dozen cases. If volume is high and the picture is already clear, use the third week for fixes and a repeat measurement of the same metrics.
At the end, compare pilot-period figures against the baseline, counted by the same rules. The output is a short document of one or two pages: the hypothesis, the criteria, the results, the log of typical errors and a recommendation.
What counts as a result: metrics and thresholds
Below is a template worth filling in during week 0. The thresholds are illustrative — set your own based on the baseline and on what matters most to you.
| What we test | How to measure it | Example threshold (illustrative) |
|---|---|---|
| Answer quality | Share of drafts sent without edits or with minor edits, from the log | Most routine enquiries pass without significant edits |
| Dangerous errors | Cases where the agent promised something untrue: a price, a date, stock | Zero during the period; otherwise, fix first |
| Speed | Time to first reply compared with the baseline | Noticeably faster, especially out of hours |
| Team workload | Time staff spend reviewing and correcting | Less than writing the replies by hand used to take |
| Customer reaction | Share of conversations where the customer asked for a person or complained | No higher than before the pilot |
| Business effect | Share of enquiries that reached an order or payment | No lower than before the pilot |
Note the last row. In three weeks the business effect often does not show up in revenue — the sample is too small and sales cycles too long. So for a pilot the sensible bar is "no worse than before", not "sales went up". Growth is tested after scaling, over months.
Three outcomes and how to choose
| Outcome | When | What next |
|---|---|---|
| Scale | Key thresholds met, no dangerous errors, the team is ready to work alongside the agent | Widen the flow or add channels; keep spot checks and a monthly report |
| Fix | Quality is close to the threshold and errors are systematic and explainable: gaps in the knowledge base, unclear instructions | Address the causes from the log and re-measure for one or two weeks |
| Stop | Errors are random and do not trace back to clear causes, or almost every case needs a human decision | Record the findings; the documented process and the log are useful for a later attempt or a different process |
Stopping is a normal pilot outcome, not a failure. Finding that out in three weeks rather than after a full rollout is the whole point. An honest supplier mentions this option up front; if nobody raises it, ask.
Checklist before the final review meeting
- Baseline and pilot figures were counted by the same rules.
- The error log is sorted by type: knowledge base, instructions, task.
- Every dangerous error has been reviewed on its own: why it happened and what stops it recurring.
- The process owner has given their view — they have read more conversations than anyone.
- It is clear what changes at scale: volume, channels, who does the checking.
FAQ
Can a pilot be done in one week?
Building a prototype — sometimes. Getting conclusions — rarely. Shadow mode and independent operation give different pictures, and one week yields too few real enquiries. A one-week pilot usually ends with that same "it seems to work".
What if very few enquiries came in during the pilot?
Extend the pilot, or run historical enquiries from your CRM and messaging channels through the agent and compare its answers with what your staff actually sent. That does not replace live traffic, but it adds to the statistics.
Should customers be told they are talking to AI?
In shadow mode the customer is talking to a member of staff, so the question does not arise. Once the agent replies on its own, it is more honest to say so and give an easy way to reach a person — that causes far less frustration when the agent does not know something. Check the disclosure rules that apply in your market as well.
How much does a pilot cost?
It depends on the process, the volume and the state of your data. The first conversation exists precisely to understand the task and work out the numbers. Get in touch if you would like to discuss a pilot on one of your processes, or browse more articles in the Implementation section.