You’ve probably watched the pilot that doesn’t end. A vendor offers a trial, a handful of reps get licenses, the team sits through a few check-ins, and when the renewal notice lands, the honest answer to “did it work?” is a shrug. The tool renews because canceling feels like admitting the experiment went nowhere, or it quietly dies because the champion moved on. Neither outcome came from evidence.
Why do so many AI pilots end without an answer?
They start without a definition of success. In IBM’s 2025 CEO Study, released May 6, 2025, surveyed CEOs reported that only 25% of AI initiatives had delivered expected ROI over the last few years, and only 16% had scaled enterprise wide (IBM Institute for Business Value, 2025). Those are shares of initiatives, not of companies, and they are CEOs grading their own programs.
| Item | Value |
|---|---|
| Delivered expected ROI | 25% |
| Scaled enterprise wide | 16% |
Gartner expects agentic AI to follow a similar path. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls (Gartner, June 25, 2025). That’s a prediction about projects still running, not a count of projects already shut down.
Of Gartner’s three reasons, cost gets settled in a vendor negotiation and risk controls in a security review. Unclear business value is the one your pilot design controls. A baseline, a control group and a written stopping rule exist to answer it. A pilot without them can run for months and still end with an opinion.
If you haven’t checked whether your data and team are ready for any of this, start with an AI readiness assessment before you sign a pilot agreement.
Step one: pull the baseline before the tool goes live
Pick the one metric the tool is supposed to move, pull it from your CRM for the pilot reps and the control reps over a recent period, and check that the fields behind it are filled in. Do this before anyone logs in to the new tool. Once the tool is live, you can’t go back and measure “before.”
The metric depends on what you’re buying. A call-notes tool should move notes logged per deal. An AI SDR should move meetings booked per rep. A scoring model should move stage conversion, and a forecasting tool should move forecast accuracy. One metric per pilot. If the vendor’s pitch needs a whole dashboard to look good, write that down too.
Then check the plumbing underneath the metric. In Validity’s State of CRM Data Management in 2025, a survey of 602 CRM users and stakeholders, 76% said less than half of their organization’s CRM data is accurate and complete (Validity, 2025). If your baseline depends on a stage date reps fill in when they remember, or an activity type half the floor logs, the baseline is often the first thing to give way. You end up comparing a tool’s clean output against your team’s patchy history and calling the gap a win.
Run the field check now. Look at the fill rate on each field the metric touches, for both groups, across the baseline window. Where it’s thin, tighten the field rules before the pilot starts so the before and the after get measured the same way. (If you want a working definition of what “ready” looks like here, see what AI-ready data means.)
How do you set up a control group of reps?
Split reps with comparable books into a pilot team and a control team, then report on both groups side by side from the CRM. In HubSpot, teams let you group users and build team-based reports. HubSpot’s own example is grouping sales reps into regional teams to report on each region’s performance, and teams require a Professional or Enterprise subscription (HubSpot Knowledge Base, updated September 29, 2026).
Matching matters more than group size. Put reps with a similar segment, similar tenure, a similar territory and a similar mix of deal sizes on both sides. If the pilot gets your strongest closers and the control gets last quarter’s hires, the tool will look brilliant and teach you nothing.
In Salesforce, you can build the same split with a report filter on a user field or role. Whichever CRM you run, the comparison should live in a saved report your team can open on day one, built before the tool arrives.
Then keep the control group clean for the length of the pilot. No borrowing the tool for a big deal. If a control rep starts using it, pull them out of the comparison and note the date.
How do you tell what the AI did and what the rep did?
If the tool writes to your CRM, the property history shows which changes came from it. HubSpot’s property history shows, for each property on a record, the historical values, the date and time of each change and the source of the change. When a custom agent made the change, the agent’s name appears in the source along with who created the agent (HubSpot Knowledge Base, updated October 16, 2026).
That gives you two measurements most pilot plans skip. The first is the tool’s share of updates: of the changes to the fields it touches, how many came from the agent and how many came from reps. The second is accuracy. Pull a sample of agent-written changes each week and have a manager check them against the call, email or meeting they came from. A tool that writes a lot of values and gets a meaningful slice of them wrong is adding drag to your data, and you want to know that before it reaches the whole team.
Check how your tool shows up in that source field before the pilot starts. HubSpot documents custom agents by name. If your tool writes through an integration or a different feature, confirm what the history records for its changes so you aren’t guessing mid-pilot. For the wider question of what an agent should be allowed to touch, see AI agents and CRM write access.
Kill criteria: write them before day one
Kill criteria are the conditions under which you stop the pilot, written down and agreed before anyone gets a license. They have to exist before day one, because after day one each number has an owner with a reason it doesn’t count yet. You set the thresholds. They depend on your team, your cost and what the tool replaces.
The whole thing fits on one page. A pilot charter covers:
- The metric. The one number the tool should move, how you pull it from the CRM, and the baseline for both groups.
- Minimum movement against control. How much better the pilot group has to do than the control group over the same period. Last quarter is the wrong comparison.
- A data-quality floor. The tool must not degrade the fields it writes. Name those fields, and name the accuracy check you’ll run on the property-history sample.
- Rep usage. The level of use below which a result doesn’t mean anything, and how you’ll measure it.
- Cost. Licenses, implementation hours and admin time, set against the value of the movement you’re asking for.
- The groups. Who’s in the pilot, who’s in control, and how you matched them.
- The decision date. A fixed date ahead of the renewal or contract deadline, when you decide yes, no, or one extension with a written reason.
- The owner. One person who makes the call on that date.
Count how much of that list is about data. Gartner predicts that by 2028, CSOs who overhaul data, automation and user experience will be five times more likely to gain ROI from AI than those choosing quick fixes (Gartner, July 28, 2026). A pilot is a data project as much as a tool test. If your charter spends more lines on fields than on features, it’s pointed the right way.
From pilot to production: what changes?
Scale only what beat the control group on the metric you agreed before day one. Then extend the field rules and the property-history audit to each rep the tool reaches, and keep the control group running for one more measurement cycle before you retire it. Production is the pilot’s measurement, kept running at full size.
The result to worry about is the one that only holds in a small room. A group of reps who volunteered for the tool will use it differently from the whole floor. The extra measurement cycle is how you find out whether the result survives the reps who didn’t ask for it.
Keep the accuracy sample going too. Field rules that held with a pilot group drift when the tool reaches reps who weren’t part of the setup, and the property history is where you’ll see it first.
If the tool didn’t beat control, end it on the decision date and keep the baseline report. It’s the starting point for the next pilot, and the next vendor will want to see it.
The pilot that can say no
You can run a pilot that’s allowed to say no, or you can run one until the renewal date decides for you. The first takes some unglamorous setup in your CRM before the tool shows up. The second costs you the contract term and leaves you with an answer you still won’t trust.
Pick before the vendor’s start date picks for you.
If you want to check the baseline fields before the pilot starts, the free CRM Diagnostic runs 40+ automated checks across data quality, pipeline hygiene, freshness, reachability and ICP fit.