Most AI tool trials I hear about start with a signup and end with a shrug. The tool was there, some people used it, nobody can say whether it helped, and it is still there. I think the fix is a short list of questions answered before the trial starts, in writing, by the people accountable for the answers. Here is mine.
Where does the code go, and what is kept
The first question is about data. When an engineer sends a file to the tool, where does it go, how long is it kept, and who at the vendor can see it. Is it used to train anything. The answers are in a document nobody reads, and they differ between tools and between tiers of the same tool. A trial that cannot answer this has not really been approved.
What can the tool reach
The second question is about access. An agent that can run commands can read environment variables, open network connections, and call whatever the developer's credentials can call. I would want to know what the tool can reach from a developer's machine and the build system, and run the trial with less than that. The time to find out that a tool reads the whole home directory is before it does.
What would success look like
The third question is the one most trials skip. What is the tool for, which tasks, and what would change if it worked. Faster first drafts of one kind of change, or fewer review rounds on one kind of bug. Something specific enough that it could fail. A trial without a success condition cannot end, and a trial that cannot end is an adoption with no decision attached.
What happens to review
The fourth is about the team. If the tool works as advertised, more code arrives at review. Who reviews it, what are they looking for, and how much of their week will that take. I think the honest answer is often that review becomes the limiting step, and a trial that measures only the author's speed will declare victory while the reviewers quietly drown.
Who can turn it off
The fifth is about control. If the tool misbehaves, sends data somewhere it should not, or the vendor makes the news, who on the team can disable it that afternoon, and how. A tool that can only be turned off by a procurement process is a tool the team does not control.
How the trial ends
The last question is about the exit, in both directions. If the trial fails, what gets removed and who tells the people who liked it. If it succeeds, which workflows, documentation, and norms change, and who owns the changes. Either way the trial ends on a date with a decision, not by fading into permanence.
None of this is hard, and most of it takes an afternoon. It turns a trial from something that happens to a team into something the team does, with a reason, a measure, and an ending.
Photo source: https://photos.robertstowe.com/dolomites
