Every evaluation of an AI coding tool I have seen was out of date before it finished. The tool changed under it, the model behind it changed under that, and the number it produced described a version nobody was running by the time the slides were shown. These tools are unusually hard to evaluate. Here is why, and what I would do anyway.
The target moves monthly
A traditional evaluation assumes the tool holds still long enough to be measured. These do not. The models are replaced every few months, the tooling around them every few weeks, and the same product in March and June can differ more than two competing products did in the same quarter.
The task mix dominates
Every serious study this year found the same thing. The task matters more than the tool. Documentation changes, tests, and configuration go well. Feature work in a mature codebase goes worse. So the result is mostly a function of which tasks were chosen, and a team that picks tasks badly will pick the wrong tool well.
The easy metrics measure the wrong thing
Time to complete and lines produced are easy to measure and nearly meaningless, because the tools are built to raise them. The things that matter, whether the code is maintainable, whether the author understands it, whether review got slower, whether anything shipped faster, show up months later. An evaluation that runs for three weeks sees only the easy numbers.
Novelty wears off
Engineers trying a new tool are more careful and more enthusiastic than they will be in month four. Some of the measured gain in any trial is that attention, and it does not persist. The honest evaluation would run long enough for the novelty to fade, which is longer than anyone budgets.
Nobody is neutral
Some want the tool, some fear it, and the vendor wants the sale. Every evaluation I have seen was, in part, a search for support of a conclusion already held.
What I would do anyway
I would stop trying to pick the winner and start deciding what a tool would have to change for the team to care. Not output. Something downstream, review time, defect rate, time to onboard, how long a feature takes end to end. One or two of those, measured before, with a number the team agrees would mean the tool helped.
Then run the tool against that for longer than feels comfortable, on the team's real work rather than a chosen sample, and accept that the answer will be about this tool at this time on this codebase, not a verdict on the category.
And expect to do it again, because the tools will keep changing. I think evaluation here is not a decision but a practice, with the same few downstream numbers watched over time. The team that ran one bake-off in the spring is guessing by autumn.
Photo source: https://photos.robertstowe.com/northern-arizona
