The challenges of measuring developer productivity

The Niagara River winding blue-green between wooded banks toward the lake, seen from a high bluff

Every developer productivity metric I have used measured something real, and every one of them stopped meaning that thing within a quarter. Not because anyone cheated. Because people are smart and metrics are visible, and once a number matters, the work bends toward the number.

Why I think this is structural

The output of engineering is a change in a system. The value of that change is only visible later, somewhere else: in a customer who could do something they could not do before, in an outage that did not happen, in a feature next year that was easy because of a decision this year. The distance between the work and its value is long, and every metric that is quick to collect measures something on the near side of that distance.

Pull requests, commits, story points, lines of code, cycle time. All of them are measures of motion. Motion correlates with value when nobody is watching the motion. The moment it is watched, the correlation weakens, because motion is easy to produce and value is not.

I have watched pull request counts rise as pull requests shrank. I have watched cycle time fall as work got sliced into pieces that each moved quickly and together took longer than before. Nobody did anything wrong. The metric just taught people what it wanted.

What a manager is actually trying to learn

When I ask myself why I want a productivity number, the honest answers are a short list. Is this team healthy, or is something wrong that I cannot see. Is the investment we are making in this area paying off. Did the change we made, the new tool or the new process, help. Who needs support.

None of those questions is answered by a single number, and I think the search for the single number is where measurement goes wrong. They are answered by a mix of things, most of which are slow.

The measures I still trust

Delivery outcomes over time. Not how many things shipped but whether the things that shipped did what they were meant to, measured by whoever the thing was for. This is slow and it is the only measure that points at value.

The questions the team asks. A team that is stuck asks about blockers, environments, and approvals. A team that is healthy asks about the problem. Listening is not a metric, but it is the fastest signal I know.

Change failure and recovery. Of all the quantitative measures, the share of changes that cause a problem and the time it takes to recover are the hardest to game, because gaming them means actually breaking less and recovering faster, which is the point.

Whether people say the same thing in private. If the dashboard is green and the one-on-ones are full of frustration, the dashboard is wrong.

What I try not to do

I try not to compare individuals on any output measure, because individual output in engineering is a property of what the person was assigned, who they were unblocking, and what they were cleaning up, none of which the number sees. I try not to put a motion metric on a wall, because that is the moment it stops working. And I try not to promise anyone a single number that will tell them whether engineering is productive, because I do not think the number exists, and I think the honest thing is to say so.

Photo source: https://photos.robertstowe.com/niagara