Coding agents sped up the task and weakened understanding

(Im)Paired Programming: Coding Agents Improve Productivity but Harm Understanding

Coding agents (e.g., Cursor) improve developer productivity by optimizing task completion, but shifting users from writing code to prompting and reviewing may harm their understanding, impeding oversight, learning, and communication. To probe this, we have 54 students create a website with one of two AI systems: an agent that edits user code; or a chatbot where users write code alone or adapt generic code snippets. We test understanding via comprehension questions and a task where users extend their code without agents, showing: (1) While agents aid initial task completion, they harm users’ code comprehension and thus do not prepare users to extend their code; (2) Low-effort agent interaction types, like copy+paste prompts and auto-accepted edits, are linked with lower comprehension; and (3) Despite self-reported weaker understanding, users still prefer coding agents because they are quick and easy to use. While users stay in the loop for coding workflows, understanding should not be forgotten. Towards this goal, we distill our analyses into future research directions for coding agent developers: dissuading low-effort prompting, creating readable code, and promoting active engagement.

arxiv.org

A group at the University of Maryland ran a controlled experiment with 54 participants building a website. Half used a coding agent that edits the code directly. Half used a chatbot and wrote the code themselves. The agent group finished the initial task faster. Then the researchers measured how well each group understood the code they had just produced, and how prepared they were to extend it. The agent group did worse, and the gap tracked the kind of interaction: copy-paste prompts and auto-accepted edits correlated with the least comprehension. The participants still preferred the agent, for its speed.

It is a small student study. I would not generalize its numbers, but its shape matches what I see.

Every productivity case for coding agents is built on the first measurement, time to complete. Almost none of them include the second, whether the person can still reason about the system. On a real team, that second number determines who can fix the production bug at 2am, who can review the next change, and whether the junior who shipped this feature learned enough to ship the next.

I think comprehension is the hidden cost line in the agent return on investment, and I think it lands hardest on the people early in their careers, who most need to build the understanding the tool lets them skip. I think a team should decide, on purpose, which work is for learning and keep the agent out of it.

Does your team have any work that is deliberately done without the agent, and why?

Photo source: https://photos.robertstowe.com/texas