Most teams choose AI coding assistants from a demo, a keynote, or one enthusiastic engineer's weekend project. That is why the results are so inconsistent: the tool is never the variable that matters most. The task is. When we evaluate AI coding assistants against the work your team actually does — a messy codebase, a flaky test suite, a vague ticket — the ranking often differs from the marketing leaderboard. This guide gives you a framework we use to run fair, repeatable evaluations in about one afternoon, using your own repository and a simple scoring rubric you can defend in a design review.
The method has four parts: define the task, freeze the inputs, score the output blind, and make the decision with explicit thresholds. It works for any pair of assistants, and it produces evidence you can re-run next quarter when pricing or model quality changes.
The Problem With Assistant Demos
Demos are optimized for greenfield code: a fresh chat, a clean prompt, a task the presenter has practiced. Real engineering is the opposite. The repository has six years of history, the naming conventions are inconsistent, and the ticket says "make the export stop timing out for large accounts" without saying which of four services owns the export.
Three failure modes show up constantly:
- Plausible but wrong code. The assistant generates a clean function that ignores the transaction boundary the ticket implied.
- Confident dependency drift. It uses an API from a newer library version than the one pinned in your lockfile.
- Silent scope creep. It "improves" adjacent code, creating review noise exactly where you did not ask for changes.
None of these appear in a todo-app benchmark. They appear within minutes on a real branch — if you test there.
A Fair AI Coding Assistants Evaluation Workflow
The core idea is to hold everything constant except the assistant. Pick three tasks from your last sprint that were actually completed, ideally one bug fix, one small feature, and one refactor with a performance constraint. Each task needs four artifacts:
- The original ticket or issue text, unedited
- The repository state at the commit before the fix
- The human diff that shipped, used later as a reference — not as the "correct answer"
- A short acceptance checklist written by the engineer who did the work
The acceptance checklist is the part most teams skip, and it is what makes scoring defensible. For the timeout example above, a good checklist is: reproduces the failure on the 10 GB fixture, keeps the existing API contract, adds a regression test, and does not introduce a new migration. Four items, binary scoring, no vibes.
Why three tasks instead of ten? Because the goal is a decision, not a paper. Three tasks across different work types is enough to expose whether an assistant is only good at one shape of problem. If the results disagree, add a fourth task in the losing category before concluding anything.
How To Run The Test In One Afternoon
Run every assistant through the exact same procedure and keep notes as you go. The whole evaluation takes two to three hours for two assistants on three tasks.
- Create a clean worktree for each assistant at the pre-fix commit:
git worktree add ../eval-<tool> <base-sha>. This prevents cross-contamination and lets you diff results later. - Paste the ticket verbatim as the first prompt. Do not improve the wording. Part of what you are measuring is how the assistant handles ambiguity.
- Allow exactly two follow-up turns. Real usage includes iteration, but unlimited turns turns the test into a measure of your prompt patience. Two turns is a reasonable middle ground; write it down before you start.
- Record the diff without applying it. Save the assistant's suggested patch, then compare it against your acceptance checklist item by item.
- Score blind if possible. Shuffle the diffs and remove tool names before reviewing. You will be surprised how often your preference changes when you cannot tell which product produced the patch.
- Log cost and latency per task. A tool that scores 10% higher but doubles your seat cost is not automatically better; make that trade explicit in the write-up.
- Re-run once on a different day for any task where the two assistants are within one checklist point. Model output varies enough that a single sample can flip a close call.
Score each checklist item 0 or 1, then total per task. The human reference diff gets the same treatment, which keeps everyone honest: if the human fix also misses a checklist item, revise the checklist — it was either wrong or testing something the ticket never required.
Reading The Results
Look for patterns, not just totals. If one assistant wins the bug fix and feature but loses the refactor badly, that is a deployment policy, not a rejection: use it for implementation tasks and route architectural changes elsewhere. If both assistants fail the same checklist item, the task is probably underspecified in your tickets, which is its own useful finding.
Verdict And When This Changes
Adopt the winner only if it beats the alternative on at least two of three task totals and does not fail any acceptance item catastrophically — data loss, security regression, or a breaking API change. Otherwise keep both for another cycle and re-test with new tasks.
Re-run this evaluation when any of these change: a major model upgrade, a pricing change above roughly 20%, a new compliance requirement, or a repeated pattern of review pushback. Put a calendar reminder on it; assistant quality is a moving target, and last quarter's verdict is not evidence.
To explore options before you design your tasks, browse the tool directory and the monthly rankings for currently listed assistants, then build your three-task evaluation set from your own sprint history — not from anyone's demo reel.