What Cursor Learned About AI Coding Evals

Static benchmarks lie. Models ace HumanEval, ship to production, reality hits. The gap between evaluation scores and codebase performance? Often massive. Teams optimize for metrics that predict nothing about real-world success.

Naman Jain at Cursor explains in Coding Evals: From Code Snippets to Codebases—how AI coding evaluation exploded from single-line completions to multi-hour transformations in just four years. The measurement problem isn't academic—it's destroying production rollouts.

At Tier, we learned this lesson with edge ML deployment (and this sounds obvious in retrospect): clean test environments never matched the chaos of real scooter networks. The same principle wrecks code generation. Static evaluations produce systems that break when they meet messy reality. The ceiling isn't technical—it's the measurement system misleading decisions.

The Four-Year Sprint

Coding AI went from parlor tricks to production systems fast. 2020: single-line pandas completions. 2026: multi-hour codebase transformations without a single human intervention.

My first project was actually working on generating like single line panda snippets and my last project was generating an entire codebase. — Naman Jain, Cursor

Each time horizon jump breaks previous measurement approaches. Single-line metrics don't predict interview performance. Interview scores don't predict repository reasoning. Repository tasks don't predict multi-hour capability.

But here's the catch: most teams still evaluate modern systems with old metrics. They measure GPT-5 capabilities using GPT-4 benchmarks and wonder why production doesn't match scores.

The pattern repeats across domains. We build evaluation frameworks for current capability levels, then models leap ahead while benchmarks lag behind. The gap costs real money—teams deploy confident in scores that predict nothing.

The Contamination Trap

Static datasets rot fast. Models train on leaked problems, inflate scores. Actual capability? Flat.

HumanEval looks impressive until you realize how many solutions leaked into training data. Models memorize answers, not reasoning patterns. Benchmarks become achievement tests, not capability tests.

The field has like really progressed very quickly. — Naman Jain, Cursor

The speed of progress makes contamination inevitable. Create benchmark, models train on it, scores inflate, real performance stays constant. Dynamic rotation helps—barely. Most organizations lack infrastructure to generate fresh problems consistently.

I'm skeptical of any benchmark that's been public for more than six months. The memorization problem compounds with model scale. Larger models memorize more training data, game static evaluations better, and inflate scores faster than smaller models—but their actual reasoning improvement might be minimal.

When Clean Evals Fail

Well-defined problems are educational tools. Production problems are integration nightmares.

The nice thing about these interview style problems is that these problems are very well defined. you have like good natural language specifications some example in... — Naman Jain, Cursor

Interview problems come with clean specs, example inputs, expected outputs. Production work comes with incomplete requirements, shifting constraints, undefined edge cases. The evaluation disconnect is massive.

Consider the jump from snippet completion to repository work. Snippet metrics measure syntax correctness, maybe semantic accuracy. Repository generation requires architectural reasoning, integration awareness, long-term consistency. Traditional benchmarks miss this entirely.

What I find genuinely fascinating about this evolution is how each capability jump exposes new failure modes we didn't even know to test for. Single-line completion doesn't require understanding project structure. Multi-file generation doesn't require maintaining consistency across refactoring sessions. Full codebase work doesn't require reasoning about deployment constraints.

Each level needs entirely different evaluation approaches.

The Testing Bottleneck

Test quality determines evaluation reliability. Weak test suites produce false confidence.

Most benchmarks use minimal test cases—maybe 5-10 per problem. Production systems need hundreds. Comprehensive testing catches subtle bugs that sparse evaluation misses. Models pass benchmarks by satisfying basic cases, then fail in deployment on edge conditions.

This is where the economics get interesting. Generating good tests is harder than generating code—and most evaluation systems skip this complexity entirely. Teams optimize for benchmarks with inadequate test coverage, deploy systems that pass trivial cases, then discover edge case failures in production.

At ENVAIO, we learned this the hard way with IoT device firmware. Simulator tests looked perfect. Real-world deployment revealed dozens of edge cases our test suite missed. Same principle applies to code generation at scale.

The Multi-Hour Problem

Real coding isn't isolated problems. It's messy codebases, unclear requirements, integration constraints. Static benchmarks miss this entirely.

Multi-hour tasks break binary evaluation completely. Pass/fail doesn't work for complex transformations. You need partial progress metrics, intermediate correctness checks, architectural impact assessment.

Repository work requires different evaluation approaches. Context matters—code history, architectural patterns, organizational standards. Snippet benchmarks ignore all of this.

I could be wrong here—my sample size is limited to enterprise contexts—but the pattern holds. Teams using contaminated benchmarks waste deployment cycles. Those building evaluation infrastructure that matches production complexity build reliable systems.

Why This Matters

The measurement gap is expensive. Deploy based on benchmark scores and you'll spend weeks fixing what tests missed. Overconfidence leads to premature rollouts.

Data access determines competitive position. Open benchmarks level the playing field—everyone sees the same (contaminated) scores. Organizations with proprietary evaluation datasets—real production problems, dynamic task generation, comprehensive test suites—maintain competitive advantage.

The teams optimizing for public benchmarks compete on metrics that don't predict success. The teams building private evaluation infrastructure optimized for their specific production requirements pull ahead consistently.

Here's the question I keep coming back to: if benchmarks are becoming less reliable predictors of production performance, what measurement approaches actually work?

What Works

Rotate evaluation problems regularly. Add tasks after training windows close. Prevents memorization, keeps scores meaningful.

Source tasks from real commits. Grade on improvement over human baselines, not absolute correctness. Synthetic problems don't predict production performance patterns.

Build dynamic test generation. Property-based testing produces varied test cases automatically. Fuzzing approaches catch edge cases static suites miss consistently.

Use partial progress metrics for long-horizon tasks. Binary pass/fail breaks on multi-hour work. Track incremental completion, intermediate correctness, architectural consistency.

Deploy contamination detection early. LLM judges spot memorized solutions better than automated scoring. Human evaluation catches sophisticated gaming patterns that metrics miss.

This works when you have resources to generate problems, rotate datasets, detect gaming. Most teams don't. They use public benchmarks, trust inflated scores, discover the gap in production. The cost: wasted cycles and lost trust.

Full talk: Watch on YouTube