Token spend is a real, metered number, which is exactly what makes it tempting as a productivity metric. A new case-study preprint, arXiv:2611.13396, asks the question directly: what does an engineer’s token spend actually measure? The study followed 404 engineers at JetBrains, a software development company whose internal gateway meters the tokens of every coding-agent call. Everything in that study is preprint-reported, author-reported, single-company, and unreplicated, and it should be read that way. Within those limits, the authors’ conclusion is blunt: token spend “measures how much an engineer works through agents, not what they produce, so it should not be used as a measure of productivity.”
If you are an engineering lead being asked whether the rising AI bill means the team is shipping more, that is your answer, with an important refinement developed below. The bill is not noise. It is stable, interpretable telemetry about agent behavior and budget consumption. The mistake is reading it as output.
The timing pressure is real. The same preprint reports, citing another source, that Uber exhausted its 2026 budget for AI coding tools in four months, with its COO saying the link between token spend and shipped features “is not there yet.” That quote is a secondhand citation inside a preprint, not a vendor-published statement, so treat it as reported rather than established. It still captures the situation many leads face in late 2026: budgets for AI coding tools are now metered on token usage, and nobody has agreed on what the meter means.
The JetBrains case: what the meter actually recorded
The case study’s value is its setup. Because the company’s gateway meters every coding-agent call per engineer, the authors could observe token spend as a clean, per-person time series rather than a survey estimate. Their finding splits into two halves that are easy to conflate.
The first half is the negative result: spend tracks the volume of work an engineer routes through agents, not the work that ships. An engineer who re-sends a large codebase context forty times looks exactly like an engineer who shipped forty features, and an engineer who lets an agent loop on a failed task outspends one who intervenes early, with nothing extra delivered.
The second half is the part that should make managers careful, and it is the strongest counter-evidence against a lazy “spend is meaningless” reading. Five-week token totals rank engineers consistently, with a projected rank reliability of 0.87. A metric that consistent feels valid. It feels like it must be measuring something real about the engineer. It is measuring something real: how much that engineer works through agents. Consistency is not validity, and this metric is the cleanest illustration of that gap in these results. A ranking can be stable for weeks and still rank the wrong thing.
The same study also surfaces downstream costs that never appear on the meter. Thirty-one percent of technical agent users reported higher cognitive load from agents, and 55% of the 299 respondents who answered a related question in the preprint’s survey reported handling colleagues’ suspected AI-generated material, at a mean cost of 1.6 hours per week. None of that is in the token bill. If your dashboard only shows spend, it is blind to both the input (what the agent did) and the cleanup (what colleagues absorbed).
Anatomy of the bill: mostly cache, not generation
The most useful structural fact comes from a study of 2,848 paired, provider-billed coding-agent runs. Cache creation and cache reads dominated the cost composition: about 87% of a four-component reconstruction and about 80% of the actual bill, with an explicitly quantified 8.7% unattributed residual.
Read that against the KPI question. If four-fifths of the bill is cache operations, then most of what an engineer “spends” is context being re-sent to the model, not tokens of reasoning or code being produced. A long session that repeatedly re-embeds the same repository state is expensive in exactly the way a productive session is expensive. The bill cannot tell them apart, because the dominant line item is the cost of reminding the model where it is.
The same paper delivers the sharpest warning against naive cost control. Removing an estimated 38% of raw tool output, a retrieval-stage token reduction that looks like an obvious win, coexisted with a +6.8% paired cost increase. The mechanism: trajectory changes. With less tool output available, the agent spent extra turns on diagnosis, testing, and re-retrieval, and those turns cost more than the savings. Token reduction is not cost reduction, because the agent’s path through the task adapts to whatever you cut. A “spend less” mandate aimed at the visible number can raise the actual bill.
This is also why the composition figures matter more than the totals. Before interpreting any movement in a team’s token spend, decompose it into cache creation, cache reads, generation, and retry traffic. A spike in cache reads during a migration week is a different event than a spike in generation during a feature push, and both look identical in a single total.
Confounds no engineer controls
Even the non-cache portion of spend is partially a function of inputs the engineer does not choose.
Task specification is the clearest case. Across 2,700 agentic-coding runs with Kimi K3, one study found that a bare user story raised token spend by 29.7% over a full specification, while run-to-run variance was unaffected by any prompt changes. Two engineers with identical ability, assigned the same task written two ways, will post different bills. Whoever writes the ticket shapes the spend, which is often the lead, the PM, or a template, not the person being measured.
Model and infrastructure choice is the second confound. Work on quantization, a standard cost lever, found that quantized models often preserve final-answer accuracy while generating longer chains of thought than their full-precision counterparts, an effect that can offset or outweigh the per-token speedup quantization is supposed to deliver. If your platform team switches a backend model to a cheaper quantization, individual token counts can rise with no change in behavior and no change in answers. A per-engineer spend ranking will record that infrastructure decision as a change in the engineers.
The third confound is that models make poor meters of themselves. A study analyzing token consumption in agentic coding tasks found that frontier models fail to predict their own token usage, with weak-to-moderate correlations up to 0.39, and systematically underestimate real costs. Human-rated task difficulty only weakly aligned with actual token costs, which undermines the intuition that a bigger bill means a harder task. Two evaluated models, Claude Sonnet 4.5 and Kimi-K2, each consumed over 1.5 million more tokens on average than GPT-5. Swapping models reshuffles everyone’s “productivity” overnight without anyone doing anything differently.
The practical consequence: never let a model’s own estimate of its token use stand in for the metered bill, and never compare spend across periods that span a model or quantization change without normalizing for it.
Loop risk: first-attempt failure multiplies cost two to five times
The failure mode with the most direct KPI consequence is retry inflation. Research on inflation-aware routing for agentic systems quantified it: when a model fails on the first attempt, single-call cost estimates can be off by a factor of two to five. The authors call the gap token inflation, the ratio between what a workflow actually costs and what a single-call estimate predicts.
This is the mechanism behind the unbounded agent loop problem. An agent that misses on attempt one does not cost one attempt plus a little; it costs several attempts’ worth of context rebuilds, re-reads, and regenerated plans. Spend climbs steeply while delivered work stays flat. If token spend is a KPI, this failure mode is actively rewarded: the engineer whose agent loops quietly on a bad task posts a bigger number than the engineer who kills the run and rewrites the prompt.
There is a constructive flip side. Retry inflation is exactly the kind of signal spend telemetry is good at detecting, if you look for it. A run whose cost has crossed two or three times its single-call estimate is a loop candidate, and that is a budget and alerting question, not a performance question. This is where token telemetry earns its keep: anomaly detection on runaway agents, not rankings of people.
Measurement window: 0.87 at five weeks, noise at one
If you are going to track spend at all, window length is the first design decision, and the case study gives it an unusually specific answer. Five-week totals rank engineers at projected reliability 0.87; two single weeks rank-correlate at only 0.4 to 0.6. A single-week token ranking is closer to noise than signal, which makes weekly spend leaderboards worse than useless, since they are precise-looking noise with a name attached.
A companion specification study offers a cheaper forecasting trick. The same study found that a single probe run costing eleven cents on an unseen task predicted that task’s spend across every other specification and thinking-effort setting to typically within 36%. That is a bounded, preprint-reported result on one model and task set, not a general forecasting guarantee. But as a practice, running a cheap probe before committing a fleet of agent runs to a large task is a reasonable budgeting habit, and it comes straight from measured data rather than vendor guidance.
The pairing checklist: what to measure alongside spend
The consistent message across the studies cited here is that input metrics need outcome partners. Here is the checklist, with the evidence class of each item made explicit.
| Companion metric | What it catches that spend cannot | Evidence class |
|---|---|---|
| Suggestion acceptance | Whether agent output is used at all; roughly 30% Copilot acceptance reported in production | Secondhand citation of a GitHub figure inside a preprint, not a vendor-published measurement |
| Cognitive load | The 31% of agent users reporting higher load | Self-reported survey data in the case-study preprint |
| Rework / review burden | The 1.6 hours per week spent handling colleagues’ suspected AI-generated material | Self-reported survey data in the same preprint |
| Code health | Whether the codebase stays maintainable for humans and agents | Measured, benchmark-level: 5,000 Python files refactored by LLMs |
| Spend decomposition (cache vs generation vs retries) | Whether a bill movement is context re-sends, new work, or loops | ~80% cache share measured across 2,848 paired, provider-billed coding runs; 2–5x retry inflation measured on GSM8K/HotpotQA retry benchmarks, not coding tasks |
Two of these deserve extra caution. The acceptance figure is a GitHub number as quoted inside a preprint, so verify against the vendor’s current published data before putting it in a slide. The code-health work operationalizes “AI-friendly code” via LLM-based refactoring on competitive-programming Python files, which is a long way from a production monorepo; it is a candidate quality counterweight, not a validated dashboard. And none of these metrics is self-sufficient either. Acceptance rates can be gamed toward trivial suggestions, and self-reported load shifts with team culture. The point of pairing is that each metric constrains the others’ failure modes.
A bounded hypothetical makes the use case concrete. Suppose your gateway shows one engineer’s five-week spend doubling. The decomposition shows the growth is almost entirely cache reads and retry traffic, acceptance on their suggestions is unchanged, and review burden on their teammates ticked up. That pattern reads as an agent looping on poorly specified tasks, and the fix is task specs and loop timeouts, not a conversation about the engineer’s effort. Flip the decomposition to mostly generation with stable acceptance and flat rework, and the same bill increase reads as more agent-mediated work being absorbed without obvious harm. Same total, opposite interpretations, and only the paired metrics separate them.
What one preprint cannot establish
The limits here are structural, not incidental. The central result comes from 404 engineers at JetBrains, one company with one gateway configuration, author-reported and unreplicated; the paper itself notes that JetBrains builds AI coding tools, so some metered use may be testing of its own agents. The confound figures come from different populations entirely: 2,700 runs on Kimi K3 for the specification result, 2,848 provider-billed runs for the cache composition, GPT-5 comparisons for the prediction result. Cache shares near 80%, a 29.7% specification effect, and 2–5x retry multipliers are measured numbers, but they are measured on specific models and setups, and the retry multipliers come from GSM8K and HotpotQA question-answering benchmarks with a synthetic retry protocol, not from coding tasks. Your gateway’s caching behavior, your model mix, and your task distribution will move all of them.
The evidence also does not establish that token spend is useless. A stable five-week ranking of agent-mediated work volume is genuine information, about engagement, adoption, and budget consumption. What the evidence rules out is the inference from that ranking to output. And none of these studies validates a particular dashboard or threshold: the checklist above is my synthesis of measured findings, not a configuration any paper tested end to end.
Verdict: read the meter, not the engineer
Do not adopt token spend as a productivity KPI, and do not rank engineers on it at any window length. The best evidence available says it measures how much people work through agents, it is dominated by context re-sends rather than generation, it moves with task specifications and infrastructure choices nobody being measured controls, and it rewards exactly the retry behavior you want to discourage.
Do use it as budget and agent-behavior telemetry. Aggregate five or more weeks before treating any number as stable. Decompose movements into cache, generation, and retry components before interpreting them, and treat runaway retry traffic as an alerting problem. Treat model-reported estimates as unreliable for budgeting, and prefer a cheap probe run when you need a forecast. Then pair everything with outcome signals: acceptance, rework, cognitive load, code health.
If you are pushed to report one number upward, report spend alongside at least one outcome measure, and say plainly what the case study found: the link between this bill and shipped work is weak.
Frequently Asked Questions
How long should an engineering lead track token spend before treating the data as stable?
Five-week totals rank engineers at projected reliability 0.87; two single weeks rank-correlate at only 0.4 to 0.6. A single-week token ranking is closer to noise than signal, which makes weekly spend leaderboards worse than useless, since they are precise-looking noise with a name attached.
What is the primary component of the cost in AI coding agent bills?
Cache creation and cache reads dominated the cost composition: about 87% of a four-component reconstruction and about 80% of the actual bill, with an explicitly quantified 8.7% unattributed residual.
Join the discussion
Share a useful perspective or ask a question about this article.