Tokenization can be learned end-to-end with reinforcement learning rather than frozen before training begins, according to a camera-ready ICML 2026 paper that reports its method beating prior straight-through approaches at 100 million parameters1. Every result in that claim is author-reported, demonstrated only against research baselines rather than deployed BPE tokenizers, and unreplicated as of 2026-08-27. If you serve or fine-tune an existing model, nothing in your stack changes.
What does arXiv:2602.13940 actually claim?
The paper’s core claim is that token boundaries are a learnable discrete decision that belongs inside the training objective, not a compression artifact fixed before training starts.
The framing is explicit. The abstract describes tokenization as “a hardcoded compression step which remains in the training pipeline of Large Language Models (LLMs), despite a general trend towards architectures becoming increasingly end-to-end.” That is a fair description of the standard pipeline. A tokenizer, usually byte-pair encoding or a unigram model built with tooling like SentencePiece, is trained on a corpus first. The vocabulary is frozen. Only then does model training begin, and every subsequent stage of that model’s life, from pretraining through fine-tuning through serving, inherits the vocabulary as a fixed constraint. The paper’s position is that this division of labor is a historical accident rather than a requirement, and that the boundary-drawing step can be pulled into the training loop and optimized against the same loss as everything else.
The version history on the arXiv listing shows v1 posted 2026-02-15, v2 on 2026-05-31, and v3 on 2026-08-25, two days before this article. The v3 revision carries the ICML 2026 camera-ready designation, with Sam Dauncey listed as submitter and the paper categorized under cs.LG and cs.AI. The camera-ready label matters for how to weigh the evidence: it means the paper went through conference review, which puts it a tier above a raw preprint. It does not mean anyone has reproduced the results.
How do the three boundary-learning strategies differ?
The three approaches differ in where the boundary decision lives and how gradients reach it: corpus statistics computed once before training, a continuous relaxation that treats a discrete choice as if it were differentiable, or a score-function estimator that optimizes the discrete choice directly.
The heuristic track is the incumbent. BPE and unigram tokenizers derive boundaries from corpus frequency statistics in a separate preprocessing step, and the paper notes that prior work attempting to bring the compression step inside the architecture still relied on heuristics to draw token boundaries. Heuristics are cheap and deterministic, but they answer the wrong question: they optimize for compression properties of the corpus, not for the downstream loss of the model being trained. A boundary that maximizes compression is not necessarily a boundary that minimizes perplexity, and nothing in the heuristic pipeline can discover that, because the heuristic never sees the loss.
The straight-through track tries to fix this by making boundary drawing differentiable. The discrete decision, boundary or no boundary at each position, is approximated with a continuous surrogate so gradients can flow backward through it. This is a standard trick in discrete latent-variable work, and it has a known cost: the gradient of the relaxation is not the gradient of the discrete objective. The paper says these estimates “treat the problem of drawing discrete token boundaries as a continuous one,” which is precisely the mismatch. You get end-to-end training, but the gradient signal is biased by construction.
The score-function track, which the paper advocates, estimates the gradient of the expected loss directly over the discrete boundary decisions, in the style of REINFORCE; the estimator itself is specified in the paper’s full text. No continuous surrogate is involved, so there is no relaxation bias. The authors argue this yields tighter theoretical guarantees because the estimator directly optimizes the discrete boundary-drawing problem to minimize loss, rather than optimizing a stand-in for it.
Why does the score-function approach need reinforcement learning machinery?
Score-function estimators trade bias for variance, and the paper reports that the variance was severe enough that time discounting, a standard RL technique, was required to make the method practicable.
The variance problem is structural. Each boundary decision is effectively an action, and the loss that scores the quality of that action arrives at the end of a long computation. A boundary drawn early in a sequence affects everything downstream, so the credit-assignment horizon stretches across the whole forward pass. Score-function estimators are unbiased in principle, but their variance grows with the horizon over which credit must be assigned, and naive implementations produce gradient estimates so noisy that training stalls or diverges. Time discounting shrinks the effective horizon by downweighting distant outcomes when assigning credit to each decision, which cuts variance at the cost of reintroducing some bias. This is textbook reinforcement learning, and the paper’s contribution is not inventing the technique but showing it is the load-bearing piece that makes score-function tokenization train at all. The variance analysis and training runs behind that claim are in the paper’s full text.
The tradeoff is worth stating precisely because it frames what the paper had to beat. A straight-through estimator produces a usable gradient, but the gradient points at the relaxed problem, so improving it can leave the discrete objective you care about unchanged or worse. A score-function estimator points at the right objective, but each sample carries little information, and the useful signal drowns in noise unless you either take many more samples or shrink the horizon over which credit is assigned. Time discounting is the cheap option: a small, bounded distortion of the objective in exchange for a large reduction in noise, which is a deal you take when the alternative is training that does not converge at all.
That framing is worth holding onto when evaluating the result. The honest scope of the contribution is making an estimator with known-good theory work in practice, at research scale, against the estimator it was designed to beat.
What has actually been demonstrated, and what hasn’t?
The demonstrated result is narrow: the method outperforms prior straight-through estimates “both qualitatively and quantitatively at the 100 million parameter scale,” in the authors’ own training runs. Nothing has been demonstrated about production-size models, deployed BPE or SentencePiece vocabularies, multilingual token costs, or serving economics.
Each qualifier does work here. The scale, 100 million parameters, is orders of magnitude below frontier model size. Techniques that win at 100M routinely fail to survive contact with 7B, let alone anything larger, and this paper offers no scaling evidence either way. The baseline is the straight-through estimator, a research method, not the BPE vocabularies shipping in production tokenizers. Beating a relaxation-based baseline does not establish superiority over a well-tuned SentencePiece vocabulary; it establishes superiority over the previous attempt at the same idea. And the comparison is author-reported, from a single paper, with no independent replication or secondary analysis anywhere in the public record as of 2026-08-27.
There is also a baseline-selection question the paper cannot answer from inside its own framing. Straight-through estimation is the natural comparison for a method that improves on straight-through estimation, but the incumbent a replacement would eventually have to displace is a tokenizer trained once on a corpus in a preprocessing step and never touched again. That incumbent is cheap and deterministic, and it already sits behind every production model. A challenger has to beat it while paying for a noisier training loop, and nothing in the current evidence speaks to that accounting.
The evidence tier deserves one more clarification. The ICML 2026 camera-ready designation means conference reviewers evaluated the work, which is real scrutiny. But the arXiv posting itself is not peer review. arXiv’s submission guidelines state that submissions “should be topical and refereeable scientific contributions,” and that every submission is subject to a moderation process that verifies material is appropriate and topical. arXiv’s about page states that contents of submissions are wholly the responsibility of the submitter, presented “as is” without warranty. Acceptance plus unreplicated results is a normal state for new methods; it is also a state that frequently does not survive replication attempts.
When does tokenizer choice actually bind?
Tokenizer choice binds exactly once in a model’s life: before pretraining. Every later stage inherits the vocabulary as a load-bearing constraint, and this paper changes the calculus only for teams training from scratch.
Pretraining from scratch. This is the scenario the evidence touches. If you are about to spend a pretraining budget, vocabulary design has historically been a decision you locked in before run one, derived from corpus statistics, and lived with. The paper weakens that assumption in principle: token boundaries could instead be co-optimized with the model during training. In practice, the evidence sits at 100 million parameters in a single paper, so the correct posture for a team planning a real run is to treat RL-trained tokenization as a research track to monitor, with replication at larger scale as the gating event. Do not bet a pretraining budget on it.
The constraint the paper is attacking is structural, not conventional. Vocabulary size fixes the dimensions of the embedding table and the output layer; every parameter in those matrices is indexed by token identity. Change the vocabulary after training and you have a model whose input and output dimensions no longer match its weights, which is why tokenizer changes are effectively architecture changes, and why serving stacks ship the vocabulary alongside the checkpoint as a single artifact.
Fine-tuning, including on non-English corpora. The vocabulary is frozen. Whatever tokens-per-word cost the base model’s tokenizer imposes on your corpus, you inherit it, and no amount of fine-tuning changes the segmentation. The well-known complaint that many non-English languages consume more tokens per word than English is a property of how the vocabulary was built before training; this paper neither quantifies that disparity nor offers any knob to adjust it at fine-tuning time. If tokenization cost is hurting your fine-tuning economics, your available levers are data formatting, context budgeting, and model selection, all of which sit upstream or downstream of the vocabulary rather than inside it.
Serving a fixed model. There is nothing to decide. Tokens-per-word is a property of the artifact you deployed, baked into both latency and per-token pricing. The only lever is which model you deploy in the first place.
Should you change anything in your stack?
No. The correct response to this paper is monitoring, not migration.
For anyone serving or fine-tuning an existing model, the practical verdict is unchanged from before the paper existed: keep the frozen vocabulary, because it is the only vocabulary the model has. For the narrower population of teams pretraining from scratch, the paper shifts the question rather than answering it. The assumption that vocabulary design must precede training is now contestable on theoretical grounds, with a working demonstration at research scale, and that is a genuine change in what is worth paying attention to. It is not a change in what is worth deploying.
What would change the verdict is a short, specific list: independent replication of the score-function result, evidence at scales past 100 million parameters1, a head-to-head against a production-grade BPE or unigram vocabulary rather than against straight-through baselines, and actual multilingual tokens-per-word and downstream-loss measurements. None of those exist in the current record.
If that evidence does arrive, the second-order consequence is the interesting part. The token-cost disparity that inflates inference pricing for underrepresented languages is currently an inherited artifact: a side effect of corpus statistics frozen at vocabulary-build time, invisible to the training objective. A tokenizer trained end-to-end against the model’s loss makes that cost an optimizable term. Whether the optimizer would actually spend capacity on non-English compression is an open question, and it depends on training data composition more than on estimator theory. But the possibility moves the disparity from “fixed property of the pipeline” to “outcome of the objective,” which is a categorically different thing to argue with.
What’s the strongest reason to stay skeptical?
The entire basis for the claim is a single paper’s author-reported results at 100 million parameters1 against straight-through baselines, with zero independent replication, hosted on a platform that explicitly disclaims peer review for its contents.
That last point is structural, not incidental. arXiv describes itself as hosting more than three million articles across eight subject areas, curated by volunteer moderators whose job is appropriateness screening, not correctness verification. The ICML 2026 camera-ready designation moves this paper out of the unreviewed mass, and conference acceptance is a real filter. Acceptance is not replication, and the history of machine learning methods papers is littered with accepted results that shrank or inverted when a second group ran them.
The estimator question the paper answers is real, and the progression from corpus heuristics to straight-through relaxations to score-function optimization is the right way to read where tokenization research is heading. BPE’s replacement, if it comes, will arrive through replicated results at production scale. A camera-ready at 100M is the opening bid, not the settlement.
Frequently Asked Questions
Does the ICML 2026 camera-ready status imply the results have been independently replicated?
No. Conference acceptance confirms peer review of the methodology and claims, but it does not constitute independent replication. As of August 2026, no secondary analysis or reproduction of the 100M-parameter results exists in the public record, leaving the findings as author-reported single-source evidence.
How does the variance of score-function estimators compare to straight-through relaxations in practice?
Score-function estimators are unbiased but suffer from high variance that grows with the credit-assignment horizon, often causing training divergence without mitigation. Straight-through relaxations produce lower-variance gradients but introduce structural bias because they optimize a continuous surrogate rather than the discrete boundary objective.
What specific operational change is required for teams fine-tuning existing models on non-English corpora?
No operational change is required or possible. The vocabulary remains frozen, so tokenization costs for underrepresented languages are inherited from the base model’s preprocessing step. Teams must continue relying on data formatting and context budgeting, as the paper provides no mechanism to adjust segmentation during fine-tuning.
Why is time discounting necessary for the score-function approach to converge?
Time discounting reduces the effective horizon over which credit is assigned to early boundary decisions, cutting gradient variance at the cost of introducing bounded bias. Without this RL technique, the noise from long-range dependencies in the forward pass would overwhelm the signal, preventing the model from learning stable token boundaries.
