groundy
Models & Research

Coding Agents on Underrepresented Languages: What Longtail-SWE Tests

Longtail-SWE offers 202 tasks across 13 underrepresented languages, showing per-language variation that pooled scores hide. Author-reported results require internal validation

Published 9 references
A scuffed translucent green resin dinosaur looks down skeptically, one foot seated in a scalloped ivory tile and the other hovering above an angular recess. Yellow ankle bands and a hard shadow punctuate the warm ivory scene.
On this page9 sections

Do not treat a Python-centric leaderboard score as evidence that a coding agent is ready for your long-tail-language repositories. Treat it as a hypothesis to test on your own issues before routing traffic. Longtail-SWE, an unreviewed preprint observed on November 14, 2026, provides a useful new reference point: 202 issue-resolution tasks from 129 repositories across 13 underrepresented programming languages, per the preprint’s abstract. Its performance figures are author-reported, not independent measurements. Across nine evaluated configurations, the highest reported results are 81.52% Pass@1 and 89.11% Pass@3, both attributed to Opus 5.5 (high) in the paper’s Table 1. The per-language results carry more decision weight than the headline: configuration-averaged Pass@1 runs from 48.1% on Clojure to 73.5% on Erlang, a roughly 1.5-fold spread (Section 4.4).

The practical conclusion is not that agents will fail on underrepresented languages. It is that a pooled score hides variation that changes routing decisions, and Longtail-SWE’s own per-language table is now the clearest demonstration. The paper’s remaining limits, which its authors state themselves, still leave the final evaluation burden with the team considering adoption.

What Longtail-SWE actually tests

Longtail-SWE’s clearest documented contribution is coverage combined with task annotation. Its 202 tasks span 129 repositories and 13 underrepresented programming languages, according to the preprint. The languages are named outright, Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala and Swift, so a team can check directly whether its stack is covered rather than assume. That is a different evaluation target from asking whether an agent can resolve another Python issue, and it gives researchers a way to examine languages that established benchmarks seldom reach.

The task mix also matters. The authors report core feature development on 67.3% of tasks and major bug fixes on 27.2%, followed by integration features, in the paper’s composition analysis. The labels are not mutually exclusive, so these are annotations rather than a clean partition of the dataset. A team whose backlog resembles that mix will find the benchmark more relevant than one whose work consists largely of other task types. The annotations help readers assess representativeness; they do not, by themselves, demonstrate that every task has sufficient tests.

Availability and attribution are documented, not open questions. Section 3.5 states that the complete release is publicly available through a GitHub repository, with code under the MIT License and dataset contributions under CC BY 4.0. Table 1 names all nine evaluated configurations, and Table 2 reports Pass@1 by language for each of them. What is missing is independent verification: the paper is unreviewed and every score in it is author-reported.

The high reported ceiling is important counter-evidence against a simple failure story. The 81.52% Pass@1 belongs to Opus 5.5 (high) (Section 4.3), exceeding runner-up GPT 6.1 Sol by 3.14 percentage points, so a named frontier configuration achieved it rather than an anonymous one. The per-language table keeps that score from becoming a readiness claim. Averaging Pass@1 equally across configurations, Erlang has the highest mean success rate at 73.5%, followed by Haskell at 72.8% and R at 69.0% (Section 4.4); Clojure, F# and Elixir sit lowest at 48.1%, 50.2% and 50.4%. No single configuration leads every language: GPT 6.1 Sol leads on Elixir, Haskell and Julia, GPT 5.6 Sol leads on F#, and Kimi K3 leads on R. The authors also caution that with 12–27 tasks per language (Section 6.1), differences reflect the sampled tasks and evaluated configurations rather than inherent language difficulty.

That spread is the paper’s most decision-relevant result. A team routing Erlang work and a team routing Clojure work would reach different conclusions from the same benchmark, and only the per-language table separates them.

The preprint does not measure a Python-versus-long-tail gap for the same agent under the same conditions. The stronger claim that Python leaderboards overstate long-tail readiness therefore remains an inference from benchmark fragility, not a measured performance delta. The safer operational position is to regard those leaderboard scores as unvalidated for a different language environment, a caution that the verified per-language spread supports without proving.

Four ways a per-language score can mislead

Tests can certify incorrect patches

A resolve rate is only as trustworthy as the tests deciding whether a patch passed. UTBoost reports 36 SWE-bench instances with insufficient test cases and 345 erroneous patches incorrectly labeled as passed in the original SWE-bench. Its corrections affected 40.9% of SWE-Bench Lite leaderboard entries and 24.4% of SWE-Bench Verified entries, per the UTBoost leaderboard update, changing 18 and 11 rankings respectively.

That finding identifies a concrete mechanism behind inflated results: a weak test suite can accept a patch that does not actually solve the issue. The agent need not improve for its measured score to rise. Longtail-SWE’s construction pipeline is aimed at exactly this failure mode: a judge reviews the solver’s trajectory and test results alongside the task to guide revisions to tests that reject valid alternative implementations or accept incorrect solutions.

A second preprint, SWE-Bench ProMax, cites an audit finding flawed tests in nearly 60% of unsolved SWE-bench Verified instances, including 35.5% with narrow tests and 18.8% with broad tests (ProMax). It says these findings led OpenAI to deprecate the benchmark. That audit and deprecation are secondhand claims inside the ProMax preprint, not independently established events; they reinforce the need for scrutiny, but should not be repeated as directly verified.

Contamination controls need enforcement

Issue-resolution benchmarks often use public development history, which creates a route to the answer that has nothing to do with solving the task. In one harness study, an audit of early web-enabled runs found that 71% of DeepSeek and Qwen trials had fetched the upstream fix from GitHub. The authors discarded those runs and enforced the no-web condition at the network layer.

That enforcement detail matters. A statement that web access is disabled is less informative than a control that prevents the retrieval path in the evaluation environment.

Publication dates provide a different control. SWE-Bench+ admitted only issues appearing after October 2023 because the models it evaluated had training-data cutoffs through that date. This design reduces the chance that the original issue and fix were already in training data, provided the relevant cutoff is known. SWE-bench-Live presents another approach: a continuously updating benchmark rather than a permanently frozen task set.

Longtail-SWE is candid about where it stands on this axis. Its threats-to-validity section states that tasks derive from public issues and pull requests, so overlap with training data cannot be ruled out, and that undisclosed training cutoffs make it difficult to guarantee a fixed task set postdates every evaluated model’s training data (Section 6.1). The authors note the pipeline can restrict collection to tasks created on or after a specified date, a control worth asking about.

None of these controls makes contamination impossible. Together, they give readers specific questions to ask: How were task dates chosen? Could the agent browse? Was that restriction enforced and audited? Does the benchmark refresh its tasks?

The harness is part of the result

A coding-agent score belongs to a configuration, not to a model in isolation. The harness is the scaffolding that runs the model on the task. In the same harness study, swapping the harness while holding the model fixed moved SWE-bench Verified pass rate by as much as rerunning the same harness on the 45 hardest tasks, across five models.

Consequently, a per-language comparison that changes both language and scaffolding cannot attribute the difference cleanly to language. Even two scores for the same model may not be comparable if their harnesses differ. Longtail-SWE makes the principle concrete in a single table: nine named configurations, each pairing a model with a SWE-agent-like scaffold for three attempts per task, spread from 81.52% Pass@1 for Opus 5.5 (high) down to 16.67% for Qwen 3.6 27B (thinking) (Table 1). The headline is the top of a range, not a typical result.

Difficulty can dominate the headline

Cross-benchmark percentages are not exchange rates. Task construction and difficulty calibration can separate systems far more than a headline percentage suggests.

SWE-Bench Pro V2 reports that frontier agents resolve up to 99.4% of its 642 V2 tasks. Yet its 51-task Hard split spreads systems from 52.9% to 98.0%, per the same paper. A benchmark can therefore be extensively repaired and still approach saturation at the frontier, while a selected harder subset restores separation. Cleaning a benchmark does not guarantee a low ceiling.

The ProMax authors report a very different picture: on 170 large-scale refactoring instances, the best evaluated model, GPT-5.2, reaches a 41.2% resolve rate, and no model dominates across all seven languages (ProMax). Those tasks came from 70 repositories and were filtered from 29,782 initial candidates through a three-stage pipeline, according to the ProMax preprint.

Longtail-SWE is itself an example of why calibration questions matter. Its difficulty selection ran Qwen 3.8 27B three times per candidate task and excluded tasks solved in all three attempts, and the paper concedes in its internal-validity note that this may bias Qwen’s reported scores (Section 6.1). Some evaluated models also participated in task construction, the authors note, and may benefit from the resulting task interpretations.

It would be invalid to subtract 41.2% from Longtail-SWE’s 81.52% and call the difference a language penalty. The task types, construction pipelines, configurations and difficulty distributions differ. The useful lesson is that a percentage has little decision value without knowing what was hard, what was repaired and which systems the benchmark was designed to separate.

How the benchmarks line up

BenchmarkCoverage and constructionWhat it contributes to a routing decision
Longtail-SWE202 issue-resolution tasks, 129 repositories and 13 underrepresented languages from Clojure to Swift; feature-heavy task annotationsNamed configurations with per-language Pass@1 for all nine; results are author-reported and the preprint is unreviewed
SWE-bench Multilingual, described in the SWE-Trail paper300 issues across C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby and RustA named nine-language baseline that also makes clear which languages remain outside its coverage
Claw-SWE-Bench350 instances from 43 repositories across eight language groups, including Ruby, PHP and Python; tasks sourced from SWE-bench Multilingual and SWE-bench-Verified-MiniConnects multilingual evaluation with harness assessment, while reusing established task sources
SWE-Bench ProMax170 refactoring tasks across 70 repositories and seven languages, filtered from 29,782 candidatesShows how task type and difficulty can produce much lower multilingual results, with no single model leading every language
SWE-Bench Pro V2Repaired or retired 480 of 731 original public-split tasks; 23 contracted engineers spent 404 hours, with revised instructions checked by a second engineer solving from text aloneDemonstrates the labor behind task repair and the value of retaining a harder split when the main benchmark saturates

The Pro V2 construction details are especially useful as a reality check. Its engineers worked in languages they used professionally, and a second engineer verified the revised instructions by solving from the text alone. That does not establish equivalent verification for Longtail-SWE. It shows what a documented construction-rigor claim looks like, and how much work can sit behind it.

A checklist for any per-language agent claim

Before using a benchmark result to route real work, ask five questions:

  1. Which languages and repositories were actually evaluated? Require named coverage rather than a language count, and per-language results rather than a pooled score. Longtail-SWE supplies both, and its per-language table shows why the requirement matters: configuration-averaged means stretch from 48.1% on Clojure to 73.5% on Erlang (Table 2) while the headline is 81.52%. A benchmark that reports only a pooled number can conceal exactly that spread.

  2. What makes a passing patch correct? Look for test-sufficiency checks, task review and documented corrections. UTBoost’s erroneous-pass findings show why “the tests passed” cannot always be equated with “the issue was resolved.”

  3. How was contamination controlled? Check issue dates against known training cutoffs, web-access rules and whether restrictions were enforced at the network layer. For a static dataset, ask whether the tasks remain useful as models and training data change.

  4. Which complete configuration produced the score? Record the model, harness and attempt budget. Comparisons across languages should keep those conditions fixed, or state clearly when they do not. Pass@1 and Pass@3 results should not be collapsed into one readiness number.

  5. How was difficulty calibrated? Examine task types, repair procedures, ceiling effects and any harder subset. A high score on saturated tasks and a low score on refactoring-heavy tasks can both be valid within their own evaluations while answering different questions.

This checklist also clarifies where Longtail-SWE stands: the documentation burden is met, and what remains open is verification. The results are author-reported in an unreviewed preprint, and the authors themselves flag training-data overlap that cannot be ruled out, difficulty selection that may bias Qwen’s scores, and construction models that overlap with evaluated configurations. Those are verification gaps, not reasons to dismiss the dataset’s construction contribution.

Run a smoke evaluation before routing traffic

I would use Longtail-SWE as a prompt to design an internal evaluation, not as approval for a rollout. Start with representative issues from each language and repository class you actually intend to assign to an agent. Include feature development and bug-fixing work if those reflect your backlog, rather than assuming another benchmark’s task mix represents your environment.

Keep the evaluation controls explicit:

  • Preserve the exact model, harness and attempt budget for each run.
  • Set and enforce the network policy, then retain enough run information to audit unexpected retrieval.
  • Use date-based task selection when the model’s training cutoff is known; treat an unknown cutoff as a limitation of that control.
  • Review test sufficiency before counting a pass, and inspect apparent successes as well as failures.
  • Report results separately for each language instead of relying only on a pooled percentage. Longtail-SWE’s Table 2 is a working example of what that reporting looks like.

A strong result supports a limited trial on the kinds of tasks sampled. It does not establish universal performance across the language or future repository work. A weak result calls for checking the harness, tests and task representativeness before concluding that the model itself is unsuitable.

This is real evaluation work, and it falls disproportionately on polyglot teams that public benchmarks have served poorly. But skipping it does not remove the cost; it moves the cost into misrouted tasks, review and failed adoption attempts.

Longtail-SWE therefore settles a narrower point than its top score might suggest: quality-annotated issue-resolution tasks can be assembled across a broader set of underrepresented programming languages, with named configurations and per-language results attached. Its author-reported scores are useful research evidence, but the unreviewed status and the threats its own authors name, from training-data overlap to construction roles that overlap with evaluated models, keep them from being a deployment guarantee. Until a team has evidence on its own stack, the defensible decision is to treat Python leaderboard success as unvalidated elsewhere and run the smoke evaluation first.

References

Follow the links in the article for context. The supporting material is collected here for further reading.

  1. Longtail-SWEarxiv.orgAccessed
  2. UTBoostarxiv.orgAccessed
  3. SWE-Bench ProMaxarxiv.orgAccessed
  4. harness studyarxiv.orgAccessed
  5. SWE-Bench+arxiv.orgAccessed
  6. SWE-bench-Livearxiv.orgAccessed
  7. SWE-Bench Pro V2arxiv.orgAccessed
  8. SWE-Trail paperarxiv.orgAccessed
  9. Claw-SWE-Bencharxiv.orgAccessed

Join the discussion

Share a useful perspective or ask a question about this article.

Discussion guidelinesComments privacy