Meituan's General 365 Benchmark: Top Models All Score Under 63%
Meituan's General 365 benchmark caps Gemini 3 Pro at 62.8% and most of 26 models below 60%, exposing how saturated public leaderboards inflate reasoning scores.
The archive · Page 5 of 6
Models & Research
Where architecture, training tricks, and eval methodology meet the marketing layer — separating durable progress in foundation models from leaderboard theater that quietly falls apart under load.
The archive
97–120 of 139 articles · Newest first
LLM Surrogates in A/B Tests: The 39% Recovery Gap and the Silent Bias Risk
A 2026 preprint proposes LLMs as A/B test proxies for human subjects. Raw outputs captured 39% of the treatment effect, and surrogacy bias does not average out.
LLM Token Pricing vs Compute Cost: What the Tokenomics Math Shows
Token pricing decouples from GPU cost via cached discounts, reasoning markups, and per-model multipliers. No single per-token rate works as a budget unit for real workloads.
Do LLM Judges Favor Their Own Output? A Sanity Check on Self-Preference
An LLM judge's self-preference claim holds only if the study fixed generation quality. Without that control, the bias number conflates real preference with a quality gap.
Can AI Write CAD Programs? CADBench Measures the Gap
CADBench evaluates eleven systems on 18,000 CAD samples and finds vision-language models unreliable, a gap SWE-bench and LiveCodeBench were never built to measure.
ByteDance's Doubao 2.1 Pro vs GPT-5.5: Reading Self-Reported Benchmarks
ByteDance's Doubao-Seed-2.1 Pro launch deck claims parity with GPT-5.5 on benchmarks ByteDance graded itself, and no independent source has reproduced the scores yet.
Can RoboSSM's State-Space Backbone Replace Transformer Imitation Policies?
The RoboSSM preprint swaps the transformer backbone of in-context robot imitation for a Longhorn state-space model and claims LIBERO gains. The full paper is still pending.
Pruning Experts to Shrink MoE Models: Does Attribution-Guided Compression Beat Magnitude?
Routing architecture decides whether MoE experts can be pruned safely: hard routers keep calibration per expert, soft routers need an aggregate guard.
GLM-5.2 vs Kimi K2.7 Code: Two Open-Weight Bets on Agentic Coding
GLM-5.2 and Kimi K2.7 Code shipped a day apart as open-weight coding models. One leads the open-weights index; the other wins on price and a claimed token-efficiency gain.
How Linear Is a Transformer Feed-Forward Block? A New Test Says It's Learned, Not Built In
Per-block linear recoverability in transformer FFNs swings from near-linear to strongly nonlinear between layers, so compression tools must probe each block per checkpoint.
GLM-5.2 Benchmarks: What 62.1% SWE-bench Pro and 99.2% AIME Actually Mean
Every GLM-5.2 benchmark figure, from 62.1% SWE-bench Pro to 99.2% AIME, traces to Zhipu's own evaluations. This guide separates vendor-run scores from external evals.
GLM-5.2 on Terminal-Bench 2.1: Strengths, Gaps, and How to Route Real Coding Tasks
GLM-5.2 scores 81.0 on Terminal-Bench 2.1, trails Claude Opus 4.8 (85.0) on shell tasks, but wins on 1M-token monorepo context. Here is how to route tasks.
GLM-5.2 vs Claude Opus 4.8: Open-Weight Coding at Frontier Pricing
GLM-5.2 posts 62.1% on SWE-bench Pro and 81.0 on Terminal-Bench 2.1, four points behind Opus 4.8. MIT weights are self-hostable; flat plan starts at $18/month.
GLM-5.2's 753B MoE Costs More to Self-Host Than the MIT License Suggests
GLM-5.2 ships 753B parameters under MIT with strong coding benchmarks, but the full MoE weight load makes self-hosting far heavier than the license terms imply.
STAR Replaces Scalar Reward in Text-to-Image RL with Attention-Derived Spatial Maps
STAR uses attention-derived spatial maps to replace uniform scalar reward in diffusion RL, shifting the bottleneck from preference pair volume to reward localization.
Can Editing One Neuron Fix LLM Repetition Loops?
A June 2026 preprint localizes Gemma 4 repetition loops to a few MLP neurons and removes them with a one-time weight edit while benchmark scores hold.
Claude Fable 5 Benchmarks: What FrontierCode, CursorBench, and ViBench Show
Claude Fable 5 claims top benchmark scores. Verified data shows every model below 14% on FrontierCode Diamond, and partner scores lack public methodology.
Does Attribution Patching Lie? A Fix for a Common Interpretability Shortcut
A June 2026 paper traces attribution patching's errors to downstream non-linearities and proposes a Hessian-vector-product correction that costs one extra backward pass.
Why Pruning a Model Can Raise Its Out-of-Distribution Accuracy
Task-aware layer pruning removes distortion-amplifying layers and improves out-of-distribution accuracy, which means standard in-distribution benchmarks miss the real effect.
Do Unified Multimodal Models Actually Interleave Understanding and Generation?
IMUG-Bench tests whether unified multimodal models can alternate between understanding and generation in one context, exposing gaps that separate benchmarks conceal.
How LLMs Track Who Did What: The Entity Rebinding Circuit
New research isolates a compact attention-head circuit for entity rebinding in Gemma and Llama, showing tracking failures stem from a binding step, not context length.
Claude Fable 5 vs Opus 4.8: When 2x Pricing Is Worth It
Claude Fable 5 prices at $10/$50 per million tokens, 2x Opus 4.8. Frontier research, long-context agents, and molecule design clear the bar. Standard coding does not.
Claude Mythos 5 Access Rules: Who Gets Project Glasswing and Why
Claude Mythos 5 shares Fable 5's architecture but with safeguards lifted in select areas. Access requires Project Glasswing approval or a biology research designation.
Fable 5 Distillation Protection: How Anthropic Blocks Model Copying
Claude Fable 5 ships with distillation protection to prevent capability extraction. A first-principles look at what it is, how it works, and why API consumers should care.
