GPT-6.1 Sol vs GPT-6 Astra: When the 5x Cheaper Model Wins
OpenAI reports GPT-6.1 Sol matches Astra on coding at one-fifth the cost, but independent verification is lacking. Route agentic work to Sol and keep Astra for hard research.
The topic guide
Models & Research
Where architecture, training tricks, and eval methodology meet the marketing layer — separating durable progress in foundation models from leaderboard theater that quietly falls apart under load.
133 published articlesFollow this topic via RSS
Latest analysis
Start with these guides
Foundational reading and comparisons to help you get your bearings.
Chinese AI Models Compared: DeepSeek, Qwen, Kimi, Doubao, and Ernie
Compare DeepSeek, Qwen, Kimi, Doubao, ERNIE and GLM through dated benchmarks, license terms, context windows, API pricing and practical workload fit.
Open-Weight LLM Leaderboards 2026: Where DeepSeek, Qwen, and GLM Rank
DataLearner's June 2026 snapshot ranks GLM-5.2 seventh by HLE at 54.70 and places no Chinese flagship in the overall top three, undercutting launch-day claims.
Can You Serve LLMs on 2-Bit Weights? What Ultra-Low-Bit Quantization Costs
arXiv 2508.06753 reports 2-bit LLM serving with up to 7x speedups on Intel Xe2. For self-hosters, the binding cost is not memory but task-specific eval coverage to verify QAT-
Can You Trust LLM Confidence Scores? Verbalized vs Logprob Signals
A preprint reports that verbalized LLM confidence weakly tracks logprob signals, with instruction-tuned models showing worse calibration. Gate on verifiable internal signals.
What I cover
Every new foundation model arrives wrapped in a benchmark chart and a press release. This coverage exists because the chart almost never tells you what the model actually does — and the press release tells you even less. The interesting work is one layer down: the attention variant that changes the cost curve, the parameterization fix that makes scaling laws actually transfer, the eval protocol whose failure modes flip the standings once you change a decoding parameter or a codec. I cover that layer.
I treat open-weight and closed releases as the same story told from opposite ends of a distribution shift. A trillion-parameter mixture beating a dense competitor on one harness and losing on another is not a contradiction; it is evidence about what the harness measures. Training-efficiency claims, context-window claims, and reasoning-benchmark claims all get the same treatment — read the method section, find the assumption that was load-bearing, and report whether removing it changes the result. When a paper proposes a new safety mechanism, a new compression trick, or a new continual-learning split, I am interested in the assumptions and limitations behind the headline result.
The throughline is comparative and skeptical without being contrarian. Foundation-model research has become the field where the largest gap between published numbers and deployed behavior tends to live. Closing that gap — with reproduction notes, ablation reading, and honest accounting of what generalizes versus what was tuned into the eval — is the beat.
The archive
1–24 of 129 articles · Newest first
Which Languages Cost the Most Energy in LLM Inference?
A preprint reports LLM inference energy varies up to 179x by language, with Pashto costing far more than English. These author-reported findings suggest locale mix is a key, 1
Reviewing AI-Generated Code: Where 53.6K Human Edits Cluster
Author-reported DECODE data shows most AI code edits occur within 15 minutes of acceptance, with 31% of trajectories ending in removal, suggesting review should focus on the 1
Does Layer Skipping Actually Speed Up LLM Inference? A Rigor-Matched Audit
A rigor-matched audit finds layer skipping speeds LLM inference, but wall-clock rankings reverse when decision overhead is separated from pure generation cost.
Can You Trust LLM Confidence Scores? Verbalized vs Logprob Signals
A preprint reports that verbalized LLM confidence weakly tracks logprob signals, with instruction-tuned models showing worse calibration. Gate on verifiable internal signals.
Generative vs Cross-Encoder LLM Rerankers: Can Single-Pass Decoding Close the Gap?
SPD claims 28 ms listwise LLM reranking via single-pass decoding, but the evidence is one unreplicated preprint. Keep cross-encoders in production until independent benchmarks
mdlARC's 44% ARC-AGI-1 Claim: What the Small Budget Leaves Out
An author-reported 44% on ARC-AGI-1 public eval for about $0.67 of rented compute shows what small budgets can justify, and what still needs replication.
FP8 vs MXFP4 vs BF16: Why Your Quantized LLM Disagrees Across GPUs
FP8 and MXFP4 are umbrella specs, not single formats. A new preprint offers bit-exact conformance vectors to test quantized LLM portability across GPUs, exposing hidden format
Running a 104GB LLM on a 48GB Mac: What Expert Streaming Costs
Expert streaming lets 48GB Macs run 104GB MoE models by paging weights from SSD. This guide compares streaming to quantization and offload, analyzing workload fit and hardware
GPT-6 Astra on ARC-AGI-3: What the Agentic Score Actually Measures
GPT-6 Astra's ARC-AGI-3 scores vary by 97 points based on effort settings. Route agents by cost per solved task, not launch-day leaderboards.
Deepfake KYC Fraud: Tamper-Resilient Watermarks That Recover the Original Face
VeriFi preprint claims watermarks recover original faces after tampering, not just flag fakes. Robustness is untested against commercial tools and compression, so treat as a R
Detecting AI-Generated Audio: Why Decay Tails Betray Voice Clones
A new preprint shows AI audio leaks in decay tails via group delay. It offers a cheap, watermark-free filter for fraud screening, though hold-out accuracy is only 66.7%.
Reward Hacking Starts in the Verifier: Rule Checks vs LLM Judges for Math RL
Rule checkers miss format variants while LLM judges get hacked during RL. This failure map from arXiv 2505.22203 shows why hybrid verifiers and reward audits are now essential
Can LLMs Train on Their Own Problems? What Zero-Data Self-Play Changes
J-Zero claims zero-data self-play beats baselines by 8.0 points on unverifiable tasks, but judge verdict flips of 5.3 to 48.4 percent suggest the gains may be self-validated.
Why LLM Agent Benchmarks Move When the Harness Changes
A new preprint argues agent benchmark scores track the evaluation harness as much as the model. Teams should pin harness versions and ablate scaffold changes to avoid misat.
Can You Serve LLMs on 2-Bit Weights? What Ultra-Low-Bit Quantization Costs
arXiv 2508.06753 reports 2-bit LLM serving with up to 7x speedups on Intel Xe2. For self-hosters, the binding cost is not memory but task-specific eval coverage to verify QAT-
Do LLMs Still Need BPE? What RL-Trained Tokenizers Change
An ICML 2026 paper shows RL can learn token boundaries end-to-end, beating straight-through baselines at 100M parameters. For serving or fine-tuning, keep your frozen BPE or S
GLM-5.3-Flash vs Qwen3.8-Flash-Next: Which Budget LLM to Route To
GLM-5.3-Flash and Qwen3.8-Flash-Next lack confirmed pricing and independent evals. Hold your router, verify per-token costs, and run a 50-case domain holdout before switching.
Why Prompt Caching Can Change Model Outputs: A Prefix Invariance Audit
A new arXiv audit shows attention masks miss causality leaks in state-space models. Run a two-pass check before trusting prefix caches on hybrid stacks.
GDPR Deletion Requests vs LLM Weights: What Machine Unlearning Actually Removes
GDPR erasure demands hit LLM weights, but retraining is the only defensible fix. Behavior-level unlearning passes probes without proving data is gone, leaving compliance teams
Why Drift Monitors Confuse Covariate Shift With Concept Drift
A new preprint proposes CJSD, a two-discriminator test that separates covariate shift from concept drift. This distinction determines whether to retrain or recalibrate, fixing
AI-Generated Apps Look Right, but Do They Actually Work?
MobileForge shows AI apps compile but fail navigation. Shift review from screenshots to project-level state tests to ship reliable generated frontends.
Prompt Injection in 3D Scenes: The Attack Surface Multimodal Agents Ignore
arXiv 2602.07104 shows 3D object placement injects instructions into multimodal agents, bypassing text and image filters. Teams must gate modalities and scope actions to.
Why LLM Log Anomaly Detection Pages You for Nothing
arXiv 2608.17965 shows LLM log detectors are overconfident in wrong verdicts. Gate paging on calibrated confidence, not raw F1, to prevent alert fatigue and ignored critical.
DeepSeek v4 Flash Vision: Routing Images Without Verified Pricing
DeepSeek v4-flash-vision-exp lacks verified pricing and benchmarks. This guide prices Qwen-VL alternatives and outlines a fallback protocol for experimental vision endpoints.
