How to Evaluate LLM Long-Term Memory: Repeated Judging, Negative Controls
A September 2026 preprint argues that LLM memory benchmarks require repeated judging, negative controls, and ablation arms to distinguish real gains from judge noise.
The archive · Page 2 of 7
Models & Research
Where architecture, training tricks, and eval methodology meet the marketing layer — separating durable progress in foundation models from leaderboard theater that quietly falls apart under load.
The archive
25–48 of 159 articles · Newest first
How to Stop an LLM Agent From Repeating Failed Steps
The headline architecture still rests on one preprint's mechanistic simulations, with no live-model validation or independent replication.
GPT-6.1 Sol vs GPT-6 Astra: When the 5x Cheaper Model Wins
OpenAI reports GPT-6.1 Sol matches Astra on coding at one-fifth the cost, but independent verification is lacking. Route agentic work to Sol and keep Astra for hard research.
How to Evaluate an LLM for Mental Health Conversations: MentalHealthBench
OpenAI's MentalHealthBench offers a rubric-based alternative to refusal audits, though vendor-reported scores and grader bias limit independent verification.
2-Bit KV Cache Quantization: Memory You Save, Recall You Risk
2-bit KV cache quantization offers major memory savings but risks silent recall degradation. Evidence supports 4-bit defaults for accuracy-critical workloads.
Why LLM Agents Report Success on Tasks They Failed
Three 2026 preprints show LLM agents often report success despite incomplete coverage or stale state, requiring artifact-level verification gates to catch silent failures.
Which Languages Cost the Most Energy in LLM Inference?
A preprint reports LLM inference energy varies up to 179x by language, with Pashto costing far more than English. These author-reported findings suggest locale mix is a key, 1
Reviewing AI-Generated Code: Where 53.6K Human Edits Cluster
Author-reported DECODE data shows most AI code edits occur within 15 minutes of acceptance, with 31% of trajectories ending in removal, suggesting review should focus on the 1
Does Layer Skipping Actually Speed Up LLM Inference? A Rigor-Matched Audit
A rigor-matched audit finds layer skipping speeds LLM inference, but wall-clock rankings reverse when decision overhead is separated from pure generation cost.
Can You Trust LLM Confidence Scores? Verbalized vs Logprob Signals
A preprint reports that verbalized LLM confidence weakly tracks logprob signals, with instruction-tuned models showing worse calibration. Gate on verifiable internal signals.
Generative vs Cross-Encoder LLM Rerankers: Can Single-Pass Decoding Close the Gap?
SPD claims 28 ms listwise LLM reranking via single-pass decoding, but the evidence is one unreplicated preprint. Keep cross-encoders in production until independent benchmarks
mdlARC's 44% ARC-AGI-1 Claim: What the Small Budget Leaves Out
An author-reported 44% on ARC-AGI-1 public eval for about $0.67 of rented compute shows what small budgets can justify, and what still needs replication.
FP8 vs MXFP4 vs BF16: Why Your Quantized LLM Disagrees Across GPUs
FP8 and MXFP4 are umbrella specs, not single formats. A new preprint offers bit-exact conformance vectors to test quantized LLM portability across GPUs, exposing hidden format
Running a 104GB LLM on a 48GB Mac: What Expert Streaming Costs
Expert streaming lets 48GB Macs run 104GB MoE models by paging weights from SSD. This guide compares streaming to quantization and offload, analyzing workload fit and hardware
GPT-6 Astra on ARC-AGI-3: What the Agentic Score Actually Measures
GPT-6 Astra's ARC-AGI-3 scores vary by 97 points based on effort settings. Route agents by cost per solved task, not launch-day leaderboards.
Deepfake KYC Fraud: Tamper-Resilient Watermarks That Recover the Original Face
VeriFi preprint claims watermarks recover original faces after tampering, not just flag fakes. Robustness is untested against commercial tools and compression, so treat as a R
Detecting AI-Generated Audio: Why Decay Tails Betray Voice Clones
A new preprint shows AI audio leaks in decay tails via group delay. It offers a cheap, watermark-free filter for fraud screening, though hold-out accuracy is only 66.7%.
Reward Hacking Starts in the Verifier: Rule Checks vs LLM Judges for Math RL
Rule checkers miss format variants while LLM judges get hacked during RL. This failure map from arXiv 2505.22203 shows why hybrid verifiers and reward audits are now essential
Can LLMs Train on Their Own Problems? What Zero-Data Self-Play Changes
J-Zero claims zero-data self-play beats baselines by 8.0 points on unverifiable tasks, but judge verdict flips of 5.3 to 48.4 percent suggest the gains may be self-validated.
Why LLM Agent Benchmarks Move When the Harness Changes
A new preprint argues agent benchmark scores track the evaluation harness as much as the model. Teams should pin harness versions and ablate scaffold changes to avoid misat.
Can You Serve LLMs on 2-Bit Weights? What Ultra-Low-Bit Quantization Costs
arXiv 2508.06753 reports 2-bit LLM serving with up to 7x speedups on Intel Xe2. For self-hosters, the binding cost is not memory but task-specific eval coverage to verify QAT-
Do LLMs Still Need BPE? What RL-Trained Tokenizers Change
An ICML 2026 paper shows RL can learn token boundaries end-to-end, beating straight-through baselines at 100M parameters. For serving or fine-tuning, keep your frozen BPE or S
GLM-5.3-Flash vs Qwen3.8-Flash-Next: Which Budget LLM to Route To
GLM-5.3-Flash and Qwen3.8-Flash-Next lack confirmed pricing and independent evals. Hold your router, verify per-token costs, and run a 50-case domain holdout before switching.
Why Prompt Caching Can Change Model Outputs: A Prefix Invariance Audit
A new arXiv audit shows attention masks miss causality leaks in state-space models. Run a two-pass check before trusting prefix caches on hybrid stacks.
