Accelerate vs Megatron Core: The Model Size Curve for Distributed Training
Accelerate simplifies FSDP and DeepSpeed for 7B to 70B models. Megatron Core delivers higher MFU for 400B+ training. The ND-Parallel feature remains unverified.
The archive · Page 3 of 7
Infrastructure & Runtime
The serving stack, network fabric, and cloud-account substrate beneath production AI, where every throughput claim collides with rebuild windows, egress invoices, and control-plane risk.
The archive
49–72 of 145 articles · Newest first
Why cgroups, not permission prompts, bound AI agent CPU and memory
AgentCgroup shows tool calls drive 15.4x memory spikes. Framework prompts gate actions but do not limit CPU or memory. Wrap agent process trees in cgroups to prevent OOM.
vLLM Configs Shift Energy, Latency, and Accuracy: A 9,000-Run Study
A 9,000-run study shows vLLM attention kernels and prefix caching shift energy, latency, and accuracy. Benchmark configs per workload before production.
AWS Estimated Billing Was Off by $1.7B: Reconciling Actual Cloud Spend
A $1.7 billion AWS estimated billing error exposes why console projections are direction indicators, not bookable numbers. Teams must reconcile against the Cost and Usage.
Cloudflare Attribution vs Custom Logs: The Per-Path AI Crawler Decision
Cloudflare Attribution packages AI crawler traffic as business insights, but rollups hide the per-path decisions operators actually need. Compare vendor dashboards against.
Spectral Compute CUDA Translation: vLLM Procurement vs Porting Cost
Spectral Compute proposes CUDA binary translation for AMD and Intel GPUs, but vLLM already supports native HIP. The real question is whether translation overhead beats the.
Running MiniCPM-V-4.6 on Fermi: What 6 GB of VRAM Forces
MiniCPM-V-4.6 runs on a 2011 Fermi GPU with 6 GB VRAM. The study shows software staging recovers multimodal inference, shifting the constraint from hardware procurement to.
LLM Inference Without a GPU: Pure CPU vs Hybrid CPU-GPU Scheduling
ATSInfer schedules LLM tensors across CPU and integrated GPU at tensor granularity, reporting up to 3.29× decode speedup where pure-CPU inference is memory-bandwidth-bound.
RL-Found CUDA Kernels Beat cuBLAS: Kernel Tuning Shifts to Reward Design
CUDA-L2 used reinforcement learning over 1,000 kernel configurations to beat cuBLAS by 19.2% on HGEMM, shifting the scarce kernel skill from hand-tuning to reward design.
pgvector vs Pinecone vs Qdrant in 2026: How to Pick a Vector Database
pgvector, Pinecone, and Qdrant split on deployment, not on which ANN index is fastest. The call is where vectors live, who runs the cluster, and what filtered search costs.
Ollama vs LM Studio: Picking a Local LLM Runtime in 2026
Both wrap llama.cpp, so Ollama vs LM Studio is a headless MIT daemon versus a proprietary desktop GUI, with one ceiling: neither is a production serving engine.
Beyond Quantization: LLM Efficiency Is Now a Memory-Bandwidth Problem
A new survey reframes LLM efficiency as a memory-bandwidth co-design problem, pushing 2027 fleet sizing toward HBM capacity over peak GPU FLOPS.
How Sparse LLM Weights Cut GPU Inference Cost Without Quantization
A three-layer sparse matmul kernel achieves 1.64x kernel-level and 1.41x end-to-end speedups, making moderate unstructured sparsity a viable third optimization lever.
GLM-5.2: vLLM Int4 Drops MTP Without Patches, SGLang FP8/NVFP4 Keeps It
GLM-5.2's speculative decoder speeds decode, but vLLM int4 drops MTP without a community patch. SGLang FP8/NVFP4 keeps MTP intact. Format, not kernel speed, decides serving.
Serving DeepSeek on Azure: Compliance Without Owning the GPU Fleet
DeepSeek on Azure through Vercel's AI Gateway lets regulated teams route the model inside Microsoft's perimeter, turning a binary compliance ban into a per-token cost call.
Vercel Makes WAF Mitigated Traffic Free: Recompute Your Edge Cost Model
Vercel now waives CDN and bandwidth charges for WAF-mitigated traffic, removing the penalty for aggressive blocking and shifting the bottleneck to rule tuning.
GLM 5.2 Hosting Compared: Vercel AI Gateway vs Self-Hosted vLLM
GLM-5.2 self-hosting on 8 B200s costs $1.74 per million output tokens at batch 50 but $15.68 for one request. Bursty loads make a managed gateway cheaper than idle GPU node.
Cloudflare DMARC Management GA: What to Configure Before p=reject
Cloudflare DMARC Management is now generally available and free for Cloudflare DNS customers, but p=reject still breaks on SPF lookup limits and authentication misalignment.
Claude Code Permissions vs OS Privilege Isolation: What the Gap Costs
Claude Code's per-tool prompts are consent controls, not isolation. A July 2026 arXiv preprint and MCP's prompt-injection bugs show why agent runtimes need real isolation.
LLM Memory Without the RAM: What SSD-Backed Paging Actually Costs
TF-Engram pages LLM memory to SSD, moving the bottleneck from HBM to storage bandwidth, latency, and prefetch accuracy. It wins only when reads arrive before decode stalls.
Triton Kernels Pass Tests but Run Slow: The GPU Kernel Eval Gap
A July 2026 preprint shows Triton and TileLang kernels can pass correctness checks yet run hundreds of times slower than library baselines, moving validation cost to adopters.
Vercel Edge Config: What Global Feature Flags Actually Cost at the Edge
Vercel Edge Config reads feature flags in under 15ms, but its 10-second global write window means flipped flags can still route traffic to disabled regions.
Serverless GPU Inference on GCP: What the Cold Starts Actually Cost
Cloud Run GPU's 19-second cold start for Gemma 3 4B means scale-to-zero beats dedicated GPUs only for spiky, batch workloads below roughly 40 to 50 percent utilization.
Vercel In-Function Concurrency: What It Changes for Stateful Node.js
Vercel's in-function concurrency lets one instance run multiple Node.js or Python requests, cutting idle billing and cold starts but forcing handling of shared state, leaks.