groundy

infrastructure & runtime

106 articles·rss

Top in infrastructure & runtime

infra

GLM-5.2: vLLM Int4 Drops MTP Without Patches, SGLang FP8/NVFP4 Keeps It

GLM-5.2's speculative decoder speeds decode, but vLLM int4 drops MTP without a community patch. SGLang FP8/NVFP4 keeps MTP intact. Format, not kernel speed, decides serving.

jul 10
infra

Serving DeepSeek on Azure: Compliance Without Owning the GPU Fleet

DeepSeek on Azure through Vercel's AI Gateway lets regulated teams route the model inside Microsoft's perimeter, turning a binary compliance ban into a per-token cost call.

jul 10
infra

Vercel Makes WAF Mitigated Traffic Free: Recompute Your Edge Cost Model

Vercel now waives CDN and bandwidth charges for WAF-mitigated traffic, removing the penalty for aggressive blocking and shifting the bottleneck to rule tuning.

infra

GLM 5.2 Hosting Compared: Vercel AI Gateway vs Self-Hosted vLLM

GLM-5.2 self-hosting on 8 B200s costs $1.74 per million output tokens at batch 50 but $15.68 for one request. Bursty loads make a managed gateway cheaper than idle GPU node.

infra

Cloudflare DMARC Management GA: What to Configure Before p=reject

Cloudflare DMARC Management is now generally available and free for Cloudflare DNS customers, but p=reject still breaks on SPF lookup limits and authentication misalignment.

infra

Claude Code Permissions vs OS Privilege Isolation: What the Gap Costs

Claude Code's per-tool prompts are consent controls, not isolation. A July 2026 arXiv preprint and MCP's prompt-injection bugs show why agent runtimes need real isolation.

infra

LLM Memory Without the RAM: What SSD-Backed Paging Actually Costs

TF-Engram pages LLM memory to SSD, moving the bottleneck from HBM to storage bandwidth, latency, and prefetch accuracy. It wins only when reads arrive before decode stalls.

infra

Triton Kernels Pass Tests but Run Slow: The GPU Kernel Eval Gap

A July 2026 preprint shows Triton and TileLang kernels can pass correctness checks yet run hundreds of times slower than library baselines, moving validation cost to adopters.

more in this beat1–50 of 98 · page 1 / 2
  1. jul 08infraVercel Edge Config: What Global Feature Flags Actually Cost at the Edge
  2. jul 08infraServerless GPU Inference on GCP: What the Cold Starts Actually Cost
  3. jul 08infraVercel In-Function Concurrency: What It Changes for Stateful Node.js
  4. jul 08infraRunning LLMs on AMD GPUs With ROCm: What Actually Works
  5. jul 07infraCloudflare Meerkat: What Globally Distributed Consensus Costs at the Edge
  6. jul 07infraVercel CDN Now Honors External Origin Cache-Control: Audit Your Headers
  7. jul 07infraAI Found Real Bugs in Cloudflare's Circl Crypto Library
  8. jul 07infraPruning RAG Context: What to Cut Before the LLM Sees It
  9. jul 07infraCloudflare's x402 Gateway: What Per-Request API Billing Actually Needs
  10. jun 29infraDoubao 2.1 Pro: What 180 Trillion Daily Tokens Means for Inference Infrastructure
  11. jun 29infraEvery CUDA Kernel Pays a Launch Tax: The Host-to-Device Walkthrough
  12. jun 28infraVercel Montreal Region: Audit Residency Before You Migrate
  13. jun 28infraGLM-5.2 on vLLM and Ascend: Open Weights Beyond NVIDIA
  14. jun 27infraHow Vercel Runs Its Own CDN in Front of Discourse: A Self-Dogfooding Case Study
  15. jun 27infraVercel Runtime Logs Surface CDN Cache Hits, Not the Eviction Cause
  16. jun 27infraMultimodal Knowledge Graph RAG vs Vector RAG: What MKG-RAG-Bench Shows
  17. jun 27infraVercel Observability Now Tracks Redirects and Rewrites Beside Function Errors
  18. jun 27infraCloudflare Workflows Saga Rollbacks: Compensating Actions in Serverless Orchestration
  19. jun 26infraStatic Corpus RAG: The Bible Case for Separating Churn from Algorithm Complexity
  20. jun 26infraVercel's KIKO Milano Black Friday Case Study: What the Scaling Claims Skip
  21. jun 26infraVercel Postgres vs Neon vs Supabase: When the Bundled DB Wins
  22. jun 26infraFine-Tuning a 20B LLM With RLHF on a 24GB GPU: What Fits
  23. jun 26infraVercel Flat Rate CDN Beta: Break-Even Math for Spiky Workloads, Tax for the Rest
  24. jun 25infraWhere DeepSeek Weights Actually Run on Vercel's AI Gateway
  25. jun 25infraVercel's Anti-Lock-In Pitch: What the Open-Source Bet Still Locks In
  26. jun 25infraVercel Adds Tag-Based CDN Cache Invalidation: Surrogate Keys at the Edge
  27. jun 25infraGLM 5.2 Fast on Vercel AI Gateway: What Routing Through Wafer Actually Buys
  28. jun 25infraVercel CDN Cache Tags vs Path Purging: When Tag Invalidation Wins
  29. jun 25infraPrisma Joins the Vercel Marketplace: The ORM Becomes the Database Vendor
  30. jun 25infraOpenAI on AWS Bedrock: Routing Math to Run Before You Move Traffic
  31. jun 24infraVercel's Function Observability: What Native Metrics Replace and What They Don't
  32. jun 24infraAWS Databases on the Vercel Marketplace: The Cross-Cloud Latency Tax
  33. jun 24infraTurso on the Vercel Marketplace: Edge SQLite vs the Serverless Connection Pool
  34. jun 23infraVercel on the AWS Marketplace: What the Listing Does to Procurement and Lock-In
  35. jun 23infraServing Cold MoE Models: CrossPool Disaggregates KV Cache and Weights
  36. jun 23infraVercel's In-Function Concurrency: What It Does to Cold Starts and Billing
  37. jun 23infraPoisoning a RAG Retriever: How Conflict-Aware Edits Inject False Knowledge
  38. jun 23infraVercel Raised Its CDN Origin Timeout to Two Minutes: What Breaks First
  39. jun 23infraGradio-Lite Runs Model Inference in the Browser via Pyodide, No Server
  40. jun 23infraCloudflare AI Gateway Adds Spend Limits to Cap the Runaway Inference Bill
  41. jun 23infraVercel Now Honors stale-if-error: Serving Stale Cache When the Origin Dies
  42. jun 22infraVercel's Manual CDN Purge API: Cache Control Without a Redeploy
  43. jun 22infraCloudflare Now Routes Public Traffic to Private Apps via DNS, No VPN
  44. jun 22infraGitHub's AI Capacity Crunch Pushes Microsoft to Rent AWS Compute
  45. jun 20infraCloudflare's Temporary Accounts Give AI Agents Disposable Credentials
  46. jun 20infraRunning Long-Context Agents on a 4-Bit KV Cache: Where Accuracy Breaks
  47. jun 19infraWhen LLM-Generated CUDA Kernels Pass Tests but Get the Math Wrong
  48. jun 18infraRunning GLM-5.2 at Home: SGLang, vLLM, Transformers, and KTransformers Setup Guide
  49. jun 15infraAWS Bedrock Now Requires Data Sharing for Mythos: The Self-Hosting Calculus
  50. jun 14infravLLM Cold Start Latency: Why Scale-to-Zero LLM Serving Stalls
about this beateditorial framing

Production AI runs on infrastructure that was never designed for it. Inference serving is a moving target as prefill and decode pull apart onto different hardware, KV caches spill into tiered storage, and collective communication libraries get rewritten to claw back bandwidth. Every benchmark win on synthetic workloads has to survive long-context synthesis, multi-tenant interference, and the unglamorous math of tokens-per-dollar before it counts.

The fabric underneath is just as contested. Vector databases are converging with the OLTP stack, serverless runtimes are quietly absorbing what connection poolers used to own, and overlay networks keep colliding with cloud-provider NAT and egress policy in ways that turn architecture diagrams into invoices. Storage density is outrunning rebuild windows, forcing erasure-coding choices that used to be theoretical. Cheaper-inference research keeps threatening the assumption that scale must mean GPU farms, while denser GPU farms keep proving it.

This beat covers that tension on the merits. We track serving architectures, networking and peering economics, retrieval and caching layers, GPU and storage hardware, and the cloud-account dependencies that quietly underwrite the whole stack. We compare vendor claims against published numbers, flag when a throughput headline hides a quality regression, and pay attention to the boring failure modes that take down platforms more often than the exciting ones do.