Running RAG on a Snapdragon NPU: The On-Device Retrieval Tradeoff
End-to-end RAG on the Snapdragon X Elite Hexagon NPU delivers 4x lower latency and 4x less energy than CPU with no quality loss, but soldered memory caps your index size.
The archive · Page 6 of 7
Infrastructure & Runtime
The serving stack, network fabric, and cloud-account substrate beneath production AI, where every throughput claim collides with rebuild windows, egress invoices, and control-plane risk.
The archive
121–144 of 145 articles · Newest first
GraphRAG vs VectorRAG: Does the Graph Index Earn Its Cost?
A Samsung preprint finds vector retrieval matches GraphRAG on QA tasks at a fraction of the indexing cost, shifting the burden of proof to teams building graph pipelines.
MiniMax M3 Ships 1M Context and Desktop Control as Open Weights
MiniMax M3 promises open weights with 1M-token context and frontier coding, but BenchLM ranks it #29 overall and #69 on multimodal. Teams need independent verification.
DeepSeek-V4 FlashMemory: Sparse Attention for Million-Token Context
FlashMemory's learned index compresses DeepSeek-V4's KV cache to 13.5% of baseline at parity accuracy. The project is suspended; per-suite recall breakdowns are not published.
Is Cloudflare's Bot Traffic Surge Real? The Measurement Dispute
Cloudflare claims a 15x bot surge using a classifier that flags privacy browsers as bots. Audit your own logs before trusting the numbers behind Pay-Per-Crawl.
Indexing Images for RAG: kapa.ai's Approach to Multimodal Retrieval
kapa.ai's data shows indexing image captions at ingestion adds 1-6% query overhead versus 27-51% for raw query-time vision, shifting recall risk to caption fidelity.
DGX Spark: Budget memory, bandwidth and workload together
DGX Spark's 128 GB configuration offers model capacity without guaranteeing interactive throughput. Memory capacity, memory bandwidth, model architecture, precision, and concurrency all belong in the same decision.
Pod-Level Remote Attestation in Kubernetes: Confidential Workloads on dstack
dstack-capsule binds pod identity into Intel TDX hardware quotes, enabling multi-pod confidential VMs without the per-VM density tax of Confidential Containers.
Generating GPU Kernels for Moore Threads Silicon: Can LLMs Break CUDA Lock-In?
MusaCoder trains a 9B model to emit native GPU kernels for Moore Threads' MUSA architecture, claiming parity with frontier models on vendor-controlled benchmarks.
Microsoft's Azure Linux Goes General-Purpose: The Container Base-Image Play
Microsoft's Azure Linux 4.0 extends the internal CBL-Mariner into a Fedora-based server OS for VMs. Preview gaps remain, and AKS teams should test now but wait for GA.
Cloudflare Acquires VoidZero, the Company Behind Vite's Rust Toolchain
Cloudflare acquired VoidZero, putting Vite, Rolldown, and Oxc maintainers on a deploy-target vendor's payroll. MIT licensing stays. Roadmap neutrality is the open question.
Tesla V100 local LLM value depends on software support
A Tesla V100 can still run compatible CUDA workloads, but its value as a local LLM purchase depends on a specific workload, a compatible software build and safe physical integration.
Why LLMs Still Botch Kubernetes Manifests: The Training-Data Gap
A 1.5B-parameter model hits 91.5% on Kubernetes YAML generation, but the remaining failures are syntactically valid manifests that deploy and quietly violate cluster intent.
Gemma 4 31B on Cloud TPU vs GPU: The Serving Cost Crossover Point
TPU v6e Flex-start delivers 308M tokens per dollar for Gemma 4 31B prefill, undercutting H100 rates for open-weight serving, but production decode costs remain unquantified.
ObjectCache Moves KV Reuse to S3-Class Storage: Why Layerwise Retrieval Beats Full-Prefix Cache Hits
ObjectCache retrieves KV cache per-layer from S3, adding 5.6% TTFT at 64K context but 56-75 ms at 4K. Long-context deployments where DRAM is the bottleneck benefit most.
vLLM 0.21 Makes Prefill-Decode Disaggregation Actually Practical
vLLM v0.21 reportedly adds bi-directional KV cache transfers between prefill and decode nodes, making P/D ratios dynamic and requiring new NIXL transfer telemetry.
OpenRAG: The Open-Source RAG Platform Challenging Pinecone
OpenRAG combines Langflow, OpenSearch, and Docling into a single deployable RAG platform. Here's how it compares to managed services like Pinecone.
MLX vs llama.cpp on Apple Silicon: Which Runtime to Use for Local LLM Inference
MLX and llama.cpp both run quantized LLMs on Apple Silicon unified memory. Here is what each project documents, and how to measure which wins on your Mac.
When prefill and decode disaggregation helps LLM serving
Prefill and decode disaggregation lets LLM serving separate prompt processing from token generation. Its value depends on latency targets, KV transfer, and worker utilization.
Google LiteRT: Running LLMs on Your Phone Without the Cloud
Google's LiteRT (formerly TensorFlow Lite) powers on-device GenAI on Android, Chrome, and Pixel. What developers need to know about private, offline inference.
BitNet b1.58 makes CPU inference an option for specific models
BitNet b1.58 makes CPU inference an option for specific models. It does not establish that GPU farms are obsolete, or that a full-precision model can be replaced without evaluating the task.
WebAssembly AI: Running Models in the Browser
How WebAssembly, WebGPU, and frameworks like Transformers.js and WebLLM run AI models in the browser: the real performance trade-offs and when to use them.
Tailscale Peer Relays are a tailnet fallback when direct paths fail
Tailscale Peer Relays let a tailnet use an explicitly configured device as a relay when direct connectivity is unavailable.
DNS Persist 01: how persistent ACME validation works
DNS Persist 01 binds a persistent DNS authorization to a certificate authority and an ACME account.
