Serving Multimodal LLMs: Why Mixed Text and Vision Requests Stall GPUs

On an H100 serving with vLLM’s default token budget of 8192, a single prefill-only batch can consistently exceed 180 ms per iteration, according to a 2025 paper on adaptive GPU multiplexing, and every decode token queued behind that batch waits with it. That number is the mechanism behind a complaint platform engineers keep hitting: add image or video traffic to a self-hosted vision-language model, and text-only latency degrades even while GPUs look fully busy. A preprint accepted at ACM SIGOPS ATC 2026, “Rocks, Pebbles and Sand” (arXiv:2603.26498), gives the underlying cost model a memorable name and argues the fix is scheduling, not hardware. The practical finding worth taking away before any of the systems claims: if you run mixed-modality traffic through one uniform queue, you are assuming every request costs the same to admit to the GPU, and the evidence says that assumption is wrong by orders of magnitude.

This article maps that mechanism onto what you can actually check in a vLLM or SGLang deployment, and then asks the harder question: whether modality-aware scheduling is the right fix, or only part of one.

Why a video request is not a text request

LLM serving has two phases with very different shapes. Prefill processes the input prompt in one large parallel computation; decode then generates tokens one at a time, each step small but latency-sensitive. Text-only requests have modest prefills. Vision requests do not: images and especially video expand into large numbers of input tokens, and the model may spend significant time in a separate vision encoder before the language model even starts.

The RPS-Serve authors state the consequence directly: multimodal requests “differ by orders of magnitude in both time and memory,” which they map to an abstraction where “video requests behave like rocks, dominating GPU resources; image requests are pebbles, moderately heavy; and text-only requests are sand, lightweight yet latency-sensitive.” The paper names models such as InternVL-38B (and one it designates “Gemma-31B,” an unusual model name worth double-checking against the released artifact) as spending a significant portion of inference time in vision encoding for image and video inputs.

Their scheduling proposal follows from the cost model. RPS-Serve “estimates the GPU admission cost of each request using its prefill latency and memory footprint, and classifies requests into sand, pebbles, and rocks using lightweight clustering,” using prefill token count as a proxy for KV-cache footprint on the basis that cache size scales linearly with tokens. Nothing about that requires exotic infrastructure. It requires only that the scheduler stop treating a 30-second video clip and a one-line prompt as equivalent units of work.

How one heavy prefill stalls everything behind it

The stall is not mysterious once you look at how continuous batching works. Mainstream stacks iterate: each engine step runs a batch that may mix prefills of newly admitted requests with ongoing decodes. The adaptive multiplexing paper describes the failure mode precisely: “inserting long prefills into an ongoing decode batch synchronously couples the two phases, which can severely inflate TBT and lead to SLO violations.” TBT, time between tokens, is the metric your chat users feel. When a video prefill lands in the same iteration as a batch of streaming decodes, every user in that batch waits for the rock.

The same paper documents why this got worse with newer hardware. It reports that the saturation point of a 4096×4096 linear layer shifts from roughly 2048 tokens on A100 to roughly 8192 tokens on H100, and that vLLM’s default token budgets track this, 2048 on A100 and 8192 on H100. Bigger budgets are rational for throughput. But at an 8192-token budget, the authors of that paper measured prefill-only batches consistently exceeding 180 ms per iteration, and mixing such a batch with decode work inflates decode latency to a comparable level. A budget tuned for a text workload’s throughput becomes a tail-latency amplifier the moment vision prefills arrive.

The RPS-Serve paper frames the same tradeoff from the other side, noting that “monolithic execution with FCFS scheduling achieves high resource utilization and can perform better for models where LLM prefill dominates, but remains vulnerable to severe head-of-line blocking under heterogeneous workloads.” First-come-first-served is fair in the way a single-lane road is fair: everyone moves at the speed of the slowest vehicle ahead of them.

Measure before you re-architect

Every quantitative claim from the papers cited here is author-reported on a self-selected workload mix, so the first step is to reproduce the phenomenon in your own deployment rather than trust anyone’s multiplier. The measurements that matter, per the mechanism above:

• Per-modality prefill cost. Log prefill token counts and prefill latency split by request type: text-only, image, video. If vision prefills are not visibly heavier than text in your model and resolution range, the rest of this article does not apply to you.
• Per-modality TTFT and TBT. Time to first token and time between tokens, segmented the same way. The signature of a modality-induced stall is text-only TBT spiking in correlation with vision request arrivals, not with overall load.
• Prefill/decode split per iteration. How often do your batches mix a large prefill with in-flight decodes, and how long do those mixed iterations take? The 180 ms figure above is a reported measurement on specific hardware, not a universal constant; yours will differ.
• Queue depth by modality. If text requests accumulate behind video requests while GPU utilization reads high, you have a scheduling problem. If utilization is low and queues grow anyway, you have a capacity problem, and no scheduler in this article will fix it.

The papers establish that these stalls happen and why; the specific diagnostic sequence above is my operational reading of that mechanism, not a procedure any of the papers benchmarked. Treat it as a hypothesis to test against your own traces.

One config check worth doing this week

The cheapest actionable item in the entire evidence set is the token budget. If the 2025 paper’s figures are current, a deployment that moved from A100 to H100 and kept old settings, or copied a config tuned for a different GPU generation, may be running a budget that produces exactly the long prefill iterations the paper measured. Two caveats apply before you change anything. First, the 2048/8192 defaults come from a research paper’s description of vLLM, not from vendor documentation, so verify the actual defaults and flag names against the vLLM or SGLang docs for your version. Second, lowering the budget trades throughput for iteration latency; whether that trade helps depends on your traffic mix, which is why the measurement step comes first.

Four places the research says to intervene

Scheduling is one lever among several, and the recent systems literature is an argument about which lever matters most. All numbers below are author-reported, measured on each team’s own workloads and baselines; they are not comparable head-to-head.

Two things are worth noticing in this table. First, the problem generalizes beyond modality: AugServe gets its gains on augmented-LLM workloads, which suggests the underlying issue is request heterogeneity of any kind, not vision per se. Second, the interventions compose poorly as slogans but well as layers. POD-Attention works below the scheduler entirely, reducing the cost of the mixed batches that scheduling failed to prevent. Nothing in the cited research says you must choose exactly one.

The case against scheduling alone

RPS-Serve’s boldest positioning is that it achieves higher utilization “without requiring additional GPUs or disaggregation of inference stages.” Read against the rest of the table, that is a contested claim, not a settled one. The paper’s own evaluation carries the counter-evidence: the authors report around 50% higher normalized latency under vLLM than under ModServe, a disaggregated multimodal baseline, on InternVL-38B, where vision encoding dominates time to first token. On LLaVA-72B, where encoding overhead is negligible, the result flips, with around 50% lower normalized latency under vLLM. Whether disaggregation helps depends on where a model spends its prefill time. The largest single latency number in this table, Nexus’s 20× lower TTFT, comes from exactly the lever RPS-Serve declines to use: separating prefill from decode inside the GPU. ElasticMM’s authors attribute their 4.2× TTFT improvement to adapting parallelism across stages and modalities, not to queue classification, and they frame existing stacks such as vLLM as “primarily optimized for text-only workloads,” which is a competing system’s pitch, not a vendor admission.

My Read: modality-aware admission and stage disaggregation attack the same stall from different directions, and the evidence cannot tell you which one your deployment needs. If your traces show text decodes stalling whenever a video prefill runs, scheduling that keeps rocks out of decode-heavy iterations addresses the symptom directly. If they show prefill work itself starving for compute, no queue discipline creates FLOPs, and you are in disaggregation or capacity territory. The honest summary is that scheduling looks necessary under mixed traffic and may not be sufficient.

What you can do without new hardware

Between “do nothing” and “adopt an unreplicated preprint’s scheduler” sits a routing option that the evidence supports indirectly but strongly. If text-only requests are sand (lightweight, latency-sensitive) and vision requests are rocks and pebbles, then sending the two classes to different serving replicas or endpoints removes the head-of-line blocking without touching the scheduler, the kernels, or the GPU count. You already paid for the mechanism if you have more than one replica behind a load balancer. The cost is real: you lose the statistical multiplexing that lets one pool absorb the other’s idle capacity, and you now operate two capacity plans. For a deployment where text traffic carries user-facing latency SLOs and vision traffic is batch-ish or async, that trade is usually the right one. For one where vision traffic is the product, it may be backwards, and disaggregation or a parallelism-aware stack deserves the evaluation time instead.

What RPS-Serve does not prove

The limitations deserve equal billing with the mechanism. RPS-Serve is a single preprint with no independent replication, and its utilization claims are author-reported on the authors’ own workload mixes. The same is true of every multiplier in this article. No cited source establishes that a mainstream stack today exposes a modality-aware admission control you can simply enable; the token-budget figures cited here describe defaults as of a 2025 research paper and need verification against current docs. Whether any of these gains survive production traffic, with its long-tail prompt distributions and adversarial burst patterns, is an open question the preprints do not answer.

The decision, stated plainly

If mixed text and vision traffic is degrading your tail latency, the evidence supports a sequence, not a purchase. Measure per-modality prefill cost and per-modality TTFT/TBT first. Sanity-check your token budget against your GPU generation second, verifying against current vendor docs. Route text-only traffic away from vision traffic third, if your SLO structure allows it. Evaluate the heavier interventions, modality-aware scheduling, disaggregation, parallelism-aware execution, only with your own traces in hand, and treat RPS-Serve-style admission as promising and unproven until someone replicates it. The stall is real and well-explained. The best fix for your deployment is the one your own metrics can see.

Frequently Asked Questions

What is the cheapest configuration change to test for GPU stalls?

The cheapest actionable item in the entire evidence set is the token budget. If the 2025 paper’s figures are current, a deployment that moved from A100 to H100 and kept old settings, or copied a config tuned for a different GPU generation, may be running a budget that produces exactly the long prefill iterations the paper measured.

Can I separate text and vision traffic without changing the scheduler?

If text-only requests are sand (lightweight, latency-sensitive) and vision requests are rocks and pebbles, then sending the two classes to different serving replicas or endpoints removes the head-of-line blocking without touching the scheduler, the kernels, or the GPU count. You already paid for the mechanism if you have more than one replica behind a load balancer.