Cloud Run GPUs became generally available on Google Cloud in mid-2026, and the first thing the numbers make clear is that scale-to-zero GPU inference is not a cheaper version of dedicated serving: it is a different shape of cost. A Cloud Run GPU service can wake from zero in roughly five seconds for infrastructure and driver setup, but loading a 4-billion-parameter model like Gemma 3 pushes the observed Time-to-First-Token to about 19 seconds, according to Google’s GA announcement. That delay is not an implementation detail; it is the price you trade against the always-on instance tax. For the right workload the trade is excellent. For a synchronous chatbot it is disqualifying.
What Does a 19-Second Cold Start Actually Cost?
The cold start is not one delay. Google’s cold-start guide breaks it into four phases: infrastructure provisioning at around five seconds, container image streaming at one to two seconds, engine initialization at five to fifteen seconds, and finally model loading and VRAM transfer, which is where the big models spend most of their time. The five-second figure you see in announcements covers only the first phase. The 19-second TTFT for Gemma 3 4B is the end-to-end reality a user experiences, and it grows with model size because the final phase dominates.
That 19 seconds has to be valued in dollars, not milliseconds. Every cold start is a block of GPU time before revenue-bearing inference begins. If your traffic is chat-shaped, with a user waiting on the other end of a WebSocket, those 19 seconds are a churn event. If your traffic is batch-shaped, a queued job that finishes in seconds or minutes can absorb a 19-second preamble without changing the business outcome. The same number is either fatal or invisible depending on the architecture around it.
This is why the “serverless is cheaper” headline is misleading. Cloud Run GPU services scale to zero when idle, according to Google’s GA announcement, but a serverless instance is not billed only for inference milliseconds. You are paying for cold starts plus whatever idle warm window the autoscaler keeps between bursts.
Which GPU and Shape Should You Pick on Cloud Run?
Cloud Run offers two GPU tiers. The NVIDIA L4 has 24 GB of VRAM and requires at least four vCPU and 16 GiB of memory. The RTX PRO 6000 Blackwell has 96 GB of VRAM and requires at least 20 vCPU and 80 GiB of memory, according to Google’s GPU configuration docs. The choice is less about speed and more about what fits in memory without paging.
For a 4B to 7B parameter model at FP16, the L4 is already marginal on capacity but sufficient on throughput. For 70B-class models or anything running with multiple adapters, the Blackwell is the only viable option. The minimum CPU and memory quotas matter because a GPU instance is billed as a whole instance, not as a fractional accelerator. Packing a tiny model onto a Blackwell because you want “more GPU” wastes the CPU and RAM floor.
A pragmatic starting point is to match the GPU to the largest model you must serve, then measure whether inference throughput saturates the accelerator. If the model is small and request concurrency is low, the L4 is the cost-optimized default. If the model is large or you need high batch concurrency, Blackwell is the only shape that avoids model-swap hell.
When Does Scale-to-Zero Beat Always-On?
GMI Cloud’s analysis puts the crossover lower than the common rule of thumb: below roughly 40 to 50 percent sustained utilization, serverless per-token billing is cheaper than dedicated GPU-hour billing, and above that threshold dedicated infrastructure consistently wins on cost per token. Below that range, the idle waste of a dedicated instance outweighs the cold-start penalty of serverless. Above it, you are paying for enough cold-start minutes per inference hour that an always-on instance becomes cheaper.
The exact crossover depends on your request pattern, not just your average utilization. Spiky traffic with long gaps is ideal for scale-to-zero because idle periods cost nothing once the instance has scaled down. Steady traffic with short gaps is worse, because repeated idle windows and cold starts can add up to an effective always-on bill. A service that receives one request every few minutes may spend most of its budget on warm-up and idle time rather than inference.
You can model this for your own workload. Estimate the number of cold starts per hour, multiply by the average cold-start duration, and compare that wasted GPU time to the hourly cost of a dedicated Vertex AI endpoint or G2 instance. The GMI Cloud comparison treats cold-start time as lost utilization. The calculation is crude but directionally correct: when cold starts consume less than roughly 40 to 50 percent of your potential compute window, serverless wins.
How Should You Architect Around Cold Starts?
The teams that make serverless GPU work do not accept the 19-second penalty at face value. They collapse requests, tune concurrency, and keep models resident where economics allow.
Request collapsing is the most effective pattern. Instead of waking one instance per request, queue short jobs and dispatch them to a single warm instance. Cloud Run’s autoscaler tries to land bursts on warm instances before starting a cold path, so if you structure your traffic in batches rather than singletons, the warm instance absorbs the burst and rarely triggers a new cold start.
Another production pattern is to disable just-in-time compilation in the inference engine, trading a small throughput loss for faster engine initialization. On Cloud Run the engine initialization phase is the one this affects most. The throughput cost is real, but for spiky workloads it can be smaller than the cost of repeated long wakes.
Warm-call strategies are also common but should be treated as a cost, not a fix. Scheduled keep-alive requests prevent scale-to-zero, which means you are effectively running a dedicated instance through the back door. Use them only for a small pool of critical models or during known peak windows.
What About Vertex AI and the Other Clouds?
Cloud Run is not the only serverless GPU option on GCP. Vertex AI offers dedicated endpoints that stay warm as long as you pay for them, and managed endpoints that handle autoscaling with their own idle policies. The comparison is less about features and more about who owns the capacity decision.
A Vertex AI dedicated endpoint is the right default when utilization is steady and predictable. You pay for the instance, but you eliminate cold starts and you control batching, scaling, and regional placement. Cloud Run GPU is the right default when traffic is erratic, when you can tolerate wake latency, and when you want the simplest path from container to endpoint without managing a serving platform.
Across clouds the same trade-off repeats with different timeouts. A 2026 MarkAI benchmark reports that AWS Bedrock scales to zero after five minutes of inactivity. A shorter idle timeout means cold starts are more frequent for the same request pattern than on services that keep instances warm longer. The comparison is not symmetrical either: Bedrock is a managed model service, while Cloud Run lets you bring any containerized inference engine.
What Should You Actually Do?
Start by classifying your traffic shape. If users are waiting synchronously for each response, serverless GPU is the wrong tool. If jobs are asynchronous, batchable, or queue-driven, scale-to-zero can cut idle spend dramatically.
Next, measure cold-start time for your actual model, not the smallest benchmark you can find. Google’s published figure of 19 seconds is for Gemma 3 4B. Larger models will be slower because model loading dominates. If your deployment uses a 70B model, plan for a cold start measured in tens of seconds and design your UX or queue depth around it.
Then run the utilization math. Take your expected requests per hour, your average inference duration, and your cold-start duration. Compute the ratio of cold-start time to productive inference time. If that ratio sits below the roughly 40 to 50 percent threshold GMI Cloud identifies, serverless is likely cheaper. If it is well above that range, price a dedicated instance.
Finally, design for the autoscaler. Keep concurrency high enough to absorb bursts, use startup probes so traffic does not hit an instance before the model is loaded, and avoid the temptation to keep every model warm all day.
Scale-to-zero is a cost discipline, not a deployment convenience. The teams that treat it as a scheduling and queuing problem will come out ahead. The teams that treat it as a drop-in replacement for a dedicated GPU will get an unpleasant bill and slower responses.
Frequently Asked Questions
Does Cloud Run GPU bill per inference request, or per instance uptime?
Cloud Run GPU uses instance-based billing, so you pay for the whole instance while it exists, not just the milliseconds spent on tokens. That includes the roughly 15-minute idle warm window after traffic stops, plus the full duration of every cold start. A service that scales to zero is therefore cheaper only if its idle and warm-up time add up to less than the always-on alternative.
How does Cloud Run’s 15-minute idle timeout change the math against AWS Bedrock?
Bedrock scales to zero after five minutes of inactivity, so a workload with gaps of four to fifteen minutes between requests is more likely to cold-start on Bedrock than on Cloud Run. For the same traffic pattern, Cloud Run keeps the instance warm longer, which reduces wake frequency but also increases billed idle time. The cheaper platform depends on whether your gaps usually fall below or above each timeout.
What concrete knob can reduce engine initialization time in production?
Elastic’s production pattern on Cloud Run sets enforce_eager=True in vLLM to skip just-in-time CUDA graph compilation. That drops cold starts from multiple minutes to under a minute at a small throughput cost. The body mentions disabling compilation as a pattern, but the specific enforce_eager flag and the minute-level outcome are the operational detail a practitioner needs.
When does request collapsing stop helping?
Request collapsing works only while bursts arrive within the warm window and fit on one instance’s concurrency and memory budget. Once inter-arrival gaps exceed the 15-minute idle timeout, or once queued jobs overflow VRAM or batch latency limits, the autoscaler must start a cold path anyway. At that point collapsing adds queue latency without avoiding the wake-up cost.
What would make the 40-50 percent utilization crossover shift?
The crossover moves if Google changes GPU per-hour pricing, shortens the idle timeout, or reduces model-loading time. A shorter warm window raises cold-start frequency and pushes the break-even utilization lower, while faster model loads raise the threshold. The rule of thumb is anchored to 2026 pricing and wake latency, so re-run the calculation after each pricing or cold-start benchmark update.
