Situation

Want To Deploy vLLM On Cloud Run To Serve An LLM. Two Paths: GPU Mode (For Real Service) Or CPU Mode (For Tiny Model / Architecture Test Only). Both Fully Tested End To End.

Result First:

GPU Is About 19x Faster Than CPU. CPU Mode Works, But Don’t Expect Speed.

How To (GPU Mode)

Requirement:

First-Time GPU Deploy Get A Default Quota (3 GPU / Project / Region).

Deploy With This Command:

gcloud run deploy vllm-gpu-service \
  --image=us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:gemma4 \
  --region=asia-southeast1 \
  --cpu=8 \
  --memory=32Gi \
  --gpu=1 \
  --gpu-type=nvidia-l4 \
  --no-gpu-zonal-redundancy \
  --no-cpu-throttling \
  --execution-environment=gen2 \
  --concurrency=4 \
  --max-instances=1 \
  --timeout=600 \
  --port=8080 \
  --startup-probe=tcpSocket.port=8080,initialDelaySeconds=60,timeoutSeconds=240,failureThreshold=2,periodSeconds=240 \
  --no-allow-unauthenticated \
  --args="vllm,serve,google/gemma-4-E2B-it,--enforce-eager,--enable-chunked-prefill,--enable-prefix-caching,--generation-config=auto,--enable-auto-tool-choice,--tool-call-parser=gemma4,--reasoning-parser=gemma4,--dtype=bfloat16,--max-num-seqs=64,--gpu-memory-utilization=0.9,--tensor-parallel-size=1,--port=8080,--host=0.0.0.0"

Test It:

$ curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
  https://vllm-gpu-service-xxx.asia-southeast1.run.app/v1/chat/completions \
  -d '{"model":"google/gemma-4-E2B-it","messages":[{"role":"user","content":"用一句話介紹你自己"}],"max_tokens":100}'

{"choices":[{"message":{"content":"我是一個由 Google DeepMind 開發的開放權重大型語言模型，名為 Gemma 4。"}}],
 "usage":{"prompt_tokens":14,"completion_tokens":23,"total_tokens":37}}

Total Round Trip: 6.1s.

Notes

• Don’t Use The Generic docker.io/vllm/vllm-openai:latest Image. It’s Built Against A Newer CUDA Than Cloud Run L4’s Driver (535.x / CUDA 12.2) Supports, Fails With RuntimeError: The NVIDIA driver on your system is too old (found version 12020). Use Google’s Own Rebuilt Model Garden Image Above Instead — This Is Also Why The Official Codelab Never Uses The Generic Image.
• This Image’s Entrypoint Doesn’t Auto-Prepend vllm. It’s A Launcher Script (gcs_download_launcher.sh) That exec Whatever You Give It. Args Must Start With vllm,serve,..., Not Just serve,..., Or You Get serve: command not found.
• --enforce-eager Is Required, Not Optional, Unless You Extend The Startup Probe A Lot. Without It, vLLM Does torch.compile + CUDA Graph Capture Across Batch Sizes 1 To 128 On First Boot, Which Takes Over 5 Minutes — Longer Than Most Startup Probe Configs, So The Deploy Just Rolls Back. With It, Cold Start Is Under A Minute, At The Cost Of Slightly Slower Steady-State Throughput.
• Port Is 8080 Here, Not vLLM’s Usual 8000. Match --port To Whatever The Image Actually Listens On.
• --max-instances Does Not Prevent Cold Start, It’s Only A Concurrency Ceiling. Use --min-instances=1 If Cold Start Is Unacceptable — That Means Paying For A GPU Instance 24/7. Without It, The Service Scales To Zero When Idle, No Charge.

How To (CPU Mode)

Only Makes Sense For Very Small Models. pip install vllm Assumes CUDA, Won’t Run Inference On A CPU-Only Host. Official Path Is Building From docker/Dockerfile.cpu — Don’t Hand-Roll A Simplified Version, Pull The Real File:

git clone --depth 1 https://github.com/vllm-project/vllm.git
cd vllm

Build With This cloudbuild.yaml:

steps:
  - name: 'gcr.io/cloud-builders/docker'
    env:
      - 'DOCKER_BUILDKIT=1'
    args:
      - 'build'
      - '-f'
      - 'vllm/docker/Dockerfile.cpu'
      - '--target'
      - 'vllm-openai'
      - '--build-arg'
      - 'max_jobs=8'
      - '-t'
      - 'asia-southeast1-docker.pkg.dev/$PROJECT_ID/YOUR_REPO/vllm-cpu:latest'
      - 'vllm'

images:
  - 'asia-southeast1-docker.pkg.dev/$PROJECT_ID/YOUR_REPO/vllm-cpu:latest'

options:
  machineType: 'E2_HIGHCPU_32'

timeout: 7200s

gcloud builds submit --config=cloudbuild.yaml --region=asia-southeast1 .

Deploy With This Command:

gcloud run deploy vllm-cpu-service \
  --image=asia-southeast1-docker.pkg.dev/YOUR_PROJECT/YOUR_REPO/vllm-cpu:latest \
  --region=asia-southeast1 \
  --memory=10Gi \
  --cpu=4 \
  --timeout=600 \
  --port=8000 \
  --set-env-vars=VLLM_CPU_KVCACHE_SPACE=4,VLLM_ENABLE_V1_MULTIPROCESSING=0,VLLM_HOST_IP=127.0.0.1,GLOO_SOCKET_IFNAME=lo \
  --args="Qwen/Qwen2.5-0.5B-Instruct,--port=8000,--host=0.0.0.0" \
  --startup-probe=tcpSocket.port=8000,initialDelaySeconds=60,timeoutSeconds=240,failureThreshold=1,periodSeconds=240 \
  --no-allow-unauthenticated

Test It:

$ curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
  https://vllm-cpu-service-xxx.asia-southeast1.run.app/v1/chat/completions \
  -d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[{"role":"user","content":"用一句話介紹你自己"}],"max_tokens":100}'

{"choices":[{"message":{"content":"我是由阿里云开发的超大规模语言模型，我叫通义千问。我的目标是帮助用户理解和创造。"}}],
 "usage":{"prompt_tokens":34,"completion_tokens":27,"total_tokens":61}}

Total: 114s, 34 Prompt Tokens + 27 Completion Tokens, 0.5B Model.

Notes

• Cloud Build’s Default Docker Builder Doesn’t Have BuildKit Enabled. The Official Dockerfile Uses RUN --mount=type=cache,... (BuildKit-Only), Fails With the --mount option requires BuildKit. Add env: ['DOCKER_BUILDKIT=1'] To The Build Step.
• Cloud Build Machine Types Are All High-CPU-Low-Memory (e2-highcpu-32 = 32 vCPU / 32GB RAM, No High-Mem Option Exists). Default MAX_JOBS=32 OOM-Kills The Build Machine Mid-Compile (status "INTERNAL_ERROR", Not A Normal Build Failure) During The Triton-CPU Stage. Lower It With --build-arg max_jobs=8. Full Build Then Takes About 56 Min.
• Keep .git In The Build Context. vLLM Uses setuptools-scm To Derive Its Version From Git History — Strip .git To Save Upload Size And The Build Fails Hard With setuptools-scm was unable to detect version. A --depth 1 Shallow Clone Is Enough, Costs About 40MB Extra.
• vllm serve Takes The Model Name As A Positional Argument, Not A --model= Flag.
• Model Loads Fine But The Port Never Opens (shm_broadcast.py: No available shared memory broadcast block found in 60 seconds) — vLLM’s Multi-Process Architecture Needs /dev/shm For IPC, And Cloud Run’s Default Is Too Small. Mounting A Bigger In-Memory Volume At /dev/shm Does Not Work, Cloud Run Rejects It Outright (Mount paths [/dev/shm] are in one of disallowed mount points [/dev, /proc, /sys]). The Actual Fix Is Avoiding The Dependency, Not Enlarging It: --set-env-vars=VLLM_ENABLE_V1_MULTIPROCESSING=0,VLLM_HOST_IP=127.0.0.1,GLOO_SOCKET_IFNAME=lo. First Var Keeps The Engine Mostly In One Process; The Other Two Force torch.distributed’s TCP Rendezvous Onto Loopback, Because It Otherwise Times Out Retrying Inside Cloud Run’s gVisor Sandboxed Network ([c10d] The server socket ... has timed out, will retry).

Which One To Use

GPU Mode For Anything Real — Just Remember To Use Google’s Model Garden Image, Not The Generic vllm/vllm-openai. CPU Mode Hits Deeper Issues (shm Architecture Limits, gVisor Sandbox Network Compatibility) And Is Really Only Good For Validating The Architecture Or A Very-Low-Traffic Demo, Not Production.

Cloud Run Now Support GPU (NVIDIA L4 / RTX PRO 6000 Blackwell) Directly, No Need To Go Through GKE Anymore. GPU Mode Just Means Adding --gpu And --gpu-type To A Normal gcloud run deploy, Cloud Run Handles The Driver And Scheduling — That’s What Makes The GPU Path Above So Much Simpler Than It Used To Be.

Reference:

• GPU support for services | Cloud Run
• How to run LLM inference on Cloud Run GPUs with vLLM | Google Codelabs
• CPU - vLLM Documentation
• vllm/docker/Dockerfile.cpu · vllm-project/vllm (GitHub)
• Configure in-memory volume mounts for services | Cloud Run