What are the major vLLM configuration parameters for tuning

* **`MAX_MODEL_LEN="32767"`**: Sets the maximum token context window (prompts plus generated answers) the engine will handle. Restricting this to 32,767 tokens instead of the native ultra-large context (like 256K) drastically reduces the GPU memory required by the KV cache, allowing for higher concurrency and faster boot times.

* **`QUANTIZATION_TYPE="fp8"`**: Specifies 8-bit floating-point (FP8) precision for the model weights. This compresses the size of the model down significantly, cutting memory usage roughly in half and accelerating inference speed with minimal loss in model intelligence.

* **`KV_CACHE_DTYPE="fp8"`**: Compresses the Key-Value (KV) cache storage into FP8 format instead of standard FP16/BF16. Because the KV cache grows rapidly with long conversations, this halves its memory footprint, freeing up space to hold more concurrent users and longer contexts.

* **`GPU_MEM_UTIL="0.95"`**: Tells vLLM to claim and lock down **95%** of the total available GPU VRAM on your Cloud Run instance. The remaining 5% is left as breathing room for PyTorch operations and context switching to prevent out-of-memory (OOM) crashes.

* **`TENSOR_PARALLEL_SIZE="1"`**: Determines how many GPUs the model is split across. Set to `1` because your Cloud Run service instance is provisioned with a single GPU, meaning the entire model runs on that single card.

* **`MAX_NUM_SEQS="16"`**: Caps the maximum number of simultaneous sequences (requests) vLLM will batch and process together in a single step [cite: . This prevents memory spikes under heavy web traffic by queuing excess incoming chat requests.

My Cloud Run hangs while downloading Large OpenWeight model Is it normal?

The output is like below 

ravi_retheesh@cloudshell:~ (gemmabigquerymcp)$ gcloud builds submit --project="${GOOGLE_CLOUD_PROJECT}" --region="${GOOGLE_CLOUD_REGION}" --no-source \

substitutions="_MODEL_NAME=${MODEL_NAME},_GCS_MODEL_LOCATION=${GCS_MODEL_LOCATION}" \

    --config=/dev/stdin <<'EOF'

steps:

- name: 'gcr.io/google.com/cloudsdktool/google-cloud-cli:slim'

  entrypoint: 'bash'

  args:

  - '-c'

  - |

    if [[ "$_GCS_MODEL_LOCATION" == *"vertex-model-garden-public-us"* ]]; then

      echo "Using the public cache bucket."

      exit 0

    fi

    gcloud config set storage/parallel_composite_upload_enabled True

    gcloud config set storage/parallel_composite_upload_threshold 150M

    gcloud config set storage/sliced_object_download_threshold 150M

    MODEL_NAME="$_MODEL_NAME"

    SHORT_NAME="$${MODEL_NAME#*/}"

    gcloud storage cp -r -D "gs://vertex-model-garden-public-us/gemma4/$${SHORT_NAME}" "$_GCS_MODEL_LOCATION"

EOF

Created [https://cloudbuild.googleapis.com/v1/projects/gemmabigquerymcp/locations/us-central1/builds/21faa182-824e-42fb-b292-d4368222d847].

Logs are available at [ https://console.cloud.google.com/cloud-build/builds;region=us-central1/21faa182-824e-42fb-b292-d4368222d847?project=593821728960 ].

Waiting for build to complete. Polling interval: 1 second(s).

----------------------------- REMOTE BUILD OUTPUT ------------------------------

starting build "21faa182-824e-42fb-b292-d4368222d847"

FETCHSOURCE

BUILD

Pulling image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim

slim: Pulling from google.com/cloudsdktool/google-cloud-cli

6310eb16bf42: Pulling fs layer

488f94d8fb5f: Pulling fs layer

4f4fb700ef54: Pulling fs layer

3cbf6ceea8d8: Pulling fs layer

8421ab9bea9a: Pulling fs layer

488f94d8fb5f: Verifying Checksum

488f94d8fb5f: Download complete

8421ab9bea9a: Verifying Checksum

8421ab9bea9a: Download complete

4f4fb700ef54: Verifying Checksum

4f4fb700ef54: Download complete

6310eb16bf42: Verifying Checksum

6310eb16bf42: Download complete

6310eb16bf42: Pull complete

488f94d8fb5f: Pull complete

4f4fb700ef54: Pull complete

3cbf6ceea8d8: Verifying Checksum

3cbf6ceea8d8: Download complete

3cbf6ceea8d8: Pull complete

8421ab9bea9a: Pull complete

Digest: sha256:14026c08cf391812086012d6c5f14078221ce1b46be0df419f7f31ec9f69c712

Status: Downloaded newer image for gcr.io/google.com/cloudsdktool/google-cloud-cli:slim

gcr.io/google.com/cloudsdktool/google-cloud-cli:slim

Updated property [storage/parallel_composite_upload_enabled].

Updated property [storage/parallel_composite_upload_threshold].

Updated property [storage/sliced_object_download_threshold].

Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/README.md to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/README.md

Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/chat_template.jinja to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/chat_template.jinja

  

Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/config.json to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/config.json

Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/generation_config.json to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/generation_config.json

Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/model-00001-of-00002.safetensors to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/model-00001-of-00002.safetensors

Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/model-00002-of-00002.safetensors to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/model-00002-of-00002.safetensors

Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/model.safetensors.index.json to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/model.safetensors.index.json

Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/processor_config.json to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/processor_config.json

Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/tokenizer.json to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/tokenizer.json

Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/tokenizer_config.json to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/tokenizer_config.json 

Yes, hanging at this point is completely expected.

What you are looking at in your terminal output is the script successfully copying the smaller configuration files (README.md, tokenizers, json configs), and it has now reached the heavy .safetensors files.

What is happening internally?

Massive File Sizes: The Gemma 4 31B model consists of multi-gigabyte weight tensors split across multiple large files (e.g., model-00001-of-00002.safetensors and model-00002-of-00002.safetensors), each weighing anywhere from 10 GB to 15 GB+ depending on the quantization.

Silent Progress: The standard gcloud storage cp command does not stream a granular, incremental progress bar inside a Cloud Build remote execution log. Because it is processing a massive file payload over the network in total silence, the terminal appears completely frozen.

Is it actually working?

Yes. As long as the build hasn't timed out or thrown an explicit error, the Cloud Build worker is actively pulling the file blocks from Google's public model garden bucket and streaming them into your custom GCS cache bucket (gemmabigquerymcp-us-central1-hf-model-cache).

What should you do?

Wait it out: Depending on network routing inside the us-central1 data center, a multi-gigabyte copy can take anywhere from 2 to 10 minutes.

Verify via Console (Optional): If you want to check if data is actually transferring, you can open a separate Cloud Shell tab and run a check on your destination bucket to see file sizes growing:

Bash

gcloud storage du -s "gs://${MODEL_CACHE_BUCKET}"

Let it run for a few more minutes; it will finish and proceed to the next step once the tensor files are fully copied over.

How to run Large OpenWeight models on Infra like cloud run?

**Yes, this snippet is very good and standard practice** for handling large open-weights models like Gemma 4 on serverless infrastructure like Cloud Run.

### How This Works and Why It's Designed This Way

* **Bypassing Cloud Build Limits:** Cloud Build has strict storage limits and timeouts. Trying to download a massive 30GB+ model inside a standard build step or pushing it straight into a container image will typically break or exceed size thresholds.

* **Leveraging Google's Public Cache (`vertex-model-garden-public-us`):** Google hosts public copies of model weights inside specialized regional GCS buckets. If your Cloud Run service runs in a `us-` region, the snippet intelligently skips downloading entirely (`exit 0`), allowing your app to stream weights straight from Google’s ultra-fast internal network.

* **Parallel Upload Optimization:** If you are using a non-US region or deploying a custom fine-tuned variant, the `gcloud config set storage/...` commands configure parallel composite uploads and sliced downloads. This breaks large model safetensors files into chunks, downloading them concurrently to bypass single-stream bottlenecks.

---

### Your Options to Run Gemma 4 on Google Cloud Run

When deploying Gemma 4 (such as the 31B Dense or 26B MoE variants) to Cloud Run, you have two primary deployment architectures:

| Approach | Architecture | Best For |

| --- | --- | --- |

| **1. vLLM Container + GCS Cache (The Snippet Method)** | Packages vLLM inside a custom container image, pulls weights directly from GCS into GPU memory using high-performance streaming. | High throughput, production-grade OpenAI-compatible API serving, and maximum token generation speed. |

| **2. Ollama on Cloud Run** | Deploys an Ollama runtime container to Cloud Run backed by serverless GPUs, pulling model weights dynamically on start. | Lightweight setups, rapid prototyping, and simpler configuration files. |

Both approaches require provisioning **NVIDIA RTX Pro 6000 (Blackwell)** or similar serverless GPUs on Cloud Run to supply enough VRAM to hold the model weights.

Do Google Provide vLLM service? How do cache the model weights?

 This command creates a dedicated Google Cloud Storage (GCS) bucket to act as a staging cache for your large AI model weights.

Why is this step performed?

Bypassing Hugging Face Rate Limits / Slowdowns: Downloading massive multi-gigabyte open-weights models directly from Hugging Face every time a Cloud Run container spins up is slow and prone to network timeouts or rate limits.

Optimizing Cold Starts with vLLM: When combined with the Run:ai Model Streamer feature built into vLLM, the container can stream model weight tensors concurrently and directly from GCS into GPU memory.

 This drastically minimizes container boot-up and model-loading times.

Direct VPC Egress: Keeping the bucket in the same single region as your Cloud Run service ensures data flows privately and rapidly over Google's internal network rather than traversing the public internet.

Does Google by default provide a vLLM runtime?

No, Google Cloud does not provide a pre-installed, managed vLLM runtime service out of the box (like a native managed-vllm API endpoint).

Instead, Google provides the underlying infrastructure to run vLLM yourself:

Compute Support: Cloud Run supports attaching high-performance GPUs (such as NVIDIA L4 or RTX Pro 6000) and Google Cloud TPUs, alongside serverless scaling.

Bring-Your-Own-Container (BYOC): To use vLLM on Cloud Run, you package vLLM inside a custom container image (or use open-source deep learning containers/Buildpacks) that pulls the model weights from your GCS bucket upon startup and exposes an OpenAI-compatible API endpoint.

What is PodMonitoring and node-exporter in Kubernetes?

A PodMonitoring is a custom Kubernetes resource used by Google Cloud Managed Service for Prometheus to define how metric data from application pods is scraped and ingested.What It DoesTarget Scraping: Finds specific pods within a Kubernetes namespace using label selectors and scrapes their Prometheus-formatted metrics endpoints.Flow Control: Controls how metrics stream into Cloud Monitoring, allowing you to modify scrape intervals, apply time-series filtering, reduce data rates, and use Prometheus relabeling rules.Namespace Scoping: Scrapes targets strictly within the single namespace where the resource is deployed (you must deploy copies across multiple namespaces to scrape globally, or use a ClusterPodMonitoring resource instead)

Node Exporter acts as the data producer, while a monitoring custom resource acts as the data collector.When you want to monitor hardware and OS metrics (like CPU, memory, and disk usage) of your Google Kubernetes Engine (GKE) nodes, you deploy Node Exporter. To actually pull those metrics into Google Cloud, you use a target-scraping custom resource.Because Node Exporter tracks node-level infrastructure rather than isolated application workflows, it connects to Google Cloud Managed Service for Prometheus using two main strategies:1. The Global Connection (ClusterPodMonitoring)While you can use a standard PodMonitoring resource, Google Cloud recommends using a ClusterPodMonitoring resource for Node Exporter.The Problem: Node Exporter runs as a DaemonSet across all nodes, usually isolated in a system namespace (like kube-system or gmp-system). A standard PodMonitoring resource is strictly limited to looking inside its own namespace.The Connection: A ClusterPodMonitoring resource breaks past namespace boundaries. It identifies all Node Exporter pods running across your entire cluster using label selectors (e.g., app.kubernetes.io/name: node-exporter) and scrapes their metric endpoints (typically port 9100).2. The Isolated Connection (PodMonitoring)If you specifically choose to use a standard PodMonitoring resource instead, you must deploy it directly into the exact same namespace where your Node Exporter daemonset lives. It will selectively connect to and scrape only the Node Exporter pods within that specific namespace boundary.What the Configuration Looks LikeTo bridge the gap between Node Exporter and Google Cloud, you apply a manifest that explicitly targets the exporter. Here is the official setup pattern for the Google Cloud Node Exporter integration using a cluster-wide monitoring rule:yamlapiVersion: monitoring.gke.io/v1

kind: ClusterPodMonitoring

metadata:

  name: node-exporter

spec:

  selector:

    matchLabels:

      app.kubernetes.io/name: node-exporter # 1. Finds the Node Exporter pods

  endpoints:

  - port: metrics                         # 2. Targets the exporter's metric port

    interval: 30s                         # 3. Sets how frequently to scrape hardware data

Use code with caution.The Data Pipeline FlowNode Exporter gathers raw kernel, memory, and disk utilization data directly from the host GKE node.Google Cloud's Managed Collector reads your monitoring configuration, discovers the exporter pods via the labels, and scrapes the data.The metrics are forwarded to Google's Monarch database, allowing you to view node health using PromQL in Cloud Monitoring or Grafana.Are you setting up Node Exporter to build a custom cluster dashboard, or are you troubleshooting missing node-level metrics in Cloud Monitoring?

Google Cloud Collect metrics from Exporters using the Managed Service for Prometheus

Main aim is to 

Deploy a GKE instance

Configure the PodMonitoring custom resource and node-exporter tool

Build the GMP binary locally and deploy to the GKE instance

Apply a Prometheus configuration to begin collecting metrics

gcloud auth list

To ingest the metric data emitted by the example application, you use target scraping. Target scraping and metrics ingestion are configured using Kubernetes custom resources. The managed service uses PodMonitoring custom resources (CRs).

A PodMonitoring CR scrapes targets only in the namespace the CR is deployed in. To scrape targets in multiple namespaces, deploy the same PodMonitoring CR in each namespace. You can verify the PodMonitoring resource is installed in the intended namespace by running kubectl get podmonitoring -A.

Google Managed Service for Prometheus (GMP) ingestion of Node Exporter metrics links standard open-source infrastructure monitoring with Google Cloud's fully managed global monitoring backend

Node Exporter: A lightweight agent that runs on your servers or cluster nodes to collect low-level, machine-level hardware and operating system metrics (such as CPU usage, memory, disk I/O, and network traffic). It exposes these metrics locally in a standard Prometheus text format.GMP Binary / Collector: Google's drop-in replacement or forked version of the Prometheus collector/binary. Instead of relying on a locally managed Prometheus time-series database for long-term storage, this specialized binary scrapes the local /metrics endpoint of Node Exporter and forwards (pushes) those metrics directly into Google Cloud's global monitoring database (Monarch / Cloud Monitoring).Viewing the Metrics: Querying the gathered data using standard PromQL inside the Google Cloud Console, Metrics Explorer, or connected dashboards like Grafana

What is Signoz

 

SigNoz is an open-source observability platform built on OpenTelemetry. We’re building an enterprise-grade alternative to fragmented monitoring stacks, with logs, metrics, traces, alerts, and dashboards in one place.

Choose how to run SigNoz

SigNoz Cloud (Recommended)

Fully managed SigNoz with a 30-day free trial, no credit card required, usage-based pricing that starts at $49, and regional data hosting.

Start free →

Enterprise

Enterprise Cloud, BYOC, or Enterprise Self-Hosted with compliance, support, custom retention, RBAC, ingestion controls, data residency, and region selection.

Explore Enterprise →

Community

Free open-source SigNoz that runs in your own infrastructure. Deploy with Docker, Kubernetes, or Linux and keep full control of your data plane.

Install SigNoz →

What can you monitor?

SigNoz helps teams debug production issues faster by connecting logs, metrics, traces, alerts, dashboards, exceptions, and agent-native workflows in one place.

APM Overview

Monitor service latency, error rate, throughput, Apdex, top endpoints, database calls, and external calls.

Log Management

Ingest, search, aggregate, and correlate logs with traces and metrics using a visual query builder.

Metrics and Dashboards

Build dashboards for application, infrastructure, and custom metrics using Query Builder, PromQL, or ClickHouse SQL.

nfrastructure Monitoring

Monitor Kubernetes clusters, pods, nodes, workloads, and host-level CPU, memory, disk, network, logs, and traces.

LLM and AI Observability

Trace LLM apps, RAG pipelines, prompts, tool calls, tokens, latency, and costs alongside application and infrastructure telemetry.

Agent-Native Observability and MCP

Use the SigNoz MCP server to bring telemetry into coding agents, or use Noz inside SigNoz to investigate incidents, tune alerts, and build dashboards with production context. Noz is available only on SigNoz Cloud.

Distributed Tracing

Follow requests across services with flamegraphs, waterfalls, span events, filters, and trace analytics.

Trace Funnels

Create funnels from traces to understand request-flow drop-offs, failed transitions, and systemic workflow issues.

Learn more: Trace funnels documentation

Also monitor: exceptions, alerts, external APIs, and integrations for OpenTelemetry, Prometheus, Kubernetes, cloud providers, language SDKs, application frameworks, databases, and LLM tools.

Why teams use SigNoz

OpenTelemetry-native

Instrument once with open standards and keep ownership of your telemetry.

Correlated signals

Move from service charts to traces, logs, infra metrics, and exceptions without switching tools.

Single columnar database

Built for high-cardinality, high-volume observability workloads.

Predictable pricing

No per-host pricing, no user-seat pricing, and no special pricing for custom metrics.

Enterprise ready

SOC 2 Type II and HIPAA compliance, RBAC, ingestion controls, custom retention, support, BYOC, and self-hosting.

Getting started

Start on Cloud

Create a managed SigNoz workspace and get your first dashboard without running observability infrastructure.

Start free on SigNoz Cloud

Self-host SigNoz

Run SigNoz in your own infrastructure with Foundry, Docker, Kubernetes, or Linux.

Foundry · Docker · Kubernetes · Linux

Send data

Instrument applications and infrastructure with OpenTelemetry, Prometheus, language SDKs, and integrations.

Instrumentation · Integrations

Comparisons to familiar tools

SigNoz is often adopted by teams moving from a stack of single-purpose tools or commercial platforms with unpredictable pricing.

Prometheus

Good if you just need metrics. SigNoz keeps metrics, logs, traces, dashboards, and alerts together so teams can debug with correlated context.

Jaeger

Jaeger only does distributed tracing. SigNoz adds metrics, logs, trace analytics, dashboards, alerts, exceptions, and trace-to-log workflows.

Elastic

SigNoz uses columnar database for efficient observability analytics and high-cardinality log workloads, with 50% lower resource requirement compared to Elastic during ingestion. Check the detailed study.

Loki

In the linked benchmark, SigNoz indexed all keys in the test setup, while Loki hit max stream errors when more labels were added. Check the detailed study.