Sunday, August 30, 2026

My Cloud Run hangs while downloading Large OpenWeight model Is it normal?

The output is like below 

ravi_retheesh@cloudshell:~ (gemmabigquerymcp)$ gcloud builds submit --project="${GOOGLE_CLOUD_PROJECT}" --region="${GOOGLE_CLOUD_REGION}" --no-source \

substitutions="_MODEL_NAME=${MODEL_NAME},_GCS_MODEL_LOCATION=${GCS_MODEL_LOCATION}" \


    --config=/dev/stdin <<'EOF'


steps:


- name: 'gcr.io/google.com/cloudsdktool/google-cloud-cli:slim'


  entrypoint: 'bash'


  args:


  - '-c'


  - |


    if [[ "$_GCS_MODEL_LOCATION" == *"vertex-model-garden-public-us"* ]]; then


      echo "Using the public cache bucket."


      exit 0


    fi


    gcloud config set storage/parallel_composite_upload_enabled True


    gcloud config set storage/parallel_composite_upload_threshold 150M


    gcloud config set storage/sliced_object_download_threshold 150M


    MODEL_NAME="$_MODEL_NAME"


    SHORT_NAME="$${MODEL_NAME#*/}"


    gcloud storage cp -r -D "gs://vertex-model-garden-public-us/gemma4/$${SHORT_NAME}" "$_GCS_MODEL_LOCATION"


EOF


Created [https://cloudbuild.googleapis.com/v1/projects/gemmabigquerymcp/locations/us-central1/builds/21faa182-824e-42fb-b292-d4368222d847].


Logs are available at [ https://console.cloud.google.com/cloud-build/builds;region=us-central1/21faa182-824e-42fb-b292-d4368222d847?project=593821728960 ].


Waiting for build to complete. Polling interval: 1 second(s).


----------------------------- REMOTE BUILD OUTPUT ------------------------------


starting build "21faa182-824e-42fb-b292-d4368222d847"




FETCHSOURCE


BUILD


Pulling image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim


slim: Pulling from google.com/cloudsdktool/google-cloud-cli


6310eb16bf42: Pulling fs layer


488f94d8fb5f: Pulling fs layer


4f4fb700ef54: Pulling fs layer


3cbf6ceea8d8: Pulling fs layer


8421ab9bea9a: Pulling fs layer


488f94d8fb5f: Verifying Checksum


488f94d8fb5f: Download complete


8421ab9bea9a: Verifying Checksum


8421ab9bea9a: Download complete


4f4fb700ef54: Verifying Checksum


4f4fb700ef54: Download complete


6310eb16bf42: Verifying Checksum


6310eb16bf42: Download complete


6310eb16bf42: Pull complete


488f94d8fb5f: Pull complete


4f4fb700ef54: Pull complete


3cbf6ceea8d8: Verifying Checksum


3cbf6ceea8d8: Download complete


3cbf6ceea8d8: Pull complete


8421ab9bea9a: Pull complete


Digest: sha256:14026c08cf391812086012d6c5f14078221ce1b46be0df419f7f31ec9f69c712


Status: Downloaded newer image for gcr.io/google.com/cloudsdktool/google-cloud-cli:slim


gcr.io/google.com/cloudsdktool/google-cloud-cli:slim


Updated property [storage/parallel_composite_upload_enabled].


Updated property [storage/parallel_composite_upload_threshold].


Updated property [storage/sliced_object_download_threshold].


Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/README.md to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/README.md


Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/chat_template.jinja to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/chat_template.jinja


  


Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/config.json to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/config.json


Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/generation_config.json to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/generation_config.json


Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/model-00001-of-00002.safetensors to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/model-00001-of-00002.safetensors


Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/model-00002-of-00002.safetensors to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/model-00002-of-00002.safetensors


Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/model.safetensors.index.json to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/model.safetensors.index.json


Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/processor_config.json to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/processor_config.json


Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/tokenizer.json to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/tokenizer.json


Copying gs://vertex-model-garden-public-us/gemma4/gemma-4-31B-it/tokenizer_config.json to gs://gemmabigquerymcp-us-central1-hf-model-cache/model-cache/google/gemma-4-31B-it/tokenizer_config.json 







Yes, hanging at this point is completely expected.


What you are looking at in your terminal output is the script successfully copying the smaller configuration files (README.md, tokenizers, json configs), and it has now reached the heavy .safetensors files.


What is happening internally?

Massive File Sizes: The Gemma 4 31B model consists of multi-gigabyte weight tensors split across multiple large files (e.g., model-00001-of-00002.safetensors and model-00002-of-00002.safetensors), each weighing anywhere from 10 GB to 15 GB+ depending on the quantization.


Silent Progress: The standard gcloud storage cp command does not stream a granular, incremental progress bar inside a Cloud Build remote execution log. Because it is processing a massive file payload over the network in total silence, the terminal appears completely frozen.


Is it actually working?

Yes. As long as the build hasn't timed out or thrown an explicit error, the Cloud Build worker is actively pulling the file blocks from Google's public model garden bucket and streaming them into your custom GCS cache bucket (gemmabigquerymcp-us-central1-hf-model-cache).


What should you do?

Wait it out: Depending on network routing inside the us-central1 data center, a multi-gigabyte copy can take anywhere from 2 to 10 minutes.


Verify via Console (Optional): If you want to check if data is actually transferring, you can open a separate Cloud Shell tab and run a check on your destination bucket to see file sizes growing:


Bash

gcloud storage du -s "gs://${MODEL_CACHE_BUCKET}"

Let it run for a few more minutes; it will finish and proceed to the next step once the tensor files are fully copied over.


No comments:

Post a Comment