* **`MAX_MODEL_LEN="32767"`**: Sets the maximum token context window (prompts plus generated answers) the engine will handle. Restricting this to 32,767 tokens instead of the native ultra-large context (like 256K) drastically reduces the GPU memory required by the KV cache, allowing for higher concurrency and faster boot times.
* **`QUANTIZATION_TYPE="fp8"`**: Specifies 8-bit floating-point (FP8) precision for the model weights. This compresses the size of the model down significantly, cutting memory usage roughly in half and accelerating inference speed with minimal loss in model intelligence.
* **`KV_CACHE_DTYPE="fp8"`**: Compresses the Key-Value (KV) cache storage into FP8 format instead of standard FP16/BF16. Because the KV cache grows rapidly with long conversations, this halves its memory footprint, freeing up space to hold more concurrent users and longer contexts.
* **`GPU_MEM_UTIL="0.95"`**: Tells vLLM to claim and lock down **95%** of the total available GPU VRAM on your Cloud Run instance. The remaining 5% is left as breathing room for PyTorch operations and context switching to prevent out-of-memory (OOM) crashes.
* **`TENSOR_PARALLEL_SIZE="1"`**: Determines how many GPUs the model is split across. Set to `1` because your Cloud Run service instance is provisioned with a single GPU, meaning the entire model runs on that single card.
* **`MAX_NUM_SEQS="16"`**: Caps the maximum number of simultaneous sequences (requests) vLLM will batch and process together in a single step [cite: . This prevents memory spikes under heavy web traffic by queuing excess incoming chat requests.
No comments:
Post a Comment