Deploying LiteLLM Gateway oneself

 Deploying a LiteLLM Gateway (also known as the LiteLLM Proxy) allows you to establish a single, central endpoint that translates OpenAI-formatted API requests into formats compatible with over 100 upstream AI providers.

The production-ready standard way to deploy LiteLLM is using Docker Compose alongside a PostgreSQL database. The database handles critical gateway operations, including saving virtual API keys, configuring models directly from the UI, and tracking team spending budgets.

Step 1: Create Your Configuration Files

Create a fresh project folder on your server and navigate into it:

bash

mkdir litellm-gateway && cd litellm-gateway

Create a file named litellm_config.yaml. This file sets basic settings and tells the gateway to pull model routing definitions directly from your PostgreSQL database:

yaml

model_list: [] # We will dynamically add models via the Admin UI later

general_settings:

  master_key: "sk-your-secure-admin-master-key" # Must start with sk-

  store_model_in_db: true                        # Enables UI-based model management

Note: Always replace sk-your-secure-admin-master-key with a long, randomly generated secret. This key acts as your master admin password).

Step 2: Set Up Docker Compose

Create a docker-compose.yml file in the same directory to orchestrate the gateway proxy and the database backend:

yaml

version: '3.8'

services:

  litellm-db:

    image: postgres:16-alpine

    container_name: litellm-db

    environment:

      POSTGRES_USER: litellm_admin

      POSTGRES_PASSWORD: your_secure_db_password

      POSTGRES_DB: litellm_gateway_db

    volumes:

      - pgdata:/var/lib/postgresql/data

    ports:

      - "5432:5432"

    restart: always

  litellm-gateway:

    image: ghcr.io/berriai/litellm:main-latest

    container_name: litellm-gateway

    ports:

      - "4000:4000" # Exposes OpenAI-compatible endpoint & UI dashboard

    volumes:

      - ./litellm_config.yaml:/app/litellm_config.yaml

    environment:

      - DATABASE_URL=postgresql://litellm_admin:your_secure_db_password@litellm-db:5432/litellm_gateway_db

      - LITELLM_MASTER_KEY=sk-your-secure-admin-master-key

      - LITELLM_SALT_KEY=sk-permanent-salt-key-for-db-encryption

    depends_on:

      - litellm-db

    command: ["--config", "/app/litellm_config.yaml", "--detailed_debug"]

    restart: always

volumes:

  pgdata:

LITELLM_SALT_KEY: Provide a secondary unique key starting with sk-. This is used to encrypt upstream provider credentials inside your database. Never change this key after your first deployment, or your stored credentials will become unreadable.

Step 3: Launch the Gateway

Start your services in detached (background) mode:

bash

docker compose up -d

You can monitor the setup initialization logs to ensure everything resolves cleanly:

bash

docker compose logs -f litellm-gateway

Step 4: Access the Admin UI & Hook Up Providers

1. Open your web browser and navigate to http://localhost:4000/ui.

2. When prompted, log in using your LITELLM_MASTER_KEY.

3. Click on the Models tab to add your downstream configurations (e.g., adding OpenAI, Anthropic Claude, AWS Bedrock, or local Ollama instances) and paste the respective provider API keys.

4. Navigate to the Virtual Keys tab to issue unique keys to your engineering teams or software applications with explicit spend ceilings and rate limits.

Step 5: Route Application Traffic

Your applications can now treat the LiteLLM gateway exactly like an OpenAI endpoint. For instance, in Python:

from openai import OpenAI

client = OpenAI(

    api_key="sk-the-virtual-key-you-generated-in-the-ui",

    base_url="http://localhost:4000/v1"

)

response = client.chat.completions.create(

    model="gpt-4o", # Or any model name configured in your gateway

    messages=[{"role": "user", "content": "Hello LiteLLM!"}]

)

print(response.choices[0].message.content)

Production Checklist

• TLS/HTTPS: LiteLLM serves unencrypted HTTP by default. Place a reverse proxy like Nginx, Traefik, or an AWS ALB in front of port 4000 to manage SSL termination.

• Redis Caching: For production-scale throughput, add a Redis container to your Docker Compose setup and point to it using REDIS_URL to facilitate low-latency caching and accurate distributed rate-limiting.

For advanced continuous deployment setups, LiteLLM also provides Official Helm Charts for Kubernetes and native Terraform configurations