LiteLLM accessing thorugh OpenAI langchain class

 

Accessing LiteLLM Gateway using LangChain ChatOpenAI

LiteLLM Gateway provides an OpenAI-compatible API, allowing applications to access different LLM providers through a single gateway endpoint.

If your Python application already uses langchain-openai, you can directly use the ChatOpenAI class to connect to LiteLLM without installing additional LiteLLM libraries.

1. Required configuration

Your customer typically provides:

Parameter	Description	Usage
Gateway URL	LiteLLM API endpoint	base_url
API Key (sk-...)	Authentication token	api_key
Model ID	Model configured in LiteLLM	model
Key Alias	Administrative name for the key	Not required
Key Hash	Hashed key identifier	Not required
Environment	DEV / TEST / PROD	Configuration selection

2. Install dependencies

pip install langchain-openai

Configure environment variables:

export LITELLM_BASE_URL="https://gateway.customer.com/v1"
export LITELLM_API_KEY="sk-your-api-key"
export LITELLM_MODEL="customer-model-id"

The URL must be the customer's OpenAI-compatible API base URL. Do not append /v1 if it is already included.

3. Basic LLM invocation

import osfrom langchain_openai import ChatOpenAIllm = ChatOpenAI(    model=os.environ["LITELLM_MODEL"],    base_url=os.environ["LITELLM_BASE_URL"],    api_key=os.environ["LITELLM_API_KEY"],    temperature=0.2,    timeout=60,    max_retries=2,)response = llm.invoke(    "Explain the difference between BGP and OSPF.")print(response.content)

The request is sent to the LiteLLM Gateway, which routes it to the configured underlying LLM.

4. Additional examples

Example A: System and user messages

from langchain_core.messages import SystemMessage, HumanMessageresponse = llm.invoke([    SystemMessage(        content="You are an expert network troubleshooting assistant."    ),    HumanMessage(        content="Why is my BGP neighbor stuck in Idle state?"    ),])print(response.content)

Example B: Streaming responses

for chunk in llm.stream(    "Explain Kubernetes networking."):    print(chunk.content, end="", flush=True)

This works when streaming is supported by the selected model and gateway.

Example C: Async invocation

import asyncioasync def main():    response = await llm.ainvoke(        "Explain how gRPC load balancing works."    )    print(response.content)asyncio.run(main())

Example D: Tool calling

from langchain_core.tools import tool@tooldef get_device_status(device_id: str) -> str:    """Retrieve the current status of a network device."""    return f"Device {device_id} is reachable"llm_with_tools = llm.bind_tools([get_device_status])response = llm_with_tools.invoke(    "Check the status of router R1.")print(response.tool_calls)

This example demonstrates how the model generates tool calls. The application or agent must execute the requested tools. The gateway and selected model must support tool calling.

5. Using multiple models through the same gateway

You can create different ChatOpenAI instances pointing to the same LiteLLM Gateway.

def get_llm(model_name: str):    return ChatOpenAI(        model=model_name,        base_url=os.environ["LITELLM_BASE_URL"],        api_key=os.environ["LITELLM_API_KEY"],    )planner_llm = get_llm("planner-model-id")supervisor_llm = get_llm("supervisor-model-id")summarizer_llm = get_llm("summarizer-model-id")

These IDs must correspond to models exposed by the customer's gateway and authorized for your API key.

6. Architecture

7. Important considerations

• Model discovery: Use the OpenAI SDK's client.models.list() or the gateway's /models endpoint to identify permitted model IDs, if enabled.

• Authentication: Use the actual sk-... API key, not the key alias or hash.

• Compatibility: Validate tool calling, streaming, structured outputs and model-specific parameters.

• Security: Store credentials in environment variables, secret managers or Kubernetes Secrets. Never hardcode them.

• Production: Configure timeouts, retries, TLS verification and appropriate error handling.

Summary

Requirement	Recommendation
Python library	langchain-openai
LangChain class	ChatOpenAI
Gateway configuration	base_url
Authentication	api_key
Model selection	model
LangGraph compatibility	Yes, subject to model capabilities
Additional LiteLLM SDK	Not required
Existing AzureChatOpenAI	Replace with ChatOpenAI for LiteLLM connections

Final recommendation: For your existing LangChain/LangGraph application, introduce LiteLLM as a configurable provider using ChatOpenAI. This allows you to reuse the same agent orchestration, prompts, tools and streaming interfaces while routing inference requests through the customer's centralized LLM Gateway.