How to Use Hosted & Tuned Models on Vertex AI

For enterprise-grade scalability, reliability, and integration with Google Cloud's MLOps ecosystem, you can use models deployed to Vertex AI Endpoints. This includes models from Model Garden or your own fine-tuned models.

Integration Method: Pass the full Vertex AI Endpoint resource string (projects/PROJECT_ID/locations/LOCATION/endpoints/ENDPOINT_ID) directly to the model parameter of LlmAgent.

Ensure your environment is configured for Vertex AI:

Authentication: Use Application Default Credentials (ADC):

gcloud auth application-default login

Environment Variables: Set your project and location:

export GOOGLE_CLOUD_PROJECT="YOUR_PROJECT_ID"

export GOOGLE_CLOUD_LOCATION="YOUR_VERTEX_AI_LOCATION" # e.g., us-central1

Enable Vertex Backend: Crucially, ensure the google-genai library targets Vertex AI:

export GOOGLE_GENAI_USE_VERTEXAI=TRUE

Model Garden Deployments

You can deploy various open and proprietary models from the Vertex AI Model Garden to an endpoint.

from google.adk.agents import LlmAgent

from google.genai import types # For config objects

# --- Example Agent using a Llama 3 model deployed from Model Garden ---

# Replace with your actual Vertex AI Endpoint resource name

llama3_endpoint = "projects/YOUR_PROJECT_ID/locations/us-central1/endpoints/YOUR_LLAMA3_ENDPOINT_ID"

agent_llama3_vertex = LlmAgent(

    model=llama3_endpoint,

    name="llama3_vertex_agent",

    instruction="You are a helpful assistant based on Llama 3, hosted on Vertex AI.",

    generate_content_config=types.GenerateContentConfig(max_output_tokens=2048),

    # ... other agent parameters

)

Fine-tuned Model Endpoints

from google.adk.agents import LlmAgent

# --- Example Agent using a fine-tuned Gemini model endpoint ---

# Replace with your fine-tuned model's endpoint resource name

finetuned_gemini_endpoint = "projects/YOUR_PROJECT_ID/locations/us-central1/endpoints/YOUR_FINETUNED_ENDPOINT_ID"

agent_finetuned_gemini = LlmAgent(

    model=finetuned_gemini_endpoint,

    name="finetuned_gemini_agent",

    instruction="You are a specialized assistant trained on specific data.",

    # ... other agent parameters

)

Third-Party Models on Vertex AI (e.g., Anthropic Claude)¶

Some providers, like Anthropic, make their models available directly through Vertex AI.

Integration Method: Uses the direct model string (e.g., "claude-3-sonnet@20240229"), but requires manual registration within ADK.

Why Registration? ADK's registry automatically recognizes gemini-* strings and standard Vertex AI endpoint strings (projects/.../endpoints/...) and routes them via the google-genai library. For other model types used directly via Vertex AI (like Claude), you must explicitly tell the ADK registry which specific wrapper class (Claude in this case) knows how to handle that model identifier string with the Vertex AI backend.

Setup:

Vertex AI Environment: Ensure the consolidated Vertex AI setup (ADC, Env Vars, GOOGLE_GENAI_USE_VERTEXAI=TRUE) is complete.

Install Provider Library: Install the necessary client library configured for Vertex AI.

pip install "anthropic[vertex]"

Register Model Class: Add this code near the start of your application, before creating an agent using the Claude model string:

# Required for using Claude model strings directly via Vertex AI with LlmAgent

from google.adk.models.anthropic_llm import Claude

from google.adk.models.registry import LLMRegistry

LLMRegistry.register(Claude)

from google.adk.agents import LlmAgent

from google.adk.models.anthropic_llm import Claude # Import needed for registration

from google.adk.models.registry import LLMRegistry # Import needed for registration

from google.genai import types

# --- Register Claude class (do this once at startup) ---

LLMRegistry.register(Claude)

# --- Example Agent using Claude 3 Sonnet on Vertex AI ---

# Standard model name for Claude 3 Sonnet on Vertex AI

claude_model_vertexai = "claude-3-sonnet@20240229"

agent_claude_vertexai = LlmAgent(

    model=claude_model_vertexai, # Pass the direct string after registration

    name="claude_vertexai_agent",

    instruction="You are an assistant powered by Claude 3 Sonnet on Vertex AI.",

    generate_content_config=types.GenerateContentConfig(max_output_tokens=4096),

    # ... other agent parameters

)