Simplifying LLM Integrations with LiteLLM

BerriAI/litellm

Let's dive into a really neat tool called LiteLLM. This isn't just another library; it's a game-changer for anyone working with Large Language Models (LLMs). Think of it as your universal adapter for all things LLM, simplifying your life significantly.

At its core, LiteLLM is a Python SDK and a proxy server (often called an LLM Gateway) that lets you interact with over 100 different LLM APIs using a single, unified interface – the familiar OpenAI format.

Why is this a big deal for us engineers?

Imagine you're building an application that needs to leverage the power of LLMs. In the past, if you wanted to experiment with, say, OpenAI's GPT models, then also try out Anthropic's Claude, and maybe even Bedrock's offerings, you'd have to learn and implement separate API calls, authentication methods, and error handling for each one. This leads to

Code Duplication
Lots of similar but slightly different code for each LLM.

Maintenance Headaches
Keeping up with API changes for multiple providers.

Vendor Lock-in
Making it hard to switch providers if a better, cheaper, or more performant option comes along.

Increased Complexity
Your codebase becomes a tangled mess of LLM integrations.

LiteLLM swoops in to solve all these problems. It provides a consistent "OpenAI-like" API for a vast array of LLMs, including

Amazon Bedrock

Azure OpenAI

OpenAI

Google Vertex AI

Cohere

Anthropic

AWS Sagemaker

HuggingFace

Replicate

Groq

and many, many more!

In short, LiteLLM offers

Unified API
Write your LLM interaction code once, and use it across different providers.

Provider Agnosticism
Easily switch between LLMs without rewriting core logic.

Simplified Development
Focus on your application's features, not on the nuances of each LLM API.

Cost Optimization
Dynamically route requests to the most cost-effective or performant LLM based on your needs.

Reliability
Built-in features for retries, fallbacks, and load balancing (especially with the proxy server).

Getting LiteLLM up and running is straightforward.

First, you'll need to install the Python package. You can do this using pip

pip install litellm

LiteLLM needs access to your API keys for the various LLM providers. The most common and recommended way to provide these is via environment variables. Here's an example of how you might set them up (replace YOUR_..._KEY with your actual keys)

# For OpenAI
export OPENAI_API_KEY="YOUR_OPENAI_API_KEY"

# For Anthropic
export ANTHROPIC_API_KEY="YOUR_ANTHROPIC_API_KEY"

# For AWS Bedrock (if using boto3, configure AWS credentials as usual)
# For specific Bedrock models, you might need to set region and profile
export AWS_ACCESS_KEY_ID="YOUR_AWS_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="YOUR_AWS_SECRET_ACCESS_KEY"
export AWS_REGION_NAME="YOUR_AWS_REGION" # e.g., us-east-1

# For Google Vertex AI (requires gcloud authentication, or set GOOGLE_APPLICATION_CREDENTIALS)
# export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/service-account-key.json"

Important
Never hardcode your API keys directly into your code! Use environment variables or a secure secret management system.

Let's look at some practical examples of how to use LiteLLM.

This will look very familiar if you've used the OpenAI Python library.

import litellm

# Call an OpenAI model
try:
    response = litellm.completion(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Hello, how are you today?"}]
    )
    print("OpenAI Response:")
    print(response.choices[0].message.content)
except Exception as e:
    print(f"Error calling OpenAI: {e}")

Now, let's switch to an Anthropic model. Notice how the litellm.completion call remains almost identical! Only the model parameter changes.

import litellm

# Call an Anthropic Claude model
try:
    response = litellm.completion(
        model="claude-3-opus-20240229", # Or "claude-3-sonnet-20240229", "claude-3-haiku-20240307"
        messages=[{"role": "user", "content": "What is the capital of France?"}]
    )
    print("\nAnthropic Claude Response:")
    print(response.choices[0].message.content)
except Exception as e:
    print(f"Error calling Anthropic: {e}")

LiteLLM supports Bedrock models seamlessly. For Bedrock, the model names are prefixed with bedrock/.

import litellm

# Call an Amazon Bedrock model (e.g., Anthropic Claude on Bedrock)
# Ensure your AWS credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION_NAME) are set
try:
    response = litellm.completion(
        model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0", # Example Bedrock model ID
        messages=[{"role": "user", "content": "Tell me a short story about a robot."}]
    )
    print("\nBedrock (Claude) Response:")
    print(response.choices[0].message.content)
except Exception as e:
    print(f"Error calling Bedrock: {e}")

# Example with an Amazon Titan model on Bedrock
try:
    response = litellm.completion(
        model="bedrock/amazon.titan-text-express-v1",
        messages=[{"role": "user", "content": "Describe the benefits of cloud computing."}]
    )
    print("\nBedrock (Titan) Response:")
    print(response.choices[0].message.content)
except Exception as e:
    print(f"Error calling Bedrock Titan: {e}")

One of the powerful features of LiteLLM is its ability to handle fallbacks. If one model fails, you can automatically try another.

import litellm

# Configure a list of models to try, in order of preference
# litellm will try the first one, if it fails, it will try the next, and so on.
try:
    response = litellm.completion(
        model=["gpt-4o", "claude-3-sonnet-20240229"],
        messages=[{"role": "user", "content": "Explain the concept of quantum entanglement."}],
        max_retries=3 # LiteLLM also supports automatic retries
    )
    print("\nFallback Model Response:")
    print(response.choices[0].message.content)
    print(f"Model used: {response.model}") # You can see which model was ultimately used
except Exception as e:
    print(f"Error with fallback models: {e}")

Beyond the Python SDK, LiteLLM also offers a proxy server. This is super useful for

Centralized API Key Management
Your applications only talk to the proxy, and the proxy manages all your individual LLM API keys securely.

Load Balancing
Distribute requests across multiple LLM providers or multiple instances of the same model.

Cost Management & Monitoring
Log requests, track costs, and get insights into LLM usage.

Caching
Cache common LLM responses to reduce latency and cost.

Rate Limiting
Protect your LLM APIs from being overwhelmed.

Streaming
The proxy can handle streaming responses from LLMs, just like direct API calls.

To run the proxy server

Create a config.yaml file

# config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
  - model_name: claude-3-sonnet
    litellm_params:
      model: claude-3-sonnet-20240229
  - model_name: bedrock-claude-sonnet
    litellm_params:
      model: bedrock/anthropic.claude-3-sonnet-20240229-v1:0
      # Ensure AWS credentials are set as environment variables where the proxy runs

Run the proxy

litellm --port 8000 --config config.yaml

This will start the proxy server, typically on http://0.0.0.0:8000.

Call the proxy from your application
Your application will then make requests to your local proxy server, using the OpenAI-compatible API.

import openai # Yes, you can even use the openai library to talk to your local litellm proxy!

# Point the OpenAI client to your local LiteLLM proxy
openai.api_base = "http://localhost:8000"
openai.api_key = "sk-123" # A dummy key is fine, as the proxy handles real keys

try:
    response = openai.chat.completions.create(
        model="gpt-4o", # This model name maps to what you defined in config.yaml
        messages=[{"role": "user", "content": "Tell me a joke."}]
    )
    print("\nProxy Server Response (via OpenAI client):")
    print(response.choices[0].message.content)
except Exception as e:
    print(f"Error calling proxy: {e}")

Rapid Prototyping
Quickly experiment with different LLMs without changing your core code.

Future-Proofing
Easily swap out LLMs as new, better, or cheaper options emerge.

Abstraction Layer
It provides a clean abstraction over the complexities of various LLM APIs.

Microservices Friendly
The proxy server can be deployed as a dedicated service, providing LLM access to all your microservices.

Cost Control & Optimization
Implement routing logic to prioritize cost-effective models or highly performant ones based on your use case.

Reliability
Built-in retry and fallback mechanisms improve the robustness of your LLM integrations.