From Zero to Code: Integrating Local LLMs with ollama-python

ollama/ollama-python

This library is essentially a friendly Python interface for the Ollama system, which allows you to run large language models (LLMs) locally on your machine.

From a software engineer's perspective, the ollama Python library is incredibly useful because it localizes and simplifies the use of LLMs. Here are the main benefits

You can run powerful models entirely on your own hardware, meaning no data is sent to external services. This is crucial for

Handling sensitive or proprietary data.

Developing applications in highly regulated industries (e.g., finance, healthcare).

By running models locally, you eliminate API call costs associated with cloud-based LLM services. This is a huge win for

Prototyping and experimentation, where costs can quickly add up.

Applications with high volume, low-latency requirements.

Your application can use LLM capabilities without an internet connection, provided the model is downloaded locally. This is essential for

Edge computing devices or remote environments.

Desktop applications that require local processing.

The library provides easy-to-use functions for interacting with LLMs, abstracting away the complexities of model loading, data serialization, and communication with the Ollama server. You can

Generate text, chat, create embeddings, and manage models—all with simple Python function calls.

Quickly swap between different models (like Llama 3, Mistral, etc.) by just changing a string.

Getting the ollama Python library set up involves two main steps

Before using the Python library, you need to have the Ollama application running on your system. This is the background process that actually handles the LLM execution.

Download & Install Ollama
Go to the official Ollama website and follow the instructions to install the core application for your operating system (macOS, Linux, Windows).

Pull a Model
Use the Ollama CLI (command line interface) to download a model you want to use. For example, to get a popular and fast model

ollama pull llama3

This downloads the llama3 model and makes it ready for use.

Once Ollama is running, you can install the Python client in your project environment.

pip install ollama

Here are a few practical examples of how to use the library for common tasks.

This is the simplest use case
providing a prompt and getting a single response.

import ollama

# Initialize the client (connects to the local Ollama server)
client = ollama.Client()

# Define the model you pulled earlier (e.g., llama3, mistral, etc.)
MODEL_NAME = 'llama3'

# Generate a response to a prompt
prompt = "Explain the concept of 'technical debt' in a short paragraph."

response = client.generate(
  model=MODEL_NAME,
  prompt=prompt
)

# The generated text is in the 'response' dictionary
print("--- LLM Response ---")
print(response['response'])
# print(f"Total duration: {response['total_duration']} ns") # You also get useful metrics!

The chat endpoint is perfect for multi-turn conversations, as it manages the conversation history for you.

import ollama

client = ollama.Client()
MODEL_NAME = 'llama3'

messages = [
    {
        'role': 'system',
        'content': 'You are a helpful and witty programming assistant.'
    },
    {
        'role': 'user',
        'content': 'Write a Python function to calculate the factorial of a number.'
    }
]

print("--- Initial Query ---")
chat_response = client.chat(
    model=MODEL_NAME,
    messages=messages
)

# Print the model's response
print(chat_response['message']['content'])

# --- Add the history and a new prompt for the second turn ---
messages.append(chat_response['message']) # Add the assistant's previous response
messages.append({
    'role': 'user',
    'content': 'Now, make the function recursive.'
})

print("\n--- Follow-up Query (Context Maintained) ---")
follow_up_response = client.chat(
    model=MODEL_NAME,
    messages=messages
)

# Print the model's response, which is aware of the previous turn
print(follow_up_response['message']['content'])

Embeddings are crucial for many advanced applications like semantic search or Retrieval-Augmented Generation (RAG). This function lets you create them locally.

import ollama

client = ollama.Client()
# Use an embedding-optimized model, like 'nomic-embed-text'
EMBEDDING_MODEL = 'nomic-embed-text' 
# Make sure to run 'ollama pull nomic-embed-text' first!

text_to_embed = "The key to RAG is generating high-quality vector representations."

embedding_result = client.embeddings(
  model=EMBEDDING_MODEL,
  prompt=text_to_embed
)

# The 'embedding' key contains the list of floats (the vector)
embedding_vector = embedding_result['embedding']

print(f"Original Text: {text_to_embed}")
print(f"Embedding Vector Length: {len(embedding_vector)}") 
print(f"First 5 dimensions: {embedding_vector[:5]}")