Ollama: Your Local LLM Companion

ollama/ollama

Ollama is a command-line tool that makes it incredibly easy to run large language models (LLMs) locally on your own machine. Think of it as a Docker for LLMs. Instead of needing to set up complex environments, deal with dependencies, and configure models from scratch, you can just use a simple ollama run command.

From a software engineer's perspective, this is a game-changer for several reasons

Offline Development
You can develop and test applications that use LLMs without an internet connection. This is perfect for air-gapped environments or when you're on the go.

Cost Savings
No need to pay for API usage from services like OpenAI. Once you download a model, it's free to use as much as you want.

Privacy and Security
Your data never leaves your machine. This is crucial for applications that handle sensitive or proprietary information.

Rapid Prototyping
Experimenting with different models (e.g., Llama 3, DeepSeek-R1, Gemma 3) is fast and painless. You can quickly see which model is best suited for your specific task.

Customization
You can create your own custom models by fine-tuning existing ones. This is a powerful feature for building specialized applications.

Getting started is super straightforward. The process is typically just two steps.

First, you need to install the Ollama application for your operating system. Go to the official Ollama website and download the installer for macOS, Windows, or Linux. The installer sets up the command-line interface and the background service that runs the models.

Once installed, open your terminal and run a model. Let's try running Llama 3, which is a popular and powerful model.

ollama run llama3

The first time you run this command, Ollama will automatically download the llama3 model for you. This might take a few minutes depending on your internet speed, as these models can be several gigabytes in size. After the download is complete, you'll be dropped into an interactive chat session with the model. You can now start chatting with it directly in your terminal.

To exit the interactive session, just type /bye.

The real magic happens when you integrate Ollama into your software. Ollama provides a REST API that you can call from any programming language. It's easy to use and follows a familiar structure, just like many commercial LLM APIs.

Here's an example using Python, which is a popular language for AI and machine learning tasks. You'll need to install the ollama Python library first

pip install ollama

Let's write a quick script that uses Ollama to get a response from a model.

import ollama

# Create a client instance
client = ollama.Client(host='http://localhost:11434')

# Send a prompt to the model
response = client.chat(model='llama3', messages=[
  {
    'role': 'user',
    'content': 'Why is the sky blue?',
  },
])

# Print the model's response
print(response['message']['content'])

What's happening here?

We import the ollama library.

We create a Client object, which connects to the local Ollama service running on port 11434.

We use the client.chat() method to send a message to the llama3 model. The format of the messages list is a common standard in conversational AI APIs.

The model's response is a dictionary, and we access the actual content from response['message']['content'].

For a better user experience in a web application or terminal interface, you often want to stream the responses word by word, just like in a real-time chat. Ollama's API supports this out of the box.

Here's how you'd modify the Python script to stream the response

import ollama

client = ollama.Client(host='http://localhost:11434')

# Use `stream=True` to get responses as they're generated
stream = client.chat(
    model='llama3',
    messages=[{'role': 'user', 'content': 'Write a 1-paragraph story about a robot who finds a flower.'}],
    stream=True
)

# Iterate through the stream and print the content
for chunk in stream:
    # `chunk['message']['content']` holds the new part of the response
    print(chunk['message']['content'], end='', flush=True)

print() # Add a newline at the end

This code snippet demonstrates how to handle a streaming response, which is crucial for building a responsive user interface. You can adapt this same pattern to build anything from a simple command-line tool to a full-fledged web application.