Run LLM Agents Locally with Ollama in Python

You can run an LLM agent entirely on your own computer. Nothing is sent to a hosted model, you need no account or API key, and it costs nothing per request. This page gets you there with Ollama, which runs the model, and Maki, an open-source Python framework for agents.

1. Install Ollama and pull a small model

Install Ollama from ollama.com and start it. Then pull a model. A small one is the right place to start:

ollama pull qwen3.5:4b

That model is 3.4 GB on disk and runs on a modest laptop. A larger one such as gemma4:26b is 17 GB on disk and needs at least about that much free memory. Larger models write better, but the 4B model is enough to learn on.

2. Install Maki

python -m venv .venv
source .venv/bin/activate
pip install maki-framework

The package is maki-framework. The name maki on PyPI is a different project, so check the spelling.

3. Run an agent

This is the whole program:

"""Chapter 2: your first agent.

Run it with:  python ch02_first_agent/scout.py
Needs a model server running (Ollama by default, see chapter 1).
To use llama.cpp or vLLM instead:  SCOUT_BACKEND=llamacpp python ch02_first_agent/scout.py
"""
import os

from maki import init_backend
from maki.agents import Agent

DEFAULT_SPEC = "ollama:qwen3.5:4b"   # 3.4 GB. Bigger machine: "ollama:gemma4:26b".


def make_backend(spec=None):
    """The model Scout talks to, named by a "provider:model" string.

    Set SCOUT_BACKEND to use another server, for example "llamacpp" or
    "vllm:Qwen/Qwen2.5-1.5B-Instruct". Nothing else in the program changes.
    """
    spec = spec or os.environ.get("SCOUT_BACKEND", DEFAULT_SPEC)
    # think=False is an Ollama setting (explained in the chapter); other backends lack it.
    options = {"think": False} if spec.split(":")[0] == "ollama" else {}
    return init_backend(spec, **options)


def build_agent(backend=None):
    """Scout, a research assistant. Pass a backend to swap the model."""
    return Agent(
        name="Scout",
        maki_instance=backend or make_backend(),
        role="research assistant",
        instructions="Answer in three short sentences. Say so when you are unsure.",
    )


def main(backend=None):
    scout = build_agent(backend)
    answer = scout.execute_task("In plain words, what is a multi-agent LLM application?")
    print(answer)
    return answer


if __name__ == "__main__":
    main()

Save it, or take it from the course code, and run python scout.py. The first run is slow while Ollama loads the model into memory. Later runs are faster, because the model is already loaded.

Things worth knowing

Thinking models. qwen3.5 writes out its reasoning before it answers. think=False turns that off, which is faster and avoids a failure where the reasoning gets returned as the answer.
Switching models. Change DEFAULT_SPEC at the top. Nothing else in the code changes.
Other local servers. llama.cpp and vLLM work the same way, by setting SCOUT_BACKEND. Chapter 2 shows how.
Small models make mistakes. They misspell names and can state things a source does not say. Check anything that matters.
Moving to a hosted model. Maki can run the same agent on OpenAI or Anthropic by swapping the model object, so you can start local and move one task later.

The Learning course goes from this first agent to an agent that calls a tool and then a small pipeline. Start with what Maki is, or go straight to giving your agent a tool.