Build a Local AI Agent with Ollama: Private Models, Tools, and Memory



Build a Local AI Agent with Ollama: Private Models, Tools, and Memory

🚀 Quick Answer / TL;DR

To **build a local AI agent with Ollama**, you'll leverage Ollama to run large language models (LLMs) privately on your machine. This involves installing Ollama, pulling a suitable model, and using Python to program the agent's logic. Key components include defining the agent's persona, enabling it to use custom "tools" (functions) for specific tasks, and implementing "memory" to maintain conversational context across interactions. This setup allows for powerful, secure, and offline AI applications, offering a robust alternative to cloud-based LLMs.

In an era increasingly dominated by cloud-based AI, the need for privacy, cost-effectiveness, and low-latency inference is pushing developers towards local solutions. Large Language Models (LLMs) have revolutionized how we interact with technology, but deploying them locally and building intelligent agents around them often presents unique challenges. This comprehensive guide will walk you through the process to build a local AI agent with Ollama, empowering you to create powerful, private, and secure AI applications right on your own machine.

Ollama simplifies running open-source LLMs locally, providing an easy-to-use interface for downloading, managing, and interacting with various models. By combining Ollama’s capabilities with Python for agent orchestration, we can craft sophisticated AI agents capable of understanding context, utilizing external tools, and remembering past interactions, all without sending sensitive data to external servers.

What You Will Learn

Table of Contents

Setting the Stage: What is a Local AI Agent?

An AI agent is an autonomous software entity designed to perceive its environment, make decisions, and take actions to achieve specific goals. When we talk about a local AI agent, we refer to an agent whose core intelligence — the Large Language Model (LLM) — runs entirely on your local machine, rather than relying on cloud-based APIs.

The advantages of this approach are significant:

The three pillars of a sophisticated AI agent are:

  1. Private Models: An LLM running locally (e.g., via Ollama) provides the agent’s reasoning capabilities.
  2. Tools: Functions or APIs the agent can call to interact with the outside world, retrieve information, or perform specific actions (e.g., a calculator, a web search, interacting with local files).
  3. Memory: The ability to retain context from past interactions, allowing for more coherent and extended conversations or task sequences.

This combination allows us to build a local AI agent with Ollama that is not just a chatbot, but a proactive assistant. Let’s start by getting Ollama up and running.

Step 1: Install Ollama and Your First Local LLM

Ollama is a fantastic tool that makes running large language models locally incredibly simple. It handles model weights, GPU acceleration, and exposes a clean API.

1.1 Install Ollama

Visit the official Ollama website (ollama.com) and download the installer for your operating system (macOS, Windows, Linux).

After installation, Ollama will run as a background service. You can verify it by running ollama -v in your terminal.

1.2 Download Your First Model

Ollama has a model library with many popular open-source LLMs. For this tutorial, we’ll start with a smaller, efficient model like llama2 or mistral to ensure good performance on most machines.

To pull a model, open your terminal and type:

1
2
3

ollama pull mistral
# Or
ollama pull llama2

This command downloads the model weights to your machine. Once downloaded, you can interact with it directly from the terminal:

1
2

ollama run mistral
>>> Hi Ollama!

You’re now running an LLM locally! Next, we’ll connect to this model using Python.

Step 2: Building the Agent Core – Basic Interaction with Ollama

The heart of our local AI agent is its ability to communicate with the Ollama service. We’ll use the ollama Python client for this.

2.1 Set Up Your Python Environment

First, create a new project directory and a virtual environment:

1
2
3
4
5

mkdir local_ai_agent
cd local_ai_agent
python -m venv venv
source venv/bin/activate # On Windows: .\venv\Scripts\activate
pip install ollama

2.2 Basic Agent Interaction Script

Now, let’s write a simple Python script to send prompts to our local mistral model and get responses.

Create a file named agent_core.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

import ollama

def simple_chat(model_name: str, prompt: str) -> str:
    """
    Sends a single prompt to a local Ollama model and returns the response.
    """
    try:
        response = ollama.chat(model=model_name, messages=[
            {'role': 'user', 'content': prompt}
        ])
        return response['message']['content']
    except Exception as e:
        return f"Error interacting with Ollama: {e}"

if __name__ == "__main__":
    model = "mistral" # Ensure you have 'mistral' pulled with 'ollama pull mistral'
    
    print(f"--- Basic Interaction with {model} ---")
    
    user_prompt = "What is the capital of France?"
    print(f"User: {user_prompt}")
    agent_response = simple_chat(model, user_prompt)
    print(f"Agent: {agent_response}")

    user_prompt_2 = "Tell me a short, funny story about a cat."
    print(f"\nUser: {user_prompt_2}")
    agent_response_2 = simple_chat(model, user_prompt_2)
    print(f"Agent: {agent_response_2}")

Run this script:

1

python agent_core.py

You should see your local LLM responding to your prompts. This forms the foundational interaction for your agent. However, a truly intelligent agent needs to do more than just chat; it needs to act. This brings us to the concept of tools.

Step 3: Empowering Your Agent with Tools (Function Calling)

Tools are functions or APIs that your AI agent can call to extend its capabilities beyond pure text generation. Ollama, similar to other LLM frameworks, supports “function calling” (often referred to as tool use). This allows the LLM to understand when and how to invoke external functions based on the user’s request.

3.1 Define a Simple Tool

Let’s create a tool that performs a simple calculation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

# calculator_tool.py
def add(a: float, b: float) -> float:
    """Adds two numbers."""
    return a + b

def subtract(a: float, b: float) -> float:
    """Subtracts two numbers."""
    return a - b

def multiply(a: float, b: float) -> float:
    """Multiplies two numbers."""
    return a * b

def divide(a: float, b: float) -> float:
    """Divides two numbers. Handles division by zero."""
    if b == 0:
        raise ValueError("Cannot divide by zero!")
    return a / b

# A dictionary to easily access our tools by name
TOOLS = {
    "add": add,
    "subtract": subtract,
    "multiply": multiply,
    "divide": divide,
}

# The schema definition for Ollama, describing our tools
CALCULATOR_TOOL_SCHEMA = [
    {
        'type': 'function',
        'function': {
            'name': 'add',
            'description': 'Adds two numbers together.',
            'parameters': {
                'type': 'object',
                'properties': {
                    'a': {'type': 'number', 'description': 'The first number'},
                    'b': {'type': 'number', 'description': 'The second number'}
                },
                'required': ['a', 'b']
            }
        }
    },
    {
        'type': 'function',
        'function': {
            'name': 'subtract',
            'description': 'Subtracts the second number from the first.',
            'parameters': {
                'type': 'object',
                'properties': {
                    'a': {'type': 'number', 'description': 'The first number'},
                    'b': {'type': 'number', 'description': 'The second number'}
                },
                'required': ['a', 'b']
            }
        }
    },
    {
        'type': 'function',
        'function': {
            'name': 'multiply',
            'description': 'Multiplies two numbers together.',
            'parameters': {
                'type': 'object',
                'properties': {
                    'a': {'type': 'number', 'description': 'The first number'},
                    'b': {'type': 'number', 'description': 'The second number'}
                },
                'required': ['a', 'b']
            }
        }
    },
    {
        'type': 'function',
        'function': {
            'name': 'divide',
            'description': 'Divides the first number by the second.',
            'parameters': {
                'type': 'object',
                'properties': {
                    'a': {'type': 'number', 'description': 'The dividend'},
                    'b': {'type': 'number', 'description': 'The divisor'}
                },
                'required': ['a', 'b']
            }
        }
    }
]

3.2 Integrate Tools with Ollama

Now, let’s modify our agent_core.py to allow the LLM to use these tools.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

import ollama
import json
from calculator_tool import TOOLS, CALCULATOR_TOOL_SCHEMA # Import our tools

def chat_with_tools(model_name: str, messages: list) -> str:
    """
    Sends messages to an Ollama model, handling tool calls.
    """
    try:
        response = ollama.chat(model=model_name, messages=messages, tools=CALCULATOR_TOOL_SCHEMA)
        
        # Check if the model decided to call a tool
        if response['message'].get('tool_calls'):
            tool_calls = response['message']['tool_calls']
            print(f"Agent wants to call tools: {tool_calls}")

            # For simplicity, we'll process the first tool call.
            # In a real agent, you might handle multiple concurrent calls.
            tool_call = tool_calls[0] 
            tool_name = tool_call['function']['name']
            tool_args = tool_call['function']['arguments']

            if tool_name in TOOLS:
                print(f"Executing tool: {tool_name} with args: {tool_args}")
                try:
                    tool_output = TOOLS[tool_name](**tool_args)
                    print(f"Tool output: {tool_output}")
                    
                    # Add the tool call response to the messages for the LLM to see
                    messages.append(response['message']) # original message asking for tool call
                    messages.append({
                        'role': 'tool',
                        'content': json.dumps({"result": tool_output}), # Tool's actual output
                    })
                    
                    # Make another call to the LLM with the tool's result
                    final_response = ollama.chat(model=model_name, messages=messages)
                    return final_response['message']['content']
                except Exception as e:
                    error_msg = f"Error executing tool '{tool_name}': {e}"
                    messages.append(response['message'])
                    messages.append({
                        'role': 'tool',
                        'content': json.dumps({"error": error_msg}),
                    })
                    final_response = ollama.chat(model=model_name, messages=messages)
                    return final_response['message']['content']
            else:
                return f"Agent requested unknown tool: {tool_name}"
        else:
            return response['message']['content']
            
    except Exception as e:
        return f"Error interacting with Ollama: {e}"

if __name__ == "__main__":
    model = "mistral" # Ensure model supports function calling (e.g., mistral, codellama, llama3)
    # Note: Llama2 generally doesn't have strong function calling capabilities.
    # Consider 'mistral', 'codellama', or 'llama3' for better results with tools.
    
    print(f"--- Interaction with {model} and Tools ---")
    
    # Example 1: Basic calculation
    initial_messages_calc = [{'role': 'user', 'content': 'What is 123 plus 456?'}]
    print(f"User: {initial_messages_calc[0]['content']}")
    agent_response_calc = chat_with_tools(model, initial_messages_calc)
    print(f"Agent: {agent_response_calc}")

    print("\n--- Another calculation ---")
    initial_messages_divide = [{'role': 'user', 'content': 'Divide 100 by 25.'}]
    print(f"User: {initial_messages_divide[0]['content']}")
    agent_response_divide = chat_with_tools(model, initial_messages_divide)
    print(f"Agent: {agent_response_divide}")

    print("\n--- Non-tool related question ---")
    initial_messages_story = [{'role': 'user', 'content': 'Tell me a joke.'}]
    print(f"User: {initial_messages_story[0]['content']}")
    agent_response_story = chat_with_tools(model, initial_messages_story)
    print(f"Agent: {agent_response_story}")

Remember to use a model like mistral or llama3 as they have better function calling capabilities. If you get errors or the model doesn’t call the tool, try a different model (ollama pull llama3).

By defining tools and their schemas, we’ve significantly enhanced our agent’s ability to act upon user requests. The next crucial step is to enable it to remember past interactions.

Step 4: Giving Your Agent Memory (Context Management)

For an AI agent to have a meaningful conversation or complete multi-step tasks, it needs memory. This means retaining the context of previous messages and using them to inform future responses. Ollama’s chat API naturally supports this by accepting a list of messages.

4.1 Implementing Conversational Memory

We’ll store the conversation history in a Python list and pass it with each new request.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

import ollama
import json
from calculator_tool import TOOLS, CALCULATOR_TOOL_SCHEMA # Import our tools

# Redefine the chat_with_tools to accept and return the full message history
def conversational_chat_with_tools(model_name: str, message_history: list, new_user_message: str) -> tuple[str, list]:
    """
    Manages conversational memory and tool calls for an Ollama agent.
    Returns the agent's response and the updated message history.
    """
    message_history.append({'role': 'user', 'content': new_user_message})

    try:
        response = ollama.chat(model=model_name, messages=message_history, tools=CALCULATOR_TOOL_SCHEMA)
        
        # Check if the model decided to call a tool
        if response['message'].get('tool_calls'):
            tool_calls = response['message']['tool_calls']
            print(f"DEBUG: Agent wants to call tools: {tool_calls}")

            # Add the model's decision to call the tool to history
            message_history.append(response['message']) 
            
            tool_call = tool_calls[0]
            tool_name = tool_call['function']['name']
            tool_args = tool_call['function']['arguments']

            if tool_name in TOOLS:
                print(f"DEBUG: Executing tool: {tool_name} with args: {tool_args}")
                try:
                    tool_output = TOOLS[tool_name](**tool_args)
                    print(f"DEBUG: Tool output: {tool_output}")
                    
                    # Add the tool's output to the message history
                    message_history.append({
                        'role': 'tool',
                        'content': json.dumps({"result": tool_output}), 
                    })
                    
                    # Make another call to the LLM with the tool's result to get a user-friendly response
                    final_response = ollama.chat(model=model_name, messages=message_history)
                    message_history.append(final_response['message']) # Add final response to history
                    return final_response['message']['content'], message_history
                except Exception as e:
                    error_msg = f"Error executing tool '{tool_name}': {e}"
                    print(f"DEBUG: {error_msg}")
                    message_history.append({
                        'role': 'tool',
                        'content': json.dumps({"error": error_msg}),
                    })
                    final_response = ollama.chat(model=model_name, messages=message_history)
                    message_history.append(final_response['message'])
                    return final_response['message']['content'], message_history
            else:
                return f"Agent requested unknown tool: {tool_name}", message_history
        else:
            # If no tool call, just add the agent's response to history
            message_history.append(response['message'])
            return response['message']['content'], message_history
            
    except Exception as e:
        return f"Error interacting with Ollama: {e}", message_history

if __name__ == "__main__":
    model = "mistral"
    conversation_history = [] # Initialize empty history

    print(f"--- Conversational Agent with {model} ---")
    print("Type 'exit' to end the conversation.")

    while True:
        user_input = input("\nUser: ")
        if user_input.lower() == 'exit':
            break
        
        agent_response, conversation_history = conversational_chat_with_tools(
            model, conversation_history, user_input
        )
        print(f"Agent: {agent_response}")

Run this script and try a conversation:

1

python agent_with_memory.py

Example interaction:

1
2
3
4
5

User: What is 5 * 10?
Agent: 5 * 10 is 50.

User: And what is that plus 20?
Agent: 50 + 20 is 70.

Notice how the agent correctly uses the result from the previous turn (50) even though you didn’t explicitly state it again. This is the power of conversational memory! For long conversations, you might need to implement more advanced memory management, like summarizing old turns or using vector databases for long-term memory, but this basic approach is a solid start.

Step 5: Orchestrating a Full-Fledged Local AI Agent

Now that we understand how to interact with Ollama, use tools, and maintain memory, let’s combine these elements to build a local AI agent with Ollama that can handle a more complex use case. We’ll create a “Research Assistant” agent that can perform calculations and potentially search for information (if we were to implement a web search tool, for example).

For this example, we’ll keep the calculator tool, but imagine extending it with a search_web tool or a read_local_file tool.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103

import ollama
import json
from calculator_tool import TOOLS, CALCULATOR_TOOL_SCHEMA # Our calculator tools

class ResearchAgent:
    def __init__(self, model_name: str = "mistral"):
        self.model_name = model_name
        self.conversation_history = []
        self._initialize_agent_persona()

    def _initialize_agent_persona(self):
        """Sets up the initial system message for the agent's persona."""
        system_message = {
            'role': 'system',
            'content': (
                "You are a helpful Research Assistant named 'OllamaBot'. "
                "Your primary goal is to assist users with their questions, perform calculations "
                "when needed using the available tools, and maintain a friendly, informative tone. "
                "Always try to answer the user's question directly if you can, or use your tools "
                "if it's a computational task. If you don't know an answer, politely say so."
            )
        }
        self.conversation_history.append(system_message)
        print("OllamaBot Research Assistant initialized!")

    def _call_tool(self, tool_name: str, tool_args: dict):
        """Executes a specified tool with provided arguments."""
        if tool_name in TOOLS:
            try:
                print(f"DEBUG: Executing tool: {tool_name} with args: {tool_args}")
                return TOOLS[tool_name](**tool_args)
            except Exception as e:
                return f"Error: {e}"
        else:
            return f"Error: Unknown tool '{tool_name}'."

    def process_message(self, user_message: str) -> str:
        """
        Processes a user message, updates conversation history, and handles tool calls.
        """
        self.conversation_history.append({'role': 'user', 'content': user_message})

        try:
            # First LLM call with current history and tools
            response = ollama.chat(
                model=self.model_name,
                messages=self.conversation_history,
                tools=CALCULATOR_TOOL_SCHEMA,
                stream=False # For simplicity, not streaming for this example
            )
            
            # Add the model's response (potentially a tool call) to history
            self.conversation_history.append(response['message'])

            # Check for tool calls
            if response['message'].get('tool_calls'):
                tool_calls = response['message']['tool_calls']
                print(f"DEBUG: Agent identified tool calls: {tool_calls}")

                # Process all tool calls (for simplicity, we process them sequentially here)
                for tool_call in tool_calls:
                    tool_name = tool_call['function']['name']
                    tool_args = tool_call['function']['arguments']
                    
                    tool_output = self._call_tool(tool_name, tool_args)
                    print(f"DEBUG: Tool '{tool_name}' output: {tool_output}")

                    # Add tool output to history
                    self.conversation_history.append({
                        'role': 'tool',
                        'content': json.dumps({"tool_name": tool_name, "output": tool_output}),
                    })
                
                # Make a second LLM call with tool results in history
                final_response = ollama.chat(
                    model=self.model_name,
                    messages=self.conversation_history,
                    stream=False
                )
                self.conversation_history.append(final_response['message'])
                return final_response['message']['content']
            else:
                # No tool call, just return the LLM's direct response
                return response['message']['content']
                
        except Exception as e:
            error_msg = f"An error occurred: {e}"
            print(f"DEBUG: {error_msg}")
            # Optionally add the error to history for the LLM to see, or just return
            return "I encountered an error trying to process that. Could you please rephrase?"

if __name__ == "__main__":
    agent = ResearchAgent(model_name="mistral") # Use a model with good function calling, e.g., mistral, llama3
    print("Welcome! I'm OllamaBot, your local Research Assistant. Type 'exit' to quit.")

    while True:
        user_input = input("\nYou: ")
        if user_input.lower() == 'exit':
            print("OllamaBot: Goodbye! Have a great day!")
            break
        
        agent_response = agent.process_message(user_input)
        print(f"OllamaBot: {agent_response}")

To run this full agent:

1

python full_agent.py

Now you have a fully functional local AI agent that can maintain conversation context, leverage tools for specific tasks, and operate entirely offline. This demonstrates the power and flexibility of using Ollama to build a local AI agent with Ollama, providing a robust foundation for many private AI applications. From here, you can expand its capabilities by adding more complex tools (e.g., interacting with local APIs, data analysis, file operations) and refining its memory management.

Frequently Asked Questions

Further Reading

  1. Ollama Official Website: The primary resource for installation, model library, and documentation.
  2. Ollama Python Library Documentation: Deep dive into the Python client API for advanced interactions.
  3. LangChain & LlamaIndex: Explore these frameworks for more advanced agent orchestration, diverse tool integrations, and complex memory management techniques. While this guide focused on pure Ollama, these frameworks can abstract much of the agent logic.

Ready to take your local AI projects to the next level? Explore more advanced AI agent patterns and integration strategies on the CodeCrux blog, or contact us for expert consultation on building robust, private AI solutions tailored to your needs.



Empower Your Business with Our Expert Solutions

Unlock the full potential of your projects with our professional services!

Get Started Today