Build a Multi-Modal AI Agent with Vision, Documents, and Function Calling



Build a Multi-Modal AI Agent with Vision, Documents, and Function Calling

Quick Answer / TL;DR:

This guide walks you through building a powerful **multi-modal AI agent** that can interpret images, understand document content, and execute external functions. By leveraging large language models with vision capabilities, Retrieval-Augmented Generation (RAG) for document context, and robust function calling, you'll create an intelligent agent capable of tackling complex, real-world problems requiring diverse information sources and actions.

The landscape of Artificial Intelligence is rapidly evolving, moving beyond text-only interactions to embrace a richer, more human-like understanding of the world. At the forefront of this revolution are multi-modal AI agents – intelligent systems capable of processing and synthesizing information from various modalities like text, images, and even audio. When augmented with the ability to call external functions, these agents transform from passive responders into active problem-solvers, interacting with digital environments and real-world APIs.

This comprehensive tutorial will guide you step-by-step through the process of building your own sophisticated multi-modal AI agent that combines vision, document understanding, and function calling capabilities. Whether you’re a seasoned AI developer or just starting, you’ll gain practical insights and a runnable codebase to create agents that can see, read, and act.

What You Will Learn

Table of Contents

Setting the Stage: Understanding Multi-Modal Agents

A multi-modal AI agent extends the capabilities of traditional Large Language Models (LLMs) by allowing them to process and generate information across multiple data types, or “modalities.” While LLMs excel at text, a multi-modal agent can simultaneously interpret an image, read a document, and then decide on an action.

Think of an agent that can:

This combination makes agents far more versatile and powerful, enabling them to tackle tasks that closely mimic human decision-making processes. We’ll be focusing on a multi-modal AI agent with vision, documents, and function calling to unlock a wide range of applications.

Now that we understand the power of such agents, let’s prepare our workspace.

Setting Up Your Development Environment

To begin, we’ll need a Python environment with the necessary libraries. We’ll primarily use LangChain for orchestration, OpenAI for LLM capabilities (including vision and function calling), and LlamaIndex (or similar) for document processing.

Prerequisites

Step 1: Create a Virtual Environment

It’s good practice to isolate your project dependencies.

1
2

python -m venv multi-modal-agent-env
source multi-modal-agent-env/bin/activate  # On Windows, use `multi-modal-agent-env\Scripts\activate`

Step 2: Install Required Libraries

We’ll install langchain for agent orchestration, openai for accessing GPT models, python-dotenv for managing environment variables, Pillow for image processing, and chromadb for our vector store (you could also use FAISS or another).

1

pip install langchain langchain-openai python-dotenv Pillow pypdf chromadb

Step 3: Set Up Your API Key

Create a .env file in your project root to store your OpenAI API key securely.

1
2

# .env
OPENAI_API_KEY="YOUR_OPENAI_API_KEY_HERE"

Then, in your Python script, load it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# main.py
import os
from dotenv import load_dotenv

load_dotenv() # Load environment variables from .env file

openai_api_key = os.getenv("OPENAI_API_KEY")
if not openai_api_key:
    raise ValueError("OPENAI_API_KEY not found. Please set it in a .env file.")

# Verify installation (optional)
# from langchain_openai import ChatOpenAI
# llm = ChatOpenAI(model="gpt-4o")
# print(llm.invoke("Hello, multi-modal world!"))

With our environment ready, let’s dive into adding vision capabilities to our agent.

Integrating Vision Capabilities

Modern LLMs like OpenAI’s GPT-4o are inherently multi-modal, meaning they can directly process image inputs alongside text. We’ll leverage this to give our agent “eyes.”

Step 1: Prepare Your Image Input

You can provide images as base64 encoded strings or URLs. For local files, base64 encoding is convenient.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

import base64
from pathlib import Path
from langchain_core.messages import HumanMessage
from langchain_openai import ChatOpenAI

def encode_image(image_path):
    """Encodes an image to base64 string."""
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")

# Create a dummy image for demonstration
# You can replace this with a real image path
dummy_image_path = "example_image.png"
from PIL import Image
Image.new('RGB', (60, 30), color = 'red').save(dummy_image_path)

# Example image path
# image_path = "path/to/your/image.jpg"
base64_image = encode_image(dummy_image_path)

Step 2: Query the Vision-Enabled LLM

We can now construct a HumanMessage that includes both text and image content.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

llm_vision = ChatOpenAI(model="gpt-4o", max_tokens=1024)

# Create a multi-modal message
message = HumanMessage(
    content=[
        {"type": "text", "text": "What is in this image? Describe it briefly."},
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{base64_image}"}},
    ]
)

# Invoke the LLM
# response = llm_vision.invoke([message])
# print("Vision Response:", response.content)
# Expected: "The image is a plain red rectangle."

For a truly integrated agent, we’ll want to wrap this vision capability as a tool. This allows the agent to decide when to use vision.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

from langchain.tools import tool

@tool
def analyze_image(image_path: str) -> str:
    """Analyzes an image provided by a local path and describes its content."""
    try:
        base64_image = encode_image(image_path)
        message = HumanMessage(
            content=[
                {"type": "text", "text": "Describe the contents of this image in detail."},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{base64_image}"}},
            ]
        )
        response = llm_vision.invoke([message])
        return response.content
    except Exception as e:
        return f"Error analyzing image: {e}"

# Example usage of the tool (for testing)
# print(analyze_image(dummy_image_path))

This analyze_image tool can now be presented to our agent, allowing it to dynamically choose to “look” at an image when needed.

Next, we’ll equip our agent with the ability to “read” by integrating document understanding.

Incorporating Document Understanding with RAG

Retrieval-Augmented Generation (RAG) is crucial for giving our agent access to a vast, up-to-date, and domain-specific knowledge base that isn’t pre-trained into the LLM. This involves indexing documents and retrieving relevant chunks based on a query.

Step 1: Prepare Your Documents

Let’s create some dummy documents. In a real scenario, these would be PDFs, text files, web pages, etc.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# Create dummy documents
Path("data").mkdir(exist_ok=True)
with open("data/product_manual.txt", "w") as f:
    f.write("The CodeCrux AI Agent Framework supports multi-modal inputs, including vision and document processing. It integrates with various LLMs and vector databases like ChromaDB. Key features include dynamic tool orchestration and secure API integration.")
with open("data/support_faq.txt", "w") as f:
    f.write("For support with the CodeCrux AI Agent Framework, visit our documentation portal or contact our 24/7 support team. Common issues are resolved by checking API key configurations and network connectivity.")

# Load documents
from langchain_community.document_loaders import TextLoader

loader_manual = TextLoader("data/product_manual.txt")
loader_faq = TextLoader("data/support_faq.txt")

docs = loader_manual.load() + loader_faq.load()

Step 2: Split Documents into Chunks

Large documents need to be split into smaller, more manageable chunks for efficient retrieval.

1
2
3
4

from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = text_splitter.split_documents(docs)

Step 3: Create Embeddings and a Vector Store

Embeddings convert text chunks into numerical vectors, allowing us to find semantically similar chunks. A vector store (like ChromaDB) stores these embeddings and facilitates fast retrieval.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

# Initialize embeddings model
embeddings = OpenAIEmbeddings(openai_api_key=openai_api_key)

# Create a Chroma vector store from the chunks
# This will create a local persistent store in './chroma_db'
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)

# You can also load an existing one:
# vectorstore = Chroma(persist_directory="./chroma_db", embedding_function=embeddings)

retriever = vectorstore.as_retriever()

Step 4: Wrap the Retriever as a Tool

Similar to vision, we’ll make our document retriever a tool that the agent can invoke.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

from langchain.tools import tool

@tool
def retrieve_document_context(query: str) -> str:
    """
    Retrieves relevant document chunks based on a query.
    Useful for answering questions about product manuals, FAQs, or any stored knowledge.
    """
    try:
        docs = retriever.invoke(query)
        # Combine retrieved content into a single string
        context = "\n\n".join([doc.page_content for doc in docs])
        return context
    except Exception as e:
        return f"Error retrieving document context: {e}"

# Example usage (for testing)
# print(retrieve_document_context("What are the key features of the AI Agent Framework?"))

Now our agent has “eyes” and can “read” internal documents. The next step is to give it “hands” to interact with the world.

Enabling Function Calling for External Tools

Function calling allows the LLM to interact with external APIs or custom code. The agent will decide which tool to use, when, and with what arguments, based on the user’s prompt.

Step 1: Define Your Custom Tools

Let’s imagine our agent needs to perform an action, like fetching the current time or searching the web. For this example, we’ll create a simple “get current time” tool.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import datetime

@tool
def get_current_time(timezone: str = "UTC") -> str:
    """
    Returns the current time in a specified timezone.
    Defaults to UTC if no timezone is provided.
    Example: get_current_time("America/New_York")
    """
    try:
        from pytz import timezone as tz, utc
        if timezone not in ["UTC", "America/New_York", "Europe/London"]: # Simple validation
            return "Invalid timezone. Supported: UTC, America/New_York, Europe/London."
        
        target_tz = tz(timezone) if timezone != "UTC" else utc
        now_utc = datetime.datetime.now(utc)
        now_local = now_utc.astimezone(target_tz)
        return f"The current time in {timezone} is {now_local.strftime('%Y-%m-%d %H:%M:%S %Z%z')}"
    except ImportError:
        return "Please install 'pytz' library to use timezone functionality (pip install pytz)."
    except Exception as e:
        return f"Error getting time: {e}"

# Install pytz if you plan to run the time tool
# pip install pytz

Step 2: Combine All Tools

Our agent will need access to all the tools we’ve built: vision, document retrieval, and our custom functions.

1
2
3
4

tools = [analyze_image, retrieve_document_context, get_current_time]

# For testing specific tools:
# print(get_current_time("America/New_York"))

With our tools defined, the final step is to bring everything together and build the agent itself.

Orchestrating the Multi-Modal Agent

LangChain provides powerful abstractions for building agents. We’ll use the create_openai_tools_agent for its robust function-calling capabilities with GPT models.

Step 1: Initialize the LLM for Agent Reasoning

We’ll use a ChatOpenAI instance for the agent’s reasoning, capable of understanding tool definitions and planning.

1
2
3

from langchain_openai import ChatOpenAI

llm_agent = ChatOpenAI(model="gpt-4o", temperature=0, api_key=openai_api_key)

Step 2: Define the Agent Prompt

The agent needs a clear instruction set to understand its role and how to use its tools.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are a helpful multi-modal AI agent capable of answering questions about documents, analyzing images, and performing actions using tools. "
            "If asked about an image, use the `analyze_image` tool. If asked about product information or FAQs, use the `retrieve_document_context` tool. "
            "If asked about time, use the `get_current_time` tool. If you can't find an answer, admit it. Be concise and helpful.",
        ),
        MessagesPlaceholder(variable_name="chat_history"),
        ("human", "{input}"),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
    ]
)

Step 3: Create the Agent

Now, we can assemble our multi-modal AI agent using LangChain’s create_openai_tools_agent and AgentExecutor.

1
2
3
4
5
6
7
8
9
10
11

from langchain.agents import create_openai_tools_agent, AgentExecutor
from langchain_core.messages import BaseMessage

# Create the agent
agent = create_openai_tools_agent(llm_agent, tools, prompt)

# Create the agent executor
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

# Initialize chat history
chat_history = []

Step 4: Interact with Your Agent

Let’s test our multi-modal AI agent with various queries!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

# Function to run queries and update chat history
def run_agent_query(query: str):
    global chat_history
    print(f"\n--- User: {query} ---")
    
    # Check for image input in the query
    if "image:" in query.lower():
        # This is a simplified way to detect image requests. 
        # In a real app, you'd handle file uploads properly.
        image_path_in_query = query.split("image:")[1].strip()
        
        # Assume the user references the dummy image path created earlier
        if "example_image.png" in image_path_in_query:
            # When the user explicitly mentions an image, we can directly invoke the tool
            # or add a specific message type. For now, we'll let the agent decide
            # via the tool definition. The prompt guides it.
            pass # The tool will be called if relevant.
        else:
            print("Note: For image analysis, please ensure the image path exists and is accessible to the analyze_image tool.")
            
    result = agent_executor.invoke({"input": query, "chat_history": chat_history})
    chat_history.extend([HumanMessage(content=query), result["output"]])
    print(f"--- Agent: {result['output']} ---")
    return result['output']

# Test cases:

# 1. Document Understanding
run_agent_query("What are the key features of the CodeCrux AI Agent Framework?")

# 2. Function Calling
run_agent_query("What time is it in New York?")

# 3. Vision Capability (requires the dummy_image_path created earlier)
run_agent_query(f"Can you describe the image located at example_image.png?")

# 4. A multi-modal query (requires context from previous turns or a complex prompt)
# This one might be tricky without more advanced prompt engineering or a multi-turn
# conversation explicitly guiding the agent to combine info.
# For simplicity, we'll ask a document question and then a time question in sequence.
# Let's try combining implicitly through the prompt.
run_agent_query("What's the support contact for the CodeCrux framework, and what time is it in London?")

# Clean up dummy image
os.remove(dummy_image_path)

The verbose=True setting in AgentExecutor will show you the agent’s thought process, including which tools it selects and why. This is incredibly helpful for debugging and understanding how your multi-modal AI agent operates.

Our agent can now see, read, and act!

Real-World Use Cases and Further Enhancements

The multi-modal AI agent with vision, documents, and function calling you’ve just built is a powerful foundation. Here are some real-world applications and ideas for expansion:

Real-World Use Cases

Further Enhancements

Conclusion

You’ve successfully built a sophisticated multi-modal AI agent with vision, documents, and function calling, transforming a simple LLM into a powerful, intelligent system capable of understanding and interacting with the world in a more comprehensive way. By combining visual interpretation, document-based knowledge retrieval, and external tool execution, your agent can now tackle complex tasks that require diverse forms of intelligence. The possibilities for innovation across industries are immense, and this guide provides a solid foundation for you to continue exploring and building even more advanced AI solutions.

FAQ

Why are multi-modal AI agents important?

Multi-modal AI agents are crucial because they mimic human-like understanding by processing diverse data types (text, images, etc.). This allows them to solve more complex, real-world problems that require synthesizing information from various sources and interacting with digital environments.

What LLMs support multi-modal capabilities like vision and function calling?

OpenAI’s GPT-4o and Google’s Gemini Pro are leading examples of LLMs that inherently support multi-modal inputs (including vision) and robust function calling, making them ideal for building sophisticated agents.

How does Retrieval-Augmented Generation (RAG) help a multi-modal agent?

RAG enables an agent to access and leverage external, up-to-date, and domain-specific information not included in its training data. This is critical for document understanding, providing accurate answers, and reducing hallucinations by grounding responses in factual sources.

Can I use other vector databases instead of ChromaDB for document processing?

Yes, absolutely. LangChain and LlamaIndex support a wide array of vector databases such as FAISS, Pinecone, Weaviate, Milvus, Qdrant, and many others. You can choose the one that best fits your project’s scalability and deployment needs.

What are the common challenges when building a multi-modal agent?

Common challenges include effective tool orchestration (getting the agent to pick the right tool at the right time), managing context window limitations for diverse inputs, ensuring data privacy and security, handling tool errors gracefully, and optimizing for latency and cost.

Further Reading

  1. LangChain Documentation: Agents - Explore more advanced agent types and examples.
  2. OpenAI API Documentation: GPT-4o - Deep dive into the capabilities of OpenAI’s flagship multi-modal model.
  3. Introduction to LlamaIndex - Learn more about building powerful RAG applications and knowledge agents.

Build Smarter with CodeCrux

Unlock the full potential of AI for your business with CodeCrux. From custom AI agent development to integrating advanced LLM solutions, our experts can help you design, build, and deploy intelligent systems that drive real-world value. Contact us today or explore our AI/ML services to learn how we can transform your operations.