In this article, we will learn how to create a custom agent that uses an open source llm (llama3.1) that runs locally on our PC. We will also use Ollama and LangChain.

Outline

Install Ollama
Pull model
Serve model
Create a new folder, open it with a code editor
Create and activate Virtual environment
Install langchain langchain-ollama
Build Custom agent with open source model in Python
Conclusion

Install Ollama

Follow the instructions based on your OS type in its GitHub README to install Ollama:

https://github.com/ollama/ollama

I am on a Linux-based PC, so I am going to run the following command in my terminal:

curl -fsSL https://ollama.com/install.sh | sh

Pull model

Fetch the available LLM model via the following command:

ollama pull llama3.1

This will download the default tagged version of the model. Typically, the default points to the latest, smallest sized-parameter model. In this case, it will be llama3.1:8b model.

To download another version of the model, you can go to: https://ollama.com/library/llama3.1 and select the version to install, and then run the ollama pull command with the model and its version number. Example: ollama pull llama3.1:70b

On Mac, the models will be downloaded to ~/.ollama/models

On Linux (or WSL), the models will be stored at /usr/share/ollama/.ollama/models

Serve model

Run the following command to start ollama without running the desktop application.

ollama serve

All models are automatically served on localhost:11434

Create a new folder, open it with a code editor

Create a new folder on your computer and then open it with a code editor like VS Code.

Create and activate Virtual environment

Open the terminal. Use the following command to create a virtual environment .venv and activate it:

python3 -m venv .venv

source .venv/bin/activate

Install langchain langchain-ollama

Run the following command to install langchain and langchain-ollama:

pip install -U langchain langchain-ollama

The above command will install or upgrade the LangChain and LangChain-Ollama packages in Python. The -U flag ensures that the latest versions of these packages are installed, replacing any older versions that may already be present.

Build Custom agent with open source model in Python

Create a Python file for example: main.py and add the following code:

from langchain_ollama import ChatOllama
from langchain.agents import tool
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.agents.format_scratchpad.openai_tools import (
    format_to_openai_tool_messages,
)
from langchain.agents import AgentExecutor
from langchain.agents.output_parsers.openai_tools import OpenAIToolsAgentOutputParser


llm = ChatOllama(
            model="llama3.1",
            temperature=0,
            verbose=True
        )

@tool
def get_word_length(word: str) -> int:
    """Returns the length of a word."""
    return len(word)


tools = [get_word_length]



prompt = ChatPromptTemplate.from_messages(
            [
                (
                    "system",
                    "You are very powerful assistant",
                ),
                ("user", "{input}"),
                MessagesPlaceholder(variable_name="agent_scratchpad"),
            ]
        )

llm_with_tools = llm.bind_tools(tools)

agent = (
    {
        "input": lambda x: x["input"],
        "agent_scratchpad": lambda x: format_to_openai_tool_messages(
            x["intermediate_steps"]
        ),
    }
    | prompt
    | llm_with_tools
    | OpenAIToolsAgentOutputParser()
)

# Create an agent executor by passing in the agent and tools
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
result = agent_executor.invoke({"input": "How many letters in the word educa"})

if result:
    print(f"[Output] --> {result['output']}")
else:
    print('There are no result..')

The above code snippet sets up a LangChain agent using the ChatOllama model (llama3.1) to process user input and utilize a custom tool that calculates word length. It defines a prompt template for the agent, binds the tool to the language model, and constructs an agent that processes input and formats intermediate steps. Finally, it creates an AgentExecutor to invoke the agent with a specific input. We pass a simple question to ask "How many letters in the word educa" and then we print the output or indicate if no result was found.

When we run, we get the following result:

> Entering new AgentExecutor chain...

Invoking: `get_word_length` with `{'word': 'educa'}`


5The word "educa" has 5 letters.

> Finished chain.
[Output] --> The word "educa" has 5 letters.

You see the agent used the model (llama3.1) to call the tool correctly to get the count of letters in the word.

Conclusion

Thanks for reading.

Check Ollama repo here: https://github.com/ollama/ollama

Create your own Custom LLM Agent Using Open Source Models (llama3.1)