For me, this happened when I was building FinanceGPT, which is my personal open-source project that helps me with my finances. This application lets you upload your bank statements, tax forms like 1099s, and so on, and then you can ask questions in plain English like, "How much did I spend on groceries this month?" or "What was my effective tax rate last year?"

The problem is that answering these questions means sending all the sensitive transaction history, W-2s and income data to OpenAI or Anthropic or Google, which I was not comfortable with. Even after redacting PII data from these documents, I was not ok with the trade-off.

This is where Ollama comes in. Ollama lets you run large language models entirely on your own laptop. You don't need any API keys or cloud infrastructure and no data leaves your machine.

In this tutorial, I will walk you through what Ollama is, how to get started with it, and how to use it in a real Python application so that users of the application can choose to keep their data completely local.

Table of Contents

• Prerequisites

• What is Ollama

• How Ollama's API works

• How to call Ollama from Python

• How to Integrate Ollama into a LangChain App

• How to Build an LLM-Provider Agnostic App

• How to use Ollama with LangGraph

• How FinanceGPT Uses This in Practice

• Tradeoffs to be Aware Of

• Conclusion

• Check Out FinanceGPT

• Resources

Prerequisites

You will need the following at a minimum:

• Python 3.10+

• A machine with at least 8GB of RAM (16GB recommended for larger models)

• Basic familiarity with Python and pip

What is Ollama?

Ollama is an open-source tool that makes running LLMs locally very easy. You can think of it as Docker but for AI models. You can pull models using just one command and Ollama handles everything else like downloading the weights, managing memory and the serving the model through a local REST API.

The local REST API is compatible with OpenAI's API format which means any application that can talk to OpenAI, can switch to using Ollama without changing any code.

Installation

First thing you would need is to download the installer from ollama.com. Once installed, you can verify it is running:

ollama --version

The above command checks whether Ollama was installed correctly and prints the current version.

Pull and Run Your First Model

Ollama hosts a variety of models on ollama.com/library. To pull and immediately chat with one, just do:

ollama run llama3.2

This command will download the model from ollama and start an interactive chat session with it. Note: the model size would be a few GBs depending on which model is downloaded. Alternatively, if you want to download a specific model only:

ollama pull mistral

This downloads a model to your machine without starting a chat session which is useful when you want to set up models in advance.

You can run the following command to list the models you have installed:

ollama list

This shows all models you've downloaded locally along with their sizes.

I have used the following models and they have worked great for specific tasks:

How Ollama's API works

Once Ollama is running, it will be served on localhost:11434. You can call it directly using curl:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{ "role": "user", "content": "What is compound interest?" }],
  "stream": false
}'

This sends a chat message directly to Ollama's REST API from the command line, with streaming disabled so you get the full response at once. The above endpoint is to simply chat with the model. The more useful endpoint is http://localhost:11434/v1 as this is OpenAI-compatible. This is the key feature that makes it easy to drop into existing apps that use OpenAI or other LLMs.

How to Call Ollama from Python

How to Use the Ollama Python Library

Ollama has its own Python library that is pretty intuitive to use:

pip install ollama

from ollama import chat

response = chat(
    model='llama3.2',
    messages=[
        {'role': 'user', 'content': 'Explain what a Roth IRA is in simple terms.'}
    ]
)

print(response.message.content)

The above code uses Ollama's native Python SDK to send a message and print the model's reply, which is the most straightforward way to call Ollama from Python

How to Use the OpenAI SDK with Ollama as the Backend

As mentioned earlier, Ollama has an endpoint that is OpenAI compatible, so you can also use the OpenAI Python SDK and just point it to your local server:

pip install openai

from openai import OpenAI

client = OpenAI(
    base_url='http://localhost:11434/v1',
    api_key='ollama',  # Required by the SDK, but ignored by Ollama
)

response = client.chat.completions.create(
    model='llama3.2',
    messages=[
        {'role': 'user', 'content': 'Explain what a Roth IRA is in simple terms.'}
    ]
)

print(response.choices[0].message.content)

This uses the standard OpenAI Python SDK but redirects it to your local Ollama server. The api_key field is required by the SDK but ignored by Ollama. This pattern makes using Ollama seamless for existing applications. The code is nearly identical to what you would write for OpenAI.

How to Integrate Ollama into a LangChain App

Most production applications are built with an orchestration framework like LangChain, which has a native Ollama support. This means swapping providers is just a one-line change.

Install the integration:

pip install langchain-ollama

How to Create a Chat Model

from langchain_ollama import ChatOllama

llm = ChatOllama(model="llama3.2")

response = llm.invoke("What is the difference between a W-2 and a 1099?")
print(response.content)

This creates a LangChain-compatible chat model backed by a local Ollama model, a one-line swap from ChatOpenAI.

Compare this to the OpenAI version and you will see that the interface is almost identical:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o")

How to Build an LLM-Provider Agnostic App

The real power of the application comes from the abstraction of LLM providers. Applications like Perplexity lets users choose the LLM they want to use for their tasks. Here's a simple factory pattern that returns the right LLM based on the configuration:

from langchain_openai import ChatOpenAI
from langchain_ollama import ChatOllama
from langchain_anthropic import ChatAnthropic

def get_llm(provider: str, model: str):
    """
    Return the appropriate LangChain LLM based on the provider.
    
    Args:
        provider: One of "openai", "ollama", "anthropic"
        model: The model name (e.g. "gpt-4o", "llama3.2", "claude-3-5-sonnet")
    
    Returns:
        A LangChain chat model ready to use
    """
    if provider == "openai":
        return ChatOpenAI(model=model)
    elif provider == "ollama":
        return ChatOllama(model=model)
    elif provider == "anthropic":
        return ChatAnthropic(model=model)
    else:
        raise ValueError(f"Unknown provider: {provider}")

The above snippet shows a helper that returns the right LangChain model based on a provider string, so the rest of your app never needs to know which LLM is running underneath.

Now the rest of your code does not need to know about the provider who's LLM is running underneath. This includes your chains, your agents and your tools. You pass llm around and it just works.

How to use Ollama with LangGraph

If you're using LangGraph to build agents (as I covered in my previous article on AI agents), plugging in Ollama is equally seamless:

from langgraph.prebuilt import create_react_agent
from langchain_ollama import ChatOllama
from langchain_core.tools import tool

@tool
def get_spending_summary(category: str) -> str:
    """Get total spending for a given category this month."""
    # In a real app, this would query your database
    return f"You spent $342.50 on {category} this month."

llm = ChatOllama(model="llama3.2")

agent = create_react_agent(
    model=llm,
    tools=[get_spending_summary]
)

response = agent.invoke({
    "messages": [{"role": "user", "content": "How much did I spend on groceries?"}]
})

print(response["messages"][-1].content)

This snippet builds a ReAct agent that uses a locally-running model to decide when to call tools while keeping all data on-device even during agentic workflows.

The agent will decide to call the get_spending_summary tool when needed and get the result using the locally running model instead of sending your data over the internet to OpenAI.

How FinanceGPT Uses This in Practice

FinanceGPT is built to support OpenAI, Anthropic, Google and Ollama as LLM providers. The user sets their preference on the UI or in a config file and the application instantiates the right model using a pattern very similar to the factory pattern above.

When the user chooses Ollama, here's what happens:

1. Their bank statements and other sensitive documents are parsed locally

2. Sensitive fields like SSNs are masked before any LLM call

3. The masked data and query goes to the local Ollama server running on their own machine

4. The response comes back locally and nothing ever leaves their network

To run FinanceGPT locally with Ollama, the setup looks like this:

# 1. Pull a capable model
ollama pull llama3.2

# 2. Clone and configure FinanceGPT
git clone https://github.com/manojag115/FinanceGPT.git
cd FinanceGPT
cp .env.example .env

# 3. In .env, set your LLM provider to Ollama
# LLM_PROVIDER=ollama
# LLM_MODEL=llama3.2

# 4. Start the full stack
docker compose -f docker-compose.quickstart.yml up -d

With this setup, the entire application including the frontend, backend and LLM, runs on your own hardware.

Tradeoffs to be Aware Of

Ollama is a great local alternative to using cloud LLMs, but it comes with its own problems.

Response Quality

Ollama models are essentially 7B parameter models running locally, so by design they will not match GPT-4o on complex reasoning tasks. For simple Q&A and summarization tasks, the results would be comparable, but for multi-step reasoning or nuanced judgement calls, the gap is noticeable.

Speed

Inference speed depends on the hardware that is running the model. Without a GPU, the Ollama models can take several seconds to respond. On Apple Silicon (M1/M2/M3), the performance is surprisingly good even without a dedicated GPU.

Hardware Requirements

Small models (7B parameters) need around 8GB of RAM, however larger models (13B+) need 16GB or more. If you are building your application for end users, you cannot guarantee they have the hardware.

Tool Use and Function Calling

Not all local models support function calling reliably. If your agent depends heavily on tool use, test your chosen model carefully. Models like qwen2.5 and mistral generally handle this better than others.

The right mental model: use cloud models when you need maximum capability, and local models when privacy or cost constraints make cloud models impractical.

Conclusion

In this tutorial, you learned what Ollama is, how to install it and pull models, and three different ways to call it from Python: the native Ollama library, the OpenAI-compatible SDK, and LangChain. You also saw how to build a provider-agnostic factory pattern so your app can switch between cloud and local models with a single config change.

Ollama makes local LLMs genuinely practical for production apps. The OpenAI-compatible API means integration is nearly zero-friction, and LangChain's native support means you can build provider-agnostic apps from the start.

The finance domain is an obvious fit — but the same principle applies anywhere sensitive data is involved: healthcare, legal tech, HR, personal productivity. If your app processes data that users wouldn't want stored on someone else's server, giving them a local option isn't just a nice-to-have. It's a trust feature.

Check Out FinanceGPT

All the code examples here came from FinanceGPT. If you want to see these patterns in a complete app, poke around the repo. It's got document processing, portfolio tracking, tax optimization – all built with LangGraph.

If you find this helpful, give the project a star on GitHub – it helps other developers discover it.

Resources

• Ollama Documentation

• Ollama Model Library

• LangChain Ollama Integration

• How to Build AI Agents with LangGraph (my previous article)

Having a powerful tool like this available offline is a game-changer. These Local LLMs keep high-level intelligence at your fingertips, even when you're offline. By the end of this guide, you’ll understand what local LLMs are, why they matter, and how to run them yourself, both the easy way and the more technical way.

This guide is suited but not limited to:

• Developers, technical writers, or curious engineers.

• Anyone comfortable with the terminal.

• People with some exposure to AI tools (ChatGPT, Claude, and so on).

• Anyone with little or no experience running LLMs locally.

Table of Contents

• What Are Local LLMs?

• What Running “Locally” Means

• Why Run LLMs Locally?

• How to Set Up a Local LLM

• What Is Ollama?

• How Ollama Operates

• How to Install Ollama

• How to Pull an LLM

• How to Run Your LLM

• How to Customize Local LLMs in Ollama with Modelfiles

  • What Are Modelfiles?

  • How to Customize a Model

  • What Modelfiles Do and Don't Do

• Conclusion

What are Local LLMs?

Local Large Language Models (LLMs) bring AI off the cloud and onto your personal hardware. While standard models are originally too large for consumer devices, a process called quantization reduces their numerical precision, much like compressing a large high-resolution video file so it can stream smoothly on a mobile phone. This allows powerful intelligence to run locally on your laptop without needing massive server farms.

Running models such as Meta’s Llama 3.3, Google’s Gemma 3, or Alibaba’s Qwen series locally ensures full data privacy and eliminates subscription costs. Because the AI lives on your machine, you get a fast, offline-capable workspace that keeps your code secure and under your direct control.

What Running “Locally” Means

To understand how local LLMs run on your machine, you have to look into the physical components of your computer. When you run a model like Llama 3 or Mistral locally, your hardware transforms from a general-purpose machine into a specialized AI engine.

The process relies on a tight coordination between four key hardware pillars: Storage, RAM, the GPU, and the CPU.

Storage (The model's permanent home)

Before you can chat, you must download the model. Unlike a standard app, an LLM is primarily a massive file of "weights", numerical values that represent everything the AI knows.

• The Files: You’ll likely see formats like .gguf or .safetensors. These files are large: a "small" 7B (7 billion parameter) model usually occupies 5GB to 10GB of disk space.

• SSD vs. HDD: An SSD is mandatory. Because the computer must move several gigabytes of data into memory every time you launch the model, a traditional hard drive will leave you waiting minutes for the "brain" to wake up.

VRAM and RAM (The Model’s Workspace)

This is the most critical bottleneck. For an AI to respond quickly, its entire "brain" must fit into high-speed memory.

• VRAM (Video RAM): This is the memory physically attached to your graphics card (GPU). It is significantly faster than regular system RAM. If your model fits entirely in VRAM, the AI will likely type faster than you can read.

• System RAM: If your model is too big for your GPU, the software will "spill over" into your computer’s regular RAM. While this allows you to run massive models on modest hardware, the speed penalty is severe—often dropping from 50 words per second to just one or two.

The GPU (The Mathematical Engine)

While your CPU is the "manager" of your computer, the GPU (Graphics Processing Unit) is the "mathematician."

• Parallel Power: LLMs work by performing billions of simple math problems (matrix multiplications) at the same time. A CPU has a few powerful cores, but a GPU has thousands of smaller cores designed specifically for this parallel math.

• Unified Memory (Apple Silicon): On modern Macs (M1/M2/M3), the CPU and GPU share the same pool of memory. This "Unified Memory" is a game-changer for local AI, allowing even thin laptops to handle relatively large models that would typically require a chunky desktop GPU.

For optimal performance, always compare your computer's specs with the model’s requirements to see which models you can comfortably run.

Why Run LLMs Locally?

Running an LLM locally isn't just for tech enthusiasts, it’s a strategic move for anyone who wants full control over their AI. Core benefits of running an LLM locally are:

1. Offline Usage: You're not limited to the cloud. You can explore and use your data wherever you go. Whether you're on a plane or in a remote area, your AI works without an internet connection.

2. Privacy and data ownership: Also, because you are not connected to the cloud, there is no risk of your data and prompts being exploited by a third party remotely or used to train a company’s next model.

3. Cost control: No need for monthly subscriptions and API tokens. Once you have the hardware, running the model is essentially free, given its capabilities and your configurations.

4. Customization & Experimentation: If you have multiple models downloaded, you can "swap brains" instantly. Try different models, fine-tune them for specific tasks, and tweak settings that big providers keep locked.

5. Faster iteration for dev workflows: For developers, local hosting eliminates network latency, allowing for near-instant responses and faster testing loops.

Tradeoffs

Local LLMs have certain tradeoffs to consider:

• Hardware Requirements: You’ll need a decent setup—specifically, a GPU with a good amount of VRAM (usually 8GB+) or a Mac with Apple Silicon (M1/M2/M3)—to achieve smooth performance.

• Performance Limitations: Local models are getting better every day, but they might not yet match the sheer "reasoning power" of a massive, billion-dollar cloud cluster like GPT-4.

• Initial Setup Friction: It isn’t always "plug and play." If you want to get hands-on with specific features, you will have to spend some time configuring software, downloading large model files, and troubleshooting your environment.

Even with these trade-offs, having such a tool at your disposal and under your control remains a significant advantage in everyday life.

How to Set Up a Local LLM

There are many ways to get and set up a local LLM, but for this guide, you will use Ollama, a user-friendly tool that brings private, secure AI directly to your desktop. You will learn to pull and deploy high-performance models with a single command, optimize them for your specific CPU/GPU configuration, and use the powerful Modelfile system to "program" custom AI personalities tailored to your exact needs.

What We’ll Cover:

• The Basics: Understanding how Ollama turns your PC into an AI powerhouse.

• Installation & Setup: Getting up and running in under five minutes.

• Model Management: How to find, "pull" (download), and run models like Llama 3 or Mistral.

• Customization: Writing your first Modelfile to give your AI a specific job or personality.

By the end of this, you will have a fully independent AI workstation, capable of sophisticated reasoning without ever sending a byte of data to the cloud.

What is Ollama?

Ollama is a free, open-source tool that makes running Large Language Models (LLMs) on your own hardware as easy as opening a web browser. It strips away the technical complexity that usually comes with AI research, giving you a clean, simple way to chat with, manage, and even customize your own AI models.

Before Ollama, running a local AI was a headache. You had to hunt for the right "weights" files on the internet, set up complex coding environments, and hope your hardware doesn't crash. Now, instead of spending hours configuring software, Ollama handles the heavy lifting. It automatically finds your graphics card (GPU) and tunes the settings for you.

How Ollama Operates

Ollama follows a simple "Mental Model" that mimics how you handle apps on a phone or music on a streaming service.

The Model Registry (The Library)

Ollama maintains a massive "Library", a central library of prepackaged AI models such as Llama 3, Mistral, and Gemma. You don't have to worry about file formats, you just pick a name from the list, and Ollama "pulls" it down to your machine.

The Local Runtime (The Engine)

Once you have a model, Ollama acts as the engine. It wakes the model up, loads it into your computer's memory (RAM/VRAM), and starts the mathematical "thinking" process. It is smart enough to use your GPU for speed, but it can also run on a standard CPU if that's all you have.

The CLI (The Control Centre)

Ollama uses a Command Line Interface (CLI). While that sounds technical, it just means you type simple, human-like instructions into a terminal window. Want to talk to a model? You just tell it to run. Want to see what you've downloaded? You ask it to list them.

How to Install Ollama

Go to the Ollama download page. For Windows and Mac, click the download button.

For Linux, run this command:

curl -fsSL https://ollama.com/install.sh | sh

After downloading, open the file, follow the setup instructions, and install it.

On Windows and Mac, after installation, the Ollama native Desktop Application should open.

This GUI is most beneficial for those who feel the CLI is intimidating; you don't have to be a coder to use Ollama. Instead of typing commands, you can manage your models and start conversations through a sleek window that feels just like any other chat app.

How to Pull an LLM

As mentioned earlier, Ollama has a vast library of Large Language Models for different specs and uses. To download one to your computer, use the pull command followed by the name of the LLM. For example:

ollama pull gemma3:1b

To see the models you downloaded or have, use the list command, like:

ollama list

How to Run Your LLM

You now have your LLM on your computer. To use it, you use the run command, followed by the name of the LLM. For example:

ollama run gemma3:1b

The LLM will load up, and you can prompt it.

To exit the LLM, use Ctrl + d or type in /bye.
You can perform other operations like deleting a model, copying a model, show information on a model, and so on. Type in ollama help to see all these commands.

How to Customize Local LLMs in Ollama with Modelfiles

One of Ollama’s most powerful features is the ability to customize how a local model behaves using Modelfiles. Rather than treating models as fixed black boxes, Modelfiles allow you to define how a model should respond, what role it should play, and how it should generate text, without retraining or fine-tuning.

This makes Modelfiles ideal for creating reusable, task-specific local models such as technical writers, code reviewers, research assistants, internal developer tools, or even character-driven assistants.

What are ModelFiles?

A Modelfile is a plain-text configuration file used by Ollama to create a new model based on an existing one. It describes how a base model should be wrapped, prompted, and configured at runtime.

Essentially, a Modelfile:

• Starts from a base model

• Applies a set of instructions

• Produces a new, named model that can be run like any other

Modelfiles do not modify the underlying model weights. Instead, they define behavioral rules, how the model should be prompted, how it should generate text, and how it should respond to user input.

Modelfile Syntax and Structure

Modelfiles are line-based and declarative. Each directive defines a specific aspect of the model’s behavior.

A minimal Modelfile looks like this:

FROM llama3

SYSTEM """
You are a senior technical writer.
"""

PARAMETER temperature 0.2

• FROM: This is the foundation. It tells the system which base architecture (like llama3) to inherit its intelligence and tokenizer from.

• SYSTEM: This sets the "permanent" instructions. By assigning the Senior Technical Writer role, we ensure that every response maintains a professional, structured tone without needing to remind the AI in every prompt.

• PARAMETER: These are the model's dials and knobs. In this case, we use the temperature 0.2 parameter to set a low "creativity dial," forcing the model to be more deterministic and precise, which is ideal for the consistent, factual output.

Advanced users can also use TEMPLATE for custom prompt formatting and additional MESSAGE directives to include specific conversation history, though these aren't required for this basic setup.

Quick reference cheat sheet:

How to Customize a Model

To create a model using a Modelfile, Ollama performs the following steps:

• Loads the specified base model

• Applies system-level instructions

• Configures generation parameters

• Registers the result as a new local model

For this article, you will be creating a technical writing assistant from any local LLM of your choice. You can use the LLM you downloaded earlier, or download another one you feel is a better fit for this model.

1. Set up your environment: Create a folder named my-writing-assistant, then open it in your preferred IDE or text editor.

2. Create a Modelfile: Create a file named Modelfile in your folder. Populate it with the following:

FROM llama3 

SYSTEM """
You are a senior technical writer.
Write clear, concise explanations.
Use headings and bullet points where appropriate.
Avoid marketing language.
"""

PARAMETER temperature 0.2
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.1
PARAMETER num_ctx 4096

1. Create your model: Open the terminal in your IDE, or if you are using a text editor without a built-in terminal, open your Command Prompt and navigate into the my-writing-assistant directory. Run this command:

   ollama create tech-writer -f Modelfile
   

   You should see a response like this:

   

2. Run your model: You can run your model like any other Ollama model, with the run command:

   ollama run tech-writer
   

   

   Try a documentation-based prompt and see your model behave exactly how your Modelfile designed it.

You can also interact with your models(downloaded and modified) using the Desktop App. Simply open the application, select your preferred model from the chatbox dropdown menu, and start prompting.

What Modelfiles Do and Don't Do

Modelfiles are powerful, but it’s important to understand their scope.

They:

• Customize model behavior

• Enforce consistent prompting

• Tune generation characteristics

• Create reusable local models

They do not:

• Retrain or fine-tune model weights

• Add new knowledge

• Change the model’s architecture

A Modelfile shapes how a model responds, not what it knows.

Conclusion

Running large language models locally is no longer limited to researchers or high-end machines. With Ollama and Modelfiles, you can download capable models, run them on your own device, and tailor their behavior to fit your workflow.

In this guide, we covered what local LLMs are, why they matter, how Ollama simplifies setup, and how Modelfiles let you control tone, structure, and generation settings. Instead of relying on a generic chatbot, you can build assistants that feel intentional and purpose-built.

More importantly, running models locally changes how you interact with AI. You move from simply consuming an API to understanding and shaping the system itself. As AI continues to influence software, business, and everyday tools, hands-on experience with local models gives you a clearer view of where the technology is heading. The best way to understand that shift is to experiment, pull a model, refine a Modelfile,

Now imagine you had a team of specialized assistants that worked overnight — one to read your inputs, one to summarize the key facts, one to rank what matters most, and one to format everything into a clean daily brief waiting in your inbox.

That is exactly what this handbook walks you through building. You will create a multi-agent AI system where four Python-based agents each handle one job. You will containerize each agent with Docker so the whole thing runs reliably on any machine. And you will wire it all together with Docker Compose so you can launch the entire pipeline with a single command.

This handbook assumes you are comfortable reading Python code, but it does not assume you have used Docker before. If you have never written a Dockerfile or run a container, that is fine — the fundamentals are covered as we go.

By the end, you will have a working system that turns digital noise into an organized daily digest, and you will understand the patterns behind it well enough to adapt them to your own projects.

Table of Contents

• What is a Multi-Agent System (and Why Build One)?

  • How Traditional Scripts Work

  • How AI Agents are Different

  • Why Use Multiple Agents Instead of One?

• What is Docker (and Why Does It Matter Here)?

  • The Environment Problem

  • How Docker Solves This

  • How Docker Layers Work

  • Docker vs. No Docker

• How to Plan the Architecture

• Prerequisites and Environment Setup

  • How to Install Python

  • How to Install Docker

  • How to Verify Your Setup

  • How to Set Up the Project Structure

• How to Build Each Agent Step by Step

  • The Ingestor Agent

  • The Summarizer Agent

  • The Prioritizer Agent

  • The Formatter Agent

• How to Handle Secrets and API Keys

  • Using .env Files for Development

  • How to Use Docker Secrets for Production

• How to Orchestrate Everything with Docker Compose

• How to Run the Pipeline

• How to Test the Pipeline

  • Unit Tests

  • Integration Tests

• How to Add Logging and Observability

• Cost, Rate Limits, and Graceful Degradation

• Security and Privacy Considerations

• How to Use a Local LLM for Full Privacy (Ollama)

• Example Seed Data and Expected Output

• How to Automate Daily Execution

• How to Use Cron on Linux or macOS

• How to Use Task Scheduler on Windows

• How to Add Delivery Notifications

• Troubleshooting Common Errors

• Production Deployment Options

  • Docker Swarm

  • Kubernetes

• Cloud Platforms

• Conclusion and Next Steps

What is a Multi-Agent System (and Why Build One)?

How Traditional Scripts Work

A traditional Python script follows a fixed path. It reads some input, processes it through a series of hard-coded steps, and writes the output. If the input format changes even slightly, the script often breaks. Think of it like a train on a track. Trains are fast and efficient, but they can only go where the rails take them. If the track is blocked, the train stops.

How AI Agents are Different

An AI agent is more like a bus driver. It has a destination (a goal), but it can decide which route to take based on current conditions (the data). If one road is blocked, it finds another.

Agents typically follow a loop called the ReAct pattern, which stands for Reasoning plus Acting. At each step, the agent thinks about what to do, takes an action, observes the result, and decides whether it has reached its goal. If not, it loops back and tries again. If so, it finishes.

In practice, this means an LLM-based agent can handle messy, unpredictable input much better than a traditional script. If a newsletter changes its format, the summarizer agent can still extract the key points because it reasons about the content rather than parsing a rigid structure.

Why Use Multiple Agents Instead of One?

You might wonder: why not just use one powerful agent that does everything? That approach is called the "God Model" pattern, and it has real problems. When you ask a single LLM to ingest data, summarize it, prioritize it, and format it all in one prompt, you are giving it too much to think about at once. LLMs have a limited context window and limited attention. The more tasks you pile on, the more likely the model is to hallucinate, skip steps, or produce inconsistent output.

A multi-agent system solves this through separation of concerns. Each agent has one narrow job. The Ingestor reads and combines raw files, with no LLM needed. The Summarizer calls the LLM with a focused prompt: just summarize this text. The Prioritizer scores lines by keyword with no LLM needed. And the Formatter writes Markdown output, also with no LLM.

This design has several advantages. Each agent is simpler to build, test, and debug. You can swap out the Summarizer for a better model without touching anything else. And you can scale individual agents independently — for example, running multiple Summarizers in parallel if you have a lot of input.

What is Docker (and Why Does It Matter Here)?

The Environment Problem

If you have ever shared a Python project with someone and heard "it does not work on my machine," you already understand the problem Docker solves. Every Python project depends on specific versions of Python itself, plus libraries like openai, requests, or beautifulsoup4. These dependencies live in your operating system's environment. When you install a new library or upgrade Python, you might break a different project that depends on the old version.

Virtual environments help, but they only isolate Python packages. They do not isolate the operating system, system libraries, or other tools your code might need. And they do not guarantee that someone else can recreate your exact environment. For a multi-agent system, this problem gets worse. Each agent might need different dependencies. If they share an environment, their dependencies can conflict.

How Docker Solves This

Docker packages your code, its dependencies, and a minimal operating system into a single unit called a container. When you run that container, it behaves exactly the same way regardless of what machine it is running on — your laptop, a coworker's computer, or a cloud server. Think of a Docker container like a shipping container for software. The contents are sealed inside, protected from the outside environment.

There are a few key Docker concepts to understand:

Image — A read-only template that contains your code, dependencies, and a minimal OS. You build an image from a Dockerfile. Think of it as a recipe.

Container — A running instance of an image. When you "run" an image, Docker creates a container from it. Think of it as a dish made from the recipe.

Dockerfile — A text file with instructions for building an image. It specifies the base OS, what to install, what code to copy in, and what command to run when the container starts.

Volume — A way to share files between your computer and a container, or between multiple containers. Our agents will use a shared volume to pass data to each other.

Docker Compose — A tool for defining and running multiple containers together. You describe all your containers in a single YAML file, and Compose handles building, networking, and ordering them.

How Docker Layers Work

Docker builds images in layers. Each instruction in a Dockerfile creates a new layer. Docker caches these layers, so if a layer has not changed since the last build, Docker reuses the cached version instead of rebuilding it. This is why Dockerfiles are structured in a specific order: the base OS layer rarely changes, the dependency installation layer changes when requirements.txt changes, and the application code layer changes on every code edit. By putting dependency installation before the code copy, Docker only re-runs pip install when your requirements actually change, making rebuilds much faster — seconds instead of minutes.

Docker vs. No Docker

To be clear, you do not strictly need Docker for this tutorial. You can run all four agents as plain Python scripts. But without Docker you face dependency conflicts from a shared environment, manual process management for scaling, having to redo all setup on every new machine, complex orchestration for testing, and painful Python version management when one agent needs 3.8 and another needs 3.10. With Docker, each agent has its own isolated environment, you run multiple containers in parallel with one command, docker compose up produces identical results everywhere, and each container runs its own Python version independently.

For a personal project, either approach works. But if you ever want to share this system, deploy it to a server, or run it in the cloud, Docker makes the difference between "here is a README with 15 setup steps" and "run docker compose up."

How to Plan the Architecture

Before writing any code, it is worth mapping out how the pieces fit together. The full system consists of four agents arranged in a sequential pipeline, all orchestrated by Docker Compose. Data flows through the Ingestor Agent, the Summarizer Agent, the Prioritizer Agent, and the Formatter Agent in that order. Each agent reads from a shared volume, processes its input, writes the result, and exits. Docker Compose enforces execution order by waiting for each container to finish successfully before starting the next one.

This is a synchronous pipeline: agents run one at a time, in sequence. It is the simplest multi-agent pattern to implement and understand. For more complex systems, you could replace the shared volume with a message broker like Redis or RabbitMQ, which lets agents run asynchronously and react to events. But for this daily-digest use case, the sequential approach is exactly right.

In terms of responsibilities:

• Ingestor — Reads and combines raw files from /data/input/ into ingested.txt. No LLM required.

• Summarizer — Distills key points from ingested.txt into summary.txt. The only agent that requires an LLM.

• Prioritizer — Scores items by urgency keywords, turning summary.txt into prioritized.txt. No LLM.

• Formatter — Produces the final Markdown report, daily_digest.md. No LLM.

Notice that only one of the four agents actually calls an LLM. The others are plain Python. This is intentional — you should only use an LLM when you need reasoning or language understanding. Everything else should be deterministic code. It is cheaper, faster, and more predictable.

Prerequisites and Environment Setup

You need the following tools installed before starting:

• Python 3.10 or higher — the language for the agents

• Docker Desktop (Engine 20.10+) — the container runtime

• Docker Compose v2 (included with Docker Desktop) — multi-container orchestration

• Git 2.30+ — version control

• OpenAI Python SDK (openai >= 1.0) — LLM API access

• Redis or RabbitMQ (optional) — async message queuing

• PostgreSQL (optional) — persistent data storage

How to Install Python

Download Python from python.org. On Windows, check the "Add Python to PATH" box during installation. On macOS, you can use Homebrew:

brew install python@3.12

On Linux (Ubuntu/Debian), use your package manager:

sudo apt update && sudo apt install python3 python3-pip

How to Install Docker

Docker Desktop is the easiest way to get started on Windows and macOS. Download it from docker.com and follow the prompts. On Windows, Docker Desktop requires WSL2 — the installer will guide you through enabling it. On Linux, install Docker Engine directly:

# Ubuntu/Debian
sudo apt update
sudo apt install docker.io docker-compose-v2
sudo usermod -aG docker $USER  # So you don't need sudo for docker commands

After installing, log out and back in for the group change to take effect.

How to Verify Your Setup

Open your terminal and run these commands. Each should print a version number without errors:

python --version        # Should show 3.10 or higher
docker --version        # Should show 20.10 or higher
docker compose version  # Should show v2.x
git --version           # Should show 2.30 or higher

If any command fails, go back to the installation step for that tool. The most common issue is that the command is not in your PATH.

How to Set Up the Project Structure

Each agent lives in its own directory with its own code, Dockerfile, and requirements file. This isolation means you can build, test, and update each agent independently. Create the following structure:

multi-agent-digest/
├── agents/
│   ├── ingestor/
│   │   ├── app.py
│   │   ├── Dockerfile
│   │   └── requirements.txt
│   ├── summarizer/
│   │   ├── app.py
│   │   ├── Dockerfile
│   │   └── requirements.txt
│   ├── prioritizer/
│   │   ├── app.py
│   │   ├── Dockerfile
│   │   └── requirements.txt
│   └── formatter/
│       ├── app.py
│       ├── Dockerfile
│       └── requirements.txt
├── data/
│   └── input/          # Your raw files go here
├── output/              # The final digest appears here
├── tests/               # Unit and integration tests
├── .env                 # API keys (gitignored!)
├── .gitignore
├── docker-compose.yml
└── README.md

You can create the folders quickly from the terminal:

mkdir -p multi-agent-digest/agents/{ingestor,summarizer,prioritizer,formatter}
mkdir -p multi-agent-digest/{data/input,output,tests}
cd multi-agent-digest

How to Build Each Agent Step by Step

Every agent follows the same simple pattern: read an input file from the shared volume, do its job, and write an output file. This consistency makes the system easy to understand and extend.

The Ingestor Agent

The Ingestor is the entry point of the pipeline. Its job is to read all text files from the input folder and combine them into a single file that the Summarizer can process. This is the simplest agent — no external libraries, no API calls, just file reading and writing.

agents/ingestor/app.py

import os
import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
)
logger = logging.getLogger("ingestor")

INPUT_DIR = "/data/input"
OUTPUT_FILE = "/data/ingested.txt"

def ingest():
    content = ""
    files_processed = 0
    for filename in sorted(os.listdir(INPUT_DIR)):
        filepath = os.path.join(INPUT_DIR, filename)
        if os.path.isfile(filepath):
            try:
                with open(filepath, "r", encoding="utf-8") as f:
                    content += f"\n--- {filename} ---\n"
                    content += f.read()
                    content += "\n"
                    files_processed += 1
            except Exception as e:
                logger.error(f"Failed to read {filename}: {e}")

    if files_processed == 0:
        logger.warning("No input files found in /data/input/")

    with open(OUTPUT_FILE, "w", encoding="utf-8") as out:
        out.write(content)
    logger.info(f"Ingested {files_processed} files -> {OUTPUT_FILE}")

if __name__ == "__main__":
    ingest()

The logging.basicConfig block sets up structured logging. Every agent uses the same log format, so when Docker Compose runs them together, you get a clean, consistent timeline. The sorted(os.listdir()) call ensures files are processed in alphabetical order — without it, the order depends on the filesystem and can vary between machines. The try/except block around each file read means a single corrupted file will not crash the entire pipeline. And if no files are found at all, the agent writes an empty output file rather than crashing, so downstream agents can handle empty input gracefully.

agents/ingestor/Dockerfile

FROM python:3.10-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
CMD ["python", "app.py"]

FROM python:3.10-slim starts with a minimal Linux image that has Python pre-installed. The -slim variant is about 120 MB versus 900 MB for the full image. WORKDIR /app sets the working directory inside the container. COPY requirements.txt and RUN pip install handle dependencies at build time, not runtime. COPY app.py copies the application code last because it changes most often, and Docker caches previous layers. CMD specifies the command to run when the container starts.

Since the Ingestor uses only standard library modules, its requirements.txt can be empty:

# No external dependencies needed

The Summarizer Agent

The Summarizer is the most complex agent in the pipeline. It reads the ingested text and calls an LLM API to produce a concise summary. This is the only agent that makes a network call, which means it is the only one that can fail due to external factors: the API might be down, you might hit rate limits, or your key might be invalid.

agents/summarizer/app.py:

import os
import logging
import time
from openai import OpenAI, RateLimitError, APIError

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
)
logger = logging.getLogger("summarizer")

INPUT_FILE = "/data/ingested.txt"
OUTPUT_FILE = "/data/summary.txt"

client = OpenAI()  # reads OPENAI_API_KEY from environment

SYSTEM_PROMPT = (
    "You are a helpful assistant that summarizes long text "
    "into key bullet points. Each bullet should be one "
    "concise sentence capturing a core insight."
)

MAX_RETRIES = 3
RETRY_DELAY = 5  # seconds

def summarize(text, retries=MAX_RETRIES):
    """Call the LLM API with retry logic for rate limits."""
    for attempt in range(retries):
        try:
            response = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[
                    {"role": "system", "content": SYSTEM_PROMPT},
                    {"role": "user", "content": text[:8000]}
                ],
                max_tokens=1000,
                temperature=0.3,
            )
            return response.choices[0].message.content
        except RateLimitError:
            wait = RETRY_DELAY * (attempt + 1)
            logger.warning(f"Rate limited. Retrying in {wait}s...")
            time.sleep(wait)
        except APIError as e:
            logger.error(f"API error: {e}")
            raise
    raise RuntimeError("Max retries exceeded for LLM API call")

def main():
    with open(INPUT_FILE, "r", encoding="utf-8") as f:
        raw_text = f.read()

    if not raw_text.strip():
        logger.warning("Empty input. Writing fallback summary.")
        summary = "No content to summarize."
    else:
        try:
            summary = summarize(raw_text)
        except Exception as e:
            logger.error(f"Summarization failed: {e}")
            summary = f"Summarization failed: {e}"

    with open(OUTPUT_FILE, "w", encoding="utf-8") as f:
        f.write(summary)
    logger.info(f"Summary written to {OUTPUT_FILE}")

if __name__ == "__main__":
    main()

The OpenAI() client automatically reads the OPENAI_API_KEY environment variable — you do not need to pass the key explicitly in code, which is both cleaner and safer. The text[:8000] slice limits how much text is sent to the API. Sending fewer tokens means faster responses and lower cost. For production, you would want smarter chunking that splits on sentence or paragraph boundaries rather than a raw character count.

Temperature 0.3 makes the output more focused and deterministic, which is ideal for summarization. The retry logic catches RateLimitError specifically and waits longer each time (5, 10, then 15 seconds) — this is called exponential backoff. Other API errors raise immediately because retrying them will not help. If the input is empty or the API fails completely, the agent writes a fallback message instead of crashing, so the downstream agents can still run.

agents/summarizer/requirements.txt:

openai>=1.0

The Dockerfile is identical to the Ingestor's.

The Prioritizer Agent

The Prioritizer takes the LLM-generated summary and scores each line based on urgency keywords. This is a rule-based agent — no LLM call needed. It is fast, deterministic, and free.

agents/prioritizer/app.py:

import os
import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
)
logger = logging.getLogger("prioritizer")

INPUT_FILE = "/data/summary.txt"
OUTPUT_FILE = "/data/prioritized.txt"

PRIORITY_KEYWORDS = [
    "urgent", "today", "asap", "important",
    "deadline", "critical", "action required"
]

def score_line(line):
    """Count how many priority keywords appear in a line."""
    lower = line.lower()
    return sum(1 for kw in PRIORITY_KEYWORDS if kw in lower)

def prioritize():
    with open(INPUT_FILE, "r", encoding="utf-8") as f:
        lines = [line.strip() for line in f if line.strip()]

    scored = [(line, score_line(line)) for line in lines]
    scored.sort(key=lambda x: x[1], reverse=True)

    with open(OUTPUT_FILE, "w", encoding="utf-8") as out:
        for line, score in scored:
            out.write(f"[{score}] {line}\n")

    logger.info(f"Prioritized {len(scored)} items -> {OUTPUT_FILE}")

if __name__ == "__main__":
    prioritize()

The scoring function counts how many priority keywords appear in each line. A line containing "urgent deadline" scores 2, and a line with no keywords scores 0. The scored lines are sorted in descending order, so the most urgent items appear first. Each line is prefixed with its score in brackets, like [2] Urgent: quarterly report due today. In a more advanced system, you could replace this keyword scorer with an LLM-based ranker, but for a daily digest, simple keyword matching works surprisingly well.

This agent has no pip dependencies, so the Dockerfile skips the requirements step:

agents/prioritizer/Dockerfile:

FROM python:3.10-slim
WORKDIR /app
COPY app.py .
CMD ["python", "app.py"]

The Formatter Agent

The Formatter is the final agent in the pipeline. It reads the scored lines and writes a clean Markdown document to the output directory.

agents/formatter/app.py:

import os
import logging
from datetime import datetime

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
)
logger = logging.getLogger("formatter")

INPUT_FILE = "/data/prioritized.txt"
OUTPUT_FILE = "/output/daily_digest.md"

def format_to_markdown():
    with open(INPUT_FILE, "r", encoding="utf-8") as f:
        lines = [line.strip() for line in f if line.strip()]

    today = datetime.now().strftime('%Y-%m-%d')

    with open(OUTPUT_FILE, "w", encoding="utf-8") as out:
        out.write("# Your Daily AI Digest\n\n")
        out.write(f"**Date:** {today}\n\n")
        out.write("## Top Insights\n\n")
        for line in lines:
            if '] ' in line:
                score = line.split(']')[0][1:]
                content = line.split('] ', 1)[1]
                out.write(f"- **Priority {score}**: {content}\n")
            else:
                out.write(f"- {line}\n")

    logger.info(f"Digest written to {OUTPUT_FILE}")

if __name__ == "__main__":
    format_to_markdown()

Notice that the Formatter writes to /output instead of /data. This is a separate volume mount in Docker Compose. The /data volume is internal plumbing that agents use to communicate, while the /output volume maps to a folder on your host machine where you can access the final result. The split('] ', 1) with maxsplit=1 ensures that bracket characters inside the actual content do not break the parsing.

The Dockerfile is the same as the Prioritizer's (no external dependencies).

How to Handle Secrets and API Keys

⚠️ Warning: Never commit API keys or secrets to version control. A leaked OpenAI key can rack up thousands of dollars in charges before you notice.

Using .env Files for Development

Create a .env file in your project root:

# .env -- DO NOT COMMIT THIS FILE
OPENAI_API_KEY=sk-your-key-here

Then immediately add it to your .gitignore:

# .gitignore
.env
output/
data/ingested.txt
data/summary.txt
data/prioritized.txt
__pycache__/
*.pyc

Docker Compose reads .env files automatically when it starts. In your docker-compose.yml, you reference the variable with ${OPENAI_API_KEY}, and Compose substitutes the real value at runtime. The key never appears in your Dockerfile, your code, or your version history.

How to Use Docker Secrets for Production

For production deployments on Docker Swarm or Kubernetes, environment variables are visible in process listings and inspect commands. Docker secrets are more secure:

# Create the secret
echo "sk-your-key-here" | docker secret create openai_key -

# Reference in docker-compose.yml (Swarm mode only)
services:
  summarizer:
    secrets:
      - openai_key

secrets:
  openai_key:
    external: true

The secret gets mounted as a read-only file at /run/secrets/openai_key inside the container. Your code reads the key from that file instead of from an environment variable.

How to Orchestrate Everything with Docker Compose

With all four agents built, Docker Compose ties them together. It builds each container, mounts the shared volumes, passes environment variables, and enforces the correct execution order.

docker-compose.yml:

version: "3.9"

services:
  ingestor:
    build: ./agents/ingestor
    container_name: agent_ingestor
    volumes:
      - ./data:/data
    restart: "no"

  summarizer:
    build: ./agents/summarizer
    container_name: agent_summarizer
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
    depends_on:
      ingestor:
        condition: service_completed_successfully
    volumes:
      - ./data:/data
    deploy:
      resources:
        limits:
          memory: 512M
    restart: "no"

  prioritizer:
    build: ./agents/prioritizer
    container_name: agent_prioritizer
    depends_on:
      summarizer:
        condition: service_completed_successfully
    volumes:
      - ./data:/data
    restart: "no"

  formatter:
    build: ./agents/formatter
    container_name: agent_formatter
    depends_on:
      prioritizer:
        condition: service_completed_successfully
    volumes:
      - ./data:/data
      - ./output:/output
    restart: "no"

The depends_on with condition: service_completed_successfully is the key to the sequential pipeline. This setting (available in Compose v2) tells Docker to wait until the previous container exits with a zero exit code before starting the next one. Without this condition, depends_on only waits for the container to start, not to finish — which would cause race conditions where the Summarizer tries to read a file the Ingestor has not written yet.

The volume mounts (./data:/data) map your local data folder into each container. All agents share this volume, which is how they pass files to each other. The Formatter also gets ./output:/output so the final digest lands on your host machine. The memory limit of 512M on the Summarizer prevents it from consuming too much RAM. And restart: "no" ensures Docker does not restart the agents after they finish, since they are batch jobs.

How to Run the Pipeline

docker compose up --build

The --build flag tells Compose to rebuild the images before running. You will see structured logs from each agent in sequence:

agent_ingestor    | 2025-01-20 07:00:01 [INFO] ingestor: Ingested 3 files
agent_summarizer  | 2025-01-20 07:00:04 [INFO] summarizer: Summary written
agent_prioritizer | 2025-01-20 07:00:05 [INFO] prioritizer: Prioritized 8 items
agent_formatter   | 2025-01-20 07:00:05 [INFO] formatter: Digest written

When all four containers finish, open output/daily_digest.md to see your morning brief.

How to Test the Pipeline

Unit Tests

Because each agent's core logic is a plain Python function, you can test it in isolation without Docker.

tests/test_prioritizer.py

import sys
sys.path.insert(0, 'agents/prioritizer')
from app import score_line

def test_urgent_keyword_scores_one():
    assert score_line("This is urgent") == 1

def test_multiple_keywords_stack():
    assert score_line("Urgent and important deadline") == 3

def test_no_keywords_scores_zero():
    assert score_line("Regular project update") == 0

def test_scoring_is_case_insensitive():
    assert score_line("URGENT DEADLINE ASAP") == 3

Run the tests with pytest:

pip install pytest
python -m pytest tests/ -v

Writing tests for each agent's core function means you can catch bugs before you build any Docker images, saving a lot of time compared to debugging inside running containers.

Integration Tests

To test the full pipeline end-to-end, create known input files and verify the expected output:

# Create test data
mkdir -p data/input
echo "Urgent: quarterly report due today" > data/input/test.txt
echo "Regular standup notes, no blockers" >> data/input/test.txt

# Run the pipeline
docker compose up --build

# Verify the output exists and contains expected content
test -f output/daily_digest.md && echo "File exists: PASS" || echo "File missing: FAIL"
grep -q "Priority" output/daily_digest.md && echo "Content check: PASS" || echo "Content check: FAIL"

How to Add Logging and Observability

Every agent uses Python's logging module with a consistent format. When Docker Compose runs all four containers, it interleaves their logs with container name prefixes, giving you a unified timeline of the entire pipeline.

For production systems, consider switching to JSON-formatted logs. They are easier to parse with log aggregation tools like the ELK Stack, Grafana Loki, or AWS CloudWatch:

import json
import logging

class JSONFormatter(logging.Formatter):
    def format(self, record):
        return json.dumps({
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "agent": record.name,
            "message": record.getMessage(),
        })

To use this formatter, replace the basicConfig call with a handler:

handler = logging.StreamHandler()
handler.setFormatter(JSONFormatter())
logger = logging.getLogger("summarizer")
logger.addHandler(handler)
logger.setLevel(logging.INFO)

The most useful metrics to track include the number of files ingested per run, Summarizer latency (time from API call to response), LLM token usage for cost tracking, the number of errors and retries per agent, and whether daily_digest.md was successfully generated. A simple approach for personal use is to write a JSON metrics file alongside the digest in the output directory. For team or production use, consider adding Prometheus metrics or sending data to a monitoring service.

Cost, Rate Limits, and Graceful Degradation

The Summarizer is the only agent that calls a paid API. Here is what you can expect to pay:

For a daily personal digest processing a few thousand tokens of input, gpt-4o-mini costs less than a penny per run. That works out to roughly three dollars per year.

To protect against unexpected bills, set a monthly spending cap in your OpenAI dashboard. You can also set per-minute rate limits to prevent runaway usage if a bug causes repeated API calls.

Beyond the retry logic already built into the Summarizer, you can cache LLM responses so that if the same input text appears again you reuse the previous summary instead of calling the API. Use the cheapest model that gives acceptable results — for summarization, gpt-4o-mini usually works as well as gpt-4o at a fraction of the cost. And batch requests when possible by combining many small texts into one API call.

The Summarizer already writes a fallback message when the API fails. This is the most important form of graceful degradation: the pipeline keeps running, and you get a less useful digest instead of nothing at all. If the digest is critical for your workflow, add an alerting step — for example, you could extend the Formatter to send a Slack notification when the Summarizer falls back.

Security and Privacy Considerations

When you feed personal data emails, meeting notes, private newsletters into an LLM, you need to think carefully about where that data goes.

Text you send to OpenAI or similar providers leaves your machine and is processed on their servers. As of early 2025, OpenAI's API does not use submitted data for model training by default, but policies can change. Always check your provider's current data retention and usage policies. If your input contains personally identifiable information like names, email addresses, or phone numbers, consider stripping it before calling the API, or use a local model.

The intermediate files created during the pipeline (ingested.txt, summary.txt, prioritized.txt) contain processed versions of your raw input. For personal use, keep them for debugging and delete manually. For automated pipelines, add a cleanup step that deletes intermediate files after the digest is generated. If you operate in the EU, review GDPR requirements around data minimization, right to deletion, and records of processing.

To secure your containers, use minimal base images like python:3.10-slim to reduce the attack surface, run containers as a non-root user by adding a USER directive to your Dockerfiles, update base images regularly (at least monthly) to pick up security patches, and scan your images for vulnerabilities using docker scout or Trivy.

How to Use a Local LLM for Full Privacy (Ollama)

If you want to keep all data on your machine and avoid sending anything to external APIs, you can swap the OpenAI API for a local model running through Ollama. Ollama lets you run open-source LLMs locally, handling model weight downloads, memory management, and serving an API.

To set up Ollama:

# Install Ollama (macOS or Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model (llama3 is a good general-purpose choice)
ollama pull llama3

# Verify it is running
ollama list

Replace the OpenAI API call in the Summarizer with a request to Ollama's local API:

import requests

def summarize_locally(text):
    """Call a local Ollama instance from inside a Docker container."""
    url = "http://host.docker.internal:11434/api/generate"
    payload = {
        "model": "llama3",
        "prompt": (
            "Summarize the following text into key "
            f"bullet points:\n\n{text}"
        ),
        "stream": False
    }
    try:
        resp = requests.post(url, json=payload, timeout=120)
        resp.raise_for_status()
        return resp.json().get('response', 'No response')
    except requests.exceptions.RequestException as e:
        return f"Ollama error: {e}"

The host.docker.internal hostname lets a container communicate with services running on the host machine. Ollama runs on your host (not inside a container), so this is how the Summarizer reaches it.

Note: On Linux, host.docker.internal may not resolve by default. Add this to your docker-compose.yml under the summarizer service: extra_hosts: ["host.docker.internal:host-gateway"]

Local models are slower than cloud APIs and require decent hardware (at least 8 GB of RAM for smaller models, 16 GB or more for larger ones). But they are free, fully private, and work without an internet connection.

Example Seed Data and Expected Output

To test the full pipeline without real newsletters, create these sample input files:

data/input/newsletter_ai.txt

AI Weekly Roundup - January 2025
OpenAI released a new reasoning model this week.
URGENT: New EU AI Act regulations take effect in March.
Google announced updates to their Gemini model family.
A startup raised $50M for AI-powered code review tools.

data/input/meeting_notes.txt:

Team Standup Notes - Monday
IMPORTANT: Deadline for Q1 report is this Friday.
Action required: Review the updated API documentation.
Sprint velocity is on track. No blockers reported.

Expected output in output/daily_digest.md:

# Your Daily AI Digest

**Date:** 2025-01-20

## Top Insights

- **Priority 3**: IMPORTANT: Deadline for Q1 report due Friday
- **Priority 2**: URGENT: New EU AI Act regulations in March
- **Priority 1**: Action required: Review the updated API docs
- **Priority 0**: OpenAI released a new reasoning model
- **Priority 0**: Sprint velocity is on track

The exact summary text will vary depending on your LLM model and settings, but the structure and priority ordering should remain consistent.

How to Automate Daily Execution

Now that the pipeline works end-to-end with a single command, you can schedule it to run automatically every morning.

How to Use Cron on Linux or macOS

Open your crontab with crontab -e and add this line to run the pipeline every day at 7:00 AM:

0 7 * * * cd /path/to/multi-agent-digest && docker compose up --build >> cron.log 2>&1

The >> cron.log 2>&1 part redirects all output (including errors) to a log file so you can check it later. Make sure your machine is running at the scheduled time and Docker Desktop is started.

How to Use Task Scheduler on Windows

Open Task Scheduler and create a new task. Under "Actions," set the program to:

wsl -e bash -c 'cd /mnt/c/path/to/multi-agent-digest && docker compose up --build'

Set the trigger to fire every morning at your preferred time.

How to Add Delivery Notifications

For the digest to be truly useful, you want it delivered to you rather than sitting in a folder. Here are three options:

Email — Extend the Formatter to send the digest via Python's smtplib module. You will need SMTP credentials for a service like Gmail, SendGrid, or Amazon SES.

Slack — Create an incoming webhook in your Slack workspace and POST the digest as a message. This takes about 10 lines of code.

Notion or Obsidian — Use their APIs to create a new page or note with the digest content each morning.

Troubleshooting Common Errors

Container exits with OOM error — Large files or LLM processing are exceeding memory. Increase the memory limit in docker-compose.yml under deploy > resources > limits > memory. Try 1G.

Rate limit errors from OpenAI — The retry logic handles temporary rate limits automatically. Check your OpenAI dashboard for usage caps.

depends_on does not wait for completion — Make sure you are using condition: service_completed_successfully, which requires Docker Compose v2.

Permission denied on /output — Volume mount permissions mismatch. Run chmod -R 777 ./output on the host, or add a USER directive to your Dockerfiles.

OPENAI_API_KEY not found — The .env file may be missing or not in the right directory. Create .env in the same folder as docker-compose.yml and verify with docker compose config.

Cannot reach Ollama from container — host.docker.internal may not be resolving on Linux. Add extra_hosts: ["host.docker.internal:host-gateway"] to the service in docker-compose.yml.

Production Deployment Options

The docker compose up approach works well for personal use and development. When you are ready to deploy to a server or the cloud, here are your main options.

Docker Swarm

Docker Swarm is the simplest step up from Compose. It lets you deploy across multiple machines with minimal changes to your existing Compose file:

docker swarm init
docker stack deploy -c docker-compose.yml morning-brief

Kubernetes

For production at scale, Kubernetes gives you more control over scheduling, scaling, and fault tolerance. Use Kubernetes Jobs (not Deployments) for batch agents that run once and exit. Set resource requests and limits on each container so the cluster scheduler can allocate resources efficiently. Store API keys in Kubernetes Secrets, and use CronJobs for scheduled daily execution — they work like cron but are managed by the cluster.

Cloud Platforms

All major cloud providers offer managed container services that can run this pipeline:

AWS — ECS Fargate with scheduled tasks for serverless execution, or EKS for managed Kubernetes.

Azure — Azure Container Instances for simple runs, or AKS for managed Kubernetes.

GCP — Cloud Run Jobs for serverless batch processing, or GKE for managed Kubernetes.

Conclusion and Next Steps

In this handbook, you built a multi-agent AI system from scratch. You created four specialized Python agents, containerized each one with Docker, orchestrated them with Docker Compose, and added secrets handling, structured logging, retry logic, and graceful fallbacks.

The core patterns you learned — separation of concerns, containerized agents, shared-volume communication, and defensive coding against external APIs — apply far beyond this specific use case. Any time you need a reliable, modular, and reproducible AI workflow, these patterns are a solid foundation.

Here are some directions to explore next:

Agent collaboration frameworks — Tools like CrewAI and LangGraph let you build agents that delegate tasks to each other, negotiate priorities, and collaborate in more sophisticated ways.

Local and fine-tuned models — Experiment with Ollama or vLLM to run models locally. Fine-tune a small model specifically for summarization to get better results at lower cost.

Event-driven architectures — Replace the shared volume with Redis or RabbitMQ so agents react to events in real time rather than running on a schedule.

Feedback loops — Add an agent that evaluates the quality of the daily digest and adjusts the Summarizer's prompts over time. This is how production agent systems learn and improve.

In this tutorial, you’ll learn how to run an LLM locally and privately, so you can search and chat with sensitive journals and business docs on your own machine. We’ll install Ollama and OpenWebUI, pick a model that fits your hardware, enable private document search with nomic-embed-text, and create a local knowledge base so everything stays on-disk.

Table of Contents

• Prerequisites

• Installation

• Settings for Documents

• How to Upload Your Documents

  • (Optional) Adding a system prompt

• How to Run Your LLM Locally

• Conclusion

Prerequisites

You’ll need a terminal (all systems—Windows, Mac, Linux—include one, and you can find yours with a quick search), and either Python and pip or Docker, depending on your preferred installation method for OpenWebUI.

Installation

You’ll need Ollama and OpenWebUI. Ollama runs the models, while OpenWebUI gives you a browser interface to interact with your local LLM, like you would with ChatGPT.

Step 1: Install Ollama

Download and install Ollama from its official site. Installers are available for macOS, Linux, and Windows. Once installed, verify it’s running by opening a terminal and executing:

ollama list

If Ollama is running, this will return a list of active models (or an empty list).

Step 2: Install OpenWebUI

You can install OpenWebUI either with Python (pip) or with Docker. Here, we will show how to do it with pip, but you can find instructions for Docker on the official openwebui docs.

Install OpenWebUI with the following command:

pip install open-webui

This works on macOS, Linux, and Windows, as long as you have Python ≥ 3.9 installed.

Next, start the server:

open-webui serve

Then open your browser and go to:

http://localhost:8080

Step 3: Install a Model

Choose a model from the Ollama model list and pull it locally by copying the command provided.

For example:

ollama pull gemma3:4b

If you’re unsure which model your machine can handle, ask an AI to recommend one based on your hardware. Smaller models (1B–4B) are safer on laptops.

I would recommend Gemma3 as a starter (you can download multiple models and easily switch between them). Pick the parameter number at the end (“:4b”, “:1b”, and so on) based on this guide:

• Tier 1 (small laptops or weak computers): RAM ≤8 GB or no GPU → 1B–2B.

• Tier 2: RAM 16 GB, weak GPU → 2B–4B.

• Tier 3: RAM ≥16 GB, 6–8 GB VRAM → 4B–9B.

• Tier 4: RAM ≥32 GB, 12 GB+ VRAM → 12B+.

Once you have installed Ollama and your desired model, confirm that they are active by running ollama list in the terminal:

Run WebOpenUI to launch the browser interface with:

open-webui serve

Then head over to http://localhost:8080/. Now you are ready to start using your LLM locally!

Note: it will ask you for login credentials, but these don’t really matter if you only intend to use it locally.

Settings for Documents

Now we are going to set up everything we need to interact with our local documents. First of all, we need to install the “nomic-embed-text” model to process our documents. Install it with:

ollama pull nomic-embed-text

Note: If you are wondering why we need another model (nomic-embed-text) besides our main one:

• The embedding model (nomic-embed-text) maps each text chunk from your documents to a numerical vector so OpenWebUI can quickly find semantically similar chunks when you ask a question.​

• The chat model (for example gemma3:1b) receives your question plus those retrieved chunks as context and generates the natural-language response.

Next, you should enable the “memory” feature if you want the LLM to remember the context of your past conversations in your future ones.

Download the adaptive memory function here. Functions are like plug-ins.

Now we will update our settings to enable these features. Click on your name in the bottom-left corner, then “Settings”.

Click on the first one, then go to “Personalization” and enable “Memory”.

Now we are going to access the other settings panel (“Admin Panel”). Click again on your name in the bottom-left corner and go to Admin panel → Settings → Documents.

In this section (Admin Panel → Settings → Documents), find the “Embedding” section, go to “Embedding Model Engine” and choose Ollama (find the selectable to the right). Leave the API Key blank.

Now, under “Embedding Model” write nomic-embed-text. Then go to “Retrieval” → enable “Full Context Mode”.

Chunking settings

You should also set the chunk size and overlap. OpenWebUI splits documents into smaller chunks before indexing them, since models can’t embed or retrieve very long texts in one piece.

A good default is 128–512 tokens per chunk, with 10–20% overlap. Larger chunks preserve more context but are slower and more memory-intensive, while smaller chunks are faster but can lose higher-level meaning. Overlap helps prevent important context from being cut off when text is split.

Here’s a guiding table, but I recommend obtaining the recommended values for your specific use case and setup by sharing them (including GPU or laptop model, storage, RAM, and so on) with an LLM like ChatGPT or Claude, as changing the chunking/overlap values later on requires reuploading the documents.

Suggested chunk/overlap by tier

How to Upload Your Documents

Now, the final step: uploading your documents! Go to “Workspace” in the side panel, then “Knowledge”, and create a new collection (database). You can start uploading files here.

Then, within “Workspace”, switch to the “Models” tab and create a new custom model. Creating a custom model and attaching your knowledge base tells OpenWebUI to automatically search your document collection and include the most relevant chunks as context whenever you ask a question.

Here, make sure to select your model (in my case “gemma3:1b”) and attach your knowledge base.

(Optional) Adding a system prompt

When creating your custom model in Workspace → Models, you can define a system prompt that the model will use for context throughout all your conversations.

Here are some examples of information you might want to add:

• context about yourself (“I am a 20-year-old student in bioengineering interested in…”)

• your preferred communication style (“no fluff", “be direct”, “be analytical”…)

• context about how your data is structured

Example system prompt:

You are a thoughtful, analytical assistant helping me explore patterns and insights in my personal journals. Be direct, avoid speculation, and clearly distinguish between facts from the documents and interpretation.

This prompt will automatically apply to every chat using this custom model, helping keep responses consistent and aligned with your goals.

How to Run Your LLM Locally

Now open a new chat and make sure to select your custom model:

Now you are ready to chat with your own docs in a private local environment!

Conclusion

You now have a private LLM stack (Ollama for models, OpenWebUI for the UI, and nomic-embed-text for embeddings) wired to your on-disk knowledge base. Your journals and business docs stay local; nothing is sent to third parties. The main dials are simple: pick a model that fits your hardware, enable memory and full-context retrieval, use sensible chunk/overlap, and check the console when runs stall.

If you need more headroom, deploy the same setup on your own server and keep the privacy guarantees. From here, iterate on model choice, chunking, and prompts, and add the optional functions if you need deeper visibility during long jobs.

This guide will show you how to set up an open-source LLM locally, explain the tools involved, and walk you through both the UI and command-line installation methods.

What We’ll Cover

• Understanding Open Source LLMs

• Choosing a Platform to Run LLMs Locally

• How to Install Ollama

• How to Install and Run LLMs via the Command Line

• How to Manage Models and Resources

• How to Use Ollama with Other Applications

• Troubleshooting and Common Issues

• Why Running LLMs Locally Matters

• Conclusion

Understanding Open Source LLMs

An open-source large language model is a type of AI that can understand and generate text, much like ChatGPT, but it can function without depending on external servers.

You can download the model files, run them on your machine, and even fine-tune them for your use cases.

Projects like Llama 3, Mistral, Gemma, and Phi have made it possible to run models that fit well on consumer hardware. You can choose between smaller models that run on CPUs or larger ones that benefit from GPUs.

Running these models locally gives you privacy, control, and flexibility. It also helps developers integrate AI features into their applications without relying on cloud APIs.

Choosing a Platform to Run LLMs Locally

To run an open source model, you need a platform that can load it, manage its parameters, and provide an interface to interact with it.

Three popular choices for local setup are:

1. Ollama — a user-friendly system that runs models like OpenAI GPT OSS, Google Gemma with one command. It has both a Windows UI and CLI version.

2. LM Studio — a graphical desktop application for those who prefer a point-and-click interface.

3. Gpt4All — another popular GUI desktop application.

We’ll use Ollama as the example in this guide since it’s widely supported and integrates easily with other tools.

How to Install Ollama

Ollama provides a one-click installer that sets up everything you need to run local models. Visit the official Ollama website and download the Windows installer.

Once downloaded, double-click the file to start installation. The setup wizard will guide you through the process, which only takes a few minutes.

When the installation finishes, Ollama will run in the background as a local service. You can access it either through its graphical desktop interface or using the command line.

After installing Ollama, you can open the application from the Start Menu. The UI makes it easy for beginners to start interacting with local models.

On the Ollama interface, you’ll see a simple text box where you can type prompts and receive responses. There’s also a panel that lists available models.

To download and use a model, just select it from the list. Ollama will automatically fetch the model weights and load them into memory.

The first time you ask a question, it will download the model if it does not exist. You can also choose the model from the models search page.

I’ll use the gemma 270m model which is the smallest model available in Ollama.

You can see the model being downloaded when used for the first time. Depending on the model size and your system’s performance, this might take a few minutes.

Once loaded, you can start chatting or running tasks directly within the UI. It’s designed to look and feel like a normal chat window, but everything runs locally on your PC.

You don’t need an internet connection after the model has been downloaded.

How to Install and Run LLMs via the Command Line

If you prefer more control, you can use the Ollama command-line interface (CLI). This is useful for developers or those who want to integrate local models into scripts and workflows.

To open the command line, search for “Command Prompt” or “PowerShell” in Windows and run it. You can now interact with Ollama using simple commands.

To check if the installation worked, type:

ollama --version

If you see a version number, Ollama is ready. Next, to run your first model, use the pull command:

ollama pull gemma3:270m

This will download the Gemma model to your machine.

When the process finishes, start it with:

ollama run gemma3:270m

Ollama will launch the model and open an interactive prompt where you can type messages.

Everything happens locally, and your data never leaves your computer.

You can stop the model anytime by typing /bye.

How to Manage Models and Resources

Each model you download takes up disk space and memory. Smaller models like Phi-3 Mini or Gemma 2B are lighter and suitable for most consumer laptops. Larger ones such as Mistral 7B or Llama 3 8B require more powerful GPUs or high-end CPUs.

You can list all installed models using:

ollama list

And remove one when you no longer need it:

ollama rm model_name

If your PC has limited RAM, try running smaller models first. You can experiment with different ones to find the right balance between speed and accuracy.

How to Use Ollama with Other Applications

Once you’ve installed Ollama, you can use it beyond the chat interface. Developers can connect to it using APIs and local ports.

Ollama runs a local server on http://localhost:11434. This means you can send requests from your own scripts or applications.

For example, a simple Python script can call the local model like this:

import requests, json

# Define the local Ollama API endpoint
url = "http://localhost:11434/api/generate"

# Send a prompt to the Gemma 3 model
payload = {
    "model": "gemma3:270m",
    "prompt": "Write a short story about space exploration."
}

# stream=True tells requests to read the response as a live data stream
response = requests.post(url, json=payload, stream=True)

# Ollama sends one JSON object per line as it generates text
for line in response.iter_lines():
    if line:
        data = json.loads(line.decode("utf-8"))
        # Each chunk has a "response" key containing part of the text
        if "response" in data:
            print(data["response"], end="", flush=True)This setup turns your computer into a local AI engine. You can integrate it with chatbots, coding assistants, or automation tools without using external APIs.

Troubleshooting and Common Issues

If you face issues running a model, check your system resources first. Models need enough RAM and disk space to load properly. Closing other apps can help free up memory.

Sometimes, antivirus software may block local network ports. If Ollama fails to start, add it to the list of allowed programs.

If you use the CLI and see errors about GPU drivers, ensure that your graphics drivers are up to date. Ollama supports both CPU and GPU execution, but having updated drivers improves performance.

Why Running LLMs Locally Matters

Running LLMs locally changes how you work with AI. You’re no longer tied to API costs or rate limits. It’s ideal for developers who want to prototype fast, researchers exploring fine-tuning, or hobbyists who value privacy.

Local models are also great for offline environments. You can experiment with prompt design, generate content, or test AI-assisted apps without an internet connection.

As hardware improves and open source communities grow, local AI will continue to become more powerful and accessible.

Conclusion

Setting up and running an open-source LLM on Windows is now simple. With tools like Ollama and LM Studio, you can download a model, run it locally, and start generating text in minutes.

The UI makes it friendly for beginners, while the command line offers full control for developers. Whether you’re building an app, testing ideas, or exploring AI for personal use, running models locally puts everything in your hands, making it fast, private, and flexible.

Hope you enjoyed this article. Signup for my free newsletter TuringTalks.ai for more hands-on tutorials on AI. You can also visit my website.

You can enhance the performance of an LLM by providing context — structured or unstructured data, such as documents, articles, or knowledge bases — tailored to the domain or information you want the model to specialize in. Using techniques like prompt engineering and context injection, you can build an intelligent chatbot capable of navigating extensive datasets, retrieving relevant information, and delivering responses.

Whether it's storing recipes, code documentation, research articles, or answering domain-specific queries, an LLM-based chatbot can adapt to your needs with customization and privacy. You can deploy it locally to create a highly specialized conversational assistant that respects your data.

In this article, you will learn how to build a local Retrieval-Augmented Generation (RAG) application using Ollama and ChromaDB in R. By the end, you'll have a custom conversational assistant with a Shiny interface that efficiently retrieves information while maintaining privacy and customization.

Table of Contents

• What is RAG?

• Project Overview

• Project Setup

• Ollama Installation

• Data Collection and Cleaning

• How to Create Chunks

• How to Generate Sentence Embeddings

• How to Set Up the Vector Database for Embedding Storage

• How to Write the User Input Query Embedding Function

• Tool Calling

• How to Initialize the Chat System, Design Prompts, and Integrate Tools

• How to Interact with Your Chatbot Using a Shiny App

• Complete Code

• Conclusion

What is RAG?

Retrieval-Augmented Generation (RAG) is a method that integrates retrieval systems with generative AI, enabling chatbots to access recent and specific information from external sources.

By using a retrieval pipeline, the chatbot can fetch up-to-date, relevant data and combine it with the generative model’s language capabilities, producing responses that are both accurate and contextually enriched. This makes RAG particularly useful for applications requiring fact-based, real-time knowledge delivery.

Project Overview

Project Setup

Prerequisites

Before you begin, ensure you have installed the latest version of the items listed here:

1. RStudio: The IDE – RStudio is the primary workspace where you'll write and test your R code. Its user-friendly interface, debugging tools, and integrated environment make it ideal for data analysis and chatbot development.

2. R: The Programming Language – R is the backbone of your project. You'll use it to handle data manipulation, apply statistical models, and integrate your recipe chatbot components seamlessly.

3. Python – Some libraries, like the embedding library you'll use for text vectorization, are built on Python. It’s vital to have Python installed to enable these functionalities alongside your R code.

4. Java – Java serves as a foundational element for certain embedding libraries. It ensures efficient processing and compatibility for text embedding tasks required to train your chatbot.

5. Docker Desktop – Docker Desktop allows you to run ChromaDB, the vector database, locally on your machine. This enables fast and reliable storage of embeddings, ensuring your chatbot retrieves relevant information quickly.

6. Ollama – Ollama brings powerful Large Language Models (LLMs) directly to your local computer, removing the need for cloud resources. It lets you access multiple models, customize outputs, and integrate them into your chatbot effortlessly.

Ollama Installation

Ollama is an open-sourced tool you can use to run and manage LLMs on your computer. Once installed, you can access various LLMs as per your needs. You will be using llama3.2:3b-instruct-q4_K_M model to build this chatbot.

A quantized model is a version of a machine learning model that has been optimized to use less memory and computational power by reducing the precision of the numbers it uses. This enables you to use an LLM locally, especially when you don’t have access to a GPU (Graphics Processing Unit – a specialized processor that perform complex computations).

To start, you can download and install the Ollama software here.

Then you can confirm installation by running this command:

ollama --version

Run the following command to start Ollama:

ollama serve

Next, run the following command to pull the Q4_K_M quantization of llama3.2:3b-instruct:

ollama pull llama3.2:3b-instruct-q4_K_M

Then confirm that the model was extracted with this:

ollama list

If the model extraction was successful, a list containing the model’s name, ID, and size will be returned, like so:

Now you can chat with the model:

ollama run llama3.2:3b-instruct-q4_K_M

If successful, you should receive a prompt that you can test by asking a question and getting an answer. For example:

Then you can exit the console by typing /bye or ctrl + D

Data Collection and Cleaning

The chatbot you are building will be a cooking assistant that suggests recipes given your available ingredients, what you want to eat, and how much food a recipe yields.

You first have to get the data to train the model. You will be using a dataset that contains recipes from Kaggle.

To start, load the necessary libraries:

# loading required libraries
library(xml2) #read, parse, and manipulate XML,HTML documents
library(jsonlite) #manipulate JSON objects

library(RKaggle) # download datasets from Kaggle 
library(dplyr)   # data manipulation

Then download and save recipe dataset:

# Download and read the "recipe" dataset from Kaggle
recipes_list <- RKaggle::get_dataset("thedevastator/better-recipes-for-a-better-life")

Inspect the dataframe and extract the first element like this:

# inspect the dataset
class(recipes_list)
str(recipes_list)
head(recipes_list)
# extract the first tibble
recipes_df <- recipes_list[[1]]

A quick inspection of the recipes_list object shows that it contains two objects of type tibble. You will be using only the first element for this project. A tibble is a type of data structure used for storing and manipulating data. It’s similar to a traditional dataframe, but it’s designed to enforce stricter rules and perform fewer automatic actions compared to traditional dataframes.

We’ll use a regular dataframe in this project because more people are likely familiar with it. It can also efficiently handle row indexing, which is crucial for accessing and manipulating specific rows in our recipe dataset.

In the code block below, you’ll convert the tibble to a dataframe and then drop the first column, which is the index column. Then you’ll inspect the newly converted dataframe and drop unnecessary columns.

Unnecessary columns are best removed to streamline the dataset and focus on relevant features. In this project, we’ll drop certain columns that aren’t particularly useful for training the chatbot. This ensures that the model concentrates on meaningful data to improve its accuracy and functionality.

# convert to dataframe and drop the first column
recipes_df <- as.data.frame(recipes_df[, -1])
# inspect the converted dataframe
head(recipes_df)
class(recipes_df)
colnames(recipes_df)
# drop unnecessary columns
cleaned_recipes_df <- subset(recipes_df, select = -c(yield,rating,url,cuisine_path,nutrition,timing,img_src))

Now you need to identify rows with NA (missing) values, which you can do like this:

# Identify rows and columns with NA values
which(is.na(cleaned_recipes_df), arr.ind = TRUE)

# a quick inspection reveals columns [2:4] have missing values
subset_column_names <- colnames(cleaned_recipes_df)[2:4]
subset_column_names

It is important to handle NA values to ensure that your data is complete, to prevent errors, and to preserve context.

Now, replace the NA values and confirm that there are no missing values:

# Replace NA values dynamically based on conditions
cols_to_modify <- c("prep_time", "cook_time", "total_time")
cleaned_recipes_df[cols_to_modify] <- lapply(
  cleaned_recipes_df[cols_to_modify],
  function(x, df) {
    # Replace NA in prep_time and cook_time where both are NA
    replace(x, is.na(df$prep_time) & is.na(df$cook_time), "unknown")
  },
  df = cleaned_recipes_df  # Pass the whole dataframe for conditions
)
cleaned_recipes_df <- cleaned_recipes_df %>%
  mutate(
    prep_time = case_when(
      # If cooktime is present but preptime is NA, replace with "no preparation required"
      !is.na(cook_time) & is.na(prep_time) ~ "no preparation required",
      # Otherwise, retain original value
      TRUE ~ as.character(prep_time)
    ),
    cook_time = case_when(
      # If prep_time is present but cook_time is NA, replace with "no cooking required"
      !is.na(prep_time) & is.na(cook_time) ~ "no cooking required",
      # Otherwise, retain original value
      TRUE ~ as.character(cook_time)
    )
  )
# confirm there are no missing values
any(is.na(cleaned_recipes_df))
)

# confirm the replacing NA logic works by inspecting specific rows
cleaned_recipes_df[1081,]
cleaned_recipes_df[1,]
cleaned_recipes_df[405,]

For this tutorial, we’ll subset the dataframe to the first 250 rows for demo purposes. This saves on time when it comes to generating embeddings.

# recommended for demo/learning purposes
cleaned_recipes_df <- head(cleaned_recipes_df,250)

How to Create Chunks

To understand why chunking is important before embedding, you need to understand what an embedding is.

An embedding is a vectoral representation of a word or a sentence. Machines don’t understand human text – they understand numbers. LLMs work by transforming human text to numerical representations in order to give answers. The process of generating embeddings requires a lot of computation, and breaking down the data to be embedded optimizes the embedding process.

So now we’re going to split the dataframe into smaller chunks of a specified size to enable efficient batch processing and iteration.

# Define the size of each chunk (number of rows per chunk)
chunk_size <- 1

# Get the total number of rows in the dataframe
n <- nrow(cleaned_recipes_df)

# Create a vector of group numbers for chunking
# Each group number repeats for 'chunk_size' rows
# Ensure the vector matches the total number of rows
r <- rep(1:ceiling(n/chunk_size), each = chunk_size)[1:n]

# Split the dataframe into smaller chunks (subsets) based on the group numbers
chunks <- split(cleaned_recipes_df, r)

How to Generate Sentence Embeddings

As previously mentioned, embeddings are vector representations of words or sentences. Embeddings can be generated from both words and sentences. How you choose to generate embeddings depends on your intended application of the LLM.

Word embeddings are numerical representations of individual words in a continuous vector space. They capture semantic relationships between words, allowing similar words to have vectors close to each other.

Word embeddings can be used in search engines as they support word-level queries by matching embeddings to retrieve relevant documents. They can also be used in text classification to classify documents, emails, or tweets based on word-level features (for example, detecting spam emails or sentiment analysis).

Sentence embeddings are numerical representations of entire sentences in a vector space, designed to capture the overall meaning and context of the sentence. They are used in settings where sentences provide better context like question answering systems where user queries are matched to relevant sentences or documents for more precise retrieval.

For our recipe chatbot, sentence embedding is the best choice.

First, create an empty dataframe that has three columns.

#empty dataframe
recipe_sentence_embeddings <-  data.frame(
  recipe = character(),
  recipe_vec_embeddings = I(list()),
  recipe_id = character()
)

The first column will hold the actual recipe in text form, the recipe_vec_embeddings column will hold the generated sentence embeddings, and the recipe_id holds a unique id for each recipe. This will help in indexing and retrieval from the vector database.

Next, it’s helpful to define a progress bar, which you can do like this:

# create a progress bar
pb <- txtProgressBar(min = 1, max = length(chunks), style = 3)

Embedding can take a while, so it’s important to keep track of the progress of the process.

Now it’s time to generate embeddings and populate the dataframe.

Write a for loop that executes the code block as long as the length of the chunks.

for (i in 1:length(chunks)) {}

The recipe field is the text at the chunk that is currently being executed and the unique chunk id is generated by pasting the index of the chunk and the text “chunk”.

for (i in 1:length(chunks)) {
    recipe <- as.character(chunks[i])
    recipe_id <- paste0("recipe",i)
}

The text embed function from the text library generates either sentence or word embeddings. It takes in a character variable or a dataframe and produces a tibble of embeddings. You can read loading instructions here for smooth running of the text library.

The batch_size defines how many rows are embedded at a time from the input. Setting the keep_token_embeddings discards the embeddings for individual tokens after processing, and aggregation_from_layers_to_tokens “concatenates” or combines embeddings from specified layers to create detailed embeddings for each token. A token is the smallest unit of text that a model can process.

for (i in 1:length(chunks)) {
    recipe <- as.character(chunks[i])
    recipe_id <- paste0("recipe",i)
    recipe_embeddings <- textEmbed(as.character(recipe),
                                layers = 10:11,
                                aggregation_from_layers_to_tokens = "concatenate",
                                aggregation_from_tokens_to_texts = "mean",
                                keep_token_embeddings = FALSE,
                                batch_size = 1
  )
}

In order to specify sentence embeddings, you need to set the argument to the aggregation_from_tokens_to_texts parameter as "mean".

aggregation_from_tokens_to_texts = "mean"

The "mean" operation averages the embeddings of all tokens in a sentence to generate a single vector that represents the entire sentence. This sentence-level embedding captures the overall meaning and semantics of the text, regardless of its token length.

# convert tibble to vector
  recipe_vec_embeddings <- unlist(recipe_embeddings, use.names = FALSE)
  recipe_vec_embeddings <- list(recipe_vec_embeddings)

The embedding function returns a tibble object. In order to obtain a vector embedding, you need to first unlist the tibble and drop the row names and then list the result to form a simple vector.

  # Append the current chunk's data to the dataframe
  recipe_sentence_embeddings <- recipe_sentence_embeddings %>%
    add_row(
      recipe = recipe,
      recipe_vec_embeddings = recipe_vec_embeddings,
      recipe_id = recipe_id
    )

Finally, update the empty dataframe after each iteration with the newly generated data.

  # track embedding progress
  setTxtProgressBar(pb, i)

In order to keep track of the embedding progress, you can use the earlier defined progress bar inside the loop. It will update at the end of every iteration.

Complete Code Block:

# load required library
library(text)
# # ensure to read loading instructions here for smooth running of the 'text' library
# # https://www.r-text.org/
# embedding data
for (i in 1:length(chunks)) {
  recipe <- as.character(chunks[i])
  recipe_id <- paste0("recipe",i)
  recipe_embeddings <- textEmbed(as.character(recipe),
                                layers = 10:11,
                                aggregation_from_layers_to_tokens = "concatenate",
                                aggregation_from_tokens_to_texts = "mean",
                                keep_token_embeddings = FALSE,
                                batch_size = 1
  )

  # convert tibble to vector
  recipe_vec_embeddings <- unlist(recipe_embeddings, use.names = FALSE)
  recipe_vec_embeddings <- list(recipe_vec_embeddings)

  # Append the current chunk's data to the dataframe
  recipe_sentence_embeddings <- recipe_sentence_embeddings %>%
    add_row(
      recipe = recipe,
      recipe_vec_embeddings = recipe_vec_embeddings,
      recipe_id = recipe_id
    )

  # track embedding progress
  setTxtProgressBar(pb, i)

}

How to Set Up the Vector Database for Embedding Storage

A vector database is a special type of database that stores embeddings and allows you to query and retrieve relevant information. There are numerous vector databases available, but for this project, you will use ChromaDB, an open-source option that integrates with the R environment through the rchroma library.

ChromaDB runs locally in a Docker container. Just make sure you have Docker installed and running on your device.

Then load the rchroma library and run your ChromaDB instance:

# load rchroma library
library(rchroma)
# run ChromaDB instance.
chroma_docker_run()

If it was successful, you should see this in the console:

Next, connect to a local ChromaDB instance and check the connection:

# Connect to a local ChromaDB instance
client <- chroma_connect()

# Check the connection
heartbeat(client)
version(client)

Now you’ll need to create a collection and confirm that it was created. Collections in ChromaDB function similarly to tables in conventional databases.

# Create a new collection
create_collection(client, "recipes_collection")

# List all collections
list_collections(client)

Now, add embeddings to the collection. To add embeddings to the recipes_collection, use the add_documents function.

# Add documents to the collection
add_documents(
  client,
  "recipes_collection",
  documents = recipe_sentence_embeddings$recipe,
  ids = recipe_sentence_embeddings$recipe_id,
  embeddings = recipe_sentence_embeddings$recipe_vec_embeddings
)

The add_documents() function is used to add recipe data to the recipes_collection. Here's a breakdown of its arguments and how the corresponding data is accessed:

1. documents: This argument represents the recipe text. It is sourced from the recipe column of the recipe_sentence_embeddings dataframe.

2. ids: This is the unique identifier for each recipe. It is extracted from the recipe_id column of the same dataframe.

3. embeddings: This contains the sentence embeddings, which were previously generated for each recipe. These embeddings are accessed from the recipe_vec_embeddings column of the dataframe.

All three arguments—documents, ids, and embeddings—are obtained by subsetting their respective columns from the recipe_sentence_embeddings dataframe.

How to Write the User Input Query Embedding Function

In order to retrieve information from a vector database, you must first embed your query text. The database compares your query's embedding with its stored embeddings to find and retrieve the most relevant document.

It's important to ensure that the dimensions (rows × columns) of your query embedding match those of the database embeddings. This alignment is achieved by using the same embedding model to generate your query.

Matching embeddings involves calculating the similarity (for example, cosine similarity) between the query and stored embeddings, identifying the closest match for effective retrieval.

Let’s write a function that allows us to embed a query which then queries similar documents using the generated embeddings. Wrapping it in a function makes it reusable.

  #sentence embeddings function and query
  question <- function(sentence){
    sentence_embeddings <- textEmbed(sentence,
                                     layers = 10:11,
                                     aggregation_from_layers_to_tokens = "concatenate",
                                     aggregation_from_tokens_to_texts = "mean",
                                     keep_token_embeddings = FALSE
    )

    # convert tibble to vector
    sentence_vec_embeddings <- unlist(sentence_embeddings, use.names = FALSE)
    sentence_vec_embeddings <- list(sentence_vec_embeddings)

    # Query similar documents using embeddings
    results <- query(
      client,
      "recipes_collection",
      query_embeddings = sentence_vec_embeddings ,
      n_results = 2
    )
    results

  }

This chunk of code is similar to how we have previously used the text_embed() function. The query() function is added to enable querying the vector database, particularly the recipes' collection, and returns the top two documents that closely match a user’s query.

Our function thus takes in a sentence as an argument and embeds the sentence to generate sentence embeddings. It then queries the database and returns two documents that match the query most.

Tool Calling

To interact with Ollama in R, you will utilize the ellmer library. This library streamlines the use of large language models (LLMs) by offering an interface that enables seamless access to and interaction with a variety of LLM providers.

To enhance the LLM’s usage, we need to provide context to it. You can do this by tool calling. Tool calling allows an LLM to access external resources in order to enhance its functionality.

For this project, we are implementing Retrieval-Augmented Generation (RAG), which combines retrieving relevant information from a vector database and generating responses using an LLM. This approach improves the chatbot's ability to provide accurate and contextually relevant answers.

Now, define a function that links to the LLM to provide context using the tool() function from the ellmer library.

# load ellmer library
library(ellmer)

# function that links to llm to provide context
  tool_context  <- tool(
    question,
    "obtains the right context for a given question",
    sentence = type_string()

  )

The tool() function takes the question function that returns the relevant documents that we’ll use as context as the first argument. We’ll use the documents to help the LLM answer questions accordingly.

The text, "obtains the right context for a given question", is a description of what the tool will be doing.

Finally, the sentence = type_string() defines what type of object the question() function expects.

How to Initialize the Chat System, Design Prompts, and Integrate Tools

Next, you’ll set up a conversational AI system by defining its role and functionality. Using system prompt design, you will shape the assistant’s behavior, tone, and focus as a culinary assistant. You’ll also integrate external tools to extend the chatbot’s capabilities by registering tools. Let’s dive in.

First, you need to initialize a Chat Object:

#  Initialize the chat system with propmpt instructions.
  chat <- chat_ollama(system_prompt = "You are a knowledgeable culinary assistant specializing in recipe recommendations. 
                      You provide tailored meal suggestions based on the user's available ingredients and the desired amount of food or servings.
                      Ensure the recipes align closely with the user's inputs and yield the expected quantity.",
                      model = "llama3.2:3b-instruct-q4_K_M")

You can do that using the chat_ollama() function. This sets up a conversational agent with the specified system prompt and model.

The system prompt defines the conversational behavior, tone, and focus of the LLM while the model argument specifies the language model (llama3.2:3b-instruct-q4_K_M) that the chat system will use to generate responses.

Next, you need to register a tool.

 #register tool
  chat$register_tool(tool_context)

We need to tell our chat object about our tool_context() function. Do this by registering a tool using the register_tool() function.

How to Interact with Your Chatbot Using a Shiny App

To interact with the chatbot you’ve just created, we’ll use Shiny, a framework for building interactive web applications in R. Shiny provides a user-friendly graphical interface that allows seamless interaction with the chatbot.

For this purpose, we’ll use the shinychat library, which simplifies the process of building a chat interface within a Shiny app. This involves defining two key components:

1. User Interface (UI):

   • Responsible for the visual layout and what the user sees.

   • In this case, chat_ui("chat") is used to create the interactive chat interface.

2. Server Function:

   • Handles the functionality and logic of the application.

   • It connects the chatbot to external tools and manages processes like embedding queries, retrieving relevant responses, and handling user inputs.

# load the required library
library(shinychat)

# wrap the chat code in a Shiny App
ui <- bslib::page_fluid(
  chat_ui("chat")
)

server <- function(input, output, session) {
  # Connect to a local ChromaDB instance running on docker with embeddings loaded
  client <- chroma_connect()

  #sentence embeddings function and query
  question <- function(sentence){
    sentence_embeddings <- textEmbed(sentence,
                                     layers = 10:11,
                                     aggregation_from_layers_to_tokens = "concatenate",
                                     aggregation_from_tokens_to_texts = "mean",
                                     keep_token_embeddings = FALSE
    )

    # convert tibble to vector
    sentence_vec_embeddings <- unlist(sentence_embeddings, use.names = FALSE)
    sentence_vec_embeddings <- list(sentence_vec_embeddings)

    # Query similar documents using embeddings
    results <- query(
      client,
      "recipes_collection",
      query_embeddings = sentence_vec_embeddings ,
      n_results = 2
    )
    results

  }


  # function that provides context
  tool_context  <- tool(
    question,
    "obtains the right context for a given question",
    sentence = type_string()

  )

  #  Initialize the chat system with the first chunk
  chat <- chat_ollama(system_prompt = "You are a knowledgeable culinary assistant specializing in recipe recommendations. 
                      You provide tailored meal suggestions based on the user's available ingredients and the desired amount of food or servings.
                      Ensure the recipes align closely with the user's inputs and yield the expected quantity.",
                      model = "llama3.2:3b-instruct-q4_K_M")
  #register tool
  chat$register_tool(tool_context)

  observeEvent(input$chat_user_input, {
    stream <- chat$stream_async(input$chat_user_input)
    chat_append("chat", stream)
  })
}

shinyApp(ui, server)

Alright, let’s understand how this is working:

1. User input monitoring with observeEvent(): The observeEvent() block monitors user inputs from the chat interface (input$chat_user_input). When a user sends a message, the chatbot processes it, retrieves relevant context using the embeddings, and streams the response dynamically to the chat interface.

2. Tool calling for context: The chatbot employs tool calling to interact with external resources (like the vector database) and enhance its functionality. In this project, Retrieval-Augmented Generation (RAG) ensures the chatbot provides accurate and context-rich responses by integrating retrieval and generation seamlessly.

This approach brings the chatbot to life, enabling users to interact with it dynamically through a responsive Shiny app.

Complete Code

The R scripts have been split in two, with data.R containing code that handles data gathering and cleaning, text chunking, sentence embeddings generation, creating a vector database, and loading documents to it.

The chat.R script contains code that handles user input querying, context retrieval, chat initialization, system prompt design, tool integration, and a chat Shiny app.

data.R

# install and load required packages
# install devtools from CRAN
install.packages('devtools')
devtools::install_github("benyamindsmith/RKaggle")

library(text)
library(rchroma)
library(RKaggle)
library(dplyr)

# run ChromaDB instance.
chroma_docker_run()

# Connect to a local ChromaDB instance
client <- chroma_connect()

# Check the connection
heartbeat(client)
version(client)


# Create a new collection
create_collection(client, "recipes_collection")

# List all collections
list_collections(client)

# Download and read the "recipe" dataset from Kaggle
recipes_list <- RKaggle::get_dataset("thedevastator/better-recipes-for-a-better-life")

# extract the first tibble
recipes_df <- recipes_list[[1]]

# convert to dataframe and drop the first column
recipes_df <- as.data.frame(recipes_df[, -1])

# drop unnecessary columns
cleaned_recipes_df <- subset(recipes_df, select = -c(yield,rating,url,cuisine_path,nutrition,timing,img_src))

## Replace NA values dynamically based on conditions
# Replace NA when all columns have NA values
cols_to_modify <- c("prep_time", "cook_time", "total_time")
cleaned_recipes_df[cols_to_modify] <- lapply(
  cleaned_recipes_df[cols_to_modify],
  function(x, df) {
    # Replace NA in prep_time and cook_time where both are NA
    replace(x, is.na(df$prep_time) & is.na(df$cook_time), "unknown")
  },
  df = cleaned_recipes_df  
)

# Replace NA when either or columns have NA values
cleaned_recipes_df <- cleaned_recipes_df %>%
  mutate(
    prep_time = case_when(
      # If cook_time is present but prep_time is NA, replace with "no preparation required"
      !is.na(cook_time) & is.na(prep_time) ~ "no preparation required",
      # Otherwise, retain original value
      TRUE ~ as.character(prep_time)
    ),
    cook_time = case_when(
      # If prep_time is present but cook_time is NA, replace with "no cooking required"
      !is.na(prep_time) & is.na(cook_time) ~ "no cooking required",
      # Otherwise, retain original value
      TRUE ~ as.character(cook_time)
    )
  )

# chunk the dataset
chunk_size <- 1
n <- nrow(cleaned_recipes_df)
r <- rep(1:ceiling(n/chunk_size),each = chunk_size)[1:n]
chunks <- split(cleaned_recipes_df,r)

#empty dataframe
recipe_sentence_embeddings <-  data.frame(
  recipe = character(),
  recipe_vec_embeddings = I(list()),
  recipe_id = character()
)

# create a progress bar
pb <- txtProgressBar(min = 1, max = length(chunks), style = 3)

# embedding data
for (i in 1:length(chunks)) {
  recipe <- as.character(chunks[i])
  recipe_id <- paste0("recipe",i)
  recipe_embeddings <- textEmbed(as.character(recipe),
                                layers = 10:11,
                                aggregation_from_layers_to_tokens = "concatenate",
                                aggregation_from_tokens_to_texts = "mean",
                                keep_token_embeddings = FALSE,
                                batch_size = 1
  )

  # convert tibble to vector
  recipe_vec_embeddings <- unlist(recipe_embeddings, use.names = FALSE)
  recipe_vec_embeddings <- list(recipe_vec_embeddings)

  # Append the current chunk's data to the dataframe
  recipe_sentence_embeddings <- recipe_sentence_embeddings %>%
    add_row(
      recipe = recipe,
      recipe_vec_embeddings = recipe_vec_embeddings,
      recipe_id = recipe_id
    )

  # track embedding progress
  setTxtProgressBar(pb, i)

}

# Add documents to the collection
add_documents(
  client,
  "recipes_collection",
  documents = recipe_sentence_embeddings$recipe,
  ids = recipe_sentence_embeddings$recipe_id,
  embeddings = recipe_sentence_embeddings$recipe_vec_embeddings
)

chat.R

# Load required packages
library(ellmer)
library(text)
library(rchroma)
library(shinychat)

ui <- bslib::page_fluid(
  chat_ui("chat")
)

server <- function(input, output, session) {
  # Connect to a local ChromaDB instance running on docker with embeddings loaded 
  client <- chroma_connect()

  # sentence embeddings function and query
  question <- function(sentence){
    sentence_embeddings <- textEmbed(sentence,
                                     layers = 10:11,
                                     aggregation_from_layers_to_tokens = "concatenate",
                                     aggregation_from_tokens_to_texts = "mean",
                                     keep_token_embeddings = FALSE
    )

    # convert tibble to vector
    sentence_vec_embeddings <- unlist(sentence_embeddings, use.names = FALSE)
    sentence_vec_embeddings <- list(sentence_vec_embeddings)

    # Query similar documents
    results <- query(
      client,
      "recipes_collection",
      query_embeddings = sentence_vec_embeddings ,
      n_results = 2
    )
    results

  }


  # function that provides context
  tool_context  <- tool(
    question,
    "obtains the right context for a given question",
    sentence = type_string()

  )

  #  Initialize the chat system 
  chat <- chat_ollama(system_prompt = "You are a knowledgeable culinary assistant specializing in recipe recommendations. 
                      You provide tailored meal suggestions based on the user's available ingredients and the desired amount of food or servings.
                      Ensure the recipes align closely with the user's inputs and yield the expected quantity.",
                      model = "llama3.2:3b-instruct-q4_K_M")
  #register tool
  chat$register_tool(tool_context)

  observeEvent(input$chat_user_input, {
    stream <- chat$stream_async(input$chat_user_input)
    chat_append("chat", stream)
  })
}

shinyApp(ui, server)

You can find the complete code here.

Conclusion

Building a local Retrieval-Augmented Generation (RAG) application using Ollama and ChromaDB in R programming offers a powerful way to create a specialized conversational assistant.

By leveraging the capabilities of large language models and vector databases, you can efficiently manage and retrieve relevant information from extensive datasets.

This approach not only enhances the performance of language models but also ensures customization and privacy by running the application locally.

Whether you're developing a cooking assistant or any other domain-specific chatbot, this method provides a robust framework for delivering intelligent and contextually aware responses.

This guide will explain how to host an open source LLM on your computer. Doing this helps make sure you don’t compromise your data to third-party companies through cloud-based AI solutions.

Prerequisites

• A little AI knowledge. I’ll cover the main concepts related to what we’ll be doing in the article, but some basic knowledge about LLMs will help you understand this better. No worries if you don’t know anything though – you should still find this interesting.

• A decent computer: A system with at least 16GB of RAM, a multi-core CPU, and preferably a GPU for optimal performance. (If you have lesser specs, it may be quite slow)

• Internet connection: Required to download and install the models.

• Time and patience

What is an LLM?

LLMs, or Large Language Models, are advanced AI systems that are trained to understand and generate natural human-readable language. They use algorithms to process and understand natural language and are trained on large amounts of information to understand patterns and relationships in the data.

Companies like OpenAI, Anthropic, and Meta have created LLMs that you can use to perform tasks such as generating content, analyzing code, planning trips, and so on.

Cloud-Based AI vs. Self-Hosted AI

Before deciding to host an AI model locally, it’s important to understand how this approach differs from cloud-based solutions. Both options have their strengths and are suited to different use cases.

Cloud-Based AI Solutions

These services are hosted and maintained by providers like OpenAI, Google, or AWS. Examples include OpenAI’s GPT models, Google Bard, and AWS SageMaker. You access these models over the internet using APIs or their endpoints.

Key Characteristics:

• Easy to use: Setup is minimal – you simply integrate with an API or access through the web pages.

• Scalability: Handles large workloads and concurrent requests better since they’re managed by companies.

• Cutting-edge models: Often the latest and most powerful models are available in the cloud.

• Data dependency: Your data is sent to the cloud for processing, which may raise privacy concerns.

• Ongoing costs: Though some models are free, others are typically billed per request or usage on certain models like the more powerful or latest ones, making it an operational expense.

Self-Hosted AI

With this approach, you run the model on your own hardware. Open-source LLMs like Llama 2, GPT-J, or Mistral can be downloaded and hosted using tools like Ollama.

Key Characteristics:

• Data privacy: Your data stays on your infrastructure, giving you full control over it.

• More cost-effective over the long-term: Requires an upfront investment in hardware, but avoids recurring API fees.

• Customizability: You can fine-tune and adapt models to specific needs.

• Technical requirements: Requires powerful hardware, setup effort, and technical know-how.

• Limited scalability: Best suited for personal or small-scale use.

Which Should You Choose?

If you need quick and scalable access to advanced models and don’t mind sharing data with a third party, cloud-based AI solutions are likely the better option. On the other hand, if data security, customization, or cost savings are top priorities, hosting an LLM locally could be the way to go.

How Can You Run LLMs Locally on Your Machine?

There are various solutions out there that let you run certain open source LLMs on your own infrastructure.

While most locally-hosted solutions focus on open-source LLMs—such as Llama 2, GPT-J, or Mistral—there are cases where proprietary or licensed models can also be run locally, depending on their terms of use.

• Open-Source Models: These are freely available and can be downloaded, modified, and hosted without licensing restrictions. Examples include Llama 2 (Meta), GPT-J, and Mistral.

• Proprietary Models with Local Options: Some companies may offer downloadable versions of their models for offline use, but this often requires specific licensing or hardware. For instance, NVIDIA’s NeMo framework provides tools for hosting their models on your infrastructure, and some smaller companies may offer downloadable versions of their proprietary LLMs for enterprise customers.

Just remember that if you run your own LLM, you’ll need a powerful computer (with a good GPU and CPU). In case your computer is not very powerful, you can try running smaller and more lightweight models, though it can still be slow.

Here’s an example of a suitable system setup that I am using for this guide:

• CPU: Intel Core i7 13700HX

• RAM: 16GB DDR5

• STORAGE: 512GB SSD

• GPU: Nvidia RTX 3050 (6GB)

In this guide, you’ll be using Ollama to download and run AI models on your PC.

What is Ollama?

Ollama is a tool designed to simplify the process of running open-source large language models (LLMs) directly on your computer. It acts as a local model manager and runtime, handling everything from downloading the model files to setting up a local environment where you can interact with them.

Here’s what Ollama helps you do:

• Manage your models: Ollama provides a straightforward way to browse, download, and manage different open-source models. You can view a list of supported models on their official website.

• Deploy easily: With just a few commands, you can set up a fully functional environment to run and interact with LLMs.

• Host locally: Models run entirely on your infrastructure, ensuring that your data stays private and secure.

• Integrate different models: It includes support for integrating models into your own projects using programming languages like Python or JavaScript.

By using Ollama, you don’t need to dive deep into the complexities of setting up machine learning frameworks or managing dependencies. It simplifies the process, especially for those who want to experiment with LLMs without needing a deep technical background.

You can install Ollama very easily through the Download button in their website.

How to Use Ollama to Install/Run Your Model

After you have installed Ollama, follow these steps to install and use your model:

1. Open your browser and go to localhost:11434 to make sure Ollama is running.

2. Now, open the command prompt, and write ollama run <model_name>. Add your desired model name here which is supported by Ollama, say, Llama2 (by Meta) or Mistral.

   

3. Wait for the installation process to finish.

4. In the prompt that says >>> Send a message (/? for help), write a message to the AI and press Enter.

You have successfully installed your model and now you can chat with it!

Building a Chatbot with Your Newly Installed Model

With open source models running in your own infrastructure, you have a lot of freedom to alter and use the model any way you like. You can even use it to build local chatbots or applications for personal use by using the ollama module in Python, JavaScript, and other languages.

Now let’s walk through how you can build a chatbot with it in Python in just a few minutes.

Step 1: Install Python

If you don’t already have Python installed, download and install it from the official Python website. For best compatibility, avoid using the most recent Python version, as some modules may not yet fully support it. Instead, select the latest stable version (generally the one before the most recent release) to ensure smooth functioning of all required modules.

While setting up Python, make sure to give the installer admin privileges and check the Add to PATH checkbox.

Step 2: Install Ollama

Now, you need to open a new terminal window in the directory where the file is saved. You can open the directory in the File Explorer and right click, then click on Open in Terminal (Open with Command Prompt or Powershell if you’re using Windows 10 or a previous version).

Type pip install ollama and press Enter. This will install the ollama module for Python, so you can access your models and the functions provided by the tool from Python. Wait until the process finishes.

Step 3: Add the Python Code

Go ahead and create a Python file with the .py extension somewhere in your File System, where you can access it easily. Open the file with your favourite Code Editor, and if you have none installed, you can use the online version of VS Code from your browser.

Now, add this code in your Python File:

from ollama import chat

def stream_response(user_input):
    """Stream the response from the chat model and display it in the CLI."""
    try:
        print("\nAI: ", end="", flush=True)
        stream = chat(model='llama2', messages=[{'role': 'user', 'content': user_input}], stream=True)
        for chunk in stream:
            content = chunk['message']['content']
            print(content, end='', flush=True)
        print() 
    except Exception as e:
        print(f"\nError: {str(e)}")

def main():
    print("Welcome to your CLI AI Chatbot! Type 'exit' to quit.\n")
    while True:
        user_input = input("You: ")
        if user_input.lower() in {"exit", "quit"}:
            print("Goodbye!")
            break
        stream_response(user_input)

if __name__ == "__main__":
    main()

If you don’t understand Python code, here’s what it basically does:

• First, the chat module is imported from the ollama library, which contains pre-written code to integrate with the Ollama application on your computer.

• Then a stream_response function is declared, which passes oyur prompt to the specified model, and streams (provides the response chunk by chunk as it is generated) the live response back to you.

• Then in the main function, a Welcome text is printed to the terminal. It gets the user input which is passed to the stream_response function, all wrapped in a while True or infinite loop. This lets us ask the AI questions without the execution process breaking. We also specify that if the user input contains either exit or quit, the code will stop executing.

Step 4: Write Prompts

Now go back to the terminal window and type python filename.py, replacing filename with the actual file name that you set, and press Enter.

You should see a prompt saying You:, just like we mentioned in the code. Write your prompt and press Enter. You should see the AI Response being streamed. To stop executing, enter the prompt exit, or close the Terminal window.

You can even install the module for JavaScript or any other supported language and integrate the AI in your code. Feel free to check the Ollama Official Documentation and understand what can you code with the AI Models.

How to Customize Your Models with Fine-Tuning

What is Fine-Tuning?

Fine-tuning is the process of taking a pre-trained language model and training it further on a specific and custom dataset for a specific purpose. While LLMs are trained on massive datasets, they may not always perfectly align with your needs. Fine-tuning allows you to make the model better suited for your particular use case.

How to Fine-Tune a Model

Fine-tuning requires:

• A pre-trained model: I’d suggest starting with a powerful open-source LLM like LLaMA, Mistral, or Falcon.

• A quality dataset: A dataset is a collection of data that is used for training, testing, or evaluating machine learning models, including LLMs. The quality and relevance of the dataset directly influence how well the model performs on a given task. Use a dataset relevant to your domain or task. For example, if you want the AI to write blog posts, train it on high-quality blog content.

• Sufficient resources: Fine-tuning involves re-training the model, which requires significant computational resources (preferably a machine with a powerful GPU).

For fine tuning your model, there are several tools you can use. Unsloth is a fast option to fine-tune a model with any datasets.

What Are the Benefits of Self-hosted LLMs?

As I’ve briefly discussed above, there are various reasons to self-host an LLM. To summarize, here are some of the top benefits:

• Enhanced data privacy and security, as your data does not leave your computer, and you have complete control over it.

• Cost savings, as you do not need to pay for API subscriptions regularly. Instead, it’s a one-time-investment to get powerful-enough infrastructure to help you get going in the long run.

• Great customizability, as you get to tailor the models to your specific needs through fine-tuning or training on your own datasets.

• Lower latency

When Should You NOT Use a Self-hosted AI?

But this might not be the right fit for you for several reasons. First, you may not have the system resources required to be able to run the models – and perhaps you don’t want to or can’t upgrade.

Second, you may not have the technical knowledge or time to set up your own model and fine tune it. It’s not terribly difficult, but it does require some background knowledge and particular skills. This can also be a problem if you don’t know how to troubleshoot errors that may come up.

You also may need your models to be up 24/7, and you might not have the infrastructure to handle it.

None of these issues are insurmountable, but they may inform your decision as to whether you use a cloud-based solution or host your own model.

Conclusion

Hosting your own LLMs can be a game-changer if you value data privacy, cost-efficiency, and customization.

Tools like Ollama make it easier than ever to bring powerful AI models right to your personal infrastructure. While self-hosting isn't without its challenges, it gives you control over your data and the flexibility to adapt models to your needs.

Just make sure you assess your technical capabilities, hardware resources, and project requirements before deciding to go this way. If you need reliability, scalability, and quick access to cutting-edge features, cloud-based LLMs might still be the better fit.

If you liked this article, don’t forget to show your support, and follow me on X and LinkedIn to get connected. Also, I create short but informative tech content on YouTube, so don’t forget to check out my content.

Thanks for reading this article!

We just published a course on the freeCodeCamp.org YouTube channel that will teach you all about setting up and using Ollama to build powerful AI applications locally. This comprehensive course, created by Paulo Dichone, takes a hands-on approach to exploring Ollama, a tool designed for running LLMs efficiently on your local machine. You’ll learn everything from installing and customizing models to using REST APIs and Python libraries to build real-world AI applications. By the end of the course, you'll have the skills to develop projects like a Grocery List Organizer, a Retrieval-Augmented Generation (RAG) system, and an AI-powered Recruiter Agency.

What You’ll Learn

This course is packed with practical knowledge and real-world applications. Here's a glimpse of the topics covered:

1. Getting Started with Ollama

   • What Ollama is and why it's a game-changer for local AI development.

   • Step-by-step setup instructions for your development environment.

   • Pulling and testing different Ollama models using basic CLI commands.

2. Deep Dive into LLMs

   • Understanding parameters, benchmarks, and how to optimize models for specific use cases.

   • Customizing models using the Modelfile for tasks like summarization and sentiment analysis.

3. Ollama REST API

   • Learn how to interact with Ollama models programmatically using JSON requests.

   • Overview of integration techniques, including a Python library for building local LLM applications.

4. Hands-On Projects

   • Grocery List Organizer: Automate and optimize your grocery planning using an AI-powered assistant.

   • RAG System: Build a sophisticated Retrieval-Augmented Generation system, complete with document ingestion, vector database creation, and Streamlit integration for user-friendly interfaces.

   • AI Recruiter Agency: Develop a project that showcases how AI can streamline recruitment processes by matching candidates to job descriptions effectively.

5. Advanced Features and Tools

   • Explore multimodal models like Llava for tasks such as image captioning.

   • Use Ollama's features, such as the "show" function, for interactive model exploration.

Why Choose Ollama?

Running AI models locally provides unique advantages:

• Privacy and Security: Your data stays on your machine, offering unmatched confidentiality.

• Customization: Fine-tune models to fit your specific needs without relying on external servers.

• Cost-Effectiveness: Save on recurring API costs by working entirely offline.

With detailed lessons, engaging projects, and expert guidance from Paulo Dichone, you’ll be empowered to create AI solutions that are both innovative and impactful.

Ready to Build with Local AI?

Whether you're building practical applications or experimenting with new ideas, this course will equip you with the tools and knowledge to succeed. Watch now on the freeCodeCamp.org YouTube channel (3-hour watch).