llm - freeCodeCamp.org

How to Build Your Own MCP Server and Publish Your ChatGPT App with Supabase Auth and DigitalOcean

Abdurrahman Rajab — Thu, 09 Jul 2026 16:04:02 +0000

A new type of app is emerging with the development of LLMs and AI-native apps. It lives inside an AI chat (like ChatGPT) rather than being a fully native web or mobile app.

In this tutorial, you'll learn how to build an MCP (Model Context Protocol) server from scratch, including a UI you can use as a ChatGPT app with authentication and a database.

You'll go through the process of building, testing, adding the ChatGPT app as a connector, and submitting it to publish to the app directory. This will let you build the app on three levels:

Level one: you will build your basic MCP Server that returns textual data.
Level two: you will build a UI for your MCP Server to be used within an LLM UI.
Level three: you will add authentication and a database to your MCP Server.

To fully understand this article, you'll need to have basic knowledge of:

Web development
JavaScript
React and React Native
SQL and databases

What We'll Cover:

What is an MCP Server?
- What Can You Do with an MCP Server?
Level 1: How to Build Your Own MCP Server
How to Test Your MCP Server
Level 2: How to Build the UI
How to Test Your ChatGPT App
Level 3: How to Add Supabase (Auth and Database) to the MCP Server
How to Deploy Your MCP Server to DigitalOcean
How to Publish Your ChatGPT App
What to Do Next
Acknowledgments
References

What is an MCP Server?

A Model Context Protocol (MCP) server is a program that exposes tools, resources, and prompts to an AI application through a standard protocol. An MCP server can provide read-only context, callable tools, or reusable prompt templates that help extend what an AI application can do.

A developer builds or configures the MCP server, and an MCP client inside a host application connects to it. The application can then allow the model to discover available capabilities and, when appropriate, invoke tools or fetch resources via the MCP protocol to help complete a task.

What Can You Do with an MCP Server?

An MCP server lets an AI application work with information and systems outside the model itself. For example, it can help the model look up current information, save and retrieve user data, search documents, or trigger actions in another application.

In practice, one MCP server might connect to an online database, while another might work with files on your local machine. This makes it possible to build AI workflows that are more useful, practical, and connected to real tools.

Level 1: How to Build Your Own MCP Server

In this tutorial, you'll learn how to build an MCP server using the default HTTP server from Node.js, Supabase for the database and authentication, and the official MCP server SDK. Then you'll deploy it to DigitalOcean and publish your app on ChatGPT.

That means you'll do two steps here:

First step: connect your deployed MCP server to ChatGPT as an app/connector so it can be used within ChatGPT.
Second step: submit the app for review and, if approved, publish it to the ChatGPT app directory.

The MCP server SDK isn't the only tool or framework for building your own MCP server. You can use other SDKs and tools for that if you prefer. But to simplify the first steps, here I've decided to use the more straightforward tools.

Step 0: Prepare your project

You're going to write a full project here, so you should start by creating packages and initializing the project. To do this, follow these steps:

Create a new folder with the project name. For this example, you can use mcp_todo.
Navigate to this new folder.
Open the terminal in this folder.
Initialize the npm project with npm init --init-type=module -y to create a JavaScript package file and add the packages to the project with ES6 support.
Initialize Git with git init in the project to enable version control and track changes.
Install related packages that you're going to use in your project:
- The packages are Supabase, the MCP SDK (which we'll cover in step 2), and the zod validation package for validating LLM inputs and data.
```
npm install @modelcontextprotocol/sdk zod @supabase/supabase-js
```
Create a .gitignore file and add the node_modules to it so that it won't be tracked by Git.
Add the current state of the project to your Git tracker by writing the following:
- git add .
- git commit -m "init project"

With this, you've created a new project for yourself that you can use as a starting point for managing and following the project.

Step 1: Create a Node.js Server

To start the project, you'll need to create a simple Node.js server, which you can do by creating a new file named server.js and writing the following code:

import { createServer } from "node:http";

const port = Number(process.env.PORT ?? 8787);

const httpServer = createServer(async (req, res) => {

    console.log(`${req.method} ${req.url}`);

    if (!req.url) {

        res.writeHead(400).end("Missing URL");

        return;

    }

    const url = new URL(req.url, `http://${req.headers.host ?? "localhost"}`);

    res.writeHead(404).end("Not Found");

});

httpServer.listen(port, () => {
    console.log(`Todo MCP server listening on http://localhost:${port}, press Ctrl+C to stop`);
});

This is a simple Node server that you'll use as the base for building your MCP Server.

To build your MCP server, you'll need to set it up using the MCP Server SDK. After that, you'll need to define two things: the tools you'll show the LLM and the UI and resources the LLM will use to render.

To define the tools and UI concepts, you'll use the MCP Server SDK.

Step 2: Setting Up MCP Server SDK

To set up and start the MCP server, you need to have the following:

Tools: The functions exposed by MCP Server to an LLM, enabling the LLM to interact with the server and external systems. Like calling an API, performing a computation, or querying a database.
Resources (optional): Data the MCP Server shares with an LLM. For example, a file, database schema, or an HTML UI to use inside the LLM Chat UI as an embedded frame.

You can start the server by adding this line of code at the top of the server.js file:

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";

function createTodoServer() {
    const server = new McpServer({ name: "todo-app", version: "0.1.0" });
    return server;
}

Then add a tool and resources using the following function signature:

server.registerTool(
    "NAME",
    {},
    async (args, meta) => { }
);

You can think about the tool registrar as your endpoint to the MCP Server. The LLM will check it and, based on the name and metadata, start processing the data using the arguments and results you have in this tool.

Today, you're going to build three simple tools:

Add todo
Update todo
List todos

They all look a bit similar, but you'll see how to write them all to understand the concepts in the next sections.

Step 3: Add MCP Server Tools – Create and Add a Todo

To start with, when adding todos, you'll need a simple in-memory array to manipulate. You can create the array outside the create server function to access it throughout the server.

let todos = [];// outside the createTodoServer function block
let nextId = 1; // outside the createTodoServer function block (this is a mock id for your todos)

After the array, you'll need to have two more supporting functions: first, the validator for the tools, which specifies the expected input types from the LLM.

At the top of the file, you should import the zod library:

import { z } from "zod";

Then you can write the helper function to validate it and tell the LLM what to expect from them:

const addTodoInputSchema = {
    title: z.string().min(1),
}; // outside the createTodoServer function block

Next, you'll need the return function, which you can use with other functions to have a unified return function for the tools

const replyWithTodos = (message) => ({
    content: message ? [{ type: 'text', text: message }] : [],
    structuredContent: { tasks: todos },
}); //outside the createTodoServer function block

Then you can register the add todo function in the server, inside the createTodoServer function block, before return server:

server.registerTool(
    'add_todo',
    {
        title: 'Add todo',
        description: 'Creates a todo item with the given title.',
        inputSchema: addTodoInputSchema,
        _meta: {
            'openai/toolInvocation/invoking': 'Adding todo',
            'openai/toolInvocation/invoked': 'Added todo',
        },
    },
    async (args) => {
        const title = args?.title?.trim?.() ?? '';
        if (!title) return replyWithTodos('Missing title.');
        const todo = { id: `todo-${nextId++}`, title, completed: false };
        todos = [...todos, todo];
        return replyWithTodos(`${todo.title}`);
    },
); // inside the createTodoServer function block

In the above code, you've added the tool name and used a simple approach to add the todos to the in-memory array you already identified. The trick here is to validate the data before adding it and create the related object for it.

In the metadata, you've added the title, description, inputSchema, and _meta for OpenAI to use while rendering this. You'll get a rendering, add a todo when the AI adds it, and have the latest version of the added todo when it’s finished.

At the same time, you've added the input schema so the LLM knows what to provide when invoking your server, and you've added a reply helper function to handle your todos. It’s a simple function that shows the todos in a structured way for LLMs to understand.

Step 4: List Todos from MCP Server

To list the todos, you can use a simple list function to show the todos without any changes. In the code below, you use the same concept for naming, metadata, and description context as you provided before. You're also using the previous helper function to return the todos that you have in memory. You should write this code inside the createTodoServer function block.

server.registerTool(
  'list_todos',
  {
    title: 'List todos',
    description: 'Lists all todo items.',
    _meta: {
      'openai/toolInvocation/invoking': 'Listing todos',
      'openai/toolInvocation/invoked': 'Listed todos',
    },
  },
  async () => {
    return replyWithTodos();
  },
);

Step 5: Add Todo Complete Functions

To complete and edit todos, you can create a new tool with that name that takes the todo ID and returns the updated todos. To do this, you need to add the helper function for validating the request outside the createTodoServer:

const completeTodoInputSchema = {
    id: z.string().min(1),
};

Then inside the createTodoServer function, you can add the following:

server.registerTool(
    'complete_todo',

    {
        title: 'Complete todo',
        description: 'Marks a todo as done by id.',
        inputSchema: completeTodoInputSchema,
        _meta: {
            'openai/toolInvocation/invoking': 'Completing todo',
            'openai/toolInvocation/invoked': 'Completed todo',
        },
    },

    async (args) => {
        const id = args?.id;
        if (!id) return replyWithTodos('Missing todo id.');
        const todo = todos.find((task) => task.id === id);
        if (!todo) {
            return replyWithTodos(`Todo ${id} was not found.`);
        }
        todos = todos.map((task) =>
            task.id === id ? { ...task, completed: true } : task,
        );
        return replyWithTodos(`Completed "${todo.title}".`);
    },
);

In this tool, you used the same function definition as for list todos, while adding extra guards to check whether the LLM has returned the ID and whether that ID is correct. You should always manually check the data you have before processing it, since LLMs can hallucinate and aren't required to validate their inputs.

Now you can commit your code to Git and track it in the tree:

git add .
git commit -m "feat: add MCP todo server"

Step 6: Connect Your MCP Server with the Node.js Server

Since you have written the main functions for the MCP server, you need to connect your MCP server to the Node.js HTTP server.

To do that, you need to write the streamable function and the related code. You will use this code on top of the server code from step 1 as a replacement, since it includes more functions to handle the MCP server.

First, import the StreamableHTTPServerTransport function:

import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js";

Then you can copy the next code and replace it with the server code, which has the server's structure, to use in your project.

const port = Number(process.env.PORT ?? 8787);
const MCP_PATH = '/mcp';

const httpServer = createServer(async (req, res) => {
    if (!req.url) {
        res.writeHead(400).end('Missing URL');
        return;
    }

    const url = new URL(req.url, `http://${req.headers.host ?? 'localhost'}`);

    // handle the options call for the endpoint
    if (req.method === 'OPTIONS' && url.pathname === MCP_PATH) {
        res.writeHead(204, {
            'Access-Control-Allow-Origin': '*',
            'Access-Control-Allow-Methods': 'POST, GET, OPTIONS',
            'Access-Control-Allow-Headers': 'content-type, mcp-session-id',
            'Access-Control-Expose-Headers': 'Mcp-Session-Id',
        });
        res.end();
        return;
    }

    // handles normal get method for the main link
    if (req.method === 'GET' && url.pathname === '/') {
        res.writeHead(200, { 'content-type': 'text/plain' }).end('Todo MCP server');
        return;
    }
    // here you are handling your MCP calls with streamable HTTP
    const MCP_METHODS = new Set(['POST', 'GET', 'DELETE']);
    if (url.pathname === MCP_PATH && req.method && MCP_METHODS.has(req.method)) {
        res.setHeader('Access-Control-Allow-Origin', '*');
        res.setHeader('Access-Control-Expose-Headers', 'Mcp-Session-Id');
        const server = createTodoServer();
        const transport = new StreamableHTTPServerTransport({
            sessionIdGenerator: undefined, // stateless mode
            enableJsonResponse: true,
        });
        res.on('close', () => {
            transport.close();
            server.close();
        });
        try {
            await server.connect(transport);
            await transport.handleRequest(req, res);
        } catch (error) {
            console.error('Error handling MCP request:', error);
            if (!res.headersSent) {
                res.writeHead(500).end('Internal server error');
            }
        }
        return;
    }
    res.writeHead(404).end('Not Found');
});

httpServer.listen(port, () => {
    console.log(
        `Todo MCP server listening on http://localhost:${port}${MCP_PATH}`,
    );
});

In this code, you're running the main HTTP server to handle the requests. The server exposes a /mcp endpoint for MCP clients and connects each request to a stateless MCP server using Streamable HTTP.

Now you can commit your code to Git and track it in the tree:

git add .
git commit -m "feat: add MCP server functions"

How to Test Your MCP Server

Now you can test the basic structure of your MCP server by running the following code:

node server.js

By using this command, you'll run the server you created in the previous steps. It will make it active and listen to changes at http://localhost:8787/mcp. After running server.js, you need to open the inspector, a tool that helps you see the MCP server registration and the endpoints and tools you need to use and run in a secure environment.

npx @modelcontextprotocol/inspector@latest --server-url http://localhost:8787/mcp --transport http

When you run the previous command, you can see that you have a connection to your MCP, and you need to run it and use it through the inspector UI. Using the inspector UI will help you test your MCP server without connecting it to any external services and test the inputs and outputs locally.

To test your tools, connect to the server first, and then you can see and explore them.

After writing this code, you may wonder: what UI could I show the user through an LLM? If you run your project right now, you'll only get text results as LLM chat answers. But if you build a UI, you can improve your LLM's experience. In the next section, that's what we'll tackle.

Level 2: How to Build the UI

With the previous code, you built a simple MCP server that adds todos to a todo list and marks them as complete from the app. Now you're going to explore the registerResource tool, which registers a UI resource of your design so ChatGPT can use it.

Resources are the LLM-specific data provided by your MCP Server. You can share your UI with the LLM so it can use it to display additional data and widgets in the chat.

To share the UI, you need to have an HTML file that relies on your MCP server data and uses the MCP server. So for that, you'll create a new HTML file.

Step 1: Create the HTML File to Show the UI

The TodoHTML you provided earlier should be an HTML file that can communicate with the Server and the ChatGPT UI. The UI will look like the following image:

To build such a UI you saw previously, you need to create a public/todo-widget.html file and write the following structured code:



  
    
    Todo list

Then inside

tag, you should add the following:

      Todo list

You can see it’s just simple HTML tags that allow you to have the header, form with an input, and an unordered list with id = todo-list. But the tricky part is the JavaScript module you're going to add to it.

Step 2: Add a JavaScript Module to Handle MCP Server Data.

To add the JavaScript module and code, you'll write all the code below inside the tag.

First, you need to identify the elements by selecting the HTML tag IDs you provided to them in the HTML code:

const listEl = document.querySelector("#todo-list");
const formEl = document.querySelector("#add-form");
const inputEl = document.querySelector("#todo-input");

Then you can use these elements to extract data from the ChatGPT response using some special windows.openai code. This will allow you to receive results and responses from ChatGPT while using your MCP server.

For this case, you'll use the following:

window.openai.callTool
window.openai?.toolOutput

callTool calls the tools from your MCP server by name, and toolOutput is the result of the tools you get from your MCP.

To create the first todos and show them, you can use the toolOutput and get the output from there to use in your UI. Here's a code example:

let tasks = [...(window.openai?.toolOutput?.tasks ?? [])];

You can then loop through all tasks to add them to the list element:

const render = () => {
    listEl.innerHTML = '';

    tasks.forEach((task) => {
        const li = document.createElement('li');
        li.dataset.id = task.id;
        li.dataset.completed = String(Boolean(task.completed));
        const label = document.createElement('label');
        label.style.display = 'flex';
        label.style.alignItems = 'center';
        label.style.gap = '10px';
        const checkbox = document.createElement('input');
        checkbox.type = 'checkbox';
        checkbox.checked = Boolean(task.completed);
        const span = document.createElement('span');
        span.textContent = task.title;
        label.appendChild(checkbox);
        label.appendChild(span);
        li.appendChild(label);
        listEl.appendChild(li);
    });
};

You can call this function to loop through the tasks from the OpenAI result and print them on the screen.

You can add the update function to update tasks to be completed with the following code:

const updateFromResponse = (response) => {
    if (response?.structuredContent?.tasks) {
        tasks = response.structuredContent.tasks;
        render();
    }
};

In the code above, you received a new response from the AI and an update form via the function. This function will get the todos list from the LLM and re-render the HTML to show the todos:

const handleSetGlobals = (event) => {
    const globals = event.detail?.globals;
    if (!globals?.toolOutput?.tasks) return;
    tasks = globals.toolOutput.tasks;
    render();
};

In the next code block, you'll handle the form response in the updateFormResponse function and set event listeners to update the code when changes are detected:


window.addEventListener("openai:set_globals", handleSetGlobals, {
    passive: true,
});

const mutateTasksLocally = (name, payload) => {
    if (name === "add_todo") {
        tasks = [
            ...tasks,
            { id: crypto.randomUUID(), title: payload.title, completed: false },
        ];
    }

    if (name === "complete_todo") {
        tasks = tasks.map((task) =>
            task.id === payload.id ? { ...task, completed: true } : task
        );
    }

    if (name === "set_completed") {
        tasks = tasks.map((task) =>
            task.id === payload.id
                ? { ...task, completed: payload.completed }
                : task
        );
    }
    render();
};

const callTodoTool = async (name, payload) => {
    if (window.openai?.callTool) {
        const response = await window.openai.callTool(name, payload);
        updateFromResponse(response);
        return;
    }
    mutateTasksLocally(name, payload);
};

formEl.addEventListener("submit", async (event) => {
    event.preventDefault();
    const title = inputEl.value.trim();
    if (!title) return;
    await callTodoTool("add_todo", { title });
    inputEl.value = "";
});

listEl.addEventListener("change", async (event) => {
    const checkbox = event.target;
    if (!checkbox.matches('input[type="checkbox"]')) return;
    const id = checkbox.closest("li")?.dataset.id;
    if (!id) return;
    if (!checkbox.checked) {
        if (window.openai?.callTool) {
            checkbox.checked = true;
            return;
        }
        mutateTasksLocally("set_completed", { id, completed: false });
        return;
    }
    await callTodoTool("complete_todo", { id });
});

render();

Step 3: Styling your UI

Since you've created the HTML tags and JavaScript code for your UI, you can improve the look of it by styling it the way you like with CSS. For that, you can use the following code and add it inside the style tag in the HTML file.

 :root {
        color: #0b0b0f;
        font-family:
          "Inter",
          system-ui,
          -apple-system,
          sans-serif;
      }

      html,
      body {
        width: 100%;
        min-height: 100%;
        box-sizing: border-box;
      }

      body {
        margin: 0;
        padding: 16px;
        background: #f6f8fb;
      }

      main {
        width: 100%;
        max-width: 360px;
        min-height: 260px;
        margin: 0 auto;
        background: #fff;
        border-radius: 16px;
        padding: 20px;
        box-shadow: 0 12px 24px rgba(15, 23, 42, 0.08);
      }

      h2 {
        margin: 0 0 16px;
        font-size: 1.25rem;
      }

      form {
        display: flex;
        gap: 8px;
        margin-bottom: 16px;
      }

      form input {
        flex: 1;
        padding: 10px 12px;
        border-radius: 10px;
        border: 1px solid #cad3e0;
        font-size: 0.95rem;
      }

      form button {
        border: none;
        border-radius: 10px;
        background: #111bf5;
        color: white;
        font-weight: 600;
        padding: 0 16px;
        cursor: pointer;
      }

      input[type="checkbox"] {
        accent-color: #111bf5;
      }

      ul {
        list-style: none;
        padding: 0;
        margin: 0;
        display: flex;
        flex-direction: column;
        gap: 8px;
      }

      li {
        background: #f2f4fb;
        border-radius: 12px;
        padding: 10px 14px;
        display: flex;
        align-items: center;
        gap: 10px;
      }

      li span {
        flex: 1;
      }

      li[data-completed="true"] span {
        text-decoration: line-through;
        color: #6c768a;
      }

Now you can commit your code to Git and track it in the tree:

git add .
git commit -m "feat: add MCP server UI"

Step 4: Add the UI to your MCP Server:

Writing the HTML file isn't enough to add resources to your project. You also need to upload the HTML and resources to the MCP server and configure the server to use them using the tools you provided.

To make your MCP server aware of the UI and HTML, you need to add extra functions to the MCP server and some _meta keys to the server tools.

Here's the signature of the resources function that the MCP server will use. This signature tells the LLM what type of file to read and which resources to use when it returns the output template. You'll add this code to your server.js and your MCP server, then create your own HTML file that includes the design and UI.

registerResource(name: string, uriOrTemplate: string, config: ResourceMetadata, readCallback: ReadResourceCallback): RegisteredResource;

To use the signature function, you can use the following simple code at the top of your file, which will read the HTML file you created:

import { readFileSync } from "node:fs";

const todoHtml = readFileSync("public/todo-widget.html", "utf8");

And this resources registration code in the createTodoServer function, which will tell the LLM the type of HTML to use and where to find it.

server.registerResource(
    "todo-widget",
    "ui://widget/todo.html",
    {},
    async () => ({
        contents: [
            {
                uri: "ui://widget/todo.html",
                mimeType: "text/html+skybridge",
                text: todoHtml,
                _meta: { "openai/widgetPrefersBorder": true },
            },
        ],
    })
);

In the above code, you've added the following parameters:

The name of the resource
The sources of the resource or the template as a string

You kept the config empty to simplify the example

You only used the contents of the callback to show the information about the resources with the following details:

mimeType: the type of the file you provided. You added Skybridge, which is the OpenAI protocol that renders the HTML inside an iframe in the ChatGPT UI.
URI: a specific name of your widget
Text: Which is your HTML file
_meta: specific details for ChatGPT

Step 5: Update Your MCP Server to Handle the UI

Now that you've written the HTML pages to show a simple UI for your data and added the HTML as a resource to your MCP server, you'll add the following code to the _meta section in the MCP server tools so it can handle and render the HTML output when needed. Without this, the LLM will only return the output without returning the UI:

_meta: {
            "openai/outputTemplate": "ui://widget/todo.html",
            "openai/toolInvocation/invoking": "Listing todos",
            "openai/toolInvocation/invoked": "Listed todos",
        },

So the _meta tag in your tools functions will look like the following:

    server.registerTool(
        'list_todos',
        {
            title: 'List todos',
            description: 'Lists all todo items.',
            _meta: {
                "openai/outputTemplate": "ui://widget/todo.html",
                'openai/toolInvocation/invoking': 'Listing todos',
                'openai/toolInvocation/invoked': 'Listed todos',
            },
        },
        async () => {
            return replyWithTodos();
        },
    );

Now you can commit your code to Git and track it in the tree:

git add .
git commit -m "feat: add _meta outputTemplate tag to MCP server tools"

How to Test Your ChatGPT App

After adding the UI to your MCP server, you can run and test the project on ChatGPT by doing the following:

First, run your server normally with:

node server.js

Then run your server through ngrok to enable online access, since you need OpenAI servers to be able to access your local machine:

ngrok http 8787

Note: You need to have an ngrok account and log in to it via the CLI.

To add your resources to ChatGPT, you need to enable dev mode and add it as a connector:

Click on your profile in the ChatGPT UI
Click on Apps
Click on Advanced settings to create your own app

Then you can add your server to ChatGPT and test it thoroughly.

You'll need to write the following data in this input:

App name
Descripiton
Connection: as a server URL with your ngrok link from the terminal, with the mcp slash
No authentication, since we haven't implemented it yet

After adding the app, you can use it in the conversation by calling it with the app name by writing @app_name

Here are the examples from ChatGPT:

Here is the example of completing a step:

In the next section, you'll add authentication and a database to your project to move it to the next level.

Level 3: How to Add Supabase (Auth and Database) to the MCP Server

To add authentication and a backend, you'll need a backend/SQL server and an authentication server. The easiest current way is to use a service that can provide that. For this, you'll use Supabase.

To start, you'll create a new Supabase project for your backend. The project will include a simple table for the todos you have created in your MCP server and use it as the backend. Then you'll implement authentication.

Step 1: Create the Todos Table

To create the table, navigate through your project on Supabase and use the SQL editor to write the following code to add the todos:

-- Enable pgcrypto for gen_random_uuid
CREATE EXTENSION IF NOT EXISTS pgcrypto;

-- Create todos table
CREATE TABLE IF NOT EXISTS public.todos (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id uuid REFERENCES auth.users(id) ON DELETE CASCADE,
  title text NOT NULL,
  completed boolean NOT NULL DEFAULT false,
  created_at timestamptz NOT NULL DEFAULT now(),
  updated_at timestamptz NOT NULL DEFAULT now()
);

-- Function to keep updated_at current
CREATE OR REPLACE FUNCTION public.set_updated_at()
RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
  NEW.updated_at = now();
  RETURN NEW;
END;
$$;

-- Attach trigger
DROP TRIGGER IF EXISTS set_updated_at_trigger ON public.todos;

CREATE TRIGGER set_updated_at_trigger
BEFORE UPDATE ON public.todos
FOR EACH ROW
EXECUTE FUNCTION public.set_updated_at();

At the end, set the table to row-level security. This allows related users to see their data:

-- Enable Row Level Security
ALTER TABLE public.todos ENABLE ROW LEVEL SECURITY;

-- Users can read only their own todos
CREATE POLICY "Users can view their own todos"
ON public.todos
FOR SELECT
TO authenticated
USING (user_id = (SELECT auth.uid()));

-- Users can insert only their own todos
CREATE POLICY "Users can insert their own todos"
ON public.todos
FOR INSERT
TO authenticated
WITH CHECK ((user_id IS NOT NULL) 
AND (user_id = (SELECT auth.uid())));

-- Users can update only their own todos
CREATE POLICY "Users can update their own todos"
ON public.todos
FOR UPDATE
TO authenticated 
USING (user_id = (SELECT auth.uid())) 
WITH CHECK (user_id = (SELECT auth.uid()));

-- Users can delete only their own todos
CREATE POLICY "Users can delete their own todos"
ON public.todos
FOR DELETE
USING (auth.uid() = user_id);

-- Index for faster user-specific queries
CREATE INDEX IF NOT EXISTS idx_todos_user_id
ON public.todos(user_id);

Since your database is now ready, you can integrate authentication with your server. First, you need to authenticate the server, get the token, and use it on the server. Then you can test the app again.

To authenticate, you need to implement the following endpoints on your server (and add your own information in place of the example info):

GET: https://your-mcp.example.com/.well-known/oauth-protected-resource
OAuth 2.0 metadata: https://auth.yourcompany.com/.well-known/oauth-authorization-server
OpenID Connect metadata: https://auth.yourcompany.com/.well-known/openid-configuration

The OAuth-protected resource communicates with the server about how to use and register the tools, how to run them, and what to call them. The other two endpoints share the related metadata from the server

You'll need to implement those endpoints on your server and use them as a proxy to fetch data from Supabase, since it will be your main auth server.

Step 2: Enabling the MCP Server to Connect with Supabase Auth

For this, you need to do the following:

Enable the OAuth server at Supabase and enable the dynamic registration of tools
Implement a page for login to use for OAuth permission

To enable the OAuth server on your Supabase, you need to go to https://supabase.com/dashboard/project/_/auth/oauth-server, then follow the next steps:

Toggle Enable OAuth server
Allow dynamic apps
Create your consent page: the page that LLM tools will show users when they need to grant access to the data.

To use the consent page and see it in action, you'll need to implement the OAuth server in your MCP server first. This is what you'll do in the next section.

Step 3: Create a Proxy Server for the MCP Server to Handle the Auth.

After enabling the OAuth server in Supabase, you can start implementing the OAuth code on the MCP server. To do that, you need a proxy code on your MCP server and to create a logging endpoint to use it. The proxy server will allow your MCP server to use Supabase's OAuth server.

You'll continue by adding the next code to the MCP server you've created earlier. At the top of your code, after the imports in the server.js file, you should define the following variables:

const SUPABASE_URL = "https://YOURPORJECT.supabase.co";
const MCP_SERVER_URL = "http://localhost:8787/mcp";
const SUPABASE_AUTH_URL = `${SUPABASE_URL}/auth/v1`;

Note: Don't forget to enter your own project URL for the Supabase URL. You can find it in the Supabase UI by clicking Connect at the top of the page.

Inside the createServer function and after the if (req.method === 'OPTIONS') condition, add the following proxy code to link your Supabase project:

const OIDC_DISCOVERY_URL = `${SUPABASE_AUTH_URL}/.well-known/openid-configuration`;

if (req.method === "GET" && url.pathname === "/.well-known/openid-configuration") {
    const response = await fetch(OIDC_DISCOVERY_URL);
    const data = await response.json();
    res.writeHead(200, {
        "content-type": "application/json",
        "Access-Control-Allow-Origin": "*",
        "Access-Control-Allow-Methods": "GET, OPTIONS",
    });
    res.end(JSON.stringify(data));
    return;
}

Then you can add this code for the OAuth authorities server:

const OAUTH_DISCOVERY_URL = `${SUPABASE_URL}/.well-known/oauth-authorization-server/auth/v1`;

if (req.method === "GET" && url.pathname === "/.well-known/oauth-authorization-server") {
    const response = await fetch(OAUTH_DISCOVERY_URL);
    const data = await response.json();
    res.writeHead(200, {
        "content-type": "application/json",
        "Access-Control-Allow-Origin": "*",
        "Access-Control-Allow-Methods": "GET, OPTIONS",
    });
    res.end(JSON.stringify(data));
    return;
}

Then add this code for the well-known server:

// OPTIONS /.well-known/oauth-protected-resource/mcp
// GET /.well-known/oauth-protected-resource/mcp
if (req.method === "GET" && (url.pathname === "/.well-known/oauth-protected-resource/mcp" || url.pathname === "/.well-known/oauth-protected-resource")) {
    const metadata = {
        resource: MCP_SERVER_URL,
        authorization_servers: [SUPABASE_AUTH_URL],
        // Use standard OIDC scopes. Custom resource scopes are enforced server-side, not by Supabase.
        scopes_supported: ["openid", "profile", "email", "phone"],
    };
    res.writeHead(200, {
        "content-type": "application/json",
        "Access-Control-Allow-Origin": "*",
        "Access-Control-Allow-Methods": "GET, OPTIONS",
        "Access-Control-Allow-Headers": "content-type, MCP-Protocol-Version, mcp-protocol-version, authorization",
    });
    res.end(JSON.stringify(metadata));
    return;
}

By adding the previous code snippets, you've implemented a proxy server that fetches data from Supabase and relays it to the MCP protocol as if it were your own server.

Now you can commit your code to Git and track it in the tree:

git add .
git commit -m "feat: add supabase proxy server"

Since you've implemented your proxy server, you can use authentication and authorization from your MCP server to retrieve data in your tools.

On the OAuth server, you might have noticed a consent page. The goal of this page is to inform the user that they are authorizing the LLM to connect to a database or an external resource. In the next section, you will implement this page by making two steps:

First, create a login page that lets users log in to the app.
Second, you will create a consent page that allows the logged-in user to communicate with the LLM

You'll start by creating a new Next.js server, which gives you more flexibility when working with pages.

You can create your NextJS app with the command:

npx create-next-app@latest mcp_consent --yes

Navigate to the mcp_consent folder and add Supabase:

npm install @supabase/ssr

Add the current state of the project to your Git tracker by writing the following:

git add .
git commit -m "init nextjs project"

Add .env file from your Supabase, which will include the following code:

NEXT_PUBLIC_SUPABASE_URL=YOUR_URL
NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY=YOUR_KEY

Now you can create a login page in the next path:

app/login/page.tsx

The login page:

"use client";
import { useState } from "react";
import { useRouter, useSearchParams } from "next/navigation";
import { createBrowserClient } from "@supabase/ssr/dist/module/createBrowserClient";


export default function LoginPage() {
    const [email, setEmail] = useState("");
    const [password, setPassword] = useState("");
    const [loading, setLoading] = useState(false);
    const [error, setError] = useState(null);
    const router = useRouter();
    const searchParams = useSearchParams();
    const supabase = createBrowserClient(
        process.env.NEXT_PUBLIC_SUPABASE_URL!,
        process.env.NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY!
    );

    const handleLogin = async (e: React.FormEvent) => {
        e.preventDefault();
        setLoading(true);
        setError(null);


        try {
            const { error } = await supabase.auth.signInWithPassword({
                email,
                password,
            });


            if (error) {
                setError(error.message);
            } else {
                const redirectTo = searchParams.get("redirect") || "/";
                router.push(redirectTo);
                router.refresh();
            }
        } catch (err) {
            setError("An unexpected error occurred");
        } finally {
            setLoading(false);
        }
    };


    const handleSignUp = async (e: React.FormEvent) => {
        e.preventDefault();
        setLoading(true);
        setError(null);


        try {
            const { error } = await supabase.auth.signUp({
                email,
                password,
            });


            if (error) {
                setError(error.message);
            } else {
                setError(null);
                alert("Sign up successful! Please check your email to confirm your account.");
            }
        } catch (err) {
            setError("An unexpected error occurred");
        } finally {
            setLoading(false);
        }
    };


    return (
        
            
                
                    Authentication
                


                {error && (
                    
                        {error}
                    
                )}


                
                    
                        
                            Email address
                        
                         setEmail(e.target.value)}
                            className="mt-1 block w-full px-3 py-2 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-blue-500 focus:border-blue-500 text-black"
                            placeholder="you@example.com"
                        />
                    


                    
                        
                            Password
                        
                         setPassword(e.target.value)}
                            className="mt-1 block w-full px-3 py-2 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-blue-500 focus:border-blue-500 text-black"
                            placeholder="••••••••"
                        />
                    


                    
                        
                        
                    
                


                
                    
                        Password reset or other options available upon request.
                    
                
            
        
    );
}

The OAuth decision page:

// app/api/oauth/decision/route.ts
import { createServerClient } from '@supabase/ssr'
import { cookies } from 'next/headers'
import { NextResponse } from 'next/server'
export async function POST(request: Request) {
    const formData = await request.formData()
    const decision = formData.get('decision')
    const authorizationId = formData.get('authorization_id') as string
    if (!authorizationId) {
        return NextResponse.json({ error: 'Missing authorization_id' }, { status: 400 })
    }
    const supabase = createServerClient(
        process.env.NEXT_PUBLIC_SUPABASE_URL!,
        process.env.NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY!,
        {
            cookies: {
                getAll: async () => (await cookies()).getAll(),
                setAll: async (cookiesToSet) => {
                    const cookieStore = await cookies()
                    cookiesToSet.forEach(({ name, value, options }) => cookieStore.set(name, value, options))
                },
            },
        }
    )
    if (decision === 'approve') {
        const { data, error } = await supabase.auth.oauth.approveAuthorization(authorizationId)
        if (error) {
            return NextResponse.json({ error: error.message }, { status: 400 })
        }
        // Redirect back to the client with authorization code
        return NextResponse.redirect(data.redirect_url)
    } else {
        const { data, error } = await supabase.auth.oauth.denyAuthorization(authorizationId)
        if (error) {
            return NextResponse.json({ error: error.message }, { status: 400 })
        }
        // Redirect back to the client with error
        return NextResponse.redirect(data.redirect_url)
    }
}

The OAuth Consent page:

// app/oauth/consent/page.tsx
import { createServerClient } from '@supabase/ssr'
import { cookies } from 'next/headers'
import { redirect } from 'next/navigation'

export default async function ConsentPage({
    searchParams,
}: {
    searchParams: { authorization_id?: string }
}) {
    const authorizationId = (await searchParams).authorization_id

    if (!authorizationId) {
        return Error: Missing authorization_id
    }

    const supabase = createServerClient(
        process.env.NEXT_PUBLIC_SUPABASE_URL!,
        process.env.NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY!,
        {
            cookies: {
                getAll: async () => (await cookies()).getAll(),
                setAll: async (cookiesToSet) => {
                    try {
                        const cookieStore = await cookies()
                        cookiesToSet.forEach(({ name, value, options }) =>
                            cookieStore.set(name, value, options)
                        )
                    } catch (error) {
                        // In Server Components, cookie writes can fail during render.
                        // Route Handlers/Server Actions should handle persistence.
                        console.warn('Skipping cookie write in Server Component render context', error)
                    }
                },
            },
        }
    )

    // Check if user is authenticated
    const {
        data: { user },
    } = await supabase.auth.getUser()

    if (!user) {
        // Redirect to login, preserving authorization_id
        redirect(`/login?redirect=/oauth/consent?authorization_id=${authorizationId}`)
    }

    // Get authorization details using the authorization_id
    const { data: authDetails, error } =
        await supabase.auth.oauth.getAuthorizationDetails(authorizationId)
    console.log("Auth Details: ", authDetails)
    if (error || !authDetails) {
        return Error: {error?.message || 'Invalid authorization request'}
    }
    if ("redirect_url" in authDetails && authDetails.redirect_url && typeof authDetails.redirect_url === "string") {
        const redirectUrl = authDetails.redirect_url;
        console.log("Redirect URL:", redirectUrl);
        return redirect(redirectUrl);
    }
    if (!("client" in authDetails)) {
        return Error: Invalid authorization details format
    }
    return (
        
            {/* Animated gradient background */}
            
                
                
                
            

            {/* Main Card Container */}
            
                {/* Gradient border effect */}
                
                    

                    {/* Content Card */}
                    
                        {/* Header Section */}
                        
                            
                                
                                    
                                
                            
                            Authorization Required
                            Review and authorize access to your account
                        

                        {/* Client Information */}
                        
                            
                                
                                
                                    Application
                                    {authDetails.client.name}
                                
                            

                            
                                
                                
                                    Redirect URI
                                    {authDetails.redirect_uri}
                                
                            
                        

                        {/* Permissions Section */}
                        {authDetails.scope && authDetails.scope.length > 0 && (
                            
                                Requested Permissions
                                
                                    {authDetails.scope.split(" ").map((scope, index) => (
                                        
                                            
                                            {scope}
                                        
                                    ))}
                                
                            
                        )}

                        {/* Action Buttons */}
                        
                            

                            

                            
                        

                        {/* Security Info */}
                        
                            
                            Your data is protected with industry-standard encryption
                        
                    
                
            
        
    )
}

Now you can add the current state of the project to your Git tracker by writing the following:

git add .
git commit -m "feat: add consent page"

Step 5: Testing the OAuth Implementation with MCP Server Inspector

Since you've implemented the consent page, now you can test it and check the authorization in the MCP Server Inspector. This step will help you see how OAuth works and how to test it with the inspector.

First, create a new user in Supabase for login and authentication.

Go to: https://supabase.com/dashboard/project/_/auth/users
(Auth -> users from the UI)
Click Add User -> Create a new user, then add the new user email and password.

After creating the user, you can run the projects by typing the following in different terminals:

Run your MCP server:

node server.js

Open your inspector:

npx @modelcontextprotocol/inspector@latest --server-url http://localhost:8787/mcp --transport http

Run the Next.js project to get access to the consent page:

cd mcp_consent
npm run dev

When you run the inspector, you can connect to your MCP Server on the left and navigate to Auth on the top tab. This will show you the authentication flow to test and run.

When you click Connect, go to Auth to check your options. The guided OAuth flow will show you a step-by-step guide to how the MCP Server obtains OAuth authorization and will help you debug your code if issues arise. The Check OAuth Flow button lets you connect directly and see the latest result immediately.

For the sake of speed, you can just click on the "Check OAuth Flow". This will redirect you to the login page:

After you log in, you'll get redirected again to the consent page so that you can give consent to the LLM to access your data:

Then you'll be redirected again to the MCP Server and you can check the results of the OAuth flow:

In the next step, you'll harden your MCP Server functions to take your app to the next level by using OAuth for MCP Server.

Step 6: Adding OAuth Security to Your MCP Server Tools

Congrats on implementing your OAuth flow and getting it to work! Now you'll add this flow to your MCP Server tools so it runs only when the user is authenticated.

Before updating the tools, you'll write a few helper functions to assist you during the process. First, you'll write a verification token to process every request. Then you'll update the list of MCP server tools to use the verification function instead of implementing it for each function by itself.

Inside your server.js file, you'll implement a function that verifies the token with Supabase. First, import the Supabase client to use it:

import { SupabaseClient } from "@supabase/supabase-js";

Then add the Supabase publishable key to use it in the client at the top of the server:

const SUPABASE_PUBLISHABLE_KEY = "YOUR_KEY";

And update the reply todos list to get an argument of todos, instead of the in-memory array.

const replyWithTodos = (message, todos) => ({
    content: message ? [{ type: 'text', text: message }] : [],
    structuredContent: { tasks: todos },
}); //outside the createTodoServer function block

Then you'll need to create a helper function to verify the user tokens:

const verifyToken = async (token) => {
    if (!token || !token.startsWith("Bearer ")) {
        return { isValid: false, error: "Missing or invalid Authorization header" };
    }
    // Verify token with Supabase
    try {
        // use supabase client to verify token
        const supabase = new SupabaseClient(SUPABASE_URL, SUPABASE_PUBLISHABLE_KEY, {
            global: {
                headers: {
                    Authorization: token,
                },
            },
        });
        const { data: user, error } = await supabase.auth.getUser();
        if (error || !user) {
            return { isValid: false, error: "Token verification failed" + (error?.message || "") };
        }
        console.log("Token verified for user:", user);
        return { isValid: true, token, user, supabase };
    } catch (error) {
        console.error("Token verification failed:", error);
        return { isValid: false, error: "Token verification failed" + (error?.message || "") };
    }
};

In this function, you get the token as a string, check Supabase, and return an error if the token isn't provided. If it's correct, you return the token, user data, and Supabase client.

After this, you need to have a helper function to adhere to MCP Server specs:

/**
 * Build WWW-Authenticate header for 401/403 responses
 * Per RFC 9728 OAuth 2.1 Protected Resource Metadata specification
 */
function buildWwwAuthenticateHeader(error, errorDescription) {
    const resourceMetadataUrl = `${MCP_SERVER_URL}/.well-known/oauth-protected-resource`

    let header = `Bearer resource_metadata="${resourceMetadataUrl}"`

    if (error) {
        header += `, error="${error}"`
    }

    if (errorDescription) {
        header += `, error_description="${errorDescription}"`
    }

    return header
}

function returnAuthErrorResponse(resOrMessage, error = "unauthorized", errorDescription = "Missing or invalid authorization token.") {
    const wwwAuthenticate = buildWwwAuthenticateHeader(error, errorDescription);

    if (resOrMessage && typeof resOrMessage.writeHead === "function") {
        resOrMessage.writeHead(401, {
            "content-type": "application/json",
            "Access-Control-Allow-Origin": "*",
            "WWW-Authenticate": wwwAuthenticate,
        });
        resOrMessage.end(JSON.stringify({ error, error_description: errorDescription }));
        return;
    }

    const message = typeof resOrMessage === "string" && resOrMessage.length > 0
        ? resOrMessage
        : errorDescription;

    return {
        content: [{ type: "text", text: message }],
        isError: true,
        statusCode: 401,
        _meta: {
            "mcp/www_authenticate": wwwAuthenticate,
        },
    };
}


function returnErrorResponse(message) {
    return {
        content: [
            {
                type: "text",
                text: message
            }
        ],
        isError: true
    };
}

In these helper functions, you create unique functions for errors and the OAuth error return function. At the same time, you define the OAuth-protected resources discovery specs.

Add the current state of the project to your Git tracker by writing the following:

git add .
git commit -m "feat: add helper functions"

Now you can apply them to your MCP Server tools, making them easier to read.

Step 7: Updating the MCP Server Function to Handle the Authentication

After you've built the proxy to handle authentication requests, you need to update the MCP server functions and metadata to indicate whether the tool can be used with or without authentication.

Here you'll add two main things:

The security schema.
The logic for the function to handle.

   server.registerTool(
        "list_todos",
        {
            title: "List todos",
            description: "Lists all todo items.",
            _meta: {
                "openai/outputTemplate": "ui://widget/todo.html",
                "openai/toolInvocation/invoking": "Listing todos",
                "openai/toolInvocation/invoked": "Listed todos",
            },
            securitySchemes: [
                { type: "oauth2", scopes: ["todos.read"] }
            ],
            "annotations": {
                "readOnlyHint": true,
                "openWorldHint": false,
                "destructiveHint": false,
            }
        },
        async (meta) => {
            const authHeader = meta.requestInfo.headers?.authorization;
            const authResult = await verifyToken(authHeader);
            if (!authResult?.isValid) {
                return returnAuthErrorResponse(authResult?.error);
            }
            const { data, error } = await authResult.supabase
                .from("todos")
                .select("*")
                .eq("user_id", authResult.user.user.id)
                .order("created_at", { ascending: false });
            if (error) {
                console.error("Error listing todos:", error);
                return returnErrorResponse(error.message);
            }
            return replyWithTodos(null, data ?? []);
        }
    )

In this code, you've done the following:

Updated the metadata to have a security schema that tells the MCP server to request authentication when invoking these tools.
Added an annotation, which helps the LLM model know how this function will perform. The annotations declare three types of changes that the tool can make:
- Read Only Hint: tells the LLM whether the tool is read-only and only shows data
- Open World Hint: tells the LLM whether the tool can access external data, websites, or the internet.
- Destructive Hint: tells the LLM if this is a destructive function, like deleting data permanently for the user
Then, in the function itself, you retrieved the metadata from the callback and verified the token using the Supabase helper function. After that, you used the basic Supabase functions to retrieve the data and any errors that might occur.

You can get the authorization from the metadata in the callback function itself.

In the previous function, you used the Supabase client to access the todos table, select all columns where the user_id condition is met, and order them by creation time. If the Supabase client returns an error, you return an error. Here's the code snippet that relates to Supabase:

const { data, error } = await authResult.supabase
  .from("todos")
  .select("*")
  .eq("user_id", authResult.user.user.id)
  .order("created_at", { ascending: false });
if (error) {
  console.error("Error listing todos:", error);
  return returnErrorResponse(error.message);
}

For the other function, you have the same logic applied, yet instead of select, you'll use either insert or update.

For the other functions, you can follow the same principles and use the same code. The only change is that first you get the data, then the metadata in the callback function:

    server.registerTool(
        "add_todo",
        {
            title: "Add todo",
            description: "Creates a todo item with the given title.",
            inputSchema: addTodoInputSchema,
            _meta: {
                "openai/outputTemplate": "ui://widget/todo.html",
                "openai/toolInvocation/invoking": "Adding todo",
                "openai/toolInvocation/invoked": "Added todo",
            },
            securitySchemes: [
                { type: "oauth2", scopes: ["todos.write"] }
            ],
            "annotations": {
                "readOnlyHint": false,
                "openWorldHint": false,
                "destructiveHint": true,
            }
        },
        async (args, meta) => {
            const authorizationHeader = meta.requestInfo.headers?.authorization;
            console.log("Authorization header:", authorizationHeader);
            const authResult = await verifyToken(authorizationHeader);
            console.log("Auth result:", authResult);
            if (!authResult?.isValid) {
                return returnAuthErrorResponse(authResult?.error);
            }
            const title = args?.title?.trim?.() ?? "";
            if (!title) return returnErrorResponse("Missing title.");
            let { data, error } = await authResult.supabase
                .from("todos")
                .insert({ title, user_id: authResult.user.user.id })
                .select("*");
            if (error) {
                console.error("Error adding todo:", error);
                return returnErrorResponse(error.message);
            }
            return replyWithTodos(`"${title}"`, data);
        }
    );

Here's the updated function code:

    server.registerTool(
        "complete_todo",
        {
            title: "Complete todo",
            description: "Marks a todo as done by id.",
            inputSchema: completeTodoInputSchema,
            _meta: {
                "openai/outputTemplate": "ui://widget/todo.html",
                "openai/toolInvocation/invoking": "Completing todo",
                "openai/toolInvocation/invoked": "Completed todo",
            },
            securitySchemes: [
                { type: "oauth2", scopes: ["todos.write"] }
            ],
            "annotations": {
                "readOnlyHint": false,
                "openWorldHint": false,
                "destructiveHint": true,
            }
        },
        async (args, meta) => {
            const authorizationHeader = meta.requestInfo.headers?.authorization;
            const authResult = await verifyToken(authorizationHeader);
            if (!authResult?.isValid) {
                return returnAuthErrorResponse(authResult?.error);
            }
            const id = args?.id;
            if (!id) return replyWithTodos("Missing todo id.");
            const { data, error } = await authResult.supabase
                .from("todos")
                .update({ completed: true })
                .eq("id", id)
                .eq("user_id", authResult.user.user.id)
                .select("*");
            if (error) {
                console.error("Error completing todo:", error);
                return returnErrorResponse(error.message);
            }
            if (!data || data.length === 0) {
                return replyWithTodos(`Todo ${id} was not found.`);
            }
            return replyWithTodos(`Completed "${data[0].title}".`, data);
        }
    );

By applying this, you already have a fully functioning MCP server connected to your Supabase, and you can rely on it to run.

Add the current state of the project to your Git tracker by writing the following:

git add .
git commit -m "feat: update tools to use database"

Step 8: Testing the Server with Supabase:

To do this, you can follow the same steps as in "Testing the OAuth implementation with MCP Server Inspector." But as an extra point, keep an eye on your database table in the Supabase UI, where you can see the added and updated todos. Then you can check the tools, test your todos, and even use ngrok to test them in the ChatGPT UI.

How to Deploy your MCP Server to DigitalOcean

Since you have your MCP server running and working well, you can now deploy it to DigitalOcean using their App service.

First, upload your code to GitHub and commit it with the following command:

gh repo create todo_mcp_server --private --source=. --remote=upstream

git push

This command creates a new repo on GitHub, sets your stream to GitHub, and pushes the current branches to GitHub.

Then log in to your DigitalOcean account and go to Apps (https://cloud.digitalocean.com/apps).

Click on Create app:

Choose the source as GitHub:

Then you need to select your repository and your branch. Write the source directories as: / and mcp_consent. You're doing this because you'll be running two apps: the MCP Server and the consent and login page frontend.

Next, enable auto-deploy if you want the app to update whenever you push your code to GitHub:

Since we've created two source directories, you'll have two apps and will have to manage them separately.

You'll use the MCP server in the first app and the frontend for the second app. For that reason, you'll update the network to have the server under the /server route:

And you'll downsize the CPU to minimize the cost for this demo:

You can update the size later based on your needs for the app.

As for the last step here, you'll update the run command to node server.js to ensure the app is running correctly.

For the frontend project, you'll have to click on the second app, update the inputs as well, and add the environment variables:

First, you can downsize the app:

Update the build and run commands for the Next js server:

Then you can check the route of this web app and set it as the main one:

At the end, you need to add the .env variables from your .env file to the project. You can copy and paste them directly from your mcp_const/.env file to the project.

After setting them up, you can create and run the app, which will generate a public URL from DigitalOcean that you can use in the ChatGPT UI again to test it and run the project.

Before adding the project to test it in ChatGPT, you need to update the consent page URL in Supabase from here.

Instead of havinglocalhost:300, you can add the link from your DigitalOcean account.

At this point, you can test the server with your DigitalOcean by using the following links:

YOUR_DIGITAL_OCEAN.com/server/mcp
YOUR_DIGITAL_OCEAN.com/login
YOUR_DIGITAL_OCEAN.com/oauth/consent

How to Publish Your ChatGPT App

After running your app, you need to host it. You can simply upload it to GitHub and host it on DigitalOcean as a JavaScript app. You can get the URL from DigitalOcean, then go to your OpenAI dashboard, verify yourself as a company or a solo developer, and upload the file there.

To publish your app to ChatGPT, you'll need to provide the following information:

App Info: the basic information about your app, including the logo, description, a video demo, website, support, privacy policy, and terms of service URLs (plus a few more details about monetizing your app if you have done that).
MCP Server: the links to your MCP server, the tools you have, and how you'll use them, plus a verification token for your URL that you'll need to add to your project as a path.
Testing: you'll need to provide at least 5 test cases for your MCP server so OpenAI can test its functionality. They require you to have coverage over all the major use cases that you intend to support and include all information required to successfully run the test case.

In the tests you share:
- Scenario: where you describe the use case to test (for example, “Research flights”, “Create a slideshow”, “Find a hiking trail”).
- User prompt: The exact prompt or interaction you should conduct to begin the test.
- Tool triggered: Which tools should be called? You have already implemented them.
- Expected output: The output or experience you should expect to receive back from the MCP server.
- Then you share the negative cases with the same examples.
Screenshots: App screenshots for the directory. You can use this public Figma to help you with your design. Here, you should upload 1–4 screenshots of your app widget UI in PNG or JPG format, each with a width of 706px and a height of 400–860px (at least one must be 2× retina quality). The first three screenshots are publicly visible in install views across all screen sizes and locales. Ensure the images show only your widget UI – no ChatGPT interface, user prompts, model responses, or embedded text.
Global: shows the text and localization for your app. You can also select specific countries to publish to.
Submit: This requires you to write the release notes and run a few compliance checks on your app.

After uploading and updating your project, you can wait for OpenAI to review your project and get the results from them.

What to Do Next

You now have the basic knowledge you need to explore MCP servers and ChatGPT apps. You can dive deeper by reading the documentation and checking the related tools and platforms for building apps like Skybridge.

If you liked this tutorial, you can follow me on Twitter or YouTube and run the full project demo script on GitHub.

Acknowledgments:

Thanks to Ahmed Saleh for supporting me with the ChatGPT Apps concept, Abbey from freeCodeCamp for her patience during the editorial process, and the Supabase and OpenAI teams for their awesome work and documentation!

References:

This blog would not have been written without the hard work of the OpenAI, Supabase, and DigitalOcean teams that they put into the following documentation:

AI Paper Review: Self-Consistency Improves Chain of Thought Reasoning in Language Models

Mohammed Fahd Abrah — Wed, 08 Jul 2026 18:55:53 +0000

When Chain-of-Thought Prompting was introduced, it showed that large language models could solve many difficult reasoning problems simply by thinking step by step before producing an answer.

It was a remarkable breakthrough, but it also exposed an important limitation: What happens if the model's reasoning is wrong?

Even with Chain-of-Thought, a model follows only a single reasoning path. If that path contains a mistake, the final answer is likely to be wrong as well. Better reasoning still depends on getting the first attempt right.

This paper tackles that limitation with an idea inspired by how people solve difficult problems. Rather than trusting the first solution that comes to mind, we often consider several different approaches before deciding which answer is most convincing. The authors asked whether language models could do the same.

Their answer is Self-Consistency: a simple decoding strategy that generates multiple independent reasoning paths and selects the answer that appears most consistently among them. The model itself remains unchanged. There is no additional training, fine-tuning, or supervision. Only the decoding strategy changes.

Despite its simplicity, the approach produced remarkable improvements across arithmetic, common sense, and symbolic reasoning tasks, showing that more reliable reasoning often comes from comparing multiple lines of thought rather than committing to the first one.

This paper became a natural successor to Chain-of-Thought prompting and marked an important shift in LLM research. Instead of making models larger, it showed that substantial gains could come from making better use of the reasoning abilities they already possessed.

Paper Overview

In this review, we'll explore Self-Consistency Improves Chain of Thought Reasoning in Language Models, published by researchers at Google Research and presented at ICLR 2023.

We'll begin by examining the limitations of Chain-of-Thought prompting that motivated this work, then walk through the intuition behind Self-Consistency, how the decoding algorithm works, and why generating multiple reasoning paths leads to more reliable answers.

Next, we'll analyze the experimental results across arithmetic, common sense, and symbolic reasoning benchmarks, compare Self-Consistency with alternative decoding methods such as beam search and sample-and-rank, and discuss its strengths, limitations, and computational trade-offs.

Finally, we'll examine the paper's long-term impact on language model research and how its central idea influenced later work on test-time reasoning, verification, search-based inference, and modern reasoning-oriented language models.

If you'd like to follow along, you can also read the original paper:
Self-Consistency Improves Chain of Thought Reasoning in Language Models.

And here's a quick infographic of what we'll cover throughout this review.

Prerequisites

To get the most out of this review, it helps to be familiar with the evolution of large language models and the reasoning techniques that led to Self-Consistency.

This paper builds directly on the ideas introduced by Chain-of-Thought Prompting, so reading the earlier reviews in this series will provide valuable context.

The previous reviews are especially recommended:

Among these, the Chain-of-Thought review is the most important prerequisite. It introduced the idea that language models could dramatically improve their reasoning by generating intermediate reasoning steps before producing an answer.

Self-Consistency builds directly on that breakthrough. Instead of trusting a single chain of thought, it explores multiple independent reasoning paths and selects the answer that appears most consistently across them, showing that better reasoning can emerge from a smarter decoding strategy rather than a larger or better-trained model.

It also helps to have:

A general understanding of natural language processing (NLP) and large language models
A basic understanding of Transformer-based autoregressive models
Familiarity with prompting, few-shot learning, in-context learning, and Chain-of-Thought prompting
A high-level understanding of how language models generate text token by token
General machine learning concepts such as training, inference, scaling laws, and model evaluation
Some exposure to reasoning tasks, logic problems, and mathematical word problems
A basic understanding of benchmark datasets and how model performance is evaluated

You don't need a deep background in mathematics or machine learning research to follow this article.

I'll keep the explanations intuitive and practical, focusing on why Self-Consistency became one of the most influential inference-time reasoning techniques in modern AI, how it extended the ideas introduced by Chain-of-Thought prompting, and why a simple change in decoding fundamentally changed how researchers think about reasoning in large language models.

Abstract

The original Chain-of-Thought paper showed that large language models become much better reasoners when they generate intermediate reasoning steps before producing an answer. But it still relied on a simple assumption: the model followed a single reasoning path and trusted its first solution.

This paper asks a natural follow-up question: what if that first reasoning path is wrong?

To answer it, the authors introduce Self-Consistency, a simple decoding strategy inspired by how people often solve difficult problems. Instead of committing to the first chain of thought, the model generates multiple independent reasoning paths and selects the answer that appears most consistently among them.

The model itself remains unchanged. There's no additional training, fine-tuning, or supervision. Only the decoding process is different.

The central insight is that difficult problems rarely have just one valid route to the correct answer. Different reasoning processes may approach a problem in different ways, yet still arrive at the same conclusion. By comparing these independent solutions rather than relying on a single one, the model becomes more robust to reasoning mistakes.

Although the idea is surprisingly simple, its impact is substantial. Self-Consistency significantly improves Chain-of-Thought prompting across arithmetic, common sense, and symbolic reasoning tasks, setting new state-of-the-art results on several popular benchmarks, including GSM8K, SVAMP, AQuA, StrategyQA, and ARC-Challenge.

More importantly, it demonstrated that improving reasoning doesn't always require larger models or additional training. Sometimes, a better way of exploring a model's existing reasoning abilities is enough to produce dramatically better results.

Introduction

When Chain-of-Thought Prompting was introduced in 2022, it changed the conversation around reasoning in large language models. By encouraging models to generate intermediate reasoning steps, researchers discovered that many tasks once considered difficult could suddenly be solved much more effectively.

Yet an important limitation remained: even with Chain-of-Thought, a model still committed to a single reasoning path. If that reasoning contained a mistake, the final answer was likely to be wrong.

The infographic below illustrates the standard Chain-of-Thought reasoning pipeline, showing how a language model follows a single reasoning path using greedy decoding to produce one final answer.

This paper begins with a simple observation: complex problems often have more than one valid route to the correct answer.

People rarely rely on a single line of reasoning when solving difficult problems. Instead, they explore different possibilities and gain confidence when independent approaches lead to the same conclusion. The authors ask whether language models could benefit from the same strategy.

To explore this idea, they introduce Self-Consistency, a decoding strategy that builds directly on Chain-of-Thought prompting. Instead of accepting the first reasoning path the model generates, Self-Consistency samples multiple independent reasoning paths and selects the answer that appears most consistently across them.

The goal is no longer to find a single plausible explanation, but to identify the answer that remains consistent across diverse explanations.

One of the paper's most appealing aspects is its simplicity. Unlike approaches that require additional verifiers, re-ranking models, or extra training, Self-Consistency works entirely at inference time. It requires no new annotations, no fine-tuning, and no auxiliary models. Rather than changing the model itself, it changes only how the model's reasoning is decoded.

The authors evaluate the method across a wide range of arithmetic, common sense, and symbolic reasoning benchmarks using models from UL2 and GPT-3 to LaMDA and PaLM. Across nearly every task, Self-Consistency delivers substantial improvements over standard Chain-of-Thought prompting.

Beyond the impressive benchmark results, the paper introduced a lasting idea: stronger reasoning doesn't always require larger models or more training. Sometimes, the biggest gains come from allowing a model to explore multiple solutions before deciding on the most reliable answer.

Self-Consistency over Diverse Reasoning Paths

The central idea behind this paper begins with a simple observation about human reasoning. When solving difficult problems, people rarely rely on a single line of thought. They often consider multiple possibilities before reaching a conclusion, and although those reasoning processes may differ, they frequently converge on the same answer. The authors argue that language models can benefit from the same principle.

Chain-of-Thought prompting had already shown that generating intermediate reasoning steps could significantly improve performance on complex tasks. But it still relied on greedy decoding, which commits the model to a single reasoning path. If that path contains a mistake, the final answer is likely to be wrong, even if the model could have reached the correct answer through a different line of reasoning.

Self-Consistency replaces this "one path, one answer" strategy with a simple alternative. After receiving a Chain-of-Thought prompt, the model samples multiple reasoning paths instead of selecting only the most likely one. Some paths may contain mistakes, while others may arrive at the correct solution through different reasoning processes.

Rather than evaluating the reasoning itself, the method aggregates the final answers and selects the one that appears most consistently across the generated solutions.

The infographic below compares standard Chain-of-Thought prompting with Self-Consistency, highlighting how replacing a single reasoning path with multiple independent reasoning paths leads to more reliable answers.

The intuition is straightforward. Incorrect reasoning paths tend to make different mistakes and therefore produce different answers. But correct reasoning paths often converge on the same conclusion even when their intermediate steps differ.

By looking for agreement among independent reasoning attempts, the model becomes far less dependent on the success of any single generation.

An elegant aspect of the method is that nothing about the model itself changes. Self-Consistency works entirely at inference time, requiring no additional training, fine-tuning, or auxiliary models. In effect, it behaves like a self-ensemble: instead of combining multiple models, it combines multiple reasoning attempts from the same model to produce a more reliable prediction.

The authors also compare several ways of combining the generated answers. We might expect probability-weighted methods to outperform simpler approaches, but the experiments reveal the opposite. A straightforward majority vote over the final answers performs almost as well as more sophisticated weighting schemes, suggesting that the biggest advantage comes from exploring diverse reasoning paths rather than assigning them complex scores.

This section marks an important shift in how reasoning is viewed. Traditional decoding assumes the most likely reasoning path is also the best one. Self-Consistency shows that, for reasoning tasks, diversity can be just as valuable as confidence. Exploring multiple independent solutions before choosing an answer leads to reasoning that is consistently more robust and reliable.

Experiments

After introducing Self-Consistency, the authors turned to a key question: does this simple decoding strategy actually improve reasoning in practice?

To answer it, they conducted an extensive evaluation across arithmetic, common sense, and symbolic reasoning tasks, testing whether the benefits of Self-Consistency held across different problem types, model architectures, and model sizes.

Rather than relying on a single benchmark, the evaluation spanned a diverse collection of reasoning tasks. Arithmetic datasets measured the ability to solve multi-step math word problems, common sense benchmarks tested reasoning about everyday knowledge, and symbolic tasks evaluated whether models could consistently follow abstract rules.

This broad selection helped determine whether Self-Consistency addresses a general limitation of reasoning rather than improving performance on only a particular dataset.

The experiments also covered a wide range of language models, including UL2, GPT-3, LaMDA, and PaLM, ranging from 20 billion to 540 billion parameters. Evaluating models with different architectures and scales allowed the authors to examine whether the method could generalize beyond a single model family.

To ensure a fair comparison, all experiments remained within the original few-shot Chain-of-Thought prompting framework. The prompts were unchanged, and none of the models were retrained or fine-tuned. As a result, any improvement could be attributed directly to the decoding strategy rather than differences in training or model parameters.

Generating multiple reasoning paths required replacing deterministic greedy decoding with sampling. Instead of always selecting the most likely next token, the model explored several plausible reasoning trajectories.

Although the sampling settings varied slightly across models, the objective was always the same: encourage diverse reasoning paths while maintaining coherent solutions. The authors later investigated how sensitive Self-Consistency is to these sampling choices through a dedicated robustness study.

Overall, the experimental design closely matched the paper's central claim. Rather than introducing larger models, additional supervision, or new training procedures, the authors asked a simpler question: How much better can language models reason if we change only the way they generate and select their answers? The experiments provided a systematic way to answer that question.

The infographic below illustrates the complete Self-Consistency decoding pipeline, showing how a language model generates multiple independent reasoning paths and selects the final answer through majority voting.

Main Results

The central question of this paper is straightforward: does generating multiple reasoning paths and selecting the most consistent answer improve upon the original Chain-of-Thought approach?

The experimental results left little room for doubt. Across nearly every benchmark, model, and reasoning task, Self-Consistency consistently outperformed standard Chain-of-Thought prompting.

The largest improvements appeared in arithmetic reasoning. While Chain-of-Thought had already proven highly effective for solving mathematical word problems, the results showed that much of a model's reasoning ability remained untapped when it relied on a single reasoning path.

By exploring multiple reasoning trajectories before selecting an answer, Self-Consistency achieved substantial gains on challenging benchmarks such as GSM8K, SVAMP, and AQuA, establishing new state-of-the-art results on several of them.

Another interesting pattern emerged as model size increased. Although Self-Consistency benefitted every language model evaluated, the improvements became larger for more capable models.

This suggests that larger models already contain multiple valid reasoning strategies internally, but standard greedy decoding often fails to uncover them. Self-Consistency provides a simple mechanism for making better use of those latent reasoning capabilities.

The improvements were not limited to mathematical reasoning. On common sense reasoning benchmarks such as StrategyQA and ARC-Challenge, as well as symbolic reasoning tasks, Self-Consistency again produced consistent gains over standard Chain-of-Thought prompting.

The fact that the method succeeded across such different problem domains suggests that it addresses a general weakness of greedy decoding rather than exploiting properties of a particular benchmark.

Equally noteworthy is how these improvements were achieved. Unlike many earlier approaches that relied on task-specific fine-tuning, additional verifiers, or auxiliary ranking models, Self-Consistency changed only the decoding process. The language model, prompts, and training remained exactly the same. Yet this simple modification frequently matched or surpassed methods that required additional supervision and specialized training.

Taken together, these results revealed an important insight about reasoning in language models. A model's most likely reasoning path is not necessarily its most reliable one. Allowing several independent reasoning processes to explore the same problem before choosing the answer on which they agree produces reasoning that is consistently more accurate.

More broadly, the paper demonstrates that meaningful improvements in reasoning don't always come from larger models or more training. They can also come from making better use of the reasoning abilities the model already possesses.

Common Sense and Symbolic Reasoning

The strong results on arithmetic reasoning naturally raise a broader question: is Self-Consistency mainly helping with mathematical calculations, or does it improve reasoning more generally?

To answer this, the authors evaluated the method on common sense and symbolic reasoning tasks, two domains that require very different reasoning abilities.

On the common sense benchmarks, Self-Consistency consistently outperformed standard Chain-of-Thought prompting. These tasks require models to reason about everyday situations, make logical inferences, and apply background knowledge rather than perform calculations. The consistent improvements suggested that the method was enhancing the reasoning process itself rather than exploiting properties of mathematical problems.

The symbolic reasoning tasks provided an even tougher test. Instead of relying on world knowledge, models had to follow abstract rules and manipulate symbols correctly. The authors evaluated these tasks in an out-of-distribution setting, where the test problems required longer reasoning chains than those shown in the prompt examples.

Even under these more challenging conditions, Self-Consistency continued to improve performance, particularly for larger language models.

The paper also examined how the number of sampled reasoning paths affected performance. Rather than producing diminishing returns immediately, the results showed a steady improvement as more reasoning paths were generated.

Sampling additional solutions gave the model more opportunities to recover from individual reasoning errors and identify the answer that received the strongest agreement across independent reasoning processes.

To illustrate this behavior, the authors presented several qualitative examples. In one case, greedy decoding confidently produced an incorrect answer after following a flawed reasoning path. When multiple reasoning paths were sampled, however, different solutions independently converged on the correct answer, allowing Self-Consistency to recover from the original mistake.

These examples made the method's intuition tangible: success came not from trusting a single explanation, but from comparing several independent attempts before making a decision.

Together, these experiments reinforced one of the paper's central conclusions. The benefits of Self-Consistency extend well beyond arithmetic reasoning. Whether the task involves everyday knowledge, logical inference, or abstract rule following, allowing multiple reasoning processes to compete before selecting an answer consistently produces more reliable results than relying on a single chain of thought.

Self-Consistency Helps When Chain-of-Thought Hurts Performance

One of the paper's most interesting findings challenged an assumption established by earlier Chain-of-Thought research. Although reasoning traces often improved performance, later studies showed that they were not universally helpful. On some natural language processing tasks, asking a model to explain its reasoning can actually reduce accuracy compared to standard prompting.

This raised an important question: if Chain-of-Thought sometimes hurts performance, can Self-Consistency still help?

To answer this, the authors evaluated Self-Consistency on a collection of question answering and natural language inference benchmarks. Unlike arithmetic reasoning, these tasks often required short, direct responses rather than extended reasoning chains. In such settings, generating a rationale could occasionally distract the model instead of improving its answer.

The results confirmed this behavior. On several benchmarks, standard Chain-of-Thought prompting performed worse than conventional prompting, reinforcing the idea that more reasoning doesn't necessarily lead to better reasoning.

What makes the results particularly compelling is that Self-Consistency largely reversed this trend. Even when individual reasoning paths were imperfect, aggregating multiple independent solutions consistently improved performance. Instead of relying on a single rationale that may have been misleading, the model benefitted from comparing several reasoning attempts before selecting its final answer.

These findings broadened the significance of Self-Consistency. The method isn't limited to mathematical reasoning or tasks that naturally require long chains of thought. It also makes reasoning-based prompting more reliable in situations where generating a rationale can be risky, demonstrating that the value lies not in producing more explanations, but in evaluating multiple independent ones before making a decision.

More broadly, this experiment reinforced one of the paper's central ideas: the effectiveness of Self-Consistency doesn't depend on every reasoning path being correct. It succeeds because correct reasoning paths tend to agree more often than incorrect ones, allowing the model to recover from mistakes that would otherwise determine the final answer.

Comparison to Other Existing Approaches

Once the authors established that Self-Consistency improved reasoning performance, a natural question followed: were these gains simply another manifestation of existing decoding techniques, or did Self-Consistency offer something fundamentally different?

To answer this, the paper compared it with several established approaches for improving generation quality, including sample-and-rank, beam search, and ensemble methods.

Sample-and-Rank

The first comparison was with sample-and-rank, a strategy that generates multiple candidate solutions before selecting the one the model considers most likely.

At first glance, this appears similar to Self-Consistency because both methods generate multiple outputs. The difference lies in how the final answer is chosen. Sample-and-rank still trusts a single reasoning path, whereas Self-Consistency looks for agreement across many independent reasoning paths.

The experiments showed that this distinction mattered: selecting the most consistent answer consistently outperformed selecting the most probable one.

Beam Search

The authors also compared Self-Consistency with beam search, one of the most widely used decoding algorithms in natural language generation.

Beam search explores multiple candidate sequences but favors those with the highest probabilities, often producing reasoning paths that are very similar to one another. Self-Consistency, by contrast, relies on sampling to encourage genuinely different reasoning strategies. This additional diversity proves crucial for reasoning tasks, allowing Self-Consistency to outperform beam search across the evaluated benchmarks.

Ensemble-Based Approaches

The final comparison considers ensemble-based approaches, where diversity is introduced by varying prompt order, using different prompt templates, or combining multiple predictions.

Although these methods provided modest improvements over standard Chain-of-Thought prompting, they fell well short of the gains achieved by Self-Consistency. Remarkably, Self-Consistency accomplished this while using only a single language model and a single prompt.

This comparison highlights one of the paper's most important ideas. Traditional ensembles create diversity by changing prompts or combining multiple models. Self-Consistency discovers diversity within the model itself by allowing it to explore multiple reasoning paths for the same problem. The paper described this as a form of self-ensemble, where different reasoning attempts from a single model collectively determined the final answer.

Taken together, these experiments showed that Self-Consistency is more than another decoding heuristic. Its advantage comes not from generating more outputs or ranking them more carefully, but from exploiting a simple observation: difficult reasoning problems often have multiple valid solution paths, and the answer that consistently emerges across those paths is usually the most reliable one.

Additional Studies

Having established that Self-Consistency improves reasoning performance and outperforms competing decoding methods, the authors devoted the final experimental section to a deeper question: why does the method work so reliably?

Rather than introducing new benchmarks, they investigated how Self-Consistency behaved under different sampling strategies, prompting conditions, and reasoning formats to better understand its robustness.

One of the first findings was that the method remained effective across a variety of sampling strategies. Whether the model used temperature sampling, top-k sampling, or nucleus sampling, the overall improvements remained remarkably consistent.

This suggested that Self-Consistency isn't tied to a particular decoding configuration but instead benefits from the broader idea of exploring multiple reasoning paths before making a decision.

The authors also revisited the relationship between reasoning and model scale. Although models of all sizes benefitted from Self-Consistency, the gains became increasingly pronounced as models grew larger.

This reinforced an important theme throughout the paper: Self-Consistency doesn't create new reasoning abilities. Instead, it helps larger models make better use of reasoning capabilities they already possess.

Another interesting experiment examined imperfect prompts. To simulate realistic conditions, the authors deliberately introduced mistakes into the reasoning demonstrations used for prompting. As expected, greedy decoding became less accurate. Self-Consistency, however, recovered much of the lost performance, showing that it was considerably more robust to flawed reasoning examples than standard Chain-of-Thought prompting.

One of the paper's most intriguing observations concerned the relationship between consistency and correctness. When many sampled reasoning paths converged on the same answer, that answer was much more likely to be correct. Conversely, widespread disagreement among the sampled solutions often signaled uncertainty.

This suggested that Self-Consistency offers more than improved accuracy. It also provides a simple way to estimate the model's confidence by measuring agreement among its own reasoning attempts.

The authors further showed that the method wasn't limited to natural-language reasoning. Replacing verbal reasoning traces with intermediate equations still improved performance, although the gains were smaller because shorter reasoning paths provided less opportunity for diversity.

They also demonstrated that Self-Consistency integrated naturally with Zero-Shot Chain-of-Thought prompting, producing substantial improvements even without manually written reasoning examples.

Taken together, these studies show that Self-Consistency is far more than a decoding trick that works on a handful of benchmarks. Across different sampling strategies, model scales, prompting styles, and reasoning formats, the same pattern continues to emerge: allowing a model to explore multiple reasoning paths before choosing an answer consistently produces reasoning that is both more accurate and more reliable.

Self-Consistency didn't emerge in isolation. It has built on several research directions that were already shaping reasoning in language models, combining ideas from prompting, decoding, and consistency into a remarkably simple inference-time strategy.

The most direct influence is Chain-of-Thought prompting, which showed that language models become much better reasoners when they generate intermediate reasoning steps before producing an answer.

Self-Consistency extends that idea by shifting the focus from how a model reasons to how many times it reasons before making a decision. Rather than trusting a single chain of thought, it compares multiple independent reasoning paths and selects the answer on which they agree.

The paper also draws on earlier work in decoding strategies. Techniques such as temperature sampling, top-k sampling, nucleus sampling, and beam search were originally developed to improve text generation by balancing quality and diversity.

Self-Consistency reuses these sampling methods for a different purpose. Instead of generating diverse outputs for creativity, it generates diverse reasoning paths to improve the reliability of a single final answer.

Another closely related area is verification and reranking. Previous approaches often generated multiple candidate solutions and relied on additional verifier models or rerankers (sometimes trained with extra human annotations) to identify the best answer.

Self-Consistency reaches a similar goal without any additional models or supervision. Rather than learning to evaluate reasoning paths, it simply identifies the answer that emerges most consistently across independent reasoning attempts.

Finally, the paper connects to broader research on consistency in language models. Earlier studies examined consistency in conversation, factual knowledge, and generated explanations.

Self-Consistency introduces a different perspective: consistency among multiple reasoning paths. The key insight is that when independent reasoning processes repeatedly converge on the same answer, that agreement itself becomes a strong signal of correctness.

Viewed together, these connections highlight why the paper had such a lasting impact. Self-Consistency didn't require a new model, additional training, or a complex reasoning framework. Instead, it combined existing ideas in a way that fundamentally changed how researchers thought about inference-time reasoning, demonstrating that significant gains could come simply from allowing a model to explore several solutions before choosing the most reliable one.

Discussion

One of the most important ideas in this paper is that better reasoning doesn't necessarily require larger models or more training data. Sometimes, the biggest improvement comes from changing how a model arrives at its final answer.

Rather than trusting the first reasoning path it generates, Self-Consistency allows the model to explore several independent solutions before selecting the answer that receives the strongest agreement. This simple shift changes the role of decoding from choosing the most likely response to identifying the most reliable one.

The experiments suggested that many reasoning failures weren't caused by missing knowledge. Instead, they suggested that a model may already possess the information needed to solve a problem but it fails because it follows an incorrect reasoning path.

By generating multiple reasoning attempts, Self-Consistency gives the model additional opportunities to recover from these mistakes and uncover reasoning capabilities that would otherwise remain hidden.

The paper also highlighted several practical advantages beyond improved benchmark scores. Multiple reasoning paths make it easier to inspect how a model reaches its conclusions, while the level of agreement among those paths provides a useful estimate of confidence.

When independent reasoning processes consistently produce the same answer, that agreement becomes a strong indicator of reliability. Conversely, widespread disagreement can signal uncertainty and identify problems that deserve closer inspection.

Of course, these benefits come with a trade-off. Generating multiple reasoning paths requires additional computation, making inference more expensive than standard Chain-of-Thought prompting. Although the authors showed that much of the improvement could be achieved with a relatively small number of samples, the extra computational cost remains one of the method's primary limitations.

They also noted that incorrect or nonsensical reasoning paths can still be generated. Self-Consistency reduces the impact of these errors, but it can't eliminate them entirely.

More broadly, this paper marked an important shift in how researchers approached reasoning in language models. Earlier work largely focused on improving models through larger architectures, more data, or additional training. Self-Consistency demonstrated that substantial gains could also come from better inference strategies.

That insight has influenced much of the subsequent research on test-time reasoning, search, verification, and the reasoning-oriented language models that followed, making this paper one of the key milestones in the evolution of modern LLM reasoning.

Conclusion

Self-Consistency is a natural continuation of the ideas introduced by Chain-of-Thought prompting.

What appears to be a small change in decoding turns out to have a surprisingly large impact. By replacing a single reasoning path with multiple independent ones and selecting the answer on which they agree, Self-Consistency consistently improves performance across arithmetic, common sense, and symbolic reasoning tasks.

More importantly, it demonstrates that better reasoning doesn't always require larger models or additional training. Sometimes, it simply requires asking the model to think in more than one way.

Looking back, this paper marked an important turning point in the evolution of reasoning in large language models. It shifted the focus from generating the most likely reasoning path to identifying the most reliable answer through agreement among multiple reasoning processes.

That simple idea became the foundation for many later advances in test-time reasoning, search, verification, and the reasoning-oriented language models that followed, securing Self-Consistency's place as one of the most influential papers in modern LLM reasoning.

The infographic below summarizes the key papers that laid the foundation for modern prompting, reasoning, and agentic AI.

Starting with GPT-3's demonstration of in-context learning, it follows the rapid evolution of reasoning techniques, including Zero-Shot Chain-of-Thought, Chain-of-Thought, Self-Consistency, Least-to-Most Prompting, PAL, Program-of-Thoughts, Tree-of-Thoughts, ReAct, and Reflexion.

Collectively, these contributions show how research shifted from simply prompting language models to building systems capable of structured reasoning, planning, tool use, self-reflection, and increasingly autonomous problem solving.

Resources:

Contact Me

How to Export a Claude Conversation as a PDF

Vikram Aruchamy — Mon, 06 Jul 2026 13:56:14 +0000

Whether you're documenting research, sharing AI-generated content with colleagues, creating reports, or keeping an offline backup, saving Claude conversations as PDFs is one of the easiest ways to preserve your work.

While Claude lets you export your account data for archival purposes, it doesn't currently include a built-in option to export an individual conversation directly as a PDF. As a result, users often rely on browser printing, document editors, Claude Artifacts, share links, or dedicated Claude to PDF tools depending on their workflow.

In this guide, you'll learn the most effective ways to convert Claude conversations into PDFs, including the advantages, limitations, and best use cases for each method.

Whether you need to save a single conversation, export a Claude Artifact, archive your entire conversation history, or preserve formatting in code- and image-heavy conversations, you'll find the approach that best fits your needs.

How to Save Claude Conversations as a PDF Using the Browser Print Option
How to Copy Claude Responses into Google Docs and Save Them as PDFs
How to Convert Claude Share Links into PDFs
How to Export Claude Artifacts as PDFs
How to Download All Claude Conversations from Settings
How to Choose the Best Export Method
Video Tutorial: How to Export a Claude Conversation as PDF
Conclusion

How to Save Claude Conversations as a PDF Using the Browser Print Option

The browser's built-in Print feature is the quickest way to convert a Claude conversation to PDF. It works in all modern browsers, requires no additional software, and is suitable for most one-time exports of conversations that are text-heavy, with limited images and interactive content.

Depending on your preferred workflow, you can rely on this native method or use a simple Claude to PDF Chrome Extension to export your conversation.

How Browsers Generate PDFs From Web Pages:

When you use your browser's Print feature, it doesn't take a screenshot of the page. Instead, the browser renders the page specifically for printing by processing its HTML and CSS.

Websites can also provide a print stylesheet — a set of CSS rules that changes how the page appears on paper or in a PDF.

A print stylesheet can hide navigation menus, buttons, sidebars, advertisements, and other interactive elements while optimizing the layout for printing. If a website doesn't define print-specific styles for certain elements, the browser prints them as they appear on the page.

This is why buttons such as Copy, Share, and other Claude interface controls may appear in the exported PDF when you use this option to export the conversation as pdf.

Now, lets see the steps to print the conversation to PDF.

Step 1: Open the Browser's Print Dialog

Open the Claude conversation you want to export.
Scroll through the conversation to ensure all responses have finished loading.
Press Ctrl + P (Windows/Linux) or ⌘ + P (macOS), or select Print from your browser's menu.

Step 2: Save the Conversation as a PDF

In the print dialog:

Set the destination to Save as PDF.
Choose the pages you want to export (optional).
Select a location to save the PDF.
Click Save.

Step 3: Adjust the Print Settings

Before saving the PDF, review the available print settings. Most browsers provide these options under More settings. The following image shows the print settings.

Let's go over a few of these:

Margins

Leave the margins set to None for most conversations. If wide code blocks or tables are clipped, switch to Minimum margins to use more of the page width.

Scale

Keep the Scale as Actual size If long lines of code extend beyond the page width, reduce the scale slightly so the content fits on the page.

Background graphics

By default, browsers don't print background colors. If you want to preserve the background styling used for code blocks and other interface elements, enable Background graphics.

Headers and footers

This option is disabled by default. If you'd like the PDF to include the page title, URL, date, and page numbers, enable Headers and footers.

Advantages:

Available in every modern browser.
Requires no additional software.
Works entirely on your device.
Suitable for quickly exporting individual conversations.

Limitations:

Long conversations may generate very large PDFs with awkward page breaks.
Long code blocks can wrap or split across pages.
Wide tables may be compressed or clipped.
Large images may be resized or moved across pages.
Interface elements such as Copy, Share, and other Claude controls may appear in the exported PDF if they are not hidden by Claude's print stylesheet.
Embedded Artifacts may not be fully captured and often need to be exported separately.

For short conversations, browser printing is usually sufficient. For conversations containing extensive code, large images, complex tables, or Artifacts, the other methods we'll discuss next generally produce better results.

How to Copy Claude Responses into Google Docs and Save Them as PDFs

If you only need to export a single Claude response, you can use Claude's built-in Copy button. Unlike browser printing, this method copies the response as Markdown, preserving headings, lists, tables, code blocks, links, and other formatting.

Click the Copy button located below the response. Claude copies it to your clipboard as Markdown, making it easy to import into applications that support the Markdown format.

Then open a Google Docs document. If this is your first time using Markdown import, go to Tools → Preferences and enable Markdown. This option is disabled by default.

Once enabled, select Edit → Paste from Markdown (or right-click and choose Paste from Markdown) to import the copied content.

Google Docs automatically converts the Markdown into a formatted document, preserving most elements such as:

Headings
Bullet and numbered lists
Tables
Code blocks
Blockquotes
Hyperlinks

Review the imported document before exporting it, especially if it contains complex tables, nested lists, or long code blocks. Minor formatting adjustments may be required depending on the content.

Once you're satisfied with the document, select File → Download → PDF Document (.pdf) to generate the PDF.

Advantages:

Produces a clean document without Claude's interface elements.
Preserves document structure better than browser printing.
Allows you to edit the content before exporting.
Uses built-in features available in Claude and Google Docs.

Limitations:

Suitable for exporting individual Claude responses, not entire conversations.
Images and interactive content may require manual adjustments.
Complex layouts may need minor formatting cleanup before exporting.

If you don't need to edit the response, you can also convert the copied Markdown directly using a Markdown to PDF converter online tools, eliminating the need to import it into Google Docs first.

Claude lets you create a public Share Link for any conversation. Once a conversation is shared, anyone with the link can view it in a web browser without signing in to your account.

Share Links are a convenient way to convert conversations into PDFs using free online tools, such as a Claude to PDF converter that accept a Claude Share Link and generate a downloadable PDF. They automate the conversion process and produce cleaner page layouts with fewer manual adjustments.

To create a Share Link:

Open the conversation you want to export.
Click the Share button from the top right.
Choose the Create public link option.
Copy the generated URL.
Enter the URL in the free tool text box, and your entire conversation will be downloaded as a PDF file.

This method is most appropriate when you want to generate a cleaner PDF from a publicly accessible conversation.

Note: Because Share Links are publicly accessible, avoid using this method for conversations containing confidential, personal, or sensitive information. Anyone with the link can view the shared conversation until the Share Link is revoked or deleted from your Claude account.

How to Export Claude Artifacts as PDFs

Claude Artifacts are standalone outputs that Claude generates alongside a conversation. Unlike regular chat messages, Artifacts open in a dedicated panel and are designed for working with larger pieces of content such as documents, code, web pages, and diagrams.

Common Artifact types include:

Documents
Markdown files
HTML pages
Source code
SVG graphics

If an Artifact supports PDF export, this is the easiest way to create a PDF. Open the Artifact and click Download as PDF option from the toolbar as shown in the following image:

Claude generates the PDF directly from the Artifact, producing a cleaner result than printing the entire conversation.

This approach is particularly useful for content that is intended to be read as a standalone document, such as reports, articles, technical documentation, or Markdown files.

Keep the following considerations in mind:

The PDF contains only the Artifact, not the surrounding conversation.
If a conversation contains multiple Artifacts, each one must be exported separately.
Interactive HTML Artifacts are exported as their rendered output, so interactive behavior isn't preserved in the PDF.
Code Artifacts retain their formatting, although very long lines may wrap depending on the page width.
Large SVG graphics may be scaled to fit the page size.

If your goal is to preserve the conversation itself, including prompts, responses, and the generated Artifact, you'll need to use one of the conversation export methods covered in this guide.

How to Download All Claude Conversations from Settings

If you want to archive your entire Claude account instead of exporting individual conversations, Claude's Export Data feature is the most comprehensive option. Rather than generating PDFs, Claude exports your account as a ZIP archive containing JSON files that preserve your complete conversation history.

To request an export:

Open Claude.
Go to Settings.
Select Export Data.
Request the export.
Download the ZIP archive when you receive the email.

The exported archive may contain:

Conversations
Projects (if applicable)
Account information
Other account data

Unlike browser printing, the conversations are stored as structured JSON rather than formatted documents.

A typical conversation file has the following structure:

Conversation
├── uuid
├── name
├── summary
├── chat_messages
│   ├── sender
│   ├── created_at
│   ├── content
│   │   ├── type
│   │   └── text
│   └── attachments

The fields at the top of the file contain metadata about the conversation, while the actual conversation is stored inside the chat_messages array. Each message records:

sender: Whether the message was written by the user or Claude.
created_at: When the message was created.
content: One or more content blocks.
type: The content type, such as text.
text: The actual conversation text.

If your goal is simply to read or archive the conversation, you can ignore most of the metadata and extract only the text field from each message.

The following Python script converts an exported conversation into a simple Markdown document by extracting only the conversation text.

import json

with open("conversation.json", "r", encoding="utf-8") as f:
    conversation = json.load(f)

print(f"# {conversation['name']}\n")

for message in conversation["chat_messages"]:
    sender = message["sender"].capitalize()

    for block in message["content"]:
        if block.get("type") == "text":
            print(f"## {sender}\n")
            print(block["text"])
            print()

The generated Markdown can then be:

Imported into Google Docs using Paste from Markdown.
Converted with a Markdown-to-PDF converter.
Archived in a Git repository or knowledge base.
Indexed by documentation tools.

This method is the best choice when you want to preserve your entire Claude history in a formatted document. It isn't intended for quickly exporting individual conversations as PDFs, but it provides the highest-fidelity archive of your data.

How to Choose the Best Export Method

Each export method serves a different purpose. The right choice depends on whether you're exporting a single response, an entire conversation, a Claude Artifact, or your complete account history.

Method	Best For	Advantages	Limitations
Browser Print	Quick one-time exports	Built into every browser, no additional tools required	Includes Claude interface elements, limited formatting control
Google Docs	Editing before exporting	Produces a clean, editable document with good formatting	Best suited for individual Claude responses
Claude Artifacts	Exporting generated documents, code, or HTML	Preserves the original artifact content	Doesn't export the entire conversation
Claude Share Links	Converting publicly shared conversations	Cleaner output than printing the Claude interface	Requires creating a public Share Link
Account Data Export	Backing up all conversations	Exports your complete conversation history for archival	Produces JSON files rather than readable PDFs

Use the following recommendations to choose the most appropriate method:

Quickly saving a single conversation: Use the Browser Print option.
Editing the content before exporting: Copy the response into Google Docs and export it as a PDF.
Saving a Claude Artifact: Export or print the Artifact directly.
Backing up your entire Claude account: Use Account Data Export from Claude Settings.
Preserving formatting for long or complex conversations: Use a dedicated Claude to PDF tool designed for exporting conversations.

Video Tutorial: How to Export a Claude Conversation as PDF

Conclusion

Although Claude doesn't currently offer a native option to export individual conversations as PDFs, it's possible to achieve the same result using browser printing, Google Docs, Claude Artifacts, Share Links, or the built-in account export feature. Each method has its own trade-offs in terms of formatting, convenience, and intended use.

If you're looking for a more streamlined workflow, especially for exporting conversations with code blocks, tables, images, and long responses, you can also use a dedicated Claude to PDF tool that automates the process and produces cleaner PDFs with minimal manual effort.

Codex vs Claude Code: Which AI Coding Assistant to Choose

Manish Shivanandhan — Thu, 02 Jul 2026 16:54:03 +0000

AI coding assistants have evolved from simple autocomplete tools into capable development agents that can write code, debug applications, refactor projects, and even execute complex workflows.

Among the newest generation of tools, OpenAI's Codex and Anthropic's Claude Code have emerged as two of the strongest options for developers.

Both platforms promise to improve productivity, reduce repetitive work, and help teams ship software faster. But they approach software development differently.

Choosing between them depends less on finding a universal winner and more on understanding which tool aligns with your workflow, team structure, and development goals.

Understanding Codex

Codex is OpenAI's dedicated coding agent designed to assist developers throughout the software development lifecycle.

Unlike earlier code generation tools that focused mainly on snippets and autocomplete, modern Codex operates more like an autonomous development partner.

It can understand large codebases, generate new features, fix bugs, review existing implementations, and work on multiple tasks simultaneously.

OpenAI has expanded Codex beyond a simple command-line experience, introducing desktop and cloud-based environments that allow developers to delegate work while continuing with other responsibilities.

According to OpenAI, Codex can read, edit, and run code while operating in its own environment to complete assigned tasks. This makes it particularly useful for teams that want an AI assistant capable of handling longer-running assignments independently.

Understanding Claude Code

Claude Code takes a different approach. Rather than emphasising autonomous execution, Anthropic has focused heavily on developer collaboration and reasoning quality.

Claude Code functions as a terminal-native assistant that integrates directly into existing workflows. Developers can interact with it conversationally while maintaining close oversight of the coding process.

The tool is particularly strong at explaining architectural decisions, reviewing unfamiliar codebases, and helping developers work through complex implementation challenges. Instead of simply generating solutions, Claude Code often provides context that helps engineers understand why a particular approach may be preferable.

This makes Claude Code attractive for developers who view AI as an intelligent collaborator rather than an independent coding agent.

Codex vs Claude Code: Direct Comparison

The Difference in Philosophy

The biggest distinction between Codex and Claude Code lies in their approaches to autonomy.

Codex is designed to execute delegated work efficiently. Developers describe objectives, and the system attempts to complete them with minimal intervention. It excels in situations where productivity and task completion are the primary objectives.

Claude Code, on the other hand, prioritises interaction. It keeps developers closely involved in the decision-making process and often produces explanations alongside implementation suggestions.

Neither philosophy is inherently better.

Teams building products under tight deadlines may benefit from Codex's autonomous capabilities. Developers working on complex systems that require thoughtful design discussions may prefer Claude Code's collaborative style.

Code Quality and Reasoning

When evaluating coding assistants, raw output quality matters.

Claude Code has earned a reputation for producing clean, maintainable code with strong architectural awareness. It often breaks larger problems into logical components and provides reasoning that helps developers understand the trade-offs involved.

Codex tends to optimise for execution and efficiency. Its outputs frequently focus on accomplishing the requested task with minimal overhead while maintaining practical production considerations.

Comparative testing has shown that Claude Code often excels in documentation tasks and feature design. Codex demonstrates strong consistency across multiple categories of development work. Research analysing thousands of pull requests found that no single agent dominated every software engineering task, reinforcing the idea that context matters when selecting a tool.

Workflow Integration

The way an AI coding assistant fits into your existing development process can significantly impact adoption and long-term value.

Claude Code is built around a terminal-first experience, allowing developers to interact with the model directly within familiar command-line environments. This makes it particularly appealing to engineers who prefer maintaining close control over implementation decisions while receiving real-time guidance and feedback.

Codex takes a different approach by emphasising automation and delegation. Developers can assign coding tasks and review the completed work later, making it well-suited for teams looking to reduce repetitive workloads and improve development velocity. This model can be especially useful in larger organisations where engineers frequently juggle multiple projects and priorities.

Ultimately, the right choice depends on how your team prefers to work. Developers seeking an interactive coding companion may gravitate toward Claude Code, while organisations focused on streamlining execution may find Codex a better fit within their existing workflows.

Deployment Options

Writing code is only part of the software development process. Once an application is complete, developers still need a reliable way to test, deploy, and maintain it in production.

Whether you use Codex or Claude Code, the deployment workflow remains largely the same. AI coding assistants can generate production-ready applications, but they don't replace the infrastructure needed to host them.

Developers still need platforms like Vercel, Hostinger and Railway that support automated deployments, scalable environments, SSL certificates, backups, monitoring, and straightforward rollback options.

For teams looking to deploy apps built with Claude, platforms like AWS and Vercel make it easier. They integrate continuous delivery pipelines while providing the reliability expected from production systems.

The same applies when you try to deploy apps built with Codex. Services such as Hostinger simplify deployments with managed Node.js hosting, Git integration, and built-in security features, allowing developers to move from AI-generated code to a live production environment with minimal configuration.

As AI coding assistants become part of everyday development workflows, selecting the right production hosting for AI coding assistants is becoming just as important as choosing the coding tool itself. The best workflow combines an intelligent development assistant with infrastructure that makes shipping software fast, reliable, and repeatable.

Productivity Considerations

One of the primary reasons organisations adopt AI coding assistants is to improve development velocity.

Codex often shines when repetitive or well-defined tasks dominate the workload. Generating boilerplate code, implementing straightforward features, writing tests, or executing multi-step workflows are scenarios where autonomy can deliver meaningful time savings.

Claude Code provides value during exploratory development. Developers can brainstorm implementation approaches, validate assumptions, and receive guidance while preserving human oversight.

The productivity gains from each tool depend heavily on how teams allocate engineering effort.

Organisations emphasising rapid delivery may prioritise Codex.

Teams prioritising knowledge sharing and architectural consistency may lean toward Claude Code.

Security and Oversight

As AI agents gain more capabilities, governance becomes increasingly important.

Claude Code's interactive design naturally encourages human review before significant actions occur. This reduces the likelihood of unintended modifications and reinforces developer accountability.

Codex introduces stronger automation capabilities, which can accelerate workflows but also require clearly defined operational safeguards. Organisations adopting autonomous coding agents should establish review processes, permission controls, and testing requirements before integrating them into production environments.

The goal is not to eliminate human involvement but to position AI appropriately within existing software development practices.

Should you Choose Codex or Claude Code?

The answer depends on how you work.

Choose Codex if your team values autonomy, wants to delegate substantial development tasks, and needs an assistant that can operate independently across multiple assignments. Organisations focused on maximising throughput may find this approach particularly compelling.

Choose Claude Code if you prefer collaborative problem-solving, appreciate detailed reasoning, and want AI assistance that remains closely integrated with human decision-making throughout the development process.

Neither assistant replaces engineering judgment. Instead, they amplify different aspects of software development.

Final Thoughts

The debate between Codex and Claude Code reflects a broader shift within software engineering. AI assistants are no longer limited to suggesting individual lines of code. They're evolving into sophisticated development partners capable of influencing planning, implementation, testing, and deployment.

Codex emphasises execution. Claude Code emphasises collaboration.

For some teams, Codex will unlock significant productivity gains by handling routine work autonomously. For others, Claude Code will enhance decision-making by serving as an intelligent coding companion.

Ultimately, the best choice is the one that complements your team's existing strengths and addresses its most significant bottlenecks.

As AI continues to reshape development practices, the organisations that succeed will not necessarily be those using the most advanced tools. They will be the ones who integrate those tools thoughtfully into well-defined engineering processes.

Hope you enjoyed this article. You can connect with me on LinkedIn.

How to Build an AI Agent That Runs its Own LLM Experiments with autoresearch

ishaan gupta — Mon, 29 Jun 2026 16:50:22 +0000

A few months ago, Andrej Karpathy released autoresearch. It's an open-source Python tool that lets an AI agent run experiments on one GPU while you sit back and wait for the results.

Lately I've still seen folks on Twitter arguing about whether AI agents can build their “million dollar idea” or something about Openclaw. But here's a repo that lets you hand an agent a real GPT training setup and ask it to do the research itself.

Basically it edits the code, trains, reads the loss, makes a decision about the result, and repeats this process. And all this happens while you sleep, or dig into something else. And surprisingly, it does actually work.

On a depth-12 nanochat baseline (more on what "depth" means later), Karpathy left it running for about two days. Over roughly 700 experiments, the agent found about 20 changes that genuinely improved the model, and those changes stacked on top of each other.

In this article, I'll walk through what autoresearch is, why the way it measures success is the whole trick, what each file in the repo actually does, what the agent tends to discover, and a step-by-step guide to running it yourself. By the end you should be able to point an agent at your own GPU and let it run.

Prerequisites
What is autoresearch?
Why This Matters
What Exactly is val_bpb?
What the Agent Actually Finds
Final Thoughts

Prerequisites

This article is a complete walkthrough of this repo. The goal is that by the end, you'll understand what autoresearch is and how you can run it on your own machine.

No prior ML research experience required, but if you have it then the deeper sections I wrote will be more meaningful to you. Just basic knowledge of GPU, VRAM and GPUs like H100/A100/4090 would suffice, but don't worry i have quoted the text below explaining every term i think a beginner needs to understand.

What is autoresearch?

Simply put, autoresearch is just one specific idea executed cleanly. You take a small but real LLM training setup, put it in a single Python file, and let an AI agent edit that file.

The agent runs the file and reads the loss. When you train a language model, "loss" is just a single number that scores how badly the model is predicting the next chunk of text. A high number means it's guessing poorly, and a number close to zero means it's predicting almost perfectly.

Training is the process of nudging the model's millions of internal weights to push that number down. So when I say the agent "reads the loss," I mean it looks at that score to judge whether the change it just made helped or hurt.

Based on that score, the agent decides whether the change helped, and then either keeps the change or reverts it. Then it tries something else.

The flow runs top to bottom like this: A human (you) writes the playbook (a Markdown file called program.md), which spells out the rules. An AI agent reads that playbook and starts an experiment loop.

In each pass of the loop, the agent edits the training code with a new idea, trains for five minutes, reads the resulting score, decides whether to keep or undo the change, and writes the outcome to a results file. Then it loops back and tries the next idea.

It does this on its own, around twelve times an hour. So a full night of sleep buys you roughly a hundred experiments and, with luck, a noticeably better model by morning.

The repo is laid out so the agent has exactly one knob to turn. It can't install new packages or change how the data is loaded or how the loss is measured. All of that is locked down on purpose. The only file the agent edits is train.py which consists of the model architecture, the optimizer, the batch size, the learning rate, and the structure of the training loop itself.

The reason this design works is the same reason a controlled experiment in any field works. If the data, the metric, and the budget are all fixed, then any change in the result must be coming from the change the agent made. The agent is doing science the way a careful researcher would, only it doesn't get tired and doesn't need lunch.

Why This Matters

It's tempting to read this as just another agent demo. But it's not, and the reason is the metric. That metric is called val_bpb, short for validation bits per byte. It's a specific way of scoring how well the model predicts text it has never seen during training (the "validation" set).

I'll break down exactly how it's calculated in the next section, but the one-line version is that it measures, on average, how many bits of information the model needs to encode each byte of text. Lower is better: a lower val_bpb means the model is surprised less often by real text, which is the whole goal.

The reason Karpathy uses bits per byte rather than the raw training loss is that bits per byte doesn't change just because you changed the vocabulary, so two very different models can still be compared fairly. The "lower is better" part and the "vocabulary-independent" part are two separate properties. The metric happens to have both.

When I say a baseline model from this repo "lands around 1.00 bpb," I mean that if you run the default untouched training script for its 5 minutes, the model it produces scores roughly 1.00 on this metric when measured on the held-out validation text. That's your starting line.

From there, an improvement of 0.005 bpb (so a score of about 0.995) is a small but real win, the kind the agent finds often. An improvement of 0.05 (a score near 0.95) would be enormous, the kind of jump you'd usually only get from a much bigger model or a much longer training run. So the numbers look tiny, but on this scale, thousandths of a bit genuinely matter.

Here's why optimizing this particular number is a big deal. The agent isn't chasing some artificial leaderboard that researchers spent years gaming. It's pushing down the same kind of validation loss curve that every major language model has been trained against since GPT-2 in 2019.

A "loss curve" is just the plot of that score dropping over the course of training, and "the wave of LLMs since GPT-2" is shorthand for the fact that essentially all of the progress, from GPT-2 to today's frontier models, came from people finding ways to make that curve drop faster or lower for the same amount of compute. The agent is working on the exact same problem, just at a small, fast cheap scale.

And that's what makes the next part surprising. When the agent finds an improvement "here," I mean on the small depth-12 model it's allowed to edit. "Depth" is the number of transformer layers stacked in the model. depth-12 is a small model, and depth-24 is a bigger one with twice as many layers.

Karpathy took the roughly 20 tweaks the agent discovered on the small depth-12 model and applied them to the bigger depth-24 model. Being stacked cleanly means two things at once: the improvements were additive (turning on all 20 together gave you the sum of their individual gains, rather than cancelling each other out), and they transferred (gains found on the small model still showed up on the big one).

That's the signal that the agent found real insights about training, not lucky quirks that only help at one specific size. Stacked together, they cut Karpathy's "Time to GPT-2" benchmark from 2.02 hours to 1.80 hours, which is about an 11% speedup on code he'd already hand-tuned for a long time.

The other thing that's significant is the budget. Each experiment runs for exactly 5 minutes of wall-clock training time, no more, no less. That gives roughly 12 experiments per hour, or about 100 in a typical 8-hour sleep cycle.

Exploring the Repo

Now if you clone the repo, you get a small handful of files. Most of them are plumbing. Three of them are the heart of the system and the difference between them is who edits what.

Only three files matter, and they differ by who edits them.

train.py is the file the agent edits. it holds the GPT model, the optimizer, and the training loop, and everything in it is fair game.
prepare.py is the fixed foundation that nobody edits during a run: it downloads the data, trains the tokenizer, and defines the metric.
program.md is the file you, the human, edit: it's the playbook of rules the agent follows.

The remaining files (README.md, pyproject.toml, uv.lock, .gitignore, .python-version, the analysis.ipynb notebook, and the progress.png image) are plumbing and documentation that neither you nor the agent needs to touch during a run.

There are a few other files in the repo which don't need attention from you or the agent during a run.

What Exactly is `val_bpb`?

Before going further, it helps to understand what val_bpb is. If you've read other LLM articles, you have probably seen terms like “perplexity” or “cross-entropy loss” thrown around.

Bits per byte is like their cousin. When a language model predicts text, it assigns probabilities to what comes next. If the model is confident and right, it gets a low loss. If it's confident and wrong, it gets a high loss, a large penalty. Add up those penalties across all the text and you get the model's total loss. Lower is better, because a lower total means the model assigned high probability to the words that actually appeared.

Cross-entropy loss is the standard scoring function for training language models. For each token, the model assigns a probability to every possible next token and the loss is the negative logarithm of the probability it gave to the token that actually came next. Predict the right token confidently and the loss is near zero. Assign low probability to the correct token and the loss is large. The model's total loss is the average of this across all tokens.

Cross-entropy loss measures this in nats. A nat is the unit you get when that logarithm is taken in base e (the natural log) instead of base 2. It measures the same quantity of "surprise" on a different scale (one nat is about 1.44 bits). Dividing the loss by the natural log of 2 is what rescales nats into bits, which is the conversion bits per byte performs.

Bits per byte takes that loss and divides it by the number of bytes the text actually contains, then converts to log base 2. The result is a number that tells you, on average, how many bits of information the model needs to encode each byte of text.

A perfect model would need close to zero, while a random model would need around 8 bits per byte (since a byte has 8 bits).

The reason Karpathy chose bpb instead of plain cross-entropy is that bpb is vocabulary-size-independent. If the agent decides to change the tokenizer or the vocabulary, the cross-entropy loss would be completely different even for the same model quality. Bits per byte normalizes that out, so a depth-8 model with vocab 8192 and a depth-12 model with vocab 16384 are directly comparable.

The function that computes this, evaluate_bpb, lives in prepare.py, which the agent is never allowed to edit. It can only touch train.py. Because the metric's definition sits in a file the agent can't modify, it can't lower its score by quietly changing how the score is calculated. The scoring rule stays identical for every experiment, which is what makes the comparison honest.

The 5 Minute Rule

There's one design choice in autoresearch that deserves its own section, because it's the choice that makes the whole thing work in practice. Every experiment runs for exactly 5 minutes of wall-clock training time regardless of what the agent is doing.

Wall-clock time means real elapsed time: what a clock on the wall measures, and not the number of training steps or tokens processed. 5 minutes of wall-clock time is 5 literal minutes regardless, of how much the model does in them.

If you trained for a fixed number of steps instead, the agent could “win” by making the model so small that it ripped through more steps than the baseline. If you trained for a fixed number of tokens, the agent could win by lowering the sequence length.

The agent isn't competing against another agent as we might think of it. Its only objective is to push val_bpb below the previous best score on this exact setup. So "winning" means producing a lower score, and the risk is that it lowers the score through a degenerate shortcut that games whichever budget you chose rather than a real efficiency gain. If you trained until convergence, the agent’s run would take wildly different amounts of time and you would never finish 100 experiments in a night.

A fixed wall clock budget cuts through all of this. The agent is forced to optimize for actual training efficiency on the actual hardware in front of it. If it makes the model slightly bigger but the per-step compute drops because of a smarter attention pattern, that's a real win. If it speeds up the per-step compute but the model now learns less per step, that shows up as a worse val_bpb. The two effects get netted out automatically in the end.

The H100 and A100 are NVIDIA datacenter GPUs and the RTX 4090 is a high-end consumer card. They differ sharply in speed and memory, and that's the whole point: in a fixed 5 minute budget, a faster card processes more data and reaches a lower val_bpb. So a score from one GPU can't be compared head-to-head with a score from another.

There's a tradeoff, though. Because the budget is wall-clock, the val_bpb you get on an H100 isn't directly comparable to the val_bpb you get on a 4090 or an A100. The system is designed to find the best model for your specific compute platform in 5 minutes, not to be a global benchmark.

If you want to compare across hardware, you would need to fix a different budget. For the autonomous research use case, this is exactly right.

Let’s get into each of the files in depth now.

1. `prepare.py`

Nobody touches this file but everything depends on it. It mainly performs three jobs.

The first job is downloading data. The training corpus is ClimbMix-400B, a high-quality web dataset hosted on HuggingFace and shuffled into 6,543 parquet shards. By default prepare.py downloads only 10 of these (about a few gigabytes), which is plenty for running thousands of 5-minute experiments.

The very last shard is always downloaded and pinned as the validation set. That pinning matters, since every experiment (no matter what changes) evaluates on the exact same held-out data.

The second job is training a tokenizer. The repo uses rustbpe, a fast Rust implementation of byte-pair encoding, to learn a vocabulary of 8,192 tokens from a sample of the training data. The result is exported as a tiktoken-compatible encoding so it integrates cleanly with PyTorch downstream. There's also a small precomputed lookup table called token_bytes.pt that maps each token id to its UTF-8 byte length. This is what makes the bpb calculation honest.

The third job is providing utilities that train.py imports at runtime. The dataloader is the interesting one. It does what's called best-fit packing: every row in the batch starts with a special BOS (beginning of sequence) token and the loader fills the row by greedily picking documents that fit in the remaining space. Only when no document fits does it crop the shortest available document to fill the gap.

The result is 100% utilization with no padding. This is meaningfully faster than the naïve approach of just truncating long documents and padding short ones. The constants at the top of prepare.py are deliberately simple. Three numbers and a sequence length define the entire experimental contract.

If you run autoresearch on different hardware and want to compare results with a friend, the only thing both of you need to share is these constants. That's the whole point of putting them here and nowhere else.

2. `train.py`

This is the file the agent lives in. It breaks naturally into four parts: the model, the optimizer (Muon for the matrix weights, AdamW for the embeddings and scalar parameters), the hyperparameters, and the training loop. We'll walk through each one with the goal of understanding why each piece exists.

The model is a fairly modern GPT written from scratch with no library dependencies beyond PyTorch and a Flash Attention 3 kernel. If you've read other GPT implementations the high-level structure will look familiar: a token embedding, a stack of transformer blocks, a normalization layer, and a linear head that projects back to vocabulary logits.

The interesting parts are in the details. I don’t think explaining the architecture or code is required for this repo, so I’ll just draw out a small architecture diagram for those of you who want to visualize it. Then I'll explain how the training loop is written.

The loop itself is short and almost pleasant to read. The skeleton is:

while True:
    # accumulate gradient over micro-batches to hit TOTAL_BATCH_SIZE
    for micro_step in range(grad_accum_steps):
        with autocast_ctx:
            loss = model(x, y)
        loss = loss / grad_accum_steps
        loss.backward()
        x, y, epoch = next(train_loader)

    # update LR / momentum / weight decay based on time elapsed
    progress = min(total_training_time / TIME_BUDGET, 1.0)
    # ... set group["lr"], group["momentum"], group["weight_decay"] ...

    optimizer.step()
    model.zero_grad(set_to_none=True)

    # log step metrics
    # ...

    if step > 10 and total_training_time >= TIME_BUDGET:
        break

There are a few things worth noticing here. First, the time budget is checked after the first 10 steps. This is so the budget doesn't include the initial PyTorch compilation (which can take 30 seconds or more). Without this, fast experiments would get penalized for spending half their budget on warmup.

Second, the loop has a fast-fail check. If the loss explodes or hits NaN it prints “FAIL” and exits. The agent then sees a crash and logs it. This is a defense against the agent doing something that diverges spectacularly.

Third, after the loop ends, there's a single final call to evaluate_bpb and then a structured summary printed to stdout.

That summary is the whole API between the training script and the agent:

---
val_bpb:          0.997900
training_seconds: 300.1
total_seconds:    325.9
peak_vram_mb:     45060.2
mfu_percent:      39.80
total_tokens_M:   499.6
num_steps:        953
num_params_M:     50.3
depth:            8

This is what the grep extracts and the agent reads. The whole experimental contract is seven lines of this plain text.

The Hyperparameters

The hyperparameters live in their own clearly-marked section near the bottom of train.py, with a comment that says "edit these directly, no CLI flags needed." They look like this:

# Model architecture
ASPECT_RATIO = 64       # model_dim = depth * ASPECT_RATIO
HEAD_DIM = 128          # target head dimension for attention
WINDOW_PATTERN = "SSSL" # sliding window pattern: L=full, S=half context

# Optimization
TOTAL_BATCH_SIZE = 2**19 # ~524K tokens per optimizer step
EMBEDDING_LR = 0.6
UNEMBEDDING_LR = 0.004
MATRIX_LR = 0.04
SCALAR_LR = 0.5
WEIGHT_DECAY = 0.2
ADAM_BETAS = (0.8, 0.95)
WARMUP_RATIO = 0.0
WARMDOWN_RATIO = 0.5
FINAL_LR_FRAC = 0.0

# Model size
DEPTH = 8
DEVICE_BATCH_SIZE = 128

Everything here is a deliberate single point of truth. The model dimension is computed from depth (depth × 64, rounded to the head dimension). The number of heads is computed from model dimension. This means that the agent can change one number DEPTH, and the model rescales itself coherently.

That kind of "one knob to scale the model" parameterization is exactly what makes a search space tractable.

3. `program.md`

program.md is the shortest of the three files and is arguably the most important. It's the file that we edit and it contains everything the agent needs to know about how to behave during a run.

The structure of program.md mirrors the lifecycle of a research session. It opens with setup, agrees on a run tag, creates a Git branch named autoresearch/, reads the in-scope files, verifies that the data exists, and initializes a results file. It then describes the experimentation rules, like what the agent can and can't modify, that VRAM is a soft constraint, and crucially a simplicity criterion that says all else being equal, simpler is better.

A 0.001 bpb improvement that adds 20 lines of hacky code isn't worth keeping. A 0.001 bpb improvement that removes 20 lines is definitely worth keeping.

Then comes the actual loop. The agent is told to run training with uv run train.py > run.log 2>&1 and never to use tee or stream the output because that would flood the agent's context window. It's also told to extract metrics with grep "^val_bpb:\|^peak_vram_mb:" run.log, which gives just the one or two lines that matter.

If the grep produces nothing, that means the run crashed and the agent is told to read the last 50 lines of the log and try to fix the issue (but it should give up after a few attempts and move on). The result of every experiment is logged to results.tsv.

The decision rule is simple: if val_bpb improved (got lower) then the agent advances the branch by keeping its commit. If it didn't improve, the agent runs git reset to undo the commit. If it crashed, the agent logs that and tries something else.

The last paragraph of program.md is the one that makes autoresearch what it is. It's titled NEVER STOP. The agent is explicitly told not to ask the human (you) if it should keep going, not to ask for any permissions, and not to pause for confirmation. If the agent runs out of ideas, it should think harder, look at the failures, combine near-misses, and try more radical changes.

The loop runs until we interrupt it. This single instruction is more interesting than any line of Python in the repo. It's the difference between an agent that does a few experiments and asks if you want to continue and an agent that genuinely does autonomous research overnight.

There is no contradiction with the 5 minute budget. 5 minutes governs a single experiment, one training run. The "Never stop" instruction governs the outer loop. The moment one run finishes and the agent logs the result, it launches the next one. It keeps starting fresh 5 minute experiments back-to-back until you interrupt it.

Nothing ever trains for more than five minutes. The agent simply never stops starting new 5 minute trainings.

Now that you understand how it works, let’s start using it.

Setup Guide

I'm assuming you have a single NVIDIA GPU with enough VRAM to run these experiments. Anything with 24GB or more should work with the default settings. Smaller GPUs need some tuning, which I'll cover later on.

Step 1: Install uv, the Python Project Manager the Repo Uses

uv is much faster than pip and handles virtual environments transparently. After you install it, then clone the repo and install dependencies:

curl -LsSf https://astral.sh/uv/install.sh | sh

git clone https://github.com/karpathy/autoresearch.git
cd autoresearch
uv sync

This will create a .venv and install pyTorch, Flash Attention, rustbpe, tiktoken, pyarrow, and a few other packages. It pulls PyTorch from the CUDA 12.8 wheel index, so make sure your driver supports that.

Step 2: Run the Data Preparation

This downloads 10 ClimbMix shards plus the validation shard and then trains our tokenizer.

uv run prepare.py

It takes about 2 minutes on a decent connection. If you have limited disk space, you can pass --num-shards 4 for a smaller download. The data and tokenizer get cached in ~/.cache/autoresearch/.

Step 3: Run a Manual Training Experiement

Now, you'll run a single training experiment manually, just to confirm that everything works end-to-end.

uv run train.py

After about 5 minutes of training, plus an evaluation pass at the end, you'll get the summary block with val_bpb printed. That's your baseline.

Step 4: Hand the Repo to an Agent

In practice, this means opening Claude Code or your tool of choice in the repo directory, ideally with permissions disabled or scoped tightly to the repo, and prompting it with something like this:

Have a look at program.md and let's kick off a new experiment.
Let's do the setup first.

The agent will read program.md, walk through the setup steps (creating the autoresearch branch and initializing results.tsv), confirm with you, and then start running. From this point on, you can leave it alone. When you come back, check results.tsv and the Git log on the autoresearch branch.

Tuning autoresearch for Smaller GPUs

The default configuration assumes an H100. If you have a 4090, 3090, or anything with less than 80GB of VRAM, you'll need to dial things down.

Lower the sequence length first: MAX_SEQ_LEN = 2048 in prepare.py is the biggest VRAM lever since attention scales quadratically with it. Try 512 or even 256 on a small GPU and bump DEVICE_BATCH_SIZE in train.py slightly to compensate. The product of these two is the tokens-per-forward-pass.
Lower the depth: DEPTH = 8 in train.py is the master knob for model size. Drop it to 4 on a small GPU and the model dimension automatically scales down with it.
Switch the window pattern: WINDOW_PATTERN = "SSSL" uses banded attention which is fast on H100 but can be slow on consumer GPUs, depending on the kernel implementation. Just "L" (always full attention) is simpler and often faster on smaller cards.
Lower the total batch size: TOTAL_BATCH_SIZE = 2**19 is roughly 524K tokens per optimizer step. On a small GPU, drop it to 2^14 (~16K) to start.
Consider switching the dataset: climbMix is a hard broad web corpus. On a tiny model, the loss curve is noisy and bpb numbers are hard to interpret. Karpathy specifically recommends his own TinyStories-GPT4-Clean dataset for small-scale experimentation. The text is narrower in scope (children’s stories) so a small model can actually learn to generate something coherent in 5 minutes.

There are already several community forks that have done the consumer-GPU tuning for you which you can check out in the repo's readme.md file.

What the Agent Actually Finds

It's one thing to describe how the loop works, and another to see what it produces. Karpathy was open about this on Twitter in his depth-12 run: the agent found about 20 changes that improved validation loss, all of which transferred to depth-24.

Specific examples from his post-run analysis include adding a learnable scalar to the parameterless QK-norm to sharpen attention, applying regularization to the value embeddings, widening the banded attention window, correcting the AdamW betas for certain parameter groups, tuning weight decay schedules, and adjusting initialization.

None of these would headline a research paper, but all of them showed up as 0.001 to 0.005 bpb improvements that stacked.

So it's not that an AI agent invented a new architecture. It's that the slow patient hill-climbing that real researchers spend months doing can be done by an agent in a couple of days. The result is the same boring detail-tuning that has always been where most of the actual progress in ML comes from.

Final Thoughts

autoresearch doesn't introduce a new model or a new optimizer or a new dataset. It just defines a kind of contract between a human researcher and an AI agent and it shows that the contract can be enough. That contract is something like “here is the fixed part of reality, the metric that judges you, a budget, and within those rules, do whatever you want and tell me what worked.”

There are two questions I still ponder that are worth thinking about. One is overfitting to the validation set. If you run hundreds of experiments against the same fixed validation shard, eventually the agent will start finding tweaks that look like wins on this shard but don't transfer. Karpathy himself called the results “fragile” in some sessions.

There's no obvious fix here yet beyond rotating validation data which would break comparability.

The other question is what the human’s role becomes. If the agent does the experiments, the human’s contribution shifts to shaping the search space and the rules. That is what program.md is. It's a pretty good preview of what research looks like when the loop is automated.

Well, that’s it for today. See you folks in my next article!

How to Build a Personal Web Research AI Agent with Ollama and Qwen

Darsh Shah — Fri, 26 Jun 2026 18:07:10 +0000

In this tutorial, I’ll show you how to build an AI web research agent using Ollama, Qwen, and Python. The agent searches the web for a topic, fetches relevant pages, and uses a local LLM to generate a concise digest.

Background
Motivation and Architecture
Step 1: Install Ollama and get an API key
Step 2: Pull the Qwen model
Step 3: Install Python dependencies
Step 4: Agent code
Step 5: Running the agent
Sample Output
Conclusion

Background

Most of us have used ChatGPT or Claude to send queries to a large language model. You've probably also seen hallucinations in the response when the model didn't know something, sometimes because its knowledge was out of date.

With the rise of tool calling, LLMs can now use tools to search the web for the latest information. They can then bring that information into context and use it to generate an output, summarize results, and extract key points from retrieved sources.

In this tutorial, I'll show you how I built a personal research agent that searches the internet for any topic and uses local LLM to summarize what it finds. It runs entirely on my own machine to preserve privacy and has no API costs. So it's completely free.

To follow this tutorial, you'll need Ollama installed on your machine and a free Ollama account. The tutorial works on macOS, Windows, and Linux. I'm using a MacBook Pro with 32 GB of RAM, but you can run this on a lower-memory machine by choosing a smaller Qwen model from Ollama.

Motivation and Architecture

The motivation behind this project is to have agents running on my machine that can handle a variety of tasks every day. I can spin off agents to create a daily digest of AI news, surface the latest world events, or look for new job postings.

Running a local LLM also means none of these queries leave my machine. My research history stays private, and there are no per-query API costs to worry about.

For this project, we'll use Ollama web search for retrieval and local Qwen LLM for summarization (rather than rely on hosted chat tools like ChatGPT or Claude). The system diagram below shows how the agent works.

When run in the terminal, the agent asks the user what they want to research. It then calls the Ollama web search API to fetch the top 5 results for the query, downloads each of those pages, and extracts the readable text.

The extracted content from all five pages is sent to the local Qwen model along with the user's prompt and a system prompt: "Use these web results and page contents to answer in Markdown format." The model's response is then saved as a Markdown file on disk.

Step 1: Install Ollama and Get an API Key

To get started, install the Ollama application and create an account to get an API key. The free tier of Ollama will suffice for this tutorial.

Once you have the key, place it in an environment variable:

export OLLAMA_API_KEY="paste-key-here"

Step 2: Pull the Qwen Model

We'll use Qwen for this tutorial, an open-weight model that's currently one of the best smaller sized models available.

I'm using the 4-billion-parameter variant because it follows structured prompts well and runs on a laptop without a dedicated GPU. There are other sizes like 2b or 9b available.

To use Qwen3.5:4b locally, install it using Ollama. The 4b model size is around 3.4 GB on my machine. If your machine has lower RAM, you can use qwen3.5:0.8b instead of the 4b model.

ollama pull qwen3.5:4b

Step 3: Install Python Dependencies

python3 -m venv venv
source venv/bin/activate
pip install ollama requests beautifulsoup4

Step 4: Write the Agent Code

The below Python code does four things: it takes a research prompt from the terminal, calls Ollama's web search API for the top 5 results, downloads the webpages using Requests and cleans each page's text using BeautifulSoup, then sends everything to a local Qwen model with an instruction to summarize in Markdown. Finally, it saves the result to a timestamped .md file.

Save the code in your research_agent.py file.

The summarization prompt is intentionally basic. Feel free to tweak it to match the kind of output you want.

import os
import json
import requests
import ollama
from bs4 import BeautifulSoup
from datetime import datetime
from pathlib import Path

API_KEY = os.getenv("OLLAMA_API_KEY")
SEARCH_URL = "https://ollama.com/api/web_search"
MODEL = "qwen3.5:4b"

# Search web using Ollama web search 
def search_web(query):
    response = requests.post(
        SEARCH_URL,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"query": query, "max_results": 5},
        timeout=30,
    )
    response.raise_for_status()
    return response.json().get("results", [])

# Fetch full web page content
def fetch_text(url):
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
    except requests.RequestException as e:
        return ""
    soup = BeautifulSoup(response.text, "html.parser")
    for tag in soup(["script", "style", "nav", "footer"]):
        tag.decompose()
    return soup.get_text(separator="\n", strip=True)


def main():
    user_prompt = input("Enter your prompt: ").strip()
    if not user_prompt:
        print("Prompt cannot be empty.")
        return

    results = search_web(user_prompt)

    # For each url in web search result, fetch full content
    pages = []
    for item in results:
        url = item.get("url")
        if not url:
            continue

        print(f"Fetching: {url}")
        page_text = fetch_text(url)

        pages.append({
            "title": item.get("title", ""),
            "url": url,
            "snippet": item.get("content", ""),
            "page_text": page_text,
        })

    # Prompt to send to Qwen model with web data
    prompt = f"""
    User request:
    {user_prompt}

    Use these web results and page contents to answer in markdown format.

    Data:
    {json.dumps(pages, ensure_ascii=False)}
    """

    # Invoke local Qwen model 
    response = ollama.chat(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
    )

    digest = response.message.content

    # Build a unique filename using today's date and time
    timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
    filename = f"digest-{timestamp}.md"

    # Save the digest to disk
    with open(filename, "w") as f:
        f.write(digest)
    
    print(f"Saved to digest")

if __name__ == "__main__":
    main()

Step 5: Run the Agent

python research_agent.py

The script will prompt you to enter the topic you'd like to research.

Sample Output

The summarized digest is saved as a timestamped Markdown file. The agent also prints the source URLs as it fetches them.

Before trusting the summary, skim it and spot-check a claim or two against the original source. Local models are smaller than hosted frontier models and tend to hallucinate more. So spot-checking can help with accuracy.

As a test run, I asked the research agent: "What's new in LLMs" and it fetched 5 web pages as seen below:

Enter your prompt: What's new in LLMs
Fetching: https://openai.com/nl-NL/index/chatgpt-memory-dreaming/
Fetching: https://pub.towardsai.net/tai-210-glm-5-2-closes-most-of-the-open-weight-gap-in-ten-weeks-2f970c5f1326
Fetching: https://www.globenewswire.com/news-release/2026/06/23/3315999/0/en/Multiverse-Computing-Launches-Pulsar-16B-in-collaboration-with-NVIDIA-Frontier-Grade-Reasoning-at-Half-the-Parameters.html
Fetching: https://thenextweb.com/news/anthropic-claude-tag-slack-always-on-ai-teammate
Fetching: https://www.aidoers.io/blog/claude-mythos-5-and-fable-5-explained-what-anthropic-actually-shipped

Saved to digest

The digest came out reasonably well-structured for a 4B local model. It's organized into sections with all the relevant data from the sources. I spot-checked the summary and it was accurate.

Here's what it produced:

# What's New in LLMs (June 2026)

The landscape of Large Language Models (LLMs) has evolved rapidly in June 2026, with significant updates in memory synthesis, new frontier models, enterprise integrations, and market dynamics.

## 1. Memory & Personalization: OpenAI’s "Dreaming" Update
OpenAI has deployed a new memory architecture for ChatGPT, referred to as **Dreaming V3**.
*   **Purpose:** Improves memory synthesis to optimize freshness, continuity, and relevance.
*   **Evolution:**
    *   **2024:** "Saved memories" (manual instruction-based).
    *   **2025:** "Dreaming V0" (background process curating memories from chat history).
    *   **2026:** **Dreaming V3** (significantly more capable and compute-efficient architecture).
*   **Impact:** Memory is now reviewable via a summary page, allowing users to update information and set instructions on topics to bring up.
*   **Availability:** Rolled out to ChatGPT Plus and Pro users in the US today, expanding to additional countries and Free/Go users over coming weeks.
*   **Capability:** The model now remembers specific user setups (e.g., photography gear preferences) and constraints (e.g., vegetarian diet, hotel AC preferences) without requiring explicit "remember" cues.

## 2. New Frontier Models & Benchmarks

### Claude Fable 5 & Mythos 5 (Anthropic)
*   **Classification:** Mythos-class tier, sitting above Opus in raw capability.
*   **Differentiation:** **Fable 5** is available to the public. **Mythos 5** is the identical model with cybersecurity safeguards removed, restricted to **Project Glasswing** partners only.
*   **Pricing:** $10 per million input tokens / $50 per million output tokens.
*   **Availability:** Included at no extra cost on Pro, Max, Team, and enterprise plans until June 22.
*   **Capabilities:** Significant jumps in **Knowledge work**, **Agentic coding**, **Vision**, **Legal reasoning**, and **Biology**.

### Z.ai GLM-5.2 (Open Weights)
*   **Release:** Z.ai (Z.AI) released GLM-5.2 under an MIT license on June 16, 2026.
*   **Performance:** Closed the open-weight gap in ten weeks. Scored **51** on the Artificial Analysis Intelligence Index.
    *   **Context:** Expanded from 200K to **1 million tokens**.
    *   **Architecture:** Utilizes "IndexShare" for long-context efficiency and "Compaction-aware reinforcement learning" for agents.
*   **Benchmarks:** Ranked third on the AA-Briefcase (91 held-out tasks), behind Fable and Opus 4.8 but ahead of GPT-5.5.
*   **Cost:** ~$0.52 per task (compared to $0.86 for GPT-5.5 and $1.80 for Opus 4.8).

### Multiverse Pulsar 16B (NVIDIA Collaboration)
*   **Parameters:** 16.15B total parameters (3.1B active).
*   **Performance:** Delivers 30B-class intelligence at half the parameter count.
*   **Validation:** Matches 30B-class architectures (e.g., Nemotron-3-Nano-30B-A3B) on reasoning, coding, and math.
*   **Deployment:** Available on Hugging Face under Apache 2.0 license. Optimized for lower-memory GPUs and single-node environments.

## 3. Enterprise Integration & Tools

*   **Claude Tag (Anthropic):**
    *   An "always-on AI teammate" available to **Claude Enterprise and Team** customers.
    *   **Features:** Lives inside Slack, follows conversations, learns context, and uses an **ambient mode** to proactively flag updates and tasks.
    *   **Scoping:** Identity-based permissions allow admins to restrict which channels/teams the AI can access.
*   **MCP Connectors (Anthropic):**
    *   Launched **Enterprise-Managed Authorization (EMA)**.
    *   Allows IT admins to provision connector access via identity providers (Okta) without individual OAuth flows.
*   **Perplexity Brain (Computer Agent):**
    *   Research preview for Max/Enterprise Max subscribers.
    *   Self-improving memory system that remembers what the agent *did* rather than user preferences.
    *   Results show 25% increase in answer correctness on repeated tasks.

## 4. Industry Trends & Personnel Moves

*   **Market Dynamics:** ChatGPT market share dropped below 50% (46.4% by May 2026). Claude leads in subscription conversion (13%).
*   **Talent Shifts:**
    *   **Noam Shazeer:** Co-inventor of Transformer (Google) joins OpenAI as Lead for Architecture Research.
    *   **John Jumper:** Nobel Laureate (DeepMind) joins Anthropic for AI-for-science infrastructure.
*   **Corporate M&A:**
    *   **SpaceX** acquires **Cursor** (Anysphere) for **$60 Billion** in a Q3 2026 deal to strengthen its AI coding division.
    *   **Alibaba** released the **Qwen-Robot Suite** (Qwen-RobotNav, Manip, World) for embodied intelligence and robotic control.

Conclusion

In this tutorial, you learned how to build a personal AI web research agent that searches the web, summarizes results with a local LLM, and saves a Markdown digest. All this runs on your own machine with no data leaving your laptop. You have full control over the model and prompts without any API costs.

From here, you can try new prompts to research different topics, tweak the system prompt to change the output, swap in other local models like Qwen 3.6 or Mistral, or extend the script to fit your own workflow. Happy tinkering!

If you enjoyed this tutorial, you can find more of my writing on my blog (recent posts include system design paper series), my work on my personal website, and updates on LinkedIn.

How to Teach a Small LLM to Suggest K12 Creative Project Ideas

Srishti Sethi — Tue, 23 Jun 2026 16:37:54 +0000

Recently, I wrote a post about an educational app I'd developed using AI tools, and the design decisions I made along the way.

When I showed the prototype of my activity-based learning app to a few educators, one suggestion came up repeatedly that was drawn from their own experience hunting for creative ideas on platforms like Pinterest and TikTok. They wanted a feature that could pull project ideas from across the internet based on practical search criteria: the materials they have access to, and what they'd like the end product to look like.

The app already has a basic search that returns results from its own activity data, but that data is still limited at this stage. Generating results from outside the app felt like something LLMs are well suited to handle.

I was also curious to learn how you actually teach a K12 LLM – not the kind that needs enormous datasets and compute (which I don't have access to), but the mechanics of it, for learning's sake. And, like in my previous post, I wanted to think through the design choices that go into it:

What are the technicalities behind teaching a small LLM to handle a K12 use case?
How, and on what data, do you train such a model?
How do you ensure the model is child friendly?
What does it take to integrate the model into your app?

In this post, I'll document everything I learned about training such a model and integrating it as a feature in my educational prototype.

Prerequisites
Dataset Preparation
Filtering the Corpus
Generating Training Pairs
Fine Tuning
Evaluating the Fine-tuned Model
Building the Index & RAG Retrieval
Integrate the Model with the Feature
Making Content Safe
Conclusion
Resources

Prerequisites

This is a hands-on tutorial, so here's what will help you follow along or train the model yourself.

Skills you'll want

Using Claude on the command line.
Basic Python: reading code, installing and using packages, calling APIs, and making sense of output like log files.
Reading a bit of TypeScript, since that's what the app's frontend is built in.
Most importantly, being comfortable following Claude's reasoning, weighing the options it lays out, and deciding what to do next. That back-and-forth, not any single command, is really the core skill this kind of project asks for.

You don't need a background in machine learning. The post tries to explain the ML concepts as it goes, in plain language.

Setup you'll need

An Apple Silicon Mac (M1/M2/M3 or newer). The fine-tuning step uses MLX, Apple's framework, which only runs on Apple Silicon.
Python 3 with a virtual environment python3 -m venv).
Ollama installed, with the Qwen 2.5 7B model pulled ollama pull qwen2.5:7b), for generating the training data locally. You'll want enough RAM to run a 7B model.
Claude on the command line, for working through the build.

Dataset Preparation

For this experiment, I wanted the activity data to be grounded in local cultures from around the world. This would help the model suggest creative project ideas that inspire the facilitation of cultural activities in educational settings.

I'd come across a lot of Wikipedia articles on local arts and traditions over the years. Wikipedia is my favorite resource for information: it's human-first, its content is updated frequently, and as an open source project its APIs are free to use. So I decided to use Wikipedia data to teach my model.

The genuinely hands-on part of this stage was seeding the right categories. In a Python script, I defined ~40 seed categories and grouped them under 9 STEAM labels with suggestions from Claude on which categories to scrape and how to avoid noise in the fetched data.

For extracting text from the sections of each article, Claude suggested a Python wrapper for the Wikipedia API. This let me fetch each article as a section-structured record. To keep noise down, I limited the crawl to one sub-category level deep and only kept articles above a certain content size.

# Seed categories grouped by STEAM domain.
SEED_CATEGORIES = {
    "Crafts & making": [
        "Category:Crafts",
        "Category:Origami",
        "Category:Pottery",
        "Category:Kites",
    ],
    "Arts": [
        "Category:Folk art",
        "Category:Textile arts",
        "Category:Indigenous art",
        "Category:Masks",
    ],
    "Science": [
        "Category:Ethnobotany",
        "Category:Food preservation",
        "Category:Gardening",
    ],                                                            
# ... Media arts, Engineering, Mathematics, Music & sky, Play & learning
}

MAX_DEPTH = 1             # descend only one sub-category level
MIN_CONTENT_CHARS = 800   # skip stubs (summary + sections)

Filtering the Corpus

The previous step wrote ~19,000 articles during scraping. This step makes sure the content stays relevant to STEAM topics. Relevance filtering itself runs in two stages: removing obvious noise, then semantic filtering.

The first stage drops obvious non-activity content like music, films, TV, biographies, plant/animal species using category, title, and section-heading patterns.

The second, semantic stage converts each article's title and summary into a vector using a small sentence-transformer model (all-MiniLM-L6-v2). It then compares it against two sets of example sentences: positive and negative anchors.

The positive anchors describe sentences relevant to STEAM activities and the negative anchors describe less relevant ones. Each article gets a score based on how close it sits to the positive examples versus the negative ones, and we keep every article that leans positive. We do this with the sentence-transformers library.

Writing these anchor sentences is the most human step in the process. With this filtering, I brought the corpus down to ~6,600 articles.

# Filtering the raw scrape to articles useful for STEAM activity suggestions.

POSITIVE_ANCHORS = [
    "a hands-on craft that children can make using simple materials and a technique",
    "a traditional cultural art or making technique such as weaving, carving, pottery or paper folding",
]
NEGATIVE_ANCHORS = [
    "a species of plant, animal or fungus",
    "a biography of a person",
    "a city, region, building or geographic place",
]

    # Embed article + anchors, then keep whatever leans positive.
    pos_sim = util.cos_sim(emb, pos).max(dim=1).values # closest positive anchor 
    neg_sim = util.cos_sim(emb, neg).max(dim=1).values # closest negative anchor
    scores = (pos_sim - neg_sim).tolist()

Generating Training Pairs

The next step is to generate input → output training pairs from the filtered corpus. We do this by distilling it through a pretrained, local open-source model (Qwen 2.5 7B, running via Ollama).

For each article, you send the model the title, summary, cultural context, and a few content sections. You also send it a system prompt that explains the task, specifies the output format (valid JSON, in this case), and includes one example training pair to anchor the format.

Constructing this prompt well is where human intervention matters most: the schema, the rules, and that single worked example are what determine the quality of every pair the model generates.

After generation, we cleaned and prepared the pairs for fine-tuning. The local model tended to invent its own category labels ("Ceramics," "Crafts & Making," "Circuits (metaphorical)"…). So this step maps every category onto the app's fixed set of 10 canonical categories (Art, Science, Coding, Circuits, Engineering, Storytelling, Drama, Film, Music, Nature), clamps each activity's age range into the K12 band, converts the pairs into chat format, and finally splits the data into three sets: train, validate, and test.

# The schema every generated training pair must match (valid JSON only).
  {
    "input": {
      "materials": ["3-6 realistic classroom materials"],
      "age_range": [min_int, max_int],
      "theme": "optional string or null"
    },
    "output": {
      "ideas": [{
        "title": "catchy, max 60 chars",
        "description": "2-3 sentences",
        "category": "one of: Art, Science, Coding, Circuits, Engineering, ...",
        "cultural_origin": "specific region or culture",
        "materials_used": ["subset of input materials"],
        "materials_missing": ["anything else needed"],
        "estimated_minutes": integer,
        "steps": ["3-6 short steps, one sentence each"],
        "learning_objectives": ["2-4 objectives"],
        "safety_note": "string or null"
      }]
    }
  }

Fine-Tuning

This is the step where the model learns how to behave and generate a desired response in the appropriate format. It involves fine-tuning a pretrained model (Qwen2.5-1.5B-Instruct-4bit in this case) via MLX on my dataset using the LoRA technique.

Fine-tuning with LoRA is a cheap and lightweight approach: it doesn't retrain the whole model, but instead adds a tiny correction layer that adjusts the final behavior while the original model stays frozen.

Given the constraints of this project, working on a personal laptop with a small dataset of ~400 pairs, full fine-tuning would have needed significantly more memory and compute, which would be overkill here. So LoRA was the right choice.

The LoRA Fine-tuning Cycle:

Training runs many iterations over the training pairs, and each iteration is the same short cycle. For each input, the model produces a prediction by assigning a probability score to every possible next word, based on the input and the model's current weights. During training it is then graded on how much probability it gave the actual correct next word from the training data.

(Note: in a neural network, weights and biases are the numbers that determine how the model processes an input, makes a prediction, and generates a response.).

From that comparison it calculates the train loss. It then updates the weights accordingly, specifically the small LoRA adapter weights, while the frozen base model stays untouched, so that next time the guess is a little closer. The lower the loss, the better the model is fitting the data.

Then it moves on to the next iteration, and the cycle repeats. At the end, the trained adapter weights are saved out to a safetensors file.

For example, here is how the validation loss moved over my run: 2.532 → 0.842 → 0.823 → 0.814 → 0.820 → 0.831 → 0.845. It dropped sharply at first (the model was genuinely learning), bottomed out at 0.814 around iteration 300, then ticked back up to 0.845 by the end. This was early sign that the model was starting to overfit, that is memorize the training data rather than continue improving.

So the sweet spot was the middle of the run, not the very end. This is where human review mattered most: I saved checkpoints at iterations 200, 400, and 600, and chose the 400 checkpoint, the one with the lowest validation loss among them, to evaluate and serve.

# Base model — small, instruction-tuned, 4-bit (runs on a laptop)
  model: "mlx-community/Qwen2.5-1.5B-Instruct-4bit"

  train: true
  data: "data/mlx"            # training data: train.jsonl + valid.jsonl
  adapter_path: "adapters"    # <- the trained LoRA weights get saved here

  fine_tune_type: lora
  num_layers: 8               # apply LoRA to the last 8 transformer layers only
  lora_parameters:
    rank: 8                   # adapter size — bigger = more capacity, more overfit risk

  # Training loop
  batch_size: 4               # 400 train examples / 4 = 100 iterations per epoch
  iters: 600                  # ~6 passes over the training set
  learning_rate: 1e-5

  # Watch validation loss to catch overfitting
  steps_per_eval: 100         # check validation loss every 100 steps
  save_every: 200             # checkpoint adapters at 200 / 400 / 600

Above is the configuration file. It shows the model used, the adapter path, the fine-tuning and LoRA settings, the training loop, and the validation pass.

Below is the command, run with MLX (Apple's machine learning framework), that kicks off the fine-tuning process:

mlx_lm.lora --config lora_config.yaml

The output below shows the result: the trained weights land in the adapters/ folder, with a checkpoint saved every 200 iterations at 200, 400, and 600.

  adapters/
  ├── 0000200_adapters.safetensors
  ├── 0000400_adapters.safetensors   <- the one you serve (lowest val loss of the three)
  ├── 0000600_adapters.safetensors
  └── adapters.safetensors           <- copy of the final (600) weights

Evaluating the Fine-tuned Model

Once fine-tuning was done, the model needed to be evaluated on the held-out test set, the 50 examples set aside during the training-pair generation step and never seen during training.

In this step, the user message is fed to the model, the model generates its own JSON answer, and that answer is compared against the gold (correct/reference) answer already stored in the file.

The evaluation checks and reports whether the JSON is valid, whether it has the expected keys, how much the predicted materials overlap with the gold answer, how often the prediction names a specific cultural origin, and so on.

It runs this for every example in the test set, printing a short per-example line and a summary at the end. It saves the full results, including each predicted idea alongside the actual (gold) idea, so you can read them side by side.

# Fine-tuned model on 50 held-out test examples:
  {
    "json_valid_rate":       1.00,   # always valid JSON
    "schema_match_rate":     1.00,   # always the right keys
    "avg_n_steps":           4.74,   # ~5 steps per idea
    "avg_materials_jaccard": 0.653,  # decent overlap with gold materials
    "pred_culture_specific_rate": 0.52,   # names a specific culture about half the time
    "culture_loose_match_rate":   0.108,  # but it's usually the WRONG one  <-- the gap RAG tries to close
  }

Building the Index & RAG Retrieval

In the previous step we found that culture_loose_match_rate_when_gold_specific was low: the model is bad at recalling the right cultural origin for a suggested activity.

In this step, we'll try to address that weakness with RAG (retrieval-augmented generation). Instead of hoping that the model has memorized that Raku is Japanese, we'll look up the real Wikipedia article at query time, hand it to the model, and then test whether retrieval actually helps.

This happens in two parts. First, we'll build a retrieval index, turning the Wikipedia corpus we collected earlier into a searchable "meaning database." For each article we compute an embedding by passing its title and summary through a small embedding model, all-MiniLM-L6-v2. An embedding is a numeric fingerprint of meaning, a row of 384 numbers, and articles with similar meaning end up with similar numbers. These are computed once, offline, and saved to disk.

Second comes the retrieval itself. At query time, we turn the query into the same kind of vector, score every article by how similar it is, and return the few with the highest scores (that is, the articles whose meaning is closest to what the user asked for). We then run the same evaluation as the previous phase, but with these retrieved articles pasted into the prompt, to answer the core question: when the model is handed the right Wikipedia article, does it do better?

In a nutshell, this phase is: retrieve the relevant articles, augment the prompt with them, and let the model generate.

def retrieve(query, embedder, embeddings, meta, k):
      # 1. turn the query into the same kind of 384-number vector
      q = embedder.encode([query], normalize_embeddings=True,
                          convert_to_numpy=True)[0]
      # 2. score every article by similarity (dot product of unit vectors = cosine)
      sims = embeddings @ q
      # 3. take the k closest, return them with their scores
      top = np.argsort(-sims)[:k]
      return [(meta[i], float(sims[i])) for i in top]

So with RAG, the materials overlap improved and the model named a specific culture more often – but the exact cultural match barely moved. This is something I would like to improve in future versions of the app.

Metric                        Plain     + RAG     Change
materials_jaccard             0.653     0.752     better
pred_culture_specific_rate    0.52      0.64      better
culture_loose_match_rate      0.108     0.135     barely

Integrate the Model with the Feature

Now it's time to integrate the fine-tuned model into the app and see what cultural activities it can generate to inspire educators.

The end-to-end flow starts on a "Suggest" screen, where an educator enters the materials they have on hand and, optionally, a theme for the activity. From there, the suggestion happens in two phases: retrieval, then generation.

First, the app does a vector search over the Wikipedia index and populates a grid of culturally-specific articles that match the educator's input. No model is involved, so the grid appears instantly.

Then, when you tap a card, you land on a detail screen where the fine-tuned model generates a full STEAM activity grounded in that single tradition: a title, description, materials, step-by-step instructions, learning objectives, and a safety note. Everything needed to guide the activity in the classroom.

 // Step 1 — RETRIEVAL: educator's materials -> grid of cultural articles.
  // Pure vector search on the server, no model, so the grid appears instantly.
  export async function fetchInspiration(materials: string[], theme?: string) {
    const res = await fetch(`${BASE_URL}/suggest`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ materials, theme: theme ?? null }),
    });
    return res.json();   // { results: [...articles] }
  }

  // Step 2 — GENERATION: runs only when the educator taps ONE card.
  // The fine-tuned model generates a full activity grounded in that article.
  export async function fetchActivity(
    articleId: number,
    materials: string[],
    ageRange: [number, number],
  ) {
    const res = await fetch(`${BASE_URL}/activity`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ article_id: articleId, materials, age_range: ageRange }),
    });
    return res.json();   // { activity: {...}, article: {...} }
  }

Splitting browsing from generation this way is both a cost and a quality choice: retrieval is essentially free, so the model runs just once on the tradition the educator actually commits to, rather than once for every card on the grid.

Making Content Safe

I wanted to talk about this topic explicitly at the end, even though many phases of the pipeline already involve steps to keep the model's content safe.

Even though the direct users of the app are educators, anything this feature produces can end up in front of kids. So we never want to surface or generate steps for intoxicants, drugs, tobacco, weapons, explosives, or poisons – basically any content that isn't age-appropriate.

This is something the model won't automatically handle on its own. The fine-tuned model was trained only on cultural-craft examples, so it has no built-in instinct to refuse an unsafe request, and the general knowledge of things like alcohol and weapons still lives in the base model's weights underneath.

As a builder, you have to put the necessary guards and checkpoints in place, and remind the model how to behave. We do this in two phases:

Pre-filter the data to reduce risk at the source, the same way we dropped unrelated categories earlier. Screening the corpus (and the generated training pairs) means we never teach the model unsafe content in the first place. This matters especially if you ever plan to publish your model or dataset somewhere like Hugging Face, where it should already be filtered. This step removed ~850 unsafe articles from the ~19,000 scraped.
Keep runtime guardrails in the ZubHub app as the actual guarantee. Because data filtering reduces risk but can't erase what the base model already knows, the live app screens every input before retrieval and every generated output before display. This means that nothing built around unsafe terms is ever retrieved or shown.

# safety.py — one shared list of what we never surface to kids...
  UNSAFE_TERMS = { 
      # ...
  }

  # ...matched whole-word, so "twine" != "wine" and "gunny sack" != "gun".
  def screen_text(text):
      """Return the first unsafe category found, or None if the text is clear."""
      for category, pattern in _PATTERNS.items():   # _PATTERNS built from UNSAFE_TERMS
          if pattern.search(text):
              return category
      return None

  # Phase 1, data: drop unsafe articles before they ever reach training.
  for article in corpus:
      if screen_text(article["title"] + article["summary"]):
          continue                      # never taught to the model

  # Phase 2, runtime: screen the educator's input AND the model's output.
  if screen_text(user_input):           # before retrieval
      return BLOCK_MESSAGE
  answer = model.generate(...)
  if screen_text(answer):               # before anything is shown
      return BLOCK_MESSAGE

Conclusion

In a nutshell, this article walked through how you teach a small LLM to suggest creative, hands-on projects for an educational app.

We started from a pretrained model, Qwen2.5-1.5B-Instruct, and taught it on a dataset we built from Wikipedia's STEAM and cultural articles.

The goal was to get it to take a simple input (the materials an educator has, the children's age range, and an optional theme) and respond with a structured JSON activity: a title, description, step-by-step instructions, learning objectives, and a safety note.

Along the way, we worked through the technicalities of adapting a small LLM for a K12 use case end to end: building the dataset with the Wikipedia API, filtering out irrelevant categories and unsafe content, generating training pairs, fine-tuning the model with LoRA, evaluating its quality, building a retrieval index and adding RAG to make the suggestions more grounded and specific, and finally integrating the model into the app.

Most importantly, building it this way as a hands-on project is what made the core ideas of the ML/LLM space click for me, rather than staying abstract. I hope it does the same for you!

Resources

Check out the source code in this specific PR.

From LLMs to LangChain: Understanding How Modern AI Applications Actually Work

Sudheesh Shetty — Tue, 23 Jun 2026 15:49:39 +0000

Typically, when we start experimenting with AI, many of us begin similarly. We try a single LLM call as the core of an app, like this:

const response = await llm.chat("Explain Kubernetes");

For a little while it feels like the whole flow is: the user asks something, and the model returns an answer. That early success often creates a false impression that building AI is just about sending prompts and getting responses.

That simplicity is seductive, but it doesn't hold up. Over time, users want the assistant to find answers in their documents and knowledge bases, call APIs, fetch live data, or trigger services or schedule meetings.

Users also expect the agent to access internal systems and interact with ERPs, CRMs, or other tools holding critical business data. They'll want agents to combine multiple steps, as workflows often require chaining queries, computations, and side effects into reliable processes.

This is where concepts like MCP (the Model Context Protocol) and tools like LangChain come in. Initially, they may seem like buzzwords, but they address different aspects of LLM production.

After experimenting with AI tools, I found that these concepts help solve different problems related to interfaces, orchestration, and system integration.

This article is a practical guide to understanding how LLMs connect with tools, orchestrate workflows, and power real AI applications.

Here’s what we’ll cover:

What Is an LLM?
Why LLMs Need Tools
Where MCP Comes In
So What Does LangChain Actually Do?
Putting It Together
What I Built While Learning This

Throughout the article we'll discuss what LLMs are and how they work, what tool-calling looks like in practice, what MCP is and how it works, how LangChain fits into the whole process, and how to put all these tools together.

To follow along, you'll need a basic understanding of Node.js, API operations, and basic JavaScript concepts.

What Is an LLM?

LLM stands for Large Language Model. It's a class of deep neural networks trained on massive amounts of text to model and generate human-like language. Popular examples you might have heard of include GPT, Claude, Gemini, and Llama.

How to Call an LLM From a Node.js Application

Before writing code, let’s understand what it means to call an LLM from a Node.js application.

Calling an LLM means sending input from your application to an AI provider’s API and receiving generated output in return. It's similar to calling any other external service.

In most real-world applications, the model isn't hosted or trained by your application. Instead, providers such as OpenAI and Groq host and maintain the models, while your application communicates with them over HTTP APIs.

In this example, we’ll build a minimal API using Node.js and Express. We’ll create a simple POST /chat endpoint that accepts a user message, sends it to the OpenAI API, receives the generated response, and returns it to the client.

Here, our Node.js server acts as the bridge between the user and the LLM provider.

For this example, create an API key from the Groq console. Since it offers a free tier, it’s a simple way to experiment and understand the concepts.

First, install the dependencies:

npm install express

import express from "express";

const app = express();
app.use(express.json());

app.post("/chat", async (req, res) => {
  const { message } = req.body;
  const response = await fetch("https://api.groq.com/openai/v1/chat/completions", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: GROQ_API_KEY,
    },
    body: JSON.stringify({
      model: "llama-3.3-70b-versatile",
      messages: [{ role: "user", content: message }],
    }),
  });

  const data = await response.json();

  if (!response.ok) {
    return res.status(response.status).json({ error: data });
  }

  const reply = data.choices[0].message.content;

  res.json({ reply });
});

const PORT = process.env.PORT || 8888;
app.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
});

Start the server and make a request. Use Postman and do a POST request to /chat using the below body:

POST /chat

{
  "message": "Explain Kubernetes"
}

Example response:

{
  "reply": "Kubernetes is a container orchestration platform..."
}

The backend receives the message, forwards it to the model provider, receives generated text, and returns it to the client.

LLMs are excellent at language-centric tasks: they understand phrasing and intent, generate coherent text, extract structured information from unstructured input, and perform basic reasoning over provided context. These capabilities make them powerful for things like summarization, drafting, and conversational QA.

But there’s an important limitation: LLMs don't automatically know about and can't access your private or live data. They don’t have implicit access to your company database, internal APIs, or the current state of your systems unless you provide that information at runtime.

Because of that limitation, you need secure mechanisms to connect models to live systems and data — which brings us to the idea of tools.

Why LLMs Need Tools

Imagine asking:

Check my order and raise support if delivery is delayed.

The model alone can't inspect your order database or create a support ticket in your system. To do that, it must call external functions — for example, a getOrderStatus(orderId) API and a createSupportTicket(orderId, issue) action.

Those callable functions are what we call tools: programmatic interfaces the AI can use to interact with systems and take concrete actions on behalf of users.

A tool is simply a function that an AI model can call to interact with external systems or perform actions.

For example, imagine we have a getOrderStatus(id) function that returns an order’s delivery status.

To expose this to the LLM, we define a tools array. Each tool includes:

type – currently "function"
function name – the function identifier
function description – helps the LLM decide when to call the tool
function parameters – a JSON Schema describing the arguments the tool expects

Here's an example:

function getOrderStatus(id) {
  const statuses = ["pending", "success", "cancelled"];
  const status = statuses[Math.floor(Math.random() * statuses.length)];
  return `Your order status is ${status}.`;
}

const tools = [
  {
    type: "function",
    function: {
      name: "getOrderStatus",
      description: "Get the status of an order by its ID",
      parameters: {
        type: "object",
        properties: {
          id: { type: "string", description: "The order ID" },
        },
        required: ["id"],
      },
    },
  },
];

The above tool format is for Grok. Different LLM providers may use different formats for defining tools, but the overall idea remains the same.

When making the API call, we pass both the user messages and the list of available tools.

body: JSON.stringify({
    model: "llama-3.3-70b-versatile",
    messages: [{ role: "user", content: message }],
    tools,
}),

After the API call, the LLM decides whether a tool is needed. If a tool call is requested, our application executes the corresponding function and sends the result back to the model.

For this example, we'll only handle the getOrderStatus tool. We can check whether the model requested a tool call like this:

const toolCall = data.choices[0].message.tool_calls[0];
const { id } = JSON.parse(toolCall.function.arguments);
const toolResult = getOrderStatus(id)

and later we can pass the message context with tool result

body: JSON.stringify({
    model: "llama-3.3-70b-versatile",
    messages: [
        { role: "user", content: message },
        assistantMessage,
        { role: "tool", tool_call_id: toolCall.id, content: toolResult },
    ],
    tools,
}),

Finally, return the response:

return res.json({ reply: followUpData.choices[0].message.content });

Here's a diagram of the flow:

LLM -> Tool Execution -> Tool Result -> Final Response" style="display:block;margin:0 auto" width="1774" height="887" loading="lazy">

The LLM decides whether a tool is needed and generates the required inputs, while your application executes the function.

Where MCP Comes In

Tools are simple. You define functions and tell the AI what it can use.

For example, getOrderStatus() works well when all tools are built inside your application. But as applications grow, tools may come from many places, like Slack, GitHub, databases, internal systems, or third-party services. Each one may expose tools differently.

This is where MCP (Model Context Protocol) helps. Think of MCP as a common language that lets AI systems connect to external tools in a consistent way.

Tools define what the AI can do. MCP standardizes how the AI connects to and uses those tools.

Now let’s extend the previous /chat API example so the LLM can use tools exposed through MCP. There are multiple ways to do this:

build and host your own MCP server and expose your application functions
connect to existing third-party MCP servers such as Slack

For this tutorial, we'll keep things simple and use a remote MCP server approach because it's easier to understand.

npm install express @modelcontextprotocol/sdk zod

Now let’s create our own MCP server and expose the same getOrderStatus function as an MCP tool:

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { createMcpExpressApp } from "@modelcontextprotocol/sdk/server/express.js";
import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js";
import { z } from "zod";

function getOrderStatus(id) {
  const statuses = ["pending", "success", "cancelled"];
  const status = statuses[Math.floor(Math.random() * statuses.length)];
  return `Your order status is ${status}.`;
}

function createOrderServer() {
  const server = new McpServer({ name: "order-server", version: "1.0.0" });

  server.registerTool(
    "getOrderStatus",
    {
      description: "Get the status of an order by its ID",
      inputSchema: { id: z.string() },
    },
    async ({ id }) => ({
      content: [{ type: "text", text: getOrderStatus(id) }],
    })
  );

  return server;
}

const app = createMcpExpressApp({ host: "0.0.0.0" });

app.post("/mcp", async (req, res) => {
  const server = createOrderServer();
  const transport = new StreamableHTTPServerTransport({
    sessionIdGenerator: undefined,
  });

  res.on("close", () => {
    transport.close();
    server.close();
  });

  await server.connect(transport);
  await transport.handleRequest(req, res, req.body);
});

const PORT = process.env.PORT || 3001;
app.listen(PORT, "0.0.0.0", () => {
  console.log(`Order MCP server running on http://0.0.0.0:${PORT}/mcp`);
});

This is useful when you want to expose your own application functions through MCP. Typically, the MCP server runs separately and is accessed by MCP clients. Now any MCP client can connect to this server and discover the available tools automatically.

The same idea applies to third-party MCP servers.

For example, if a Slack MCP server is available, we can connect to it instead of writing Slack integration code ourselves.

In that case, our application isn't directly calling Slack APIs. It connects to the Slack MCP server, which exposes Slack-related tools using the MCP standard.

So the difference is:

For our own features, we can build our own MCP server
For external systems, we can use existing MCP servers when available

Now we can pass MCP servers to the LLM request:

body: JSON.stringify({
  model: "llama-3.3-70b-versatile",
  messages: [{ role: "user", content: message }],
  tools: [
    {
      type: "mcp",
      server_label: "OrderServer",
      server_url: `http://0.0.0.0:${PORT}/mcp`,
      server_description: "Get the status of an order by its ID",
    },
    {
      type: "mcp",
      server_label: "Slack",
      server_url: "https://mcp.slack.com/mcp",
      server_description: "Send and read Slack messages",
      headers: {
        Authorization: `Bearer ${process.env.SLACK_BOT_TOKEN}`,
      },
    },
  ],
})

We can also use local MCP servers instead of remote URLs by connecting through transports such as StdioClientTransport. In that case, we connect locally, discover the available tools, and expose them to the LLM.

Now if the user sends:

{
  "message": "What is status of order 123"
}

The LLM decides whether a tool is needed, MCP exposes and executes the tool, and the final response is returned to the user.

The flow becomes:

/chat api -> LLM -> MCP Tool -> Tool Result -> Tool Response" style="display:block;margin:0 auto" width="1774" height="887" loading="lazy">

This standardization makes integrations far more reusable: instead of rewriting glue logic for each new connector, teams can register MCP-compliant tools and let the orchestrator and model handle discovery and invocation.

So What Does LangChain Actually Do?

I initially thought LangChain was simply another wrapper around LLM APIs, but it is better understood as an orchestration framework for AI workflows. Tools let an LLM perform actions. MCP standardizes how tools are exposed. LangChain helps coordinate models, tools, and application logic to build multi-step workflows.

For example:

User: Find flights, compare prices, book hotel, send confirmation.

Now the system may need to:

Check order status
Decide whether support is needed
Create a support ticket
Generate the final response

Without orchestration, you would manually control each step. LangChain helps manage this flow.

To use LangChain, Install the required packages:

npm install express langchain @langchain/groq

We'll reuse the same tool functions from earlier:

import express from "express";
import { createAgent } from "langchain";
import { ChatGroq } from "@langchain/groq";

const app = express();
app.use(express.json());

const agent = createAgent({
  model: new ChatGroq({
    model: "llama-3.3-70b-versatile",
    apiKey: GROQ_API_KEY,
  }),
  tools: [
    {
      name: "getOrderStatus",
      description:
        "Get order status",
      execute: ({ id }) =>
        getOrderStatus(id), // we have this function above
    },
    {
      name: "createSupportTicket",
      description:
        "Create support ticket",
      execute: ({ id }) =>
        createSupportTicket(id), //imagine a function that creates a support ticket
    },
  ],
});

app.post(
  "/chat",
  async (req, res) => {
    const { message } = req.body;

    const response =
      await agent.invoke({
        messages: [
          {
            role: "user",
            content: message,
          },
        ],
      });

    res.json({
      reply:
        response.messages
          ?.at(-1)
          ?.text,
    });
  }
);

app.listen(3000);

Now the flow becomes:

LangChain doesn't replace tools or MCP. It sits above them and coordinates how everything works together.

Putting It Together

A modern AI application usually has multiple layers working together. The LLM handles reasoning and language generation. Tools perform real operations such as reading data, calling APIs, or executing actions. MCP helps standardize how those tools are exposed and accessed. LangChain helps orchestrate the interaction between models, tools, and workflows.

By separating these responsibilities, applications become easier to extend, maintain, and scale.

The goal is more than just generating text. You want to be able to build systems that can reason, retrieve information, take actions, and reliably solve real user problems.

LLM -> LangChain -> MCP -> Tools -> Systems & Data" style="display:block;margin:0 auto" width="1536" height="1024" loading="lazy">

What I Built While Learning This

After understanding the concepts above, I wanted to reduce some of this setup for my own projects. As I experimented, I noticed most applications recreate the same plumbing over and over: connecting an LLM, wiring up tools, managing execution, and exposing orchestration patterns.

So I built a small open-source toolkit to reduce that setup. The goal was simple: you should be able to focus on business logic instead of wiring AI infrastructure.

Current capabilities:

LLM integration
Tool registration
Tool execution
Chat orchestration
LangChain support
Extensible architecture

Packages:

AI Chat Widget: https://www.npmjs.com/package/ai-chat-toolkit-widget

AI Chat Server: https://www.npmjs.com/package/ai-chat-toolkit-server

GitHub Repository: https://github.com/sudheeshshetty/ai-chat-toolkit

To build a server using the toolkit:

npm install express ai-chat-toolkit-server

Create the chat server:

const aiChat = new AiChatServer({
  path: "/my-chat",
  provider: "groq",
  apiKey: process.env.API_KEY,
  model: process.env.MODEL || "llama-3.3-70b-versatile",
  cors: {
    origin: "http://localhost:5174",
  },
  orchestration: "langchain",
  maxToolRounds: 6,
  systemPrompt:
    "You are a helpful operations assistant for a demo store. Keep answers concise.",
});

Add your tools:

aiChat.addTools([
  {
    name: "...",
    description: "...",
    inputSchema: { ... },
    handler: async (input) => { /* runs in Node */ },
  },
]);

Attach it to your Express app:

aiChat.attach(app);

Now /my-chat is exposed in your Express server and can be used directly.

You can also use ai-chat-toolkit-widget if you want to skip building the chat UI.

Examples are available in the repository, so you can try it out quickly.

A quick glance of one of the examples:

If you find it useful, I’d appreciate a star, feedback, or contributions on GitHub as I continue improving the developer experience and exploring new ideas.
Thanks for reading — I hope this helped make LLMs, tools, MCP, and LangChain feel a little less magical and a lot more practical.

How to Handle Small Context Window Limits in RAG Systems

Sviatoslav Barbutsa — Thu, 18 Jun 2026 00:09:31 +0000

Retrieval-augmented generation, or RAG, is a pattern where an application retrieves relevant source material and adds it to a model prompt so the model can answer from that context.

A larger context window in a RAG system shouldn't be treated as a substitute for good context management, although it can make the experience more forgiving for the end user. It's like running unoptimized graphics on a powerful GPU: the extra capacity can hide inefficiency for a while, but it doesn't eliminate the underlying optimization problem.

But even a very large context window still has a hard limit. If you keep adding tokens, you can eventually exceed it. This problem becomes more visible on consumer hardware, where limited memory and compute usually mean smaller usable context windows.

I ran into this problem while experimenting with local models on a consumer laptop with 12 GB of VRAM. RAG worked well for small tests but as soon as the documents got larger, the system would retrieve useful chunks and still fail to answer well.

The issue wasn't always retrieval. Sometimes the right chunk had been found, but the final prompt didn't have room for it.

This article walks through the solution I implemented for this problem:

Document summary → chunk summary → raw chunk → final answer

The pattern is based on three rules:

Use summaries for retrieval.
Use raw chunks for answering.
Use a context budget to decide what reaches the model.

To keep the demo simple and convenient, the companion repository uses small Python and TypeScript examples with a simplified in-memory retrieval store and a simplified answer extractor. This lets you see the article’s core ideas in practice without installing a full stack of dependencies, downloading models, running a Large Language Model (LLM) server, setting up an embedding service, or configuring a vector database.

That setup process could easily become its own dedicated article, so this tutorial keeps the runnable examples focused on the small-context RAG pattern: summaries for retrieval, raw chunks for answers, and a visible context budget.

The repo demonstrates the data flow and debugging pattern rather than production-grade model quality. In production, you'd want to replace the simplified summarizer, in-memory similarity search, and token estimator with your own model, embedding store, reranker, and tokenizer.

What You Will Implement
Prerequisites
Why Basic RAG Can Fail with a Small Context Window
How Summary Routing Works
How to Represent Documents and Chunks
How to Split Documents into Raw Chunks
How to Summarize Chunks and Documents
How to Recursively Reduce Summaries
How to Implement the Hierarchical Index
How to Retrieve Through Summaries
How to Implement a Budgeted Raw Context
How to Run the Demo
How to Interpret the 250 vs 1200 Token Test
How This Relates to Existing RAG Techniques
When to Use This Pattern
Conclusion

What You Will Implement

In this tutorial, you'll implement a small educational RAG pipeline that manages context window limitations by processing documents across three levels:

Document records contain a short summary used to choose likely documents.
Chunk records contain a short summary used to choose likely chunks inside those documents, plus the raw source text.
Raw context contains selected raw chunks packed into a fixed token budget.

The important distinction is that summaries are only used to decide where to look. They're not used as final evidence.

That matters because summaries are lossy. They compress information, and they may leave out the detail needed to answer the user's question. Raw chunks, by contrast, are larger, but they preserve the original wording.

The demo prints a trace for every question:

Document summary hits
Chunk summary hits
Raw chunks included
Raw chunks skipped
Answer

That trace is the debugging interface. It shows whether retrieval failed, or whether prompt assembly skipped useful evidence because the context budget was too small.

Prerequisites

To follow along, you need one of these:

Python 3.10 or newer

or:

Node.js 22 or newer
npm

You'll get the most out of this article if you're already comfortable with:

basic Python or TypeScript syntax
running commands in a terminal
reading small data classes, functions, and lists or maps
the general idea of an LLM prompt and context window
the basic RAG idea: retrieve relevant source text, add it to a prompt, and answer from that context

You don't need prior experience with vector databases, embedding APIs, LangChain, LlamaIndex, or local LLM setup.

The examples don't require an LLM provider, an embedding API, or a vector database. They use:

sentence extraction as a stand-in for LLM summarization
bag-of-words cosine similarity as a stand-in for embedding search
fixed character-based token estimates as a stand-in for a tokenizer

I made these implementation choices to save you time and make the examples easier to try, while preserving the original purpose. They also make the retrieval path visible.

Why Basic RAG Can Fail with a Small Context Window

The basic RAG loop usually looks like this:

Load documents → split documents into chunks → embed chunks → retrieve the top chunks → put retrieved chunks into the prompt → ask the model to answer.

This is a good starting point. But it hides two different problems inside one phrase: "retrieve the top chunks."

First, you need to find relevant material. That's retrieval quality.

Second, you need to decide which retrieved material actually fits in the final prompt. That's context budgeting.

On a large hosted model, you may not notice this problem right away. On a local model or a smaller context window, you'll notice it quickly.

The failure mode looks like this:

The retriever finds useful chunks.
The prompt builder tries to add them.
The context budget fills up.
Some chunks are skipped.
The final model never sees those skipped chunks.
The answer is incomplete or says "I do not know."

This can feel confusing when you inspect retrieval and see that the relevant chunk was returned. But retrieval returning a chunk isn't the same thing as the model seeing that chunk.

If you develop RAG systems on constrained hardware, this distinction becomes important.

How Summary Routing Works

Instead of searching all raw chunks directly, you can create a routing layer out of summaries.

At indexing time:

Load documents.
Split each document into chunks.
Summarize each chunk.
Reduce chunk summaries into one document summary.
Store document summaries in a document-summary store.
Store chunk summaries in per-document chunk-summary stores.
Keep raw chunks in a lookup table.

Here's what the indexing pipeline looks like:

At question time:

Search document summaries to choose likely documents.
Search chunk summaries only inside those documents.
Convert chunk-summary hits back to raw chunk IDs.
Optionally add neighboring chunks.
Pack raw chunks into the final context budget.
Answer from raw chunks only.

The query path uses the summaries for routing, then switches back to raw chunks before answering:

This gives you two useful properties:

Summaries make retrieval cheaper.
Raw chunks keep answers grounded.

It also gives you a place to debug. If the system gives a weak answer, inspect the trace. Did the right document summary match? Did the right chunk summary match? Did the raw chunk fit in the final context? Did it get skipped because of the budget?

How to Represent Documents and Chunks

The data structures are intentionally small because they contain only the essential information needed for this pipeline. In a real system, you would probably add more metadata.

Here's the Python version:

from dataclasses import dataclass

@dataclass(frozen=True)
class SearchDocument:
    page_content: str
    metadata: dict[str, str | int]

@dataclass(frozen=True)
class DocumentRecord:
    doc_id: str
    source: str
    text: str
    summary: str

@dataclass(frozen=True)
class ChunkRecord:
    chunk_id: str
    doc_id: str
    source: str
    index: int
    text: str
    summary: str
    previous_chunk_id: str | None
    next_chunk_id: str | None

The DocumentRecord stores the full document and a summary. The ChunkRecord stores the raw chunk, its summary, and links to the previous and next chunks.

Those neighbor links are useful because chunk boundaries are artificial. If retrieval finds chunk 4, the answer may start in chunk 3 or continue into chunk 5.

The index keeps both searchable stores and lookup maps:

@dataclass(frozen=True)
class HierarchicalIndex:
    documents_by_id: dict[str, DocumentRecord]
    chunks_by_id: dict[str, ChunkRecord]
    chunks_by_doc_id: dict[str, list[ChunkRecord]]
    document_summary_store: SimpleVectorStore
    chunk_summary_stores_by_doc_id: dict[str, SimpleVectorStore]

The most important lookup is this:

chunk = index.chunks_by_id[chunk_hit.metadata["chunk_id"]]

That line converts a retrieved summary hit back into the raw source text used for the final answer.

How to Split Documents into Raw Chunks

The demo splits Markdown files by paragraph and groups paragraphs until a target character size is reached:

CHUNK_SIZE = 420

def split_text(text: str) -> list[str]:
    chunks = []
    current_paragraphs = []
    current_size = 0

    for paragraph in re.split(r"\n\s*\n", text.strip()):
        paragraph = paragraph.strip()

        if not paragraph:
            continue

        if current_paragraphs and current_size + len(paragraph) > CHUNK_SIZE:
            chunks.append("\n\n".join(current_paragraphs))
            current_paragraphs = []
            current_size = 0

        current_paragraphs.append(paragraph)
        current_size += len(paragraph)

    if current_paragraphs:
        chunks.append("\n\n".join(current_paragraphs))

    return chunks

One important thing: this isn't the perfect splitter for every use case. It's intentionally readable.

In a production system, you might use a tokenizer-aware splitter, Markdown-aware sections, semantic chunking, or parent-child chunking. But regardless of the option you pick, the idea stays the same: keep raw chunks as the final evidence.

How to Summarize Chunks and Documents

To keep the demo easy to run, this article uses sentence extraction as a stand-in for LLM summarization. It scores sentences that include important RAG terms and keeps the top sentences.

def summarize_text(text: str, max_sentences: int = 2) -> str:
    sentences = [
        sentence.strip()
        for sentence in re.split(r"(?<=[.!?])\s+", " ".join(text.split()))
        if sentence.strip()
    ]

    if len(sentences) <= max_sentences:
        return " ".join(sentences)

    scored_sentences = []

    for position, sentence in enumerate(sentences):
        sentence_words = words(sentence)
        term_score = sum(3 for word in sentence_words if word in IMPORTANT_TERMS)
        first_sentence_bonus = 1 if position == 0 else 0
        scored_sentences.append((term_score + first_sentence_bonus, position, sentence))

    selected = sorted(scored_sentences, key=lambda item: (-item[0],item[1]))[:max_sentences]
    selected.sort(key=lambda item: item[1])

    return " ".join(sentence for _score, _position, sentence in selected)

In a real system, this function would call a small local model or a hosted model. The prompt instructions would be something like:

Summarize this chunk for retrieval.
Preserve names, constraints, decisions, errors, numbers, and domain-specific terms.
Don't answer a user question.

Note that the chunk summary isn't supposed to replace the raw chunk. Its only goal is to make retrieval easier.

How to Recursively Reduce Summaries

A common mistake is to create a document summary by putting every chunk summary into one prompt:

combined = "\n\n".join(chunk_summaries)
document_summary = summarize(combined)

That works for a few chunks, but it doesn't work for hundreds of chunks. You have only moved the context-window problem from answer time into indexing time.

A better approach is to reduce summaries in batches:

Chunk summaries → budgeted batches → batch summaries → higher-level summaries → final document summary.

The reduction process looks like this:

Here is the budgeted packing function:

def pack_summaries_by_token_budget(
    summaries: list[str],
    token_budget: int,
) -> list[list[str]]:
    batches = []
    current_batch = []
    current_tokens = 0

    for summary in summaries:
        summary_tokens = approximate_tokens(summary)

        if current_batch and current_tokens + summary_tokens > token_budget:
            batches.append(current_batch)
            current_batch = []
            current_tokens = 0

        current_batch.append(summary)
        current_tokens += summary_tokens

    if current_batch:
        batches.append(current_batch)

    return batches

And here is the recursive reduction loop:

def recursively_reduce_summaries(summaries: list[str]) -> str:
    if not summaries:
        return "No summary available."

    current_summaries = summaries
    level = 1

    while len(current_summaries) > 1:
        batches = pack_summaries_by_token_budget(
            current_summaries,
            SUMMARY_REDUCTION_INPUT_TOKEN_BUDGET,
        )

        if len(batches) == len(current_summaries):
            batches = force_summary_reduction_progress(current_summaries)

        print(
            f"Reducing {len(current_summaries)} summaries into "
            f"{len(batches)} batch summaries at level {level}"
        )

        current_summaries = [reduce_summary_batch(batch) for batch in batches]
        level += 1

    return summarize_text(current_summaries[0], max_sentences=3)

The fallback matters:

if len(batches) == len(current_summaries):
    batches = force_summary_reduction_progress(current_summaries)

If each summary is too large to fit with another summary, simple budget packing makes no progress, so pairing summaries forces the reduction to continue.

How to Implement the Hierarchical Index

Once you have document records and chunk records, create two kinds of stores:

one store for document summaries
one store for chunk summaries, grouped by document

Here's the document-summary store:

document_summary_store = SimpleVectorStore(
    [
        SearchDocument(
            page_content=record.summary,
            metadata={"doc_id": record.doc_id, "source": record.source},
        )
        for record in document_records
    ]
)

Then group chunks by document:

chunks_by_doc_id: dict[str, list[ChunkRecord]] = {}

for chunk in chunk_records:
    chunks_by_doc_id.setdefault(chunk.doc_id, []).append(chunk)

Then create one chunk-summary store per document:

chunk_summary_stores_by_doc_id = {}

for doc_id, doc_chunks in chunks_by_doc_id.items():
    chunk_summary_stores_by_doc_id[doc_id] = SimpleVectorStore(
        [
            SearchDocument(
                page_content=chunk.summary,
                metadata={
                    "chunk_id": chunk.chunk_id,
                    "doc_id": chunk.doc_id,
                    "source": chunk.source,
                    "chunk_index": chunk.index,
                },
            )
            for chunk in doc_chunks
        ]
    )

This is what makes retrieval hierarchical: the first search chooses documents, while the second search only looks inside the chosen documents.

How to Retrieve Through Summaries

At question time, search document summaries first:

document_hits = index.document_summary_store.similarity_search(
    question,
    k=min(DOC_RETRIEVAL_K, len(index.documents_by_id)),
)

In these searches, k controls how many top-ranked results the store should return.

Then search chunk summaries inside each selected document:

chunk_hits = []
seen_chunk_ids = set()

for document_hit in document_hits:
    doc_id = str(document_hit.metadata["doc_id"])
    chunk_store = index.chunk_summary_stores_by_doc_id[doc_id]
    doc_chunk_count = len(index.chunks_by_doc_id[doc_id])
    per_doc_hits = chunk_store.similarity_search(
        question,
        k=min(CHUNK_RETRIEVAL_K_PER_DOC, doc_chunk_count),
    )

    for chunk_hit in per_doc_hits:
        chunk_id = str(chunk_hit.metadata["chunk_id"])

        if chunk_id in seen_chunk_ids:
            continue

        chunk_hits.append(chunk_hit)
        seen_chunk_ids.add(chunk_id)

Notice what is being retrieved here: summaries.

The summary hit contains the chunk_id, but the final answer still uses the raw chunk text associated with that ID because the raw chunk preserves the original wording and details that the summary might have removed.

How to Implement a Budgeted Raw Context

After chunk-summary retrieval, convert the hits back to raw chunks.

The demo also adds neighbor chunks:

def candidate_raw_chunks(
    chunk_hits: list[SearchDocument],
    index: HierarchicalIndex,
) -> list[ChunkRecord]:
    candidates = []
    seen_chunk_ids = set()

    for chunk_hit in chunk_hits:
        chunk = index.chunks_by_id[str(chunk_hit.metadata["chunk_id"])]
        related_chunk_ids = [chunk.chunk_id]

        if EXPAND_NEIGHBOR_CHUNKS:
            related_chunk_ids.extend([chunk.next_chunk_id, chunk.previous_chunk_id])

        for chunk_id in related_chunk_ids:
            if chunk_id is None or chunk_id in seen_chunk_ids:
                continue

            candidates.append(index.chunks_by_id[chunk_id])
            seen_chunk_ids.add(chunk_id)

    return candidates

Then apply the final context budget:

def build_raw_context(
    chunk_hits: list[SearchDocument],
    index: HierarchicalIndex,
) -> tuple[str, list[tuple[ChunkRecord, int]], list[tuple[ChunkRecord, int]]]:
    included_chunks = []
    skipped_chunks = []
    used_tokens = 0

    for chunk in candidate_raw_chunks(chunk_hits, index):
        raw_context_part = format_raw_chunk(chunk)
        raw_context_tokens = approximate_tokens(raw_context_part)

        if used_tokens + raw_context_tokens > RAW_CONTEXT_TOKEN_BUDGET:
            skipped_chunks.append((chunk, raw_context_tokens))
            continue

        included_chunks.append((chunk, raw_context_tokens))
        used_tokens += raw_context_tokens

    included_chunks.sort(key=lambda item: (item[0].source, item[0].index))

    context = "\n\n---\n\n".join(
        format_raw_chunk(chunk)
        for chunk, _tokens in included_chunks
    )

    return context, included_chunks, skipped_chunks

This step is where many RAG bugs become visible.

If the system retrieves a useful chunk but skips it because the prompt is full, the problem isn't document search. It's context budgeting.

How to Run the Demo

The companion repository contains two versions of the same example.

From the companion repository root, run the Python version:

cd python
python3 -m small_context_rag_solution --question "Why can RAG fail when the context budget is too small?"

Run the TypeScript version:

cd typescript
npm install
npm run demo

You can also run either example interactively by leaving off the question flag. Type q, quit, or exit to leave interactive mode.

Python:

python3 -m small_context_rag_solution

TypeScript:

npm run build
npm start

The default raw context budget is small on purpose: RAW_CONTEXT_TOKEN_BUDGET=250. That makes skipped chunks visible.

How to Interpret the 250 vs 1200 Token Test

Run the same question with two budgets.

Python:

RAW_CONTEXT_TOKEN_BUDGET=250 python3 -m small_context_rag_solution --question "Why can RAG fail when the context budget is too small?"
RAW_CONTEXT_TOKEN_BUDGET=1200 python3 -m small_context_rag_solution --question "Why can RAG fail when the context budget is too small?"

TypeScript:

RAW_CONTEXT_TOKEN_BUDGET=250 npm run demo
RAW_CONTEXT_TOKEN_BUDGET=1200 npm run demo

With the 250-token budget, the raw context builder includes only two chunks:

doc-003-large_rag_notes-chunk-004 (110 approx tokens)
doc-003-large_rag_notes-chunk-005 (121 approx tokens)

It skips five other selected chunks:

doc-003-large_rag_notes-chunk-003 (117 approx tokens)
doc-003-large_rag_notes-chunk-001 (116 approx tokens)
doc-003-large_rag_notes-chunk-002 (120 approx tokens)
doc-001-context_window_notes-chunk-001 (131 approx tokens)
doc-001-context_window_notes-chunk-002 (73 approx tokens)

With the 1200-token budget, every selected raw chunk fits:

doc-001-context_window_notes-chunk-001 (131 approx tokens)
doc-001-context_window_notes-chunk-002 (73 approx tokens)
doc-003-large_rag_notes-chunk-001 (116 approx tokens)
doc-003-large_rag_notes-chunk-002 (120 approx tokens)
doc-003-large_rag_notes-chunk-003 (117 approx tokens)
doc-003-large_rag_notes-chunk-004 (110 approx tokens)
doc-003-large_rag_notes-chunk-005 (121 approx tokens)

No selected raw chunks are skipped.

This diagram shows the difference between the two context budgets:

A 1,200-token limit is still a very small context window for a real system, but it's much larger than 250. In this example, you can clearly see that the same retrieval route behaves differently when the prompt builder has more room.

This is why I like printing both included and skipped chunks. It helps answer a practical debugging question:

Did retrieval miss the evidence, or did prompt assembly drop it?

The demo uses a simplified answer step, so don't focus too much on the exact wording of the final answer. In a real LLM prompt, you would include instructions like:

Answer only from the raw chunks below.
If the raw chunks contain multiple relevant reasons, include all of them.
Prefer a concise bullet list for multi-part answers.
If the raw chunks don't contain enough evidence, say so.

More context doesn't automatically make the answer better. The prompt still has to tell the model how to use the extra evidence.

How This Relates to Existing RAG Techniques

This pattern isn't brand new research. It's a practical combination of several ideas that already exist in the RAG ecosystem.

LangChain uses a related technique in its ParentDocumentRetriever, which searches smaller child chunks and then returns their larger parent documents.

It is also related to the LlamaIndex Document Summary Index, which uses document summaries to select relevant documents and then retrieves the nodes for those documents.

And it's conceptually adjacent to RAPTOR, a retrieval method that builds a tree by recursively clustering and summarizing text.

The version in this article is intentionally simpler:

No clustering.
No framework requirement.
No vector database required for the demo.
No claim that summaries are enough for final answers.

The goal is to show a transparent pattern that's easy to understand under the hood and adapt to your own needs without relying on heavy frameworks. For my local-model work, the useful part was the separation:

Summaries for retrieval
Raw chunks for grounding
Budget trace for debugging

When to Use This Pattern

This pattern is useful when:

you run local models with limited VRAM
your context window is small or expensive
you have many documents but only a few are relevant to each question
you want inspectable retrieval traces
you want summaries for search but raw text for answers
you need to avoid unbounded prompts during both indexing and answering

It's less useful when:

your source documents are already small
your whole corpus fits comfortably in the prompt
exact keyword search is enough
you don't need multi-document routing
you can afford to retrieve and rerank many raw chunks directly

There is also a tradeoff. This pattern adds indexing work:

chunk summaries
recursive summary reduction
document summaries
extra lookup maps

That's usually acceptable for document assistants, research tools, internal knowledge bases, and local-model projects where indexing can happen once and queries happen many times.

Conclusion

Don't treat RAG as only "retrieve chunks and paste them into a prompt."

For small-context systems, retrieval needs routing and budgeting. Even on high-end hardware with very large context windows, good system design becomes fundamental as the project scales.

The pattern comes down to three practical rules:

Summaries help find relevant source material.
Raw chunks ground the answer.
Context budgeting decides what reaches the model.

This solution helped me develop more reliable local RAG systems on constrained hardware. It also made failures easier to debug, because I could see exactly which summaries matched, which raw chunks were selected, and which raw chunks were skipped.

Whether you're running RAG locally or using a hosted model, if you're working with a small model, a limited context window, or a strict prompt budget, this pattern is worth trying before you spend money on a larger context window.

AI Paper Review: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Mohammed Fahd Abrah — Mon, 15 Jun 2026 22:43:25 +0000

For the last few years, Large Language Models have been impressing researchers with their ability to generate text, answer questions, translate languages, and perform tasks they had never been explicitly trained to solve.

Each new generation seemed to confirm a simple belief: bigger models lead to better capabilities. Yet there was one area where progress appeared frustratingly limited. When problems required multiple steps of reasoning, language models often struggled in ways that were difficult to ignore.

A math word problem, a common sense question, or a symbolic puzzle could expose a surprising gap between fluent language generation and genuine problem solving. Models could frequently produce confident answers, but confidence alone wasn't enough. The challenge was whether they could reason through a problem before arriving at an answer.

Against this backdrop, the paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models introduced an idea that was both simple and unexpected. Rather than asking a model to produce an answer immediately, the authors encouraged it to work through intermediate reasoning steps first.

What followed was one of the most influential discoveries in modern AI research: many reasoning abilities that appeared absent in large language models weren't necessarily missing. In many cases, they simply hadn't been elicited in the right way.

This paper went on to reshape how researchers think about prompting, reasoning, and the capabilities of large language models. More importantly, it laid the intellectual foundation for many of the reasoning-oriented techniques and systems that emerged in the years that followed.

Paper Overview

In this article, we'll explore the paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, published by researchers at Google Research in 2022.

This paper introduced one of the most influential ideas in modern AI: Chain-of-Thought (CoT) Prompting. At a time when researchers were focused on scaling language models to ever-larger sizes, this study revealed that performance improvements were not always about building bigger models. Sometimes, the key was changing how we communicate with them.

The paper investigates a simple but powerful question: what happens if a language model is encouraged to show its reasoning process before giving an answer? Instead of responding directly, the model is guided to generate intermediate reasoning steps that lead to the final solution.

What makes this paper historically important is that it changed how researchers think about reasoning in large language models. The authors demonstrated that many reasoning capabilities can be unlocked through prompting alone, without additional training, fine-tuning, or architectural modifications.

The impact of this idea quickly extended beyond arithmetic reasoning. It influenced a new generation of research on reasoning, including Self-Consistency, Process Supervision, Verification-based methods, and the reasoning-oriented models that followed in subsequent years.

In many ways, this paper marked a shift from asking language models what the answer is to asking them how they arrived at the answer.

Here's the original paper if you'd like to explore it directly:

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

And here's a quick infographic of what we'll cover throughout this review.

Prerequisites

To get the most out of this breakdown, it helps to already be familiar with a few foundational ideas and the evolution of large language models that led to Chain-of-Thought prompting.

Reading the previous reviews in this series will be especially helpful:

The GPT-3 review is particularly important because the Chain-of-Thought paper builds directly on one of GPT-3's most surprising capabilities: in-context learning. Rather than changing the model architecture or retraining the model, the authors discovered that reasoning performance could be dramatically improved simply by changing how examples were presented in the prompt.

It also helps to have:

A general understanding of natural language processing (NLP) and large language models
A basic understanding of Transformer-based autoregressive models
Familiarity with prompting, few-shot learning, and in-context learning
A high-level understanding of how language models generate text token by token
General machine learning concepts such as training, inference, scaling laws, and model evaluation
Some exposure to reasoning tasks, logic problems, and mathematical word problems
A basic understanding of benchmark datasets and model performance evaluation

You don't need a deep background in mathematics or machine learning research to follow this article.

I'll keep the explanations intuitive and practical, focusing on why Chain-of-Thought prompting became one of the most influential reasoning techniques in modern AI and how a simple prompting strategy changed the way researchers think about language model reasoning.

Abstract

One of the long-standing challenges for large language models has been reasoning. While these models can generate fluent text and answer a wide variety of questions, they often struggle when a task requires multiple logical steps.

This paper introduces a remarkably simple idea to address that limitation: instead of prompting a model with only questions and answers, you should provide examples that also include the intermediate reasoning steps leading to the solution.

The authors call this approach Chain-of-Thought (CoT) Prompting. By showing a model a few demonstrations of step-by-step reasoning, they find that sufficiently large language models can generate their own reasoning chains and solve complex problems more effectively. Importantly, this improvement doesn't require additional training or fine-tuning, only a different style of prompting.

Through experiments on arithmetic, common sense, and symbolic reasoning tasks, the paper demonstrates that chain-of-thought prompting consistently improves performance. The gains become especially pronounced at larger model scales, suggesting that reasoning abilities emerge naturally as models grow and are given the right prompting strategy.

The paper's most striking result comes from the GSM8K math benchmark, where PaLM 540B, using only eight chain-of-thought examples, achieved state-of-the-art performance and even surpassed a fine-tuned GPT-3 system equipped with a verifier. This finding revealed that prompting alone could unlock reasoning capabilities that standard prompting often fails to expose.

The figure below compares standard prompting with Chain-of-Thought (CoT) prompting using a simple arithmetic example.

Source: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

In standard prompting, the model is shown question–answer pairs and is expected to produce an answer directly, which can lead to mistakes on multi-step problems.

In Chain-of-Thought prompting, the examples include intermediate reasoning steps before the final answer. When faced with a new problem, the model follows a similar step-by-step process, arriving at the correct solution.

This paper shows that providing reasoning demonstrations can substantially improve performance on arithmetic, common sense, and symbolic reasoning tasks, particularly in large language models.

Introduction

By 2022, large language models had already transformed natural language processing. Models such as GPT-3 demonstrated that scaling model size could unlock impressive capabilities, from text generation to few-shot learning.

But there was an important limitation: larger models weren't necessarily better at reasoning. Tasks that required multi-step arithmetic, common sense inference, or symbolic manipulation remained surprisingly difficult, even for some of the largest models available.

The authors begin by observing two promising research directions. The first comes from prior work showing that reasoning tasks can benefit from natural language explanations or intermediate solution steps. Instead of jumping directly to an answer, a model can generate a rationale that mirrors how a human might solve the problem.

The second direction is few-shot prompting, where a model learns a task from a handful of examples provided in the prompt, eliminating the need for task-specific fine-tuning.

Still, both approaches have drawbacks. Training models on large collections of human-written rationales is expensive and time-consuming, while standard few-shot prompting often struggles on tasks that require genuine reasoning.

The key insight of this paper was to combine the strengths of both ideas. Rather than providing only input-output examples, the prompt includes an additional component: the reasoning process itself. Each example follows the structure of input → chain of thought → output.

This simple modification led to Chain-of-Thought Prompting. By exposing intermediate reasoning steps, the model is encouraged to break complex problems into smaller, more manageable stages before arriving at a final answer.

To evaluate the idea, the authors tested chain-of-thought prompting across arithmetic, common sense, and symbolic reasoning benchmarks. The results showed substantial improvements over standard prompting, with some gains being remarkably large.

Chain-of-Thought Prompting

At the heart of this paper is a simple observation about how humans solve difficult problems. When faced with a multi-step reasoning task, we rarely jump directly to the answer. Instead, we break the problem into smaller pieces, solve each intermediate step, and gradually work toward a conclusion. The authors argued that large language models could benefit from a similar process.

This idea led to Chain-of-Thought (CoT) Prompting, where examples in the prompt included not only the question and answer, but also the reasoning steps connecting them. By seeing a few demonstrations of this reasoning process, sufficiently large language models learned to generate their own chains of thought before producing a final answer.

The significance of this approach extends beyond improving accuracy. First, it allows complex problems to be decomposed into manageable intermediate steps, making multi-step reasoning easier to perform.

Second, the generated reasoning process offers a degree of interpretability, giving researchers and users a glimpse into how the model arrived at its answer. While these reasoning traces don't fully reveal the model's internal computations, they can help identify where mistakes occur.

Another important aspect of chain-of-thought prompting is its generality. The authors proposed it not as a solution for a single benchmark, but as a broad reasoning framework that can be applied to arithmetic problems, common sense reasoning tasks, symbolic manipulation, and potentially many other challenges that require sequential reasoning.

Perhaps most importantly, this capability can be elicited from existing language models through prompting alone, without additional training or architectural modifications.

This section establishes the paper's central claim: reasoning abilities don't necessarily require new model architectures or specialized fine-tuning. In sufficiently large language models, these capabilities can emerge when the model is guided to generate intermediate reasoning steps rather than being asked to produce an answer immediately.

Arithmetic Reasoning

The authors begin their empirical evaluation with arithmetic reasoning, a domain that had long exposed a weakness of large language models.

Although solving math word problems is relatively straightforward for humans, it often requires a sequence of intermediate calculations and logical deductions.

Previous research had shown that even large language models struggled with these tasks, making arithmetic reasoning an ideal setting for testing whether chain-of-thought prompting could genuinely improve reasoning ability.

To evaluate their approach, the authors selected five established benchmarks covering a variety of math word problems. These datasets differ in style and difficulty, ranging from straightforward arithmetic questions to more complex problems that require multiple reasoning steps before arriving at a solution. Together, they provide a broad picture of how well language models handle mathematical reasoning.

The experiments compare two prompting strategies. The first is standard few-shot prompting, where the model is shown examples consisting only of questions and their corresponding answers. This was the dominant prompting approach at the time and serves as the baseline throughout the paper.

The second is chain-of-thought prompting, where each example is expanded to include the intermediate reasoning steps that connect the question to the final answer.

To ensure a fair comparison, the authors manually created a small set of eight reasoning demonstrations and reused them across the arithmetic benchmarks. Importantly, these examples weren't heavily optimized or engineered for specific datasets. Instead, they were intended to test whether a modest number of natural reasoning demonstrations could reliably encourage models to reason through new problems on their own.

The study also evaluates a diverse collection of language models, including GPT-3, LaMDA, PaLM, UL2, and Codex, spanning model sizes from hundreds of millions to hundreds of billions of parameters. This broad range allowed the authors to examine not only whether chain-of-thought prompting works, but also how its effectiveness changes as models become larger.

With this experimental framework in place, the paper investigated a central question: can providing a few examples of step-by-step reasoning enable large language models to solve mathematical problems that standard prompting struggles to handle?

Results

The arithmetic reasoning experiments revealed that the success of chain-of-thought prompting depends heavily on model scale.

One of the clearest patterns across the benchmarks was that smaller models gained little benefit from generating reasoning steps. In some cases, their performance even deteriorated because the models produced explanations that sounded plausible but were logically flawed.

The advantages of chain-of-thought prompting only became apparent once the models reached very large scales, suggesting that the ability to effectively use intermediate reasoning steps is itself an emergent capability.

Another important observation was that the benefits of chain-of-thought prompting grew as problems became more challenging. On simpler tasks that required only a single reasoning step, standard prompting was already sufficient and the additional reasoning process provided little value.

But as the complexity of the problems increased, the gap between standard prompting and chain-of-thought prompting widened substantially. The GSM8K benchmark provides the strongest example of this trend, where the largest GPT and PaLM models more than doubled their performance when allowed to reason step by step.

Perhaps the most significant result is that chain-of-thought prompting enabled large language models to compete with, and in some cases surpass, specialized systems trained directly for these tasks.

Using only a handful of reasoning demonstrations, PaLM 540B established new state-of-the-art results on several arithmetic benchmarks, despite relying solely on prompting rather than task-specific fine-tuning. This outcome challenged the prevailing assumption that strong performance on reasoning tasks necessarily required dedicated training datasets and specialized models.

To better understand these improvements, the authors manually inspected the reasoning traces generated by the models. When the model arrived at the correct answer, the reasoning process was usually correct as well, indicating that the model was often following a coherent sequence of logical steps rather than guessing the final answer.

Even among incorrect predictions, many reasoning chains were largely accurate and failed only because of small mistakes such as arithmetic slips, incorrect symbol mappings, or a missing intermediate step. More serious failures tended to arise from misunderstanding the problem itself or producing incoherent reasoning.

The error analysis also offered an explanation for why larger models benefited more from chain-of-thought prompting. Comparing PaLM 62B with PaLM 540B showed that increasing scale reduced many of the semantic misunderstandings and incomplete reasoning patterns that appeared in smaller models.

In other words, larger models were not merely generating longer explanations. They were producing reasoning chains that were more logically complete and more faithful to the underlying problem.

Ablation Study

Before diving into this section, it's worth briefly explaining what an ablation study is. In machine learning research, an ablation study systematically removes or modifies parts of a method to determine which components are actually responsible for its performance. Rather than asking whether a method works, an ablation study asks why it works.

In this paper, the authors use ablation experiments to identify which aspects of Chain-of-Thought prompting contribute most to its reasoning improvements.

After demonstrating that chain-of-thought prompting improved reasoning performance, the authors turned to a more fundamental question: why does it work? Simply observing higher accuracy isn't enough. To understand the source of these gains, they designed a series of ablation experiments that isolated different aspects of the prompting strategy.

One possible explanation is that chain-of-thought prompting helps because it encourages the model to generate mathematical equations before producing an answer. If this were true, then the natural language reasoning itself might not be necessary.

To test this idea, the authors replaced the reasoning steps with equations alone. The results showed that this approach provides only limited benefits on complex benchmarks such as GSM8K. While equations can help with simpler problems, they are often insufficient for tasks that require understanding the meaning of the question before translating it into mathematical operations. This suggests that the value of chain-of-thought prompting comes from more than symbolic calculation.

The authors then examined another hypothesis: perhaps chain-of-thought prompting succeeds simply because it allows the model to generate more tokens and therefore spend more computation on difficult problems.

To isolate this factor, they created a prompt that produces additional tokens without any meaningful reasoning content. Performance remained close to the standard prompting baseline, indicating that extra computation alone doesn't explain the observed improvements. What mattered wasn't the number of intermediate tokens, but the reasoning expressed within them.

A third possibility was that chain-of-thought prompts merely activated relevant knowledge already stored in the model. If that were the case, the reasoning steps wouldn't need to appear before the answer.

The authors tested this by moving the reasoning process to after the final answer. Once again, performance largely fell back to the baseline. This result suggested that the sequence of reasoning steps plays an active role in helping the model arrive at the correct solution rather than simply serving as an explanation after the fact.

Taken together, these experiments strengthen the paper's central argument. The success of chain-of-thought prompting can't be explained by equation generation, additional computation, or easier access to stored knowledge alone.

Instead, the evidence points toward the reasoning process itself as the critical ingredient. The intermediate steps aren't merely decorative explanations. They appear to guide the model through a sequence of decisions that makes complex problem solving more effective.

Robustness of Chain-of-Thought Prompting

One of the long-standing concerns with prompting methods is their sensitivity to the examples included in the prompt. Small changes in wording, example selection, or even the order of examples can sometimes produce noticeably different results.

Once they established that chain-of-thought prompting improves reasoning performance, the authors investigated whether these gains were robust or whether they depended on a particular set of carefully crafted demonstrations.

To answer this question, the researchers asked multiple authors of the paper to independently write reasoning traces for the same examples. They also experimented with a more concise writing style and tested prompts built from entirely different sets of examples.

The goal was to determine whether chain-of-thought prompting was succeeding because of a specific wording choice or because the underlying reasoning structure was genuinely useful.

The results provided reassuring evidence that the technique isn't tied to a particular author, writing style, or collection of exemplars. While some variation in performance naturally appeared across different prompts, every version of chain-of-thought prompting consistently outperformed standard prompting by a substantial margin. Whether the reasoning steps were detailed or concise, manually written or drawn from an independent dataset, the overall pattern remained remarkably stable.

The authors further broadened their analysis by varying the order and number of exemplars used in the prompt. Once again, the central finding persisted: although prompt design still influenced performance to some degree, the effectiveness of chain-of-thought prompting didn't depend on a single carefully engineered prompt.

This robustness analysis strengthens one of the paper's most important claims that the success of chain-of-thought prompting isn't an artifact of a particular phrasing or annotation style. Instead, the benefits appear to arise from exposing the model to a reasoning process itself, suggesting that the method captures a more general principle rather than a prompt-specific trick.

Common Sense Reasoning

Up to this point, the paper focused primarily on mathematical reasoning. While the results are impressive, they leave an important question unanswered: is chain-of-thought prompting useful only for arithmetic problems, or can it improve reasoning more broadly?

To investigate this, the authors turned to common sense reasoning tasks. Unlike math problems, these tasks often require background knowledge about the world, an understanding of human behavior, or the ability to connect multiple pieces of information before arriving at a conclusion. In many cases, the challenge isn't performing calculations but reasoning through situations that humans find intuitive.

The evaluation spanned a diverse collection of benchmarks, including common sense question answering, multi-hop reasoning, date understanding, sports-related reasoning, and even tasks that involved converting natural language instructions into robot actions.

Despite their differences, these tasks share a common requirement: solving them often involves a sequence of intermediate inferences rather than an immediate answer.

The results showed that the benefits of chain-of-thought prompting extend well beyond mathematics. Across most benchmarks, models consistently performed better when encouraged to generate intermediate reasoning steps before producing a final answer.

The improvements became particularly noticeable for larger models, suggesting that the same pattern observed in arithmetic reasoning also applies to common sense reasoning.

Some of the strongest gains appeared on tasks that required multi-step inference. On StrategyQA, for example, chain-of-thought prompting enabled PaLM 540B to surpass the previous state of the art. Similarly, on the Sports Understanding benchmark, the model achieved performance that exceeded that of an unaided human sports enthusiast.

These results suggest that the reasoning process encouraged by chain-of-thought prompting can help models connect facts, evaluate plausibility, and navigate more complex decision-making scenarios.

At the same time, the improvements weren't uniform across every dataset. The gains on CommonsenseQA were relatively modest, indicating that not all reasoning tasks benefit equally from explicit reasoning traces. This serves as an early reminder that chain-of-thought prompting isn't a universal solution, even though it consistently proves valuable across a wide range of settings.

More broadly, this section strengthens the paper's central argument by showing that chain-of-thought prompting isn't merely a technique for solving math word problems. Its effectiveness across diverse common sense tasks suggests that the method taps into a more general reasoning capability that emerges in sufficiently large language models.

Symbolic Reasoning

The final evaluation moves away from mathematics and real-world knowledge altogether. Instead, the authors focus on symbolic reasoning tasks, where success depends on following abstract rules rather than recalling facts or performing calculations. These tasks are simple for humans, yet they provide a useful way to test whether language models can consistently apply a sequence of reasoning steps.

To explore this question, the authors designed two controlled tasks. The first required the model to extract and concatenate the last letters of words in a name. The second asked the model to track the state of a coin after a sequence of flips and non-flips.

Although these tasks may appear simple, they required the model to perform precise symbolic manipulations without relying on memorized knowledge about the world.

What made these experiments particularly interesting was the introduction of an out-of-distribution setting. During prompting, the model only saw examples involving short reasoning chains. At evaluation time, it was asked to solve versions of the same tasks that required more steps than any example it had previously encountered.

This setup allowed the authors to test not only whether the model could follow a reasoning procedure, but also whether it could extend that procedure to longer and unfamiliar cases.

The results revealed a familiar pattern. Large models benefitted substantially from chain-of-thought prompting, while smaller models struggled even when the required reasoning process was straightforward.

On the in-domain tasks, where the evaluation closely matched the examples provided in the prompt, the largest models achieved near-perfect performance when guided by chain-of-thought reasoning. This indicated that they could successfully learn and apply the underlying procedure demonstrated in the prompt.

The more revealing results come from the out-of-distribution evaluations. Standard prompting largely fails when the reasoning chain becomes longer than those seen in the examples. In contrast, chain-of-thought prompting enabled performance to improve as model size increased, demonstrating an ability to extend learned reasoning patterns beyond the exact situations shown during prompting.

Although accuracy declines compared to the in-domain setting, the models were still able to generalize in ways that standard prompting couldn't.

This section provided some of the strongest evidence that chain-of-thought prompting is doing more than improving benchmark performance. By helping models apply reasoning procedures to longer and previously unseen inputs, it suggests that the generated reasoning steps serve as a scaffold for systematic problem solving rather than merely a mechanism for producing better answers on familiar examples.

Discussion

The most important contribution of this paper wasn't a new model architecture, a new training objective, or a larger dataset. Instead, it demonstrated that a simple change in prompting could unlock capabilities that standard prompting often failed to reveal.

Across arithmetic, common sense, and symbolic reasoning tasks, chain-of-thought prompting consistently allowed large language models to solve problems that were previously difficult or inaccessible.

A recurring theme throughout the paper was the relationship between reasoning and scale. The authors repeatedly observed that chain-of-thought prompting became effective only once models reached a sufficient size. Smaller models generated fluent reasoning traces, but those traces were often logically inconsistent.

Larger models, in contrast, were able to use intermediate reasoning steps in a way that genuinely improved problem-solving performance.

This finding reinforced a broader lesson emerging from language model research at the time: some capabilities don't appear gradually, but emerge once a model crosses a certain scale threshold.

Perhaps the most intriguing implication was that standard prompting may significantly underestimate what large language models are capable of doing.

Before this work, many reasoning tasks appeared to have reached a performance ceiling. Chain-of-thought prompting revealed that the limitation wasn't always the model itself, but sometimes the way the model was being asked to solve the problem. In that sense, the paper shifted attention from building more capable models to discovering better ways of interacting with the capabilities that already exist within them.

At the same time, the authors were careful not to overstate their conclusions. Although chain-of-thought outputs can resemble human reasoning, the paper doesn't prove that language models reason in the same way humans do. The generated reasoning traces may reflect genuine problem-solving processes, post-hoc rationalizations, or something in between. Determining the relationship between generated reasoning and internal model computation remains an open research question.

The authors also acknowledged several practical limitations. Constructing high-quality reasoning demonstrations can require additional effort, particularly if the approach is extended beyond few-shot prompting.

Also, generating a chain of thought doesn't guarantee that the reasoning itself is correct. Models can still produce convincing but flawed reasoning paths, leading to incorrect answers.

Finally, the strongest benefits appear only in very large models, raising questions about computational cost and whether similar reasoning abilities can be induced in smaller systems.

Viewed from a historical perspective, this paper marked a turning point in research on language model reasoning. Rather than treating reasoning as something that must be explicitly trained into a model, it suggested that reasoning abilities could be elicited through the right prompting strategy.

Many influential ideas that followed, including self-consistency, reasoning supervision, process supervision, and the reasoning-focused models that emerged in later years, can trace part of their intellectual foundation back to the simple insight introduced here: sometimes a model performs better when it's encouraged to show its work.

The ideas behind Chain-of-Thought prompting didn't emerge in isolation. Instead, the paper sits at the intersection of two research directions that had been evolving independently for several years.

The first direction focused on helping models solve complex problems through intermediate reasoning steps. Earlier work had already shown that tasks such as mathematical reasoning become easier when a model generates natural language rationales rather than producing an answer directly. Researchers explored methods that trained models to generate explanations, reasoning traces, or intermediate computations before arriving at a final solution.

Other approaches relied on formal symbolic representations, translating problems into structured equations or logical forms. Despite their differences, these efforts shared a common intuition: difficult reasoning tasks are often easier to solve when they're decomposed into smaller steps.

Chain-of-thought prompting inherits this intuition but introduces an important shift. Earlier methods typically required dedicated training procedures, specialized datasets, or task-specific fine-tuning.

In contrast, this paper demonstrated that reasoning traces could be elicited through prompting alone. Rather than teaching a model to reason through additional training, the authors showed that providing a handful of reasoning examples may be enough to unlock capabilities that already exist within sufficiently large language models.

The second research direction concerns prompting itself. Following the success of GPT-3 and few-shot learning, a growing body of work explored how prompts could be used to improve model performance without retraining.

Researchers experimented with prompt engineering, prompt tuning, and natural language instructions to better communicate tasks to language models. Most of these techniques focused on improving the input side of the interaction by changing how a task was described to the model.

Chain-of-thought prompting takes a different approach. Instead of modifying the instructions that precede a task, it augments the examples that follow them by exposing the reasoning process that connects inputs and outputs. This distinction may seem subtle, but it represents one of the paper's key insights: the contribution goes beyond a better prompt template. It focuses on the realization that demonstrating how to reason can be just as important as describing what task should be solved.

Viewed in this broader context, the paper acts as a bridge between research on reasoning traces and research on prompting. It combines the strengths of both traditions and, in doing so, lays the foundation for many later advances in language model reasoning, including self-consistency, STaR, process supervision, and the reasoning-oriented systems that followed in subsequent years.

Conclusion

Chain-of-Thought Prompting introduced a simple idea that changed how researchers think about reasoning in large language models. Rather than modifying model architectures or relying on additional training, the authors showed that reasoning abilities could often be unlocked by encouraging models to generate intermediate reasoning steps before producing an answer.

Across arithmetic, common sense, and symbolic reasoning tasks, the results demonstrated that large language models become significantly more capable when allowed to work through a problem step by step. More importantly, the paper revealed that many of these improvements emerge at larger scales, suggesting that reasoning isn't simply a product of prompting but a capability that becomes increasingly accessible as models grow more powerful.

What made this work particularly influential wasn't the complexity of the method, but the insight behind it. A model may possess the knowledge required to solve a problem, yet still fail to use that knowledge effectively when asked for an immediate answer. By exposing the reasoning process, Chain-of-Thought prompting showed that how a model arrives at an answer can be just as important as the answer itself.

This idea helped shift the focus of AI research beyond what language models know toward how they reason, plan, and solve problems. Many of the techniques that followed (including Self-Consistency, process supervision, verification-based methods, and modern reasoning-focused systems) build upon the foundation established by this paper.

Viewed in retrospect, Chain-of-Thought Prompting was more than a prompting technique. It marked a turning point in the study of language model reasoning, demonstrating that some capabilities aren't absent from a model but simply require the right conditions to emerge.

The infographic below highlights some of the most influential papers and milestones that shaped modern AI, from the introduction of GPT-1 and the scaling era of GPT-2 and GPT-3, to instruction tuning, Chain-of-Thought reasoning, Self-Consistency, process supervision, and the latest generation of reasoning-focused models. Together, these works reveal how the field evolved from teaching models to predict language toward helping them reason, verify, and solve increasingly complex problems.

Resources

Contact Me

AI Paper Review: Training Language Models to Follow Instructions with Human Feedback (InstructGPT)

Mohammed Fahd Abrah — Wed, 03 Jun 2026 18:01:27 +0000

GPT-3 was a major breakthrough in natural language processing. With 175 billion parameters, it demonstrated remarkable few-shot learning abilities and showed that scaling large language models could unlock a wide range of capabilities.

Yet despite its impressive performance, GPT-3 revealed an important limitation: raw capability doesn't automatically create a useful assistant.

A language model can generate fluent text, answer questions, and solve complex tasks while still failing to follow what the user actually wants.

GPT-3 could produce responses that were inconsistent, overly confident, difficult to control, or misaligned with user instructions. It was a powerful prediction engine, but it wasn't designed to reliably act as a helpful assistant.

This challenge motivated one of the most influential papers in modern AI: Training Language Models to Follow Instructions with Human Feedback. Rather than making the model larger, the researchers focused on teaching it how to better follow human intent.

The result was InstructGPT, a system fine-tuned from GPT-3 that demonstrated how human feedback could transform a capable language model into a far more useful and aligned assistant.

This challenge became one of the most important problems in modern AI: alignment.

Researchers realized that building larger models was only part of the solution. While scaling improved capabilities, it didn't guarantee that models would reliably follow instructions or behave in ways that matched user expectations. The next stage of progress required teaching models how to respond in a more helpful, truthful, and safe manner.

This led to the development of instruction-following systems and Reinforcement Learning from Human Feedback (RLHF). Instead of optimizing models solely to predict the next word, researchers began training them to better align with human preferences and intentions.

This shift marked a major turning point in the evolution of large language models.

GPT-3 demonstrated the power of large-scale language modeling and introduced many people to prompting and few-shot learning.

InstructGPT built on that foundation by showing how human feedback could significantly improve instruction following and model behavior. ChatGPT then brought these ideas to a much broader audience by packaging aligned language models into an accessible conversational interface used by millions of people.

In many ways, language models became capable before they became aligned.

That's why the transition from GPT-3 to InstructGPT represents one of the most important milestones in the history of artificial intelligence. The focus was no longer only on making models more capable. It was also about making them more useful, reliable, and responsive to human intent.

The success of InstructGPT pioneered many of the alignment techniques that later became a core part of systems such as ChatGPT and GPT-4.

Paper Overview:

In this article, we’ll mainly focus on the paper Training Language Models to Follow Instructions with Human Feedback, published by OpenAI in 2022.

This paper introduced InstructGPT, one of the most important transitions in the history of large language models. While earlier GPT systems focused heavily on scaling model size and improving raw capabilities, this work shifted attention toward something equally important: alignment.

The paper explores how language models can be trained to better follow human instructions using reinforcement learning from human feedback (RLHF). Instead of optimizing only for next-token prediction, the model is further optimized to produce responses that humans actually prefer – responses that are more helpful, safer, and more aligned with user intent.

What makes this paper historically important is that it became the foundation for the modern ChatGPT alignment pipeline.

Many of the interaction patterns people now associate with ChatGPT (like instruction following, conversational behavior, refusal handling, and safer responses) can be traced directly back to the ideas introduced here.

Here’s the original paper again if you want to explore it directly: Training language models to follow instructions with human feedback

And here’s a quick infographic of what we’ll cover throughout this review:

Executive Summary
The Core Problem
Why GPT-3 Was Not Enough
InstructGPT: The Birth of Alignment-Centered LLMs
RLHF Pipeline: How InstructGPT Learned to Behave Like an Assistant
- Stage 1 — Supervised Fine-Tuning (SFT)
- Stage 2 — Reward Model Training
- Stage 3 — PPO Reinforcement Learning
Helpful, Honest, Harmless
Human Feedback as the New Scaling Factor
Why ChatGPT Exploded Globally
ChatGPT as an Interface Revolution
Benchmarks and Results
Truthfulness and Hallucinations
Safety and Refusal Behavior
Limitations
Historical Importance
Discussion: The Real Shift
Connection to GPT-4
GPT-3 vs InstructGPT vs ChatGPT vs GPT-4: Key Differences
From GPT-1 to GPT-4: A Timeline of Modern AI Systems and Alignment Evolution
Final Insight
Resources

Prerequisites

To get the most out of this breakdown, it helps to already be familiar with a few foundational ideas.

Reading the previous reviews in this series will be especially helpful:

Even though GPT-4 was released after InstructGPT, reading the GPT-4 review can still be helpful. It provides a broader view of how alignment techniques evolved and how they were combined with stronger reasoning and multimodal capabilities in later generations of GPT models.

AI Paper Review: GPT-4 Technical Report (GPT-4)

It also helps to have:

A general understanding of natural language processing (NLP) and large language models
A high-level idea of Transformer-based autoregressive models
Familiarity with prompting, few-shot learning, and in-context learning
A basic understanding of reinforcement learning and human feedback systems
General machine learning concepts like training data, fine-tuning, scaling, and inference
Some familiarity with alignment, safety, and AI behavior control concepts

You don't need to be an AI researcher to follow this article, though.

I’ll keep the explanations practical and intuitive, focusing more on understanding how InstructGPT changed modern AI systems rather than getting lost in dense mathematical details or academic terminology.

Executive Summary

The paper Training Language Models to Follow Instructions with Human Feedback marks one of the biggest turning points in the history of modern AI systems. Instead of asking only how to make language models larger or smarter, OpenAI focused on a different question: how do we make these models actually helpful for real people?

The paper introduces InstructGPT, a version of GPT-3 fine-tuned to follow human instructions more accurately using a method called Reinforcement Learning from Human Feedback (RLHF).

The core insight of the paper is simple but extremely important:

Bigger language models don't automatically become better assistants.

Even highly capable models like GPT-3 could still:

ignore instructions
hallucinate facts
generate toxic or biased outputs
produce responses that were technically fluent but not actually useful to users

To solve this problem, OpenAI built a multi-stage alignment pipeline: humans first demonstrate ideal answers, humans then rank model outputs, and finally the model learns from those preferences using reinforcement learning.

This changed the direction of modern AI development.

The paper shows that alignment and usability can matter more than raw model size itself. One of the most surprising findings was that the 1.3B InstructGPT model was often preferred by human evaluators over the original 175B GPT-3 model, despite being dramatically smaller.

The paper also demonstrates improvements in instruction following, truthfulness, toxicity reduction, conversational behavior, and general user preference.

Historically, this paper became the foundation behind modern conversational AI systems.

GPT-3 proved that language models could learn from prompts.

GPT-4 later proved that scaling and multimodal reasoning could unlock even stronger capabilities.

But InstructGPT showed something equally important: AI systems must be aligned with human intent to become truly usable products.

In many ways, this paper represents the transition from raw language modeling to aligned assistants, capability scaling to behavior shaping, and research demos to real-world conversational AI systems.

And that transition eventually led directly to ChatGPT.

The Core Problem

One of the most important ideas in this paper is that raw language modeling is not the same thing as building a useful assistant.

Before InstructGPT, models like GPT-3 were trained mainly with a simple objective: predict the next token in a sequence.

That objective made language models extremely powerful at generating fluent text, but it also created a major limitation. The model learned how to continue internet text, not necessarily how to help humans.

This became one of the defining realizations behind modern AI alignment research.

Despite its impressive capabilities, GPT-3 often struggled to behave like a reliable assistant. The model could produce fluent text, but it was not explicitly trained to follow user intent.

Here are some examples that highlight the differences between GPT-3 and InstructGPT in how they respond to user prompts:

Source: Aligning language models to follow instructions

These examples reveal the central weakness of early GPT systems. GPT-3 often continued the pattern of the prompt rather than completing the requested task. InstructGPT, by contrast, responded directly to the user's instruction. The difference wasn't a matter of raw intelligence. It was a difference in training objectives.

GPT models were trained on massive internet-scale datasets where the goal was simply to predict what text comes next. As a result, the model optimized for plausibility, continuation, and pattern completion. Not necessarily for truthfulness, safety, helpfulness, or alignment with human goals.

This created a major gap between: language capability and useful assistant behavior.

For example, if a user asked a harmful, misleading, or nonsensical question, the model might still attempt to continue the pattern naturally instead of recognizing the issue. In many cases, the model behaved more like an internet text simulator than a reliable assistant.

The paper repeatedly emphasizes that scaling alone couldn't solve this problem.

Researchers increasingly recognized that better behavior would require more than scaling alone.

Models also needed stronger instruction following, better alignment with human intent, improved safety behavior, greater truthfulness, and optimization around real user needs.

Why GPT-3 Was Not Enough

When GPT-3 was released, it felt like a massive leap forward in AI capabilities.

The model could perform few-shot learning, answer questions, summarize text, generate code, translate languages, and even solve certain reasoning tasks: all without traditional fine-tuning. For many researchers, it was the first time a language model started to feel genuinely general-purpose.

Yet using GPT-3 in practice was often less reliable than its benchmark performance suggested.

In practice, using GPT-3 often required careful prompt engineering. Small wording changes could completely change the quality of the response. Sometimes the model followed instructions well, and other times it ignored them entirely.

Users often found themselves rewriting prompts repeatedly to obtain the response they actually wanted.

This became the core motivation behind InstructGPT.

OpenAI responded by exploring ways to make model behavior more consistent, predictable, and useful for users.

InstructGPT: The Birth of Alignment-Centered LLMs

The release of InstructGPT marked one of the biggest shifts in the history of large language models.

Before InstructGPT, most advances in language models came from scaling data, compute, and model size.

The focus shifted toward alignment: building systems that could follow instructions more reliably and behave in ways users actually preferred.

This is where InstructGPT introduced one of the most important ideas in modern AI systems: Reinforcement Learning from Human Feedback (RLHF).

Instead of optimizing models only to predict internet text, OpenAI started optimizing models based on what humans actually preferred. Human labelers ranked model outputs, and those preferences became part of the training process itself.

This fundamentally changed the objective of language models.

Rather than optimizing solely for next-token prediction, the system was increasingly optimized to produce responses that humans judged to be helpful, safe, and aligned with their intentions.

That distinction may sound subtle, but it completely changed the direction of AI development.

InstructGPT combined instruction-following training with human preference optimization, creating a model whose behavior could be shaped directly through feedback rather than solely through pretraining.

The model was no longer trained only to imitate the internet. It was trained to behave more like an assistant.

RLHF Pipeline: How InstructGPT Learned to Behave Like an Assistant

At the center of the InstructGPT paper is a training pipeline that completely changed how modern AI assistants are built.

RLHF was designed to build on traditional language-model pretraining rather than replace it.

The InstructGPT paper introduced a different idea: instead of training models only on internet text, why not train them using human preferences directly?

This led to the development of the RLHF pipeline: Reinforcement Learning from Human Feedback. This approach would later become a standard component of modern conversational AI systems.

The paper’s Figure 2 is especially important because it visualizes the entire alignment pipeline introduced by OpenAI. Rather than relying on a single training stage, the system uses multiple stages where human feedback gradually shapes model behavior.

Source: Training Language Models to Follow Instructions with Human Feedback (OpenAI, 2022).

As you can see in the image above, the process happens in three major stages.

Stage 1 — Supervised Fine-Tuning (SFT)

The first stage starts with human-written demonstrations.

Labelers are given prompts and asked to write ideal responses – the kinds of answers a helpful assistant should produce. These examples become the initial training dataset for the model.

At this stage, the model learns the basic patterns of assistant-style responses.

This is still traditional supervised learning, but the goal is different from standard language modeling. Instead of learning only from web text, the model now learns from examples of preferred assistant behavior.

This stage creates what the paper calls the Supervised Fine-Tuned model (SFT model).

And while this already improves behavior significantly, OpenAI realized something important: human preferences are more complex than simple “correct answers.”

There are often many possible responses to a prompt, but humans may strongly prefer some answers over others.

That leads to the next stage.

Stage 2 — Reward Model Training

In the second stage, humans no longer write responses directly.

Instead, the model generates multiple answers for the same prompt, and human labelers rank them from best to worst.

For a given prompt, one response may be clearer, another more accurate, and another safer or more appropriate. Human labelers rank these alternatives according to their preferences

The rankings are then used to train a separate neural network called the Reward Model (RM).

This model learns something extremely important: which outputs humans prefer.

In other words, the system converts human preferences into a trainable reward signal.

This becomes one of the biggest conceptual breakthroughs in the paper. Instead of manually programming behavior rules, OpenAI trains the model to approximate human judgment itself.

The reward model captures patterns in human preferences and turns them into a training signal.

That reward signal becomes the foundation for the final training stage.

Stage 3 — PPO Reinforcement Learning

The final stage uses reinforcement learning to optimize the language model against the reward model.

More specifically, the paper uses PPO (Proximal Policy Optimization), a reinforcement learning algorithm commonly used in policy optimization tasks.

At this stage, the model generates responses, receives scores from the reward model, and gradually updates its behavior to maximize those scores.

The model gradually shifts toward responses that receive higher scores from the reward model.

The key innovation is that optimization now occurs against a learned representation of human preferences rather than only a language-modeling objective.

According to the paper, this RLHF pipeline significantly improved instruction following and user preference ratings while also reducing toxic and unsafe behavior.

And in many ways, this pipeline became the blueprint for the modern era of conversational AI systems.

Helpful, Honest, Harmless

The authors argue that evaluating language models requires more than measuring capability alone. They should also be evaluated by how they behave around humans.

At the time, this represented a significant shift in how researchers evaluated language models.

That is why the paper repeatedly emphasizes a new alignment philosophy centered around three goals:

Helpful
Honest
Harmless

These ideas became the conceptual foundation behind modern alignment research and conversational AI systems.

Helpful

The first goal is straightforward: the model should genuinely help the user accomplish what they want.

In practice, helpfulness means following instructions clearly, answering questions directly, providing relevant information, and adapting to the user's intent.

This may seem simple, but it fundamentally changes the training objective.

The model is no longer optimized only for linguistic fluency. It's optimized for usefulness.

Honest

The second goal is honesty.

One of the biggest problems with large language models is that they often produce convincing answers even when those answers are wrong. The models can hallucinate facts, invent references, or respond confidently despite uncertainty.

The paper recognizes that a useful assistant shouldn't merely sound intelligent. It should also behave truthfully and acknowledge uncertainty when necessary.

This is especially important because language models are optimized to generate plausible text, not verified truth.

As a result, earlier models sometimes prioritized sounding coherent over being accurate.

The alignment process introduced in InstructGPT attempts to reduce this behavior through human feedback and preference optimization. Human evaluators consistently prefer responses that are more accurate, transparent, and reliable, and those preferences gradually shape the model during RLHF training.

The paper doesn't claim that hallucinations disappear completely. Far from it. But it marks one of the first large-scale attempts to explicitly optimize language models for truthfulness and reliability rather than pure text generation quality.

Harmless

The third goal is harmlessness.

Large language models trained on internet data inevitably absorb toxic, biased, unsafe, or harmful patterns from that data. Without alignment, models may generate dangerous instructions, offensive content, or manipulative behavior.

The paper directly addresses this concern and treats safety as a central part of model development.

Through RLHF and human preference ranking, the model learns to refuse certain harmful requests, avoid toxic generations, produce safer responses, and behave more responsibly during interaction.

This became one of the defining characteristics of modern conversational AI systems.

Instead of maximizing unrestricted generation, the system begins balancing usefulness, safety, and alignment with human values.

But the paper is also honest about limitations.

The authors acknowledge that harmful outputs, biases, and unsafe behavior can still appear. Alignment is imperfect, and human values themselves are complex and difficult to define universally.

But historically, this paper marks the moment when safety and alignment became core engineering goals rather than secondary concerns.

Taken together, these three principles (helpful, honest, and harmless) became much more than training objectives. They became the philosophical foundation behind ChatGPT-era AI systems.

Earlier GPT papers mainly explored how to scale intelligence. But InstructGPT explored something deeper: how to make intelligence usable for humans.

Human Feedback as the New Scaling Factor

One of the most fascinating ideas behind the InstructGPT paper is that it quietly changed what “scaling” meant in modern AI.

For years, progress in language models was largely measured through scaling.

GPT-1 showed that pretraining works. GPT-2 showed that larger models develop stronger zero-shot behavior. GPT-3 pushed this idea even further by scaling to 175 billion parameters and demonstrating impressive few-shot learning abilities.

And to some extent, that was true. Larger models became better at reasoning, code generation, language understanding, translation, and generalization.

That is where human feedback became central.

Instead of relying purely on internet-scale text, OpenAI introduced a training pipeline where human preferences directly shaped model behavior. Human labelers ranked responses, evaluated quality, and guided the system toward outputs people actually preferred.

In many ways, this created a completely new scaling dimension for AI systems:

scaling human feedback
scaling preference learning
scaling alignment pipelines

Historically, this shifted attention from model scale alone toward the quality of model behavior

InstructGPT focused on scaling usability. And the results were surprisingly powerful.

According to the paper, a much smaller aligned model was often preferred over the original 175B GPT-3 model by human evaluators.

That finding changed how the industry thought about progress.

The result suggested that improving behavior could sometimes matter as much as increasing scale.

This is why RLHF became one of the defining ideas of the ChatGPT era.

After InstructGPT, modern AI systems were no longer evaluated only by benchmark scores, parameter counts, or scaling curves.

They were increasingly evaluated by usefulness, conversational quality, safety, reliability, and how well they interact with humans.

And that shift fundamentally changed the future direction of large language models.

Why ChatGPT Exploded Globally

When ChatGPT launched publicly, the reaction was immediate and unlike anything the AI industry had seen before.

Millions of people started using it within days. Developers, students, writers, researchers, businesses, and everyday users suddenly felt like they were interacting with AI in a completely different way.

What made this moment so important was that advanced AI capabilities finally became accessible to ordinary users. After all, the underlying language models were already extremely capable before ChatGPT existed. GPT-3 could generate essays, answer questions, write code, summarize text, and perform impressive few-shot learning tasks. GPT-4 later pushed reasoning and multimodal abilities even further.

The challenge was no longer whether language models could perform useful tasks, but whether people could interact with them naturally.

ChatGPT combined powerful language-model capabilities with RLHF-based alignment, conversational interaction, safer behavior, and a user-friendly chat interface.

Earlier systems often required significant prompt experimentation to achieve consistent results. Users had to carefully engineer prompts, retry questions, or work around strange outputs. The models could be brilliant one moment and confusing the next.

ChatGPT changed that experience dramatically.

Thanks to the alignment techniques introduced in the InstructGPT paper, the system became far better at following instructions, maintaining conversational flow, understanding intent, and responding in a way that felt cooperative rather than purely generative.

The conversational interface itself also mattered enormously.

Before ChatGPT, interacting with advanced AI systems often required APIs, coding knowledge, prompt experimentation, or technical understanding.

ChatGPT simplified everything into a familiar chat format: you simply typed naturally, and the system responded naturally.

That design decision may sound small, but historically it was transformative. It turned large language models from research tools into consumer products.

Although imperfect, the system felt substantially more reliable than earlier language-model interfaces.

The system was designed to communicate in ways that felt more natural and cooperative.

The breakthrough was not simply that the AI became smarter. The breakthrough was that the AI became usable.

And that usability is what transformed large language models from impressive research demonstrations into globally adopted AI assistants.

ChatGPT as an Interface Revolution

One of the most important things about ChatGPT is that it changed how humans interact with computers.

Before ChatGPT, powerful AI systems mostly lived behind APIs, research demos, developer tools, and complex prompting workflows.

Using advanced language models often required technical knowledge. Developers experimented with prompt engineering, API parameters, temperature settings, and carefully structured inputs just to get reliable outputs from the model.

Even GPT-3, despite being extremely powerful, still felt like a research system for many users. You had to learn how to “talk to the model.”

And in many cases, the interaction felt fragile. Slight changes in wording could completely change the quality of the response.

ChatGPT changed that dynamic almost overnight.

Instead of making users adapt to the AI, the AI became much better at adapting to humans.

Natural conversation became the interface.

For decades, human-computer interaction depended on commands, menus, search boxes, forms, programming languages, and specialized software interfaces.

ChatGPT introduced something different: you could simply explain what you wanted in plain language. And the system would usually understand.

This made AI feel accessible to people who had never written code, used APIs, or interacted with machine learning systems before.

In many ways, ChatGPT transformed prompting into a universal interface for computing. And that single shift affected nearly every digital field.

In education, students started using conversational AI to explain difficult concepts, summarize lessons, practice languages, and receive tutoring-style help.

In coding, developers began using AI systems for debugging, code generation, documentation, and learning new frameworks.

This eventually led to the rise of AI coding assistants integrated directly into development environments.

In writing and content creation, conversational AI became a brainstorming partner capable of drafting ideas, rewriting text, organizing articles, and helping people communicate more effectively.

Search behavior also started changing. Instead of searching through lists of links, users increasingly expected direct conversational answers. This fundamentally challenged traditional search-engine interaction models.

And across productivity tools, AI systems began acting less like software features and more like collaborative assistants.

This shift was enabled by advances in conversational AI and interaction design that made dialogue feel natural and useful.

The alignment techniques introduced by InstructGPT were an important part of making these conversational experiences practical.

Historically, this may become one of the most important consequences of the GPT era: earlier software required humans to learn interfaces. ChatGPT pushed computing toward interfaces that learn humans instead.

Benchmarks and Results

We've already discussed how one of the biggest improvement didn't come from making the model larger. Instead, it came from making the model better aligned with humans.

This is one of the central findings of the entire paper, and it changed how many researchers thought about progress in large language models.

Before this work, the dominant belief was that scaling was the main path forward, with bigger models, more parameters, more compute, and more data. And GPT-3 seemed to confirm that idea. Larger models consistently showed stronger few-shot learning, reasoning, and generalization abilities.

But the InstructGPT paper introduced a different perspective. The researchers found that a relatively small 1.3B parameter InstructGPT model was often preferred by human evaluators over the original 175B GPT-3 model.

That result was extremely important. It suggested that alignment sometimes outperformed scale.

This became one of the defining insights of the ChatGPT era.

According to the paper, human evaluators consistently preferred InstructGPT responses because they were more helpful, more accurate, safer, and better aligned with what users were actually asking for.

The improvements appeared across several important areas.

One major improvement was instruction following. Earlier GPT models often ignored instructions, drifted off-topic, or generated responses that sounded fluent but failed to solve the user’s actual task. InstructGPT behaved much more like a cooperative assistant and followed prompts more reliably.

The paper also reports improvements in truthfulness. Large language models are known for hallucinating information and confidently generating false statements. Through RLHF and preference optimization, InstructGPT reduced some of these behaviors and produced answers humans judged to be more truthful and reliable.

Another important improvement involved toxicity and harmful outputs. The researchers evaluated the system on toxicity benchmarks and found that aligned models generated fewer toxic or unsafe responses compared to earlier GPT systems.

What makes these findings historically important is that they changed the industry’s understanding of what “better AI” actually meant.

Before InstructGPT, improvement was mostly measured through benchmark scores, scaling curves, and parameter counts.

After InstructGPT, researchers increasingly focused on usability, safety, alignment, conversational quality, and human preference satisfaction.

This was a major shift in AI development philosophy.

Truthfulness and Hallucinations

A major challenge for language models is that fluent responses are not always truthful.

This behavior is now commonly called hallucination.

Hallucinations can take many forms, including invented facts, fabricated references, incorrect explanations, or confident answers that lack factual support.

And because the responses are fluent and natural, the mistakes can sometimes look believable to users. The InstructGPT paper treats this as a serious issue rather than a minor flaw.

The authors note that language models are optimized for plausibility rather than verified truth. This is an important distinction: a language model can generate text that looks correct while still being inaccurate.

This is why the paper places particular emphasis on truthfulness and factual reliability.

Through RLHF and human preference optimization, InstructGPT was trained to produce answers humans judged to be more accurate and trustworthy. Human evaluators generally preferred responses that were more transparent about uncertainty and less likely to contain misleading information.

The paper also evaluates the model on truthfulness benchmarks such as TruthfulQA, where aligned models demonstrated improvements compared to earlier GPT systems.

But the paper is also careful not to overstate the results. Hallucinations didn't disappear. The aligned models could still make reasoning mistakes, generate false information, misunderstand prompts, or produce overconfident answers.

This nuance is extremely important: the paper doesn't claim that RLHF solved factuality or reasoning completely. Instead, alignment improved behavior, not perfection.

That distinction became increasingly important as ChatGPT and later GPT-4 systems reached millions of users worldwide.

The models became more useful, more truthful, and more aligned, but they still remained probabilistic language models rather than guaranteed fact engines.

In many ways, the InstructGPT paper marks the beginning of large-scale efforts to make AI systems not only intelligent, but also trustworthy enough for real-world human interaction.

Safety and Refusal Behavior

As language models became more powerful, researchers realized that safety was becoming a deployment problem.

A model that can generate human-like language at scale can also generate harmful instructions, produce toxic content, spread misinformation, or be manipulated into unsafe behavior.

The InstructGPT paper treats these risks very seriously and frames alignment as a necessary part of deploying large language models responsibly.

One of the biggest changes introduced through RLHF was safer refusal behavior.

Earlier GPT systems often attempted to answer almost anything. As a result, they often responded to unsafe prompts rather than recognizing when a refusal was appropriate.

InstructGPT begins changing that behavior.

Through human feedback and preference optimization, the model learns that some requests shouldn't be answered directly. Human labelers consistently prefer safer responses, refusals for harmful instructions, and outputs that avoid dangerous or toxic behavior.

This leads to systems that are better at refusing unsafe requests, avoiding toxic generations, and behaving more cautiously during interaction.

The paper also evaluates toxicity reduction using safety-related benchmarks and finds that aligned models generally produce fewer harmful outputs than earlier GPT systems.

Another important issue is harmful content filtering. Large language models absorb patterns from massive internet datasets, which inevitably contain biased language, misinformation, unsafe instructions, and toxic behavior.

Without alignment, models may reproduce these patterns surprisingly easily.

RLHF acts as a corrective layer on top of pretraining. Instead of only imitating internet text, the model is further optimized toward responses humans judge to be safer and more appropriate.

Of course, the paper is also realistic about limitations.

The authors acknowledge that alignment is incomplete and that unsafe outputs can still occur. Models may still be vulnerable to adversarial prompting or attempts to bypass safety behavior (what later became widely known as jailbreaks).

This is an important nuance: alignment reduces risk, but it doesn't eliminate it.

And historically, this realization became incredibly important for the future of large-scale AI deployment.

In many ways, the InstructGPT paper marks the beginning of modern AI safety engineering inside flagship language models.

InstructGPT introduced large-scale behavior alignment. Then GPT-4 expanded this even further with red teaming, adversarial testing, deployment monitoring, and much larger safety evaluation pipelines.

So this paper becomes a direct bridge between early generative language models and the much more safety-focused AI systems that followed in the GPT-4 era.

Limitations

One of the strongest aspects of the InstructGPT paper is that it doesn't present alignment as a solved problem.

Even though the results are impressive, the authors are careful and surprisingly honest about the system’s remaining weaknesses and risks.

This balance is important because the paper isn't arguing that RLHF creates perfect AI systems. The authors consistently frame alignment as a work in progress rather than a finished solution.

One major limitation is that the models still hallucinate.

The paper acknowledges that hallucinations remain a significant challenge despite alignment improvements.

RLHF improves truthfulness and instruction adherence, but it doesn't fundamentally solve the probabilistic nature of language models. The system still predicts likely text patterns rather than verifying objective truth.

Another important issue is reward hacking.

Because the model is optimized against a learned reward signal, it can sometimes discover shortcuts that maximize reward without genuinely improving reasoning or understanding. In other words, the model may learn behaviors that look aligned to evaluators while still hiding deeper problems underneath.

This is a common challenge in reinforcement learning systems more broadly.

The paper also hints at a problem that later became widely discussed in ChatGPT-era systems: over-refusal and sycophancy.

Sometimes aligned models become too cautious and refuse harmless requests unnecessarily. In other cases, models may become overly agreeable, telling users what they appear to want to hear instead of providing more balanced or truthful responses.

This creates a difficult tension between safety, helpfulness, and honesty.

Another major limitation is bias.

Since these systems are trained on massive internet datasets and further shaped through human labeling, they inevitably inherit biases from both sources. The paper explicitly acknowledges that alignment doesn't remove all harmful or biased behavior.

And perhaps most importantly, the paper emphasizes that RLHF aligns models to labeler preferences not universal human values. This is a very important nuance.

The system learns from the judgments of specific human annotators operating within specific cultural and organizational contexts. That means alignment itself is subjective and imperfect.

There is no single universally agreed definition of helpfulness, fairness, safety, or acceptable behavior.

The paper discusses these concerns carefully and recognizes that human feedback introduces its own limitations and assumptions.

The alignment itself is also fragile. Even aligned systems can sometimes be manipulated through adversarial prompting or jailbreak-style attacks that bypass safety behavior. This later became one of the defining challenges of ChatGPT and GPT-4 deployment.

And finally, there's the practical issue of scale.

RLHF requires large amounts of human labeling, ranking, evaluation, and monitoring. Building these alignment pipelines is expensive, time-consuming, and operationally complex. Unlike raw pretraining data scraped automatically from the internet, human feedback doesn't scale nearly as easily.

In many ways, the paper reveals an important truth about modern AI systems: making models intelligent is difficult. But making them reliably aligned with humans may be even harder.

Historical Importance

Looking back now, it's difficult to overstate how important the InstructGPT paper became for the entire AI industry.

Earlier GPT papers focused mostly on one central question: How do we make language models more capable?

That era was largely driven by larger datasets, larger parameter counts, scaling laws, and benchmark performance.

The models became increasingly impressive at generating text, solving tasks, and demonstrating emergent abilities. But they still behaved primarily like prediction engines trained to continue internet text.

InstructGPT changed the focus completely. For the first time, large-scale AI development began shifting from model-centric AI to interaction-centric AI.

This was a major philosophical transition: the industry realized that users didn't only care about raw intelligence, benchmark scores, or parameter counts.

They cared about usability, conversational quality, safety, trust, and whether the system could actually help them effectively.

This is why ChatGPT felt so different to the public. The underlying language model capabilities were important, but the real breakthrough came from how those capabilities were shaped into a usable human experience.

The interface became conversational. The system became more cooperative. The AI became more aligned with user intent.

That shift fundamentally changed public perception of artificial intelligence.

Before ChatGPT, most people saw AI as research software, technical demos, or specialized tools for experts.

After ChatGPT, millions of people started interacting with AI systems conversationally on a daily basis.

And that changed everything.

Earlier GPT papers focused mainly on discovering what scaling could achieve. InstructGPT introduced a different challenge: How do we safely deploy these systems in the real world?

That shift helped create entirely new areas of research and engineering, including RLHF pipelines, safety tuning, refusal behavior, red teaming, adversarial testing, policy frameworks, and large-scale human-feedback infrastructure.

In many ways, the ChatGPT era began the moment researchers realized that building powerful models was only part of the problem.

The harder challenge was making those systems reliable enough for human interaction at global scale.

It also helps explain why later systems placed much greater emphasis on safety, alignment, deployment practices, and real-world reliability.

The industry was no longer building language models only for research papers. It was building AI systems intended to operate in the real world. And the InstructGPT paper became one of the clearest turning points in that transformation.

Discussion: The Real Shift

The transition from GPT-3 to ChatGPT represents something much deeper than a simple improvement in model performance.

It changed the central question driving the entire AI industry.

During the GPT-3 era, the big question was, “Can language models learn tasks directly from prompts?”

That was the breakthrough introduced by GPT-3.

Research attention shifted toward scaling and emergent capabilities.

But the ChatGPT era introduced a completely different challenge: the question was no longer simply “Can the model perform the task?” Instead, it became, “Can humans actually trust and use these systems every day?”

That shift changed everything.

Once millions of people began interacting with AI systems directly, raw intelligence alone was no longer sufficient. Users needed systems that were understandable, reliable, safe, conversational, and aligned with human expectations.

This is exactly why the InstructGPT paper became so historically important. It introduced the idea that large language models should not only optimize for capability, but also for human interaction quality.

In many ways, the industry moved from “How smart is the model?” to “How usable is the model?”

And that transition fundamentally changed AI development.

After ChatGPT, success was no longer measured only by benchmark scores, parameter counts, or scaling curves.

It was increasingly measured by alignment, conversational quality, safety, and real-world usability.

This also explains why alignment research suddenly became central to modern AI systems.

GPT-3 showed that models could learn from prompts. ChatGPT showed that humans needed models that could cooperate.

That was the real shift.

And it may ultimately become one of the most important turning points in the history of artificial intelligence.

Connection to GPT-4

One of the most important things to understand about GPT-4 is that it didn't appear out of nowhere.

It was built on top of the alignment ideas introduced by InstructGPT and refined through the large-scale deployment experience of ChatGPT.

GPT-4 is often discussed in terms of its reasoning, multimodal abilities, and benchmark performance.

But beneath all of those improvements is something equally important: the alignment pipeline.

Without the work introduced in the InstructGPT paper, GPT-4 would likely feel far less usable as a real-world assistant.

That distinction matters enormously.

Many of GPT-4's alignment techniques can be traced back to ideas introduced by InstructGPT, including RLHF, instruction tuning, conversational alignment, safer refusal behavior, and human preference optimization.

ChatGPT then became the large-scale real-world testing ground for these ideas.

Millions of user interactions exposed weaknesses ranging from hallucinations and jailbreak attempts to broader safety and usability issues.

Those deployment lessons became incredibly valuable.

By the time GPT-4 arrived, OpenAI was no longer simply training a larger language model. It was building a large-scale aligned conversational system shaped by RLHF pipelines, human feedback, safety engineering, adversarial testing, and real-world user interaction.

This is why GPT-4 feels fundamentally different from earlier GPT models.

In many ways, GPT-4 represents the convergence of two major ideas: scaling capability and scaling alignment.

GPT-3 proved that language models could learn tasks from prompts.
InstructGPT proved that models could be shaped through human feedback.
ChatGPT proved that aligned conversational AI could work at global scale.
GPT-4 combined all of those ideas into a much more capable multimodal system.

That historical progression is important because it shows that modern AI systems aren't built through scaling alone. They're built through the combination of intelligence, alignment, interaction design, and deployment experience.

And the InstructGPT paper became one of the key foundations that made GPT-4 possible.

GPT-3 vs InstructGPT vs ChatGPT vs GPT-4: Key Differences

By this point, we've discussed GPT-3, InstructGPT, ChatGPT, and GPT-4 individually. But it can be helpful to see them side by side.

Although these systems are closely related, each one introduced a different shift in the evolution of modern AI.

GPT-3 focused on capability through scale, InstructGPT focused on alignment through human feedback, ChatGPT focused on conversational usability, and GPT-4 combined these ideas with stronger reasoning and multimodal capabilities.

The table below summarizes the main differences between them and shows how each system built on the progress of the previous generation.

Aspect	GPT-3	InstructGPT	ChatGPT	GPT-4
Core Idea	Large-scale language model enabling few-shot and in-context learning	Align language models with human instructions using RLHF	Conversational AI assistant optimized for dialogue and usability	Aligned multimodal foundation model with stronger reasoning and deployment maturity
Main Goal	Scale capability through massive pretraining	Improve instruction following and alignment	Deliver usable conversational AI for the public	Build reliable multimodal AI systems for real-world deployment
Training Objective	Predict next token from internet-scale text	Optimize outputs using human feedback and preference learning	Conversational interaction optimized through RLHF and dialogue tuning	Large-scale multimodal pretraining combined with RLHF, safety tuning, and deployment optimization
Alignment Focus	Minimal explicit alignment	Central focus of the paper	Strong conversational alignment	Advanced alignment and safety engineering
RLHF Usage	Not central	Core innovation of the system	Major component of interaction quality	Expanded and refined at larger scale
Human Feedback Role	Limited	Human rankings shape model behavior directly	Human feedback improves conversation flow and usability	Human feedback combined with large-scale safety evaluation and red teaming
Interaction Style	Prompt-based text generation	Instruction-following assistant	Natural multi-turn conversational assistant	Advanced conversational and multimodal assistant
Prompting Style	Zero-shot, one-shot, and few-shot prompting	Instruction prompts become more reliable	Conversational prompting becomes primary interface	Conversational and multimodal prompting
Conversation Memory	Limited contextual continuity	Better instruction adherence	Maintains dialogue flow across interactions	Stronger contextual reasoning across longer interactions
Instruction Following	Often inconsistent	Significantly improved	Strong conversational instruction following	More reliable and nuanced instruction handling
Truthfulness	Frequent hallucinations and overconfidence	Improved factual alignment through RLHF	More reliable but still hallucinates	Improved reasoning and factual performance, though hallucinations remain
Safety Behavior	Weak safety control	Safer refusal behavior introduced	More robust refusal and moderation behavior	Advanced safety pipelines and adversarial testing
Harmful Output Handling	Often continues unsafe prompts	Learns safer refusals from human feedback	Stronger refusal behavior in public deployment	More sophisticated alignment and safety systems
Reasoning Ability	Strong emergent reasoning for its time	Similar base capability but behaviorally improved	Improved practical reasoning in conversation	Major leap in reasoning and problem-solving
Multimodal Capability	Text only	Text only	Primarily text-based at launch	Text and image multimodal understanding
Coding Ability	Strong code generation emergence	Improved usability for coding tasks	Widely used as coding assistant	Much stronger coding and debugging performance
Context Handling	2048-token context window	Similar GPT-3-based context limits	Improved conversational memory handling	Much larger context capabilities
Model Size	175B parameters	Fine-tuned versions of GPT-3 models	Based on aligned GPT-3.5/GPT-4 systems	Undisclosed by OpenAI
Training Data	Massive internet-scale text datasets	GPT-3 pretraining plus human demonstrations and rankings	Large conversational interaction tuning datasets	Large-scale multimodal and internet-scale datasets
Learning Paradigm	In-context learning through scale	Human preference learning through RLHF	Conversational alignment at deployment scale	Combined capability scaling and alignment scaling
Key Innovation	Emergent few-shot learning	RLHF-based alignment pipeline	Conversational AI interface revolution	Multimodal aligned foundation systems
User Experience	Powerful but difficult to control	More cooperative and instruction-aware	Feels like talking to an assistant	More reliable, capable, and multimodal interaction
Reliability	Often unstable across prompts	More stable instruction behavior	Significantly improved usability	Stronger robustness and interaction quality
Deployment Style	Research and API usage	Alignment research milestone	Mass public deployment	Large-scale multimodal deployment
Benchmark Emphasis	Capability scaling and few-shot tasks	Human preference evaluations and alignment	Real-world conversational usability	Broad multimodal benchmark dominance
Main Limitation	Poor alignment and hallucinations	Alignment still incomplete and subjective	Hallucinations and jailbreak vulnerabilities	Hallucinations, safety tradeoffs, and lack of transparency
Historical Importance	Proved scaling produces emergent abilities	Introduced modern alignment-centered LLM training	Brought conversational AI to mainstream global use	Defined the era of aligned multimodal AI systems
What Changed in AI	Prompting became central	Alignment became a core research priority	AI became a mainstream consumer interface	AI became deployable multimodal infrastructure
Legacy	Foundation of prompt-driven AI	Foundation of ChatGPT alignment pipeline	Popularized conversational AI globally	Established modern multimodal AI ecosystem

From GPT-1 to GPT-4: A Timeline of Modern AI Systems and Alignment Evolution

Before we wrap up, it's worth stepping back and looking at the bigger picture.

The InstructGPT paper didn't emerge in isolation. It was part of a much larger evolution that transformed GPT models from research-focused language models into the conversational AI systems we use today.

Each generation introduced a new idea that pushed the field forward.

GPT-1 introduced large-scale pretraining, GPT-2 demonstrated zero-shot capabilities, GPT-3 popularized prompting and in-context learning, and InstructGPT introduced alignment through human feedback. ChatGPT then brought these ideas to millions of users through a conversational interface, while GPT-4 combined alignment with stronger reasoning and multimodal capabilities.

The timeline below summarizes the key transitions that shaped the modern AI era.

Year	System	Main Transition	What Changed	Key Paper / Release	Historical Importance
2018	GPT-1	Pretraining + Fine-Tuning Era	Introduced generative pretraining using Transformers before supervised fine-tuning	Improving Language Understanding by Generative Pre-Training	Started the modern large-scale NLP pretraining paradigm
2019	GPT-2	Zero-Shot Language Modeling Era	Showed that larger language models could perform multiple tasks without task-specific fine-tuning	Language Models are Unsupervised Multitask Learners	Shifted AI toward general-purpose generative models
2020	GPT-3	In-Context Learning Era	Demonstrated few-shot, one-shot, and zero-shot learning at massive scale using prompts alone	Language Models are Few-Shot Learners	Made prompting the central interface for AI systems
March 2022	InstructGPT	Alignment and RLHF Era	Introduced reinforcement learning from human feedback (RLHF) to align models with user intent	Training Language Models to Follow Instructions with Human Feedback	Shifted AI development from raw capability to alignment and usability
Nov 2022	GPT-3.5 / ChatGPT	Conversational AI Era	Combined GPT-3.5 with RLHF and chat-based interaction for public deployment	ChatGPT public release based on GPT-3.5 family	Turned LLMs into mainstream conversational assistants used globally
2023	GPT-4	Multimodal Aligned Foundation Model Era	Expanded aligned AI into multimodal reasoning across text and images with stronger reliability and safety systems	GPT-4 Technical Report	Established the modern era of deployable multimodal AI systems
2023–Present	GPT-4 + ChatGPT Ecosystem	AI Assistant Infrastructure Era	AI systems evolved into integrated assistants for coding, education, productivity, reasoning, and multimodal interaction	GPT-4 deployment ecosystem	Transitioned AI from research products into global infrastructure platforms

Final Insight

When people look back at the history of modern AI, they often focus on the moments when models became larger, more powerful, or more capable. But the story of the GPT series is not just a story about scale. It is also a story about learning how to make that intelligence useful.

GPT-1 showed that language models could learn surprisingly rich representations from large amounts of text before being adapted to specific tasks.

GPT-2 expanded that idea and revealed that scale itself could unlock new behaviors.

GPT-3 pushed the field into entirely new territory, demonstrating that a single model could perform a wide variety of tasks simply by responding to prompts and examples.

For a moment, it seemed as though scaling might be the answer to everything.

Then InstructGPT arrived and exposed a different challenge.

The problem was no longer whether a model could generate text, answer questions, or complete tasks. Models were already becoming remarkably capable.

The real question was whether people could actually rely on them. Could they follow instructions consistently? Could they respond in ways users found helpful? Could they become something more than sophisticated prediction engines?

That was the breakthrough at the heart of InstructGPT.

Rather than focusing solely on making models smarter, the paper focused on making them behave better.

Human feedback became part of the training process itself.

Alignment moved from a research concern to a core design principle. For the first time, improving the relationship between humans and AI became just as important as improving the model's raw capabilities.

The impact of that shift extended far beyond a single paper.

It laid the groundwork for ChatGPT, which introduced millions of people to conversational AI. Suddenly, interacting with advanced language models no longer required APIs, research expertise, or carefully engineered prompts. People could simply ask questions, seek advice, explore ideas, or learn something new through natural conversation.

That change transformed AI from a research breakthrough into a widely used product.

GPT-4 would later build on this foundation, combining stronger reasoning and broader capabilities with the alignment techniques that began with InstructGPT. But by then, the industry had already learned an important lesson: capability alone was not enough. Intelligence had to be usable.

In hindsight, the lasting significance of the InstructGPT paper is not that it introduced a new training pipeline. It is that it helped redefine the goal of modern AI.

The challenge was no longer just building systems that could generate language.

It was building systems that people could work with, learn from, and trust.

And that may ultimately be the transition that defined this era of artificial intelligence.

Resources:

Contact Me

How to Run an LLM Locally on Your Mobile Phone with QVAC and Expo

Jibril-M🍀 — Wed, 03 Jun 2026 17:17:33 +0000

When I was younger, I remember my mother’s Android phone, a Samsung Galaxy Note 3 that she bought right after losing her BlackBerry. During that time, a phone with 16 GB of storage was considered cutting-edge technology. The ability to store five 720p torrented movies on a single phone honestly felt unreal.

Most flagship devices back then shipped with somewhere between 2 and 8 GB of RAM, and GPUs were nowhere near what we carry around today. My mom’s Galaxy Note 3 featured the Qualcomm Adreno 330 GPU with 32 unified shader cores running at up to 578 MHz — a complete powerhouse for its time.

Fast forward to today, and the phones in our pockets are ridiculously more powerful, more efficient, and, honestly, capable of things people would’ve considered science fiction back then.

But enough about my mom’s phone. What I’m really trying to say is this: instead of spending hundreds of dollars every month on AI subscriptions and tokens, we can take advantage of the insanely capable devices we already carry around every day.

Modern smartphones now have dedicated AI acceleration, impressive thermal efficiency, and enough compute power to run lightweight language models locally, completely offline. That means better privacy, full control over your chat history, lower latency, and the ability to use AI without depending entirely on cloud services.

In this article, we’re going to build a React Native application that interacts with an LLM running directly on the device itself. The implementation will revolve around QVAC, a family of inference tools designed specifically for running AI models locally.

Prerequisites
What is QVAC?
Environment Setup
Model Management
Custom Models
Complete Implementation
Codebase Breakdown
Conclusion
Resources & Further Reading

Prerequisites

To get the most out of this article, you should have a basic understanding of front end development and React in general. You don't have to be a mobile developer, but understanding React will help a lot.

What is QVAC?

QVAC (QuantumVerse Automatic Computer) is a local-first AI inference platform developed by Tether. It's designed to move artificial intelligence away from centralized cloud systems and bring computation back to the user’s own device.

Most modern AI tools rely heavily on remote servers, API keys, and cloud infrastructure controlled by a handful of companies. While this makes AI accessible, it also creates major concerns around privacy, censorship, vendor lock-in, internet dependency, and ownership of user data. Every prompt, conversation, or uploaded file often passes through third-party servers that users have little control over.

QVAC was designed to solve that problem by allowing AI models and agents to run directly on devices like smartphones, laptops, and embedded systems, even while completely offline. Instead of sending personal conversations and sensitive data to the cloud, users can process everything locally on their own hardware.

The platform also embraces decentralization through peer-to-peer communication, reducing reliance on centralized infrastructure and eliminating single points of failure. This approach makes AI systems more private, resilient, autonomous, and accessible, especially in environments with limited internet access or strict data privacy requirements.

In simple terms, QVAC exists to make AI truly owned by its users — local-first, private by default, and independent from centralized control.

Environment Setup

To speed up the process, I prepared a React Native starter project with all the dependencies installed. But we will install and set up QVAC in this article, since that's our main topic. Here's a link to the repository.

Or you can run the below command to clone the starter project.

git clone --branch ft-ui-implementation --single-branch https://github.com/DjibrilM/QVAC-offline-Chatbot-Article-Project-

QVAC Installation

Run the following command to install the SDK: npm i @qvac/sdk. Feel free to use any package manager of your choice. As for me, I will keep things simple with npm.

Then add the following peer dependencies to your package.json:

{
  "dependencies": {
    "@qvac/sdk": "^0.7.0",
+   "bare-rpc": "^1.0.0", 
    "expo": "~54.0.33",
    "expo-status-bar": "~3.0.9",
    "react": "19.1.0",
    "react-native": "0.81.5",
+   "react-native-bare-kit": "^0.11.5"  
  },
  "devDependencies": {
    "@types/react": "~19.1.0",
    "bare-pack": "^1.5.1", 
    "typescript": "~5.9.2"
  }
}

Install the following additional dependencies:

npx expo install expo-file-system expo-build-properties expo-device

Then configure expo-build-properties and add @qvac/sdk/expo-plugin to the plugins array in your app.json:

{
  "expo": {
    "plugins": [
      "expo-router",
      "@qvac/sdk/expo-plugin",
      [
        "expo-splash-screen",
        {
          "backgroundColor": "#208AEF",
          "android": {
            "image": "./assets/images/splash-icon.png",
            "imageWidth": 76
          }
        }
      ]
    ]
  }
}

Run the following command to build the native modules:

npx expo prebuild

Note: QVAC uses llama.cpp under the hood. Due to optimization requirements and native hardware dependencies, the QVAC SDK doesn't run on emulators. You'll have to test this with a real physical device with Developer Mode enabled.

To run the app on your physical device, execute:

# For Android:
npx expo run:android --device

# For iOS:
npx expo run:ios --device

Model Management

The QVAC model management system is completely local-first and decentralized. It handles the entire lifecycle, from downloading files to lifecycle optimization, abstracting everything behind clean utility APIs.

Resumable & Deduplicated Downloading (`downloadAsset`)

It writes temporary chunks to local disk. If a network drop occurs, the partial file is preserved and resumes automatically upon the next call. Also, if multiple components invoke a download for the same asset simultaneously, QVAC handles the streaming under a single network stream.

Memory Lifecycle (`loadModel` & `unloadModel`)

loadModel maps the asset file directly into memory, maps it to your hardware target (such as the device GPU), and exposes an ephemeral modelId. Because local inference is highly memory-intensive on mobile devices, calling unloadModel frees system RAM immediately while preserving the downloaded file on disk.

Custom Models

Because QVAC relies on an optimized branch of llama.cpp, it remains highly compatible with the open-source AI ecosystem. If you plan to load custom models, ensure they adhere to these criteria:

Format: Must be in the GGUF (.gguf) format.
Quantization: For mobile and edge deployments, always prioritize Q4_0, Q4_K_M, or Q8_0 configurations to guarantee they fit safely within mobile hardware RAM constraints.

Complete Implementation

Now let's replace your main file codebase logic with the full implementation, combining the UI container layout, user interaction state, model lifecycle setup, and real-time inference handling into a cohesive structure.

Replace your entry file with the following code:

import { ChatInput } from "@/components/chat-input";
import { ChatMessage, Message } from "@/components/chat-message";
import { ModelLoader } from "@/components/model-loader";
import { Button } from "@/components/ui/button";
import { Text } from "@/components/ui/text";

import {
  completion,
  deleteCache,
  downloadAsset,
  LLAMA_3_2_1B_INST_Q4_0,
  loadModel,
  type ModelProgressUpdate,
  VERBOSITY,
} from "@qvac/sdk";
import { SymbolView } from "expo-symbols";
import { useEffect, useRef, useState } from "react";

import {
  Clipboard,
  KeyboardAvoidingView,
  Platform,
  SafeAreaView,
  ScrollView,
  View,
} from "react-native";

const makeId = () => Math.random().toString(36).substring(2, 9);

export default function Index() {
  const [messages, setMessages] = useState([]);
  const [input, setInput] = useState("");
  const [isGenerating, setIsGenerating] = useState(false);

  // Model loading state
  const [modelId, setModelId] = useState(null);
  const [isModelLoaded, setIsModelLoaded] = useState(false);
  const [isDownloading, setIsDownloading] = useState(false);
  const [downloadProgress, setDownloadProgress] = useState(0);

  const scrollViewRef = useRef(null);
  const messagesRef = useRef([]);

  useEffect(() => {
    messagesRef.current = messages;
  }, [messages]);

  const startDownload = () => {
    setIsDownloading(true);
    setupModel();
  };

  // Automatically scroll to bottom when messages list updates
  useEffect(() => {
    if (scrollViewRef.current) {
      setTimeout(() => {
        scrollViewRef.current?.scrollToEnd({ animated: true });
      }, 100);
    }
  }, [messages, isGenerating]);

  const copyToClipboard = (text: string) => {
    if (Platform.OS === "web") {
      navigator.clipboard.writeText(text);
    } else {
      Clipboard.setString(text);
    }
  };

  const setupModel = async () => {
    try {
      setIsDownloading(true);
      setDownloadProgress(0);
      
      // 1. Local download path execution
      await downloadAsset({
        assetSrc: LLAMA_3_2_1B_INST_Q4_0,
        onProgress: (progress: ModelProgressUpdate) => {
          setDownloadProgress(progress.percentage / 100);
        },
      });

      setDownloadProgress(1);

      // 2. Load model into runtime memory
      const loadedModel = await loadModel({
        modelSrc: LLAMA_3_2_1B_INST_Q4_0,
        modelType: "llm",
        modelConfig: {
          device: "gpu",
          ctx_size: 2048,
          verbosity: VERBOSITY.ERROR,
        },
      });

      setModelId(loadedModel);
      setIsModelLoaded(true);
      setIsDownloading(false);
    } catch (e: any) {
      console.error("Error setting up model:", e);
      setIsDownloading(false);
    }
  };

  async function handleSend() {
    // Guard against sending before the model is ready or while generating.
    if (!modelId || isGenerating) return;

    const trimmed = input.trim();
    if (!trimmed) return;

    setInput("");
    setIsGenerating(true);

    // Append user message and a placeholder assistant message for streaming.
    const userMsg: Message = {
      id: makeId(),
      role: "user",
      content: trimmed,
    };

    const assistantId = makeId();

    const assistantMsg: Message = {
      id: assistantId,
      role: "assistant",
      content: "",
    };

    setMessages((prev) => [...prev, userMsg, assistantMsg]);

    try {
      // Build chat history for the completion request.
      const history = [...messagesRef.current, userMsg].map((m) => ({
        role: m.role,
        content: m.content,
      }));

      // Run a streaming completion and update the last assistant bubble.
      const result = completion({
        modelId,
        history,
        stream: true,
      });

      let acc = "";

      for await (const token of result.tokenStream) {
        acc += token;

        // Update only the last assistant message content
        setMessages((prev) =>
          prev.map((m) =>
            m.id === assistantId ? { ...m, content: acc } : m
          )
        );
      }

      // Optional: Log completion performance stats
      try {
        const stats = await result.stats;
        console.log("📊 Completion stats:", stats);
      } catch {}

    } catch (e: any) {
      // Show any error in the assistant bubble.
      setMessages((prev) =>
        prev.map((m) =>
          m.id === assistantId
            ? { ...m, content: `❌ Error: ${e?.message ?? String(e)}` }
            : m
        )
      );
    } finally {
      setIsGenerating(false);
    }
  }

  if (!isModelLoaded) {
    return (
      
    );
  }

  return (
    
      
        
          
            
            Local Llama 3.2
          
          Offline Engine
        

        
          {messages.filter(m => m.content !== "" || m.role === "assistant").map((msg) => (
             copyToClipboard(msg.content)}
            />
          ))}
        

        
      
    
  );
}

Codebase Breakdown

Let’s lift the hood on how this unified component manages local model workflows and real-time UI streaming.

1. Tracking Model State & Asynchronous Synchronization

At the root of the component, we track both user-facing interface state and underlying QVAC runtime handles:

const [messages, setMessages] = useState([]);
const [modelId, setModelId] = useState(null);
const [isModelLoaded, setIsModelLoaded] = useState(false);
const [isDownloading, setIsDownloading] = useState(false);
const [downloadProgress, setDownloadProgress] = useState(0);

Because state setters in React are asynchronous, streaming loops can accidentally capture stale representations of current chat logs.

To circumvent this, a mutable messagesRef acts as a real-time single source of truth for the active session state:

const messagesRef = useRef([]);

useEffect(() => {
  messagesRef.current = messages;
}, [messages]);

2. Orchestrating Download & Memory Instantiation

When the user strikes the download button action trigger, the application launches setupModel(). This function splits tasks clearly across local storage caching and active hardware allocation layers:

await downloadAsset({
  assetSrc: LLAMA_3_2_1B_INST_Q4_0,
  onProgress: (progress: ModelProgressUpdate) => {
    setDownloadProgress(progress.percentage / 100);
  },
});

Storage Sync: downloadAsset reaches out to pull the designated standard model signature down into mobile device disk files.
Hardware Binding: Once safe on disk, loadModel executes to wake up the engine runtime:

const loadedModel = await loadModel({
  modelSrc: LLAMA_3_2_1B_INST_Q4_0,
  modelType: "llm",
  modelConfig: {
    device: "gpu",
    ctx_size: 2048,
    verbosity: VERBOSITY.ERROR,
  },
});

Passing device: "gpu" tells QVAC to run hardware-accelerated kernels across the smartphone's graphic processing hardware structure, ensuring rapid performance metrics instead of locking execution to slower CPU loops.

3. Pipeline Ingest & Streaming Generation Loop

Once user validation confirms the prompt is ready, handleSend() sets up user bubbles and generates an empty assistant placeholder card to catch token output segments.

The application map transforms references straight out of messagesRef.current into a structured history syntax before processing:

const result = completion({
  modelId,
  history,
  stream: true,
});

With stream: true enabled, QVAC doesn't hold up your application thread waiting for long string sequences to complete. Instead, it yields an asynchronous iterable stream that spits out fresh updates instantly:

let acc = "";

for await (const token of result.tokenStream) {
  acc += token;

  setMessages((prev) =>
    prev.map((m) =>
      m.id === assistantId ? { ...m, content: acc } : m
    )
  );
}

The loop continuously concatenates token text variables into the tracking accumulator (acc), target patching state properties exclusively against our placeholder identifier (assistantId). This creates a lightning-fast typing animation experience while executing fully offline on your user's physical device hardware.

Conclusion

Building a local-first AI application is no longer a concept confined to high-end desktops or specialized research labs. As we’ve seen, the smartphones we carry in our pockets every day possess more than enough computational muscle and dedicated hardware acceleration to run highly capable language models completely offline.

By leveraging React Native and the QVAC SDK, we successfully bypassed the traditional cloud-dependent architecture. We eliminated the need for complex server infrastructure, API key management, and recurring token subscription fees, all while providing an ultra-private, low-latency, streaming chat experience directly on-device.

As open-source models continue to shrink in size and grow in capabilities, edge inference will become an essential architecture for developers prioritizing privacy, offline resilience, and cost efficiency. The power to compute is back where it belongs: in the hands of the user.

Resources & Further Reading

To dive deeper into local inference, inspect the source code, or explore advanced configurations for your mobile applications, check out the following resources:

QVAC Expo Integration Tutorial – The official step-by-step documentation for configuring QVAC within the Expo and React Native ecosystems.
Project GitHub Repository – Access the complete source code, including the UI layout components, starter themes, and full configuration files used in this guide.
Llama.cpp Official Repository – Learn more about the underlying inference engine that powers QVAC's hardware-accelerated local execution.
Hugging Face GGUF Models – Explore thousands of open-source, quantized models that you can download and experiment with inside your local application.

The LLM Gateway Pattern: Why Every Kubernetes-Based AI App Needs One

Temitope Oyedele — Wed, 03 Jun 2026 17:12:17 +0000

You ship your first LLM-powered feature. It works and the users love it. A second team adds another feature calling a different model, and a third integrates a completely different provider.

Six months later, you have fourteen microservices, each holding their own API keys, writing their own retry logic, and failing in their own unique ways.

Nobody knows how much you're spending on tokens or which service is hammering the rate limit. And when OpenAI goes down, everything goes down with it.

That scenario plays out across engineering teams every single day, and the root cause is almost always the same: moving fast with LLMs while skipping the infrastructure thinking that holds everything together at scale.

Fortunately, a well-established architectural pattern solves exactly these problems. If you already run Kubernetes, you're more than halfway to implementing it. That pattern is called the LLM Gateway Pattern, and this article walks you through what it is, why it matters, and how to put it into practice.

What Is the LLM Gateway Pattern?
- How It Works
The Problem Without a Gateway
Deploying an LLM Gateway on Kubernetes
Features of an LLM Gateway
Wrapping Up

What Is the LLM Gateway Pattern?

The LLM Gateway Pattern is an architectural approach where all LLM API traffic from your applications flows through a single, centralized proxy service before reaching any external provider. Think of it as the AI equivalent of an API gateway, except it's purpose-built for the unique challenges that come with language models: token budgets, streaming responses, model routing, semantic caching, and multi-provider fallback.

Instead of every service in your cluster talking directly to OpenAI or Anthropic, they all talk to one internal gateway. That gateway handles authentication, routing, rate limiting, logging, and failover. Your application services stay clean and focused on business logic, while the gateway takes on all the messy operational concerns of working with LLMs at scale.

The pattern itself is not new in concept. Engineers have used API gateways for years to manage REST traffic. What makes LLM gateways distinct is that they understand the specific shape of LLM requests, including token counts, model parameters, prompt structure, and streaming semantics.

How It Works

The core components of an LLM Gateway on Kubernetes are straightforward. Here is the high-level flow:

App Pods send requests to the gateway using a standard OpenAI-compatible API format. Because of this, most existing LLM client libraries work without modification — you just change the base URL to point at your internal gateway service.

The Gateway Service receives each incoming request, authenticates the caller, applies any configured rate limits, checks the cache, selects the appropriate upstream provider based on routing rules, and forwards the request. On the way back, it logs token usage and latency before returning the response to the caller.

ConfigMap holds the routing rules. Which model should handle requests tagged as fast? Which provider should the system fall back to if the primary one is unavailable? All of this lives in configuration, not code, so you can update routing behaviour without redeploying anything.

Secrets hold the actual API keys for each provider. The gateway is the only service in the cluster that needs access to them. Application pods never touch provider credentials directly.

Provider endpoints are the actual LLM APIs: OpenAI, Anthropic, a self-hosted vLLM instance running in your cluster, or any other provider that exposes an OpenAI-compatible interface.

The Problem Without a Gateway

To appreciate why this pattern matters, it helps to look at what happens when you skip it.

1. Scattered Secrets and No Central Control

Every service that calls an LLM needs an API key. In Kubernetes, this usually means creating a Secret per namespace or per deployment.

When that key rotates or gets compromised, you're hunting through dozens of manifests to update it. There's no single place to revoke access or audit who is calling what.

2. No Visibility into Cost or Usage

LLM APIs charge per token. Without a centralized layer collecting usage data, you have no reliable way to know which service is responsible for that spike in your monthly bill.

3. Provider Lock-in at the Application Level

When you hardcode https://api.openai.com into your service, switching to a different provider or routing certain requests to a cheaper model becomes a code change. You need to redeploy your application just to change which model handles a request type.

4. No Caching

Many LLM applications send semantically similar or identical prompts repeatedly. Without a shared caching layer, each one incurs full token costs and full latency. The savings from even basic caching can be significant.

All of these problems compound as your team grows and more services start calling LLMs. The gateway pattern cuts through all of them in one architectural decision.

Deploying an LLM Gateway on Kubernetes

There are several tools that can serve as an LLM gateway in a Kubernetes environment, including LiteLLM Proxy, Portkey, OpenRouter, and Envoy with custom filters.

For the rest of this walkthrough, we'll use LiteLLM Proxy. It ships with a Helm chart, supports over a hundred models across all major providers, and comes with a management UI that makes initial configuration straightforward.

Storing API Keys Securely

Start by creating a Kubernetes Secret that holds your provider API keys. Your gateway pods will consume these credentials as environment variables, which means no provider key ever needs to live inside your application containers:

apiVersion: v1
kind: Secret
metadata:
  name: llm-gateway-secrets
  namespace: ai-platform
type: Opaque
stringData:
  OPENAI_API_KEY: "sk-..."
  ANTHROPIC_API_KEY: "sk-ant-..."

Defining Routing Rules in a `ConfigMap`

The routing configuration tells the gateway which models are available and how to reach each one. Keeping this in a ConfigMap means you can update your routing rules without touching a single line of application code:

apiVersion: v1
kind: ConfigMap
metadata:
  name: llm-gateway-config
  namespace: ai-platform
data:
  config.yaml: |
    model_list:
      - model_name: gpt-4o
        litellm_params:
          model: openai/gpt-4o
          api_key: os.environ/OPENAI_API_KEY
      - model_name: claude-sonnet
        litellm_params:
          model: anthropic/claude-sonnet-4-20250514
          api_key: os.environ/ANTHROPIC_API_KEY
      - model_name: fast
        litellm_params:
          model: openai/gpt-4o-mini
          api_key: os.environ/OPENAI_API_KEY

With this configuration in place, any application in your cluster can reach the gateway at http://llm-gateway.ai-platform.svc.cluster.local using the standard OpenAI client format, regardless of which actual provider sits behind it.

Scaling the Gateway

Because the gateway is stateless, horizontal scaling is straightforward. You can attach a HorizontalPodAutoscaler to scale based on CPU utilization or request rate:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-gateway-hpa
  namespace: ai-platform
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-gateway
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

Wiring Up Observability

A gateway you can't observe is a gateway you can't trust, so wiring up monitoring before you go to production is worth the extra hour it takes.

LiteLLM exposes a /metrics endpoint in Prometheus format. You can scrape it with a standard ServiceMonitor if you run the Prometheus Operator, or configure Prometheus directly to target the gateway service.

The metrics that matter most in day-to-day operations are token throughput per model, request latency percentiles, error rates per provider, and cache hit ratio.

Once Prometheus is collecting that data, you can build Grafana dashboards that show token spend broken down by caller, model, and time period. This gives engineering managers and finance teams the cost visibility they've been asking for, and it takes surprisingly little effort to set up once the metrics pipeline is in place.

If you run an OpenTelemetry collector in your cluster, you can also configure the gateway to emit trace spans for every LLM request. This lets you see the full latency breakdown from the moment a user action triggers a call in your application all the way through to the provider response. So when something is slow, you can tell immediately whether the bottleneck sits in your service, the gateway, or upstream with the provider.

Features of an LLM Gateway

Not all gateway implementations are equal, so as your needs grow, these are the core capabilities worth evaluating.

Multi-Provider Routing

A well-built gateway routes requests to different providers based on declarative, configurable rules that live entirely outside your application code. This means that changing a model never requires a redeployment.

Semantic Caching

Rather than only caching byte-for-byte identical prompts, a semantic cache uses embedding similarity to recognise when two different prompts are asking essentially the same thing. This can cut redundant API calls dramatically.

Rate Limiting Per Consumer

The gateway should let you set token budgets and request limits per team, per namespace, or per application, so no single runaway service can starve the rest of your cluster or drive up costs unchecked.

Fallback and Failover

When a primary provider fails or exceeds acceptable latency thresholds, the gateway should automatically retry against a configured fallback. This centralizes logic that is notoriously hard to get right inside individual services.

Token Usage Tracking

Every request should produce a detailed usage record capturing input tokens, output tokens, model, caller identity, and latency. This gives engineering managers the clear, actionable picture of AI spending they need.

Wrapping Up

The LLM Gateway Pattern solves a set of operational problems that every team building on language models at scale will eventually run into. Scattered secrets, invisible costs, inconsistent failure handling, and provider lock-in are all symptoms of the same underlying issue: infrastructure concerns leaking into services that shouldn't have to deal with them.

A centralized gateway on Kubernetes gives your application teams a stable, provider-agnostic interface while giving your platform team the visibility and controls they need to manage cost and reliability effectively. When a provider goes down in the middle of the night, your configured fallback kicks in automatically instead of someone waking up to a page.

Start with LiteLLM Proxy, wire up the Prometheus metrics, build a simple Grafana dashboard, and watch how quickly the pattern pays for itself. Once you have seen what centralized LLM traffic management looks like in practice, it becomes very hard to go back to doing it any other way.

How to Design APIs for AI Agents

David Aniebo — Thu, 28 May 2026 22:12:00 +0000

APIs are designed for human developers. People read documentation, infer the intent behind an endpoint, and know how to handle edge cases when something unexpected happens.

AI agents don't have that context and understanding.

AI agent understand APIs through schemas, examples, randomized data and live responses. When a behavior or method is ambiguous and inconsistent, the model doesn't pause to “think” – it fills in the blanks (randomizing).

In production, those guesses could become blocks, retry storms, duplicated side effects, or broken workflows.

This is why APIs that are perfectly fine for humans frequently fail under AI agent use. The problem is rarely “the agent isn’t smart enough.” More often, the API was never designed for an agent/machine consumer that must plan, call tools, and recover from failure without a human in the loop.

In this guide, you’ll learn how to design APIs that agents can use reliably. We’ll anchor the discussion in three practical ideas:

Deterministic behavior: same inputs and state should yield predictable outcomes and shapes.
Strong schemas: contracts that are complete, descriptive, and testable.
Guardrails at the API boundary: authorization, validation, and safe defaults that prevent unsafe autonomy.

The aim of this article is not to build “AI-powered” APIs, but rather to build APIs that are clear, strict, and dependable, even when the caller is not an agent but a fellow developers leveraging various tools.

Prerequisites
Why “Good Enough for Devs” Is Not Good Enough for Agents
Principle 1: Deterministic Behavior
Principle 2: Strong Schemas
Principle 3: Guardrails at the API Boundary
Patterns That Bridge APIs and Agent Runtimes
A Practical Before and After Example
Checklist: Is Your API Agent-Ready?
Conclusion

Prerequisites

Before reading this guide, it helps to have:

A basic understanding of HTTP APIs and REST concepts
Familiarity with JSON and API request/response patterns
An understanding of common API concepts like authentication, pagination, and retries

Why “Good Enough for Devs” Is Not Good Enough for Agents

Human developers bring implied and contextual knowledge: they read through Slack threads, read blog posts, and recognize that “this 404 usually means you forgot the workspace ID.”

Agents mostly get whatever is in the spec, the examples, and the last response body.

That gap shows up in predictable ways:

Ambiguous semantics: wrong endpoint or wrong parameter combination.
Undocumented branches: the model invents fields or misreads optional behavior.
Inconsistent error bodies: retries that shouldn't happen, or no retry when one is safe.
Non-idempotent “do things” endpoints: duplicate charges, duplicate tickets, duplicate emails.

Industry commentary and practitioner guides converge on the same point: agents are becoming a major class of API consumer, and machine legibility matters as much as developer experience.

See for example discussions of OpenAPI as the source of truth for agents, emerging tool protocols, and traffic patterns that differ from human clients in the resources listed at the end of this article.

Principle 1: Deterministic Behavior

Determinism for agents doesn't mean “always return the same JSON forever.” It means: given the same request and the same server-side state, your API behaves in a way the agent can model and when state changes, you make that explicit.

Prefer Explicit State Over Hidden Magic

Agents struggle with “sometimes the server does X depending on internal flags.” Where humans infer intent from product copy, agents infer from patterns. If those patterns drift, autonomy breaks.

Practical habits:

Model lifecycle explicitly (draft → submitted → approved) instead of overloading a single status field with undocumented combinations.
Return what changed after mutations (updated resource, relevant IDs, next allowed actions).
Avoid silent coercion (auto-correcting bad enums, silently dropping unknown fields) unless you document and signal it.

Make Writes Safe: Idempotency and Intent Keys

For any endpoint that bills, sends messages, provisions infrastructure, or otherwise does something irreversible, assume double-submission will happen.

Support idempotency keys (header or body) for create-like operations.
Use clear HTTP semantics: POST creates, PUT replaces where appropriate, PATCH for partial updates and document what repeats mean.
Where duplicates are possible, offer a lookup-by-client-reference path so agents can reconcile.

Pagination and Sorting: One Pattern, Everywhere

Agents loop. If every resource paginates differently, the model will mix strategies.

To combat this, pick one pagination style (cursor vs offset) per API surface and stick to it.

Also, always return stable sort order or require sort explicitly. You should also include next links or cursors in a consistent envelope.

Timeouts, Partial Success, and Async Work

Agents hate “maybe it worked.” Long-running work should be explicitly async:

202 Accepted + job ID + polling or webhooks.
Clear terminal states: succeeded, failed, canceled, with structured error details on failure.

Principle 2: Strong Schemas

If determinism is about behavior, schemas are about communication. For agents, your OpenAPI (or equivalent) isn't paperwork, it's part of the runtime interface.

Treat OpenAPI as a Contract, Not a Souvenir

A specification that lags production is worse than no spec: it trains the agent to be confidently wrong. Teams increasingly treat OpenAPI as the authoritative contract and validate requests/responses against it in CI and at the edge.

Here's the minimum bar for agent-friendly OpenAPI:

Every operation has a summary and a description that explain when to use it, not only what it returns.
Every request body property has description and realistic example values.
All responses are documented including 4xx/5xx with stable JSON shapes.

Describe Intent in Natural Language, Precisely

Agents aren't offended by verbosity. They're confused by vague verbs.

Instead of:

“Gets orders.”

Prefer:

“Lists orders for the authenticated merchant. Supports filtering by status and a time window on created_at. Returns at most limit items; use cursor for the next page.”

This aligns with what multiple guides call context-aware or self-describing APIs: the schema carries semantic intent, not just types.

Examples Are Part of the Contract

You should provide a happy path example per endpoint, at least one validation error example (400) with your standard error object, and examples for optional fields when they change behavior.

Examples reduce “shape hallucination” where the model guesses field names or nesting.

JSON Schema Strictness Helps Tool-Calling Stacks

If your agent uses function calling / structured outputs, tighten schemas:

Prefer enum for small closed sets.
Mark fields required honestly.
Use format (uuid, date-time) where real.
Avoid additionalProperties: true on security-sensitive payloads if you need strict validation.

Name Things Consistently

userId in one endpoint and user_id in another is a human annoyance and an agent trap. Pick a convention and enforce it.

Principle 3: Guardrails at the API Boundary

Autonomy amplifies mistakes. Guardrails turn “oops” into blocked requests instead of incidents.

Authorization Should Be Narrow and Explicit

Agents should receive credentials scoped to least privilege. For example, use short-lived tokens, with refresh documented clearly. Use scopes that map to real actions (orders:read vs orders:write). And avoid flows that assume a human can solve (CAPTCHAs) or click (email links mid-run) or isolate those as human-in-the-loop tools.

Validate Hard, Fail Loud and Structured

Reject bad input at the edge with stable error_code values (machine-actionable), human-readable message (for logs and UI), optional field or JSON Pointer to the problem, and optional doc_url linking to documentation.

This matches guidance from several practitioner articles: opaque 500s and generic errors are where autonomous clients spiral.

RFC 7807 Problem Details (application/problem+json) is a good, widely understood pattern for HTTP APIs, a structured envelope agents can parse consistently.

Separate “Read the World” from “Change the World”

For high-impact actions (refunds, deletes, transfers), consider using a two-step pattern: first create an intent, then confirm execution.

Or you can dry-run query parameters / dedicated endpoints that validate without committing.

Also keep in mind that rate limits and quotas tuned for bursty agent behavior and autonomous loops can dwarf human traffic.

Observability is a Product Feature

Log correlation IDs, surface them in responses where safe, and monitor for retry amplification. An agent that misreads a 409 as “retry forever” becomes a denial-of-wallet attack on your own systems.

Patterns That Bridge APIs and Agent Runtimes

Workflow Documentation: Sequences, Not Just Endpoints

Agents excel when they can follow a recipe. Document common sequences (“create customer → add payment method → charge”) and consider standards meant for multi-step API flows (such as Arazzo) when your product’s complexity justifies it.

Hypermedia and “Next Steps”

Including links to plausible next actions (for example, pagination next, or related resources) reduces improvisation. This is the same spirit as HATEOAS: the response whispers what you can do next, instead of forcing the model to guess URLs.

Tool-Oriented Surfaces (For Example, MCP)

Protocols like the Model Context Protocol (MCP) are gaining traction as a way to expose curated capabilities (“tools”) with schemas agents can bind to directly.

A common pragmatic pattern is not to dump every micro-endpoint as a tool, but to expose coarse-grained tools aligned to user outcomes while keeping your underlying REST API strict and clean.

MCP isn't a substitute for good API design. It's a delivery and discovery layer. Slapping a thin wrapper on a messy API still leaves you with a messy system – it just fails faster in public.

Metadata for Discovery (`llms.txt` and Friends)

Some teams publish /llms.txt or similar lightweight discovery files for documentation sites. Treat these as optional signposts, not replacements for OpenAPI.

Ecosystem adoption is still evolving, but the underlying idea is sound: make the canonical machine-readable description easy to find.

A Practical Before/After

Weak Pattern (Agent-hostile)

POST /do-stuff

Response 200 OK:

{ "ok": true }

Problems: no idempotency, no structured error, no entity ID, no way to poll, the agent must guess whether “ok” means “created” or “ignored duplicate.”

Stronger Pattern (Agent-friendly)

POST /v1/invoices
Idempotency-Key: 7b3c-...

Response 201 Created:

{
  "invoice": {
    "id": "inv_9Qz",
    "status": "draft",
    "total": { "amount": "120.00", "currency": "USD" }
  },
  "links": {
    "finalize": "/v1/invoices/inv_9Qz/finalize"
  }
}

Conflict response 409 Conflict with Problem Details:

{
  "type": "https://api.example.com/problems/duplicate-idempotency-key",
  "title": "Duplicate idempotency key",
  "status": 409,
  "detail": "A different request body was sent with the same Idempotency-Key.",
  "error_code": "IDEMPOTENCY_KEY_REUSE_BODY_MISMATCH"
}

This tells the agent what happened and whether retrying is appropriate.

Checklist: Is Your API Agent-Ready?

Contract: Published OpenAPI 3.x, validated against real traffic, with rich descriptions and examples.
Determinism: Documented state machines, consistent pagination, explicit async for long jobs.
Safe writes: Idempotency for side effects, reconciliation endpoints where needed.
Errors: Stable codes, structured bodies, documented remediation paths.
Security: Least-privilege tokens, no “mystery” side doors agents can accidentally hit.
Operations: Rate limits, bulk endpoints where appropriate, correlation IDs, dashboards for anomalous agent traffic.

Conclusion

Designing for AI agents is, in most respects, disciplined API design — pushed to the level where machines can rely on your contract without tribal knowledge.

If you remember only three things:

Be predictable: in shapes, states, and side effects.
Be explicit: in schemas, examples, and errors.
Be protective: validate early, scope narrowly, and make dangerous actions hard to trigger by accident.

RAG Explained Simply with a Real Project

Ashutosh Krishna — Thu, 28 May 2026 16:17:22 +0000

If you have used ChatGPT, you know how magical it feels. You ask a question, and it instantly generates a highly articulate answer.

But you also probably know its biggest flaw. If you ask it about your company's internal code, your private Notion workspace, or an event that happened yesterday, it fails.

Usually, it does one of two things. It either apologizes and says it doesn't have access to that information, or worse, it confidently makes something up entirely.

This happens because Large Language Models (LLMs) are like extremely smart students who are locked in a room without internet access. They only know what they memorized before they were locked inside. If you ask them a question outside of their memorized knowledge, they have to guess.

So, how do we fix this? How do we get an AI to answer questions about our private data without retraining the entire model from scratch?

The answer is RAG, which stands for Retrieval-Augmented Generation.

RAG is the architecture behind nearly every modern AI application that interacts with private data. If you have ever used a "chat with PDF" app or a customer support bot that actually knows company policies, you have interacted with RAG.

In this article, we'll break down exactly how RAG works from first principles. Then, we'll build a working RAG application from scratch using Python.

Here's what we'll cover:

What is RAG?
Why Traditional LLMs Fail
How RAG Works Internally
How to Build a Real RAG Project
The Full Data Flow
Common RAG Problems
Advanced RAG Concepts
Final Thoughts

What is RAG?

RAG stands for Retrieval-Augmented Generation. Let's break down what those three words actually mean.

Retrieval: Finding relevant information from a database.
Augmented: Adding that information to the user's original question.
Generation: Asking the LLM to write an answer using only the added information.

The Open-Book Test Analogy

To build a mental model, think of a traditional LLM as a student taking a closed-book exam. The student has read billions of books in the past, but right now, they have to answer questions purely from memory. Sometimes they forget facts, and sometimes they make up answers to avoid leaving the page blank. Not gonna lie, I pulled the same move in quite a few university exams.

RAG turns this into an open-book exam.

When you ask a question, the system first runs to a massive library (your database), finds the exact pages that contain the answer, and hands those pages to the student. The student then reads those specific pages and writes a perfect answer.

Instead of relying on the AI's memory, we're only relying on its reading comprehension skills.

Why Traditional LLMs Fail

Before we dive into how to build RAG, we need to understand exactly why prompting an LLM on its own isn't enough.

Training cutoffs: Training an LLM takes months and costs millions of dollars. Because of this, models are trained on data up to a specific date. If an LLM was trained in 2025, it has absolutely no idea what happened in 2026.
No access to private data: Your company's Jira tickets, internal wikis, and Slack messages are private. OpenAI, Google, and Anthropic don't have them in their training datasets.
Hallucinations: LLMs are essentially advanced autocomplete engines. They predict the next most likely word based on patterns. If they don't know a fact, they'll string together words that sound highly plausible but may be completely incorrect. We call this hallucinating.
Context window limitations: You might be thinking, "Why not just copy and paste my entire company wiki into the ChatGPT prompt?" Well, every LLM has a "context window", which is the maximum amount of text it can process at once. Even with modern models that have massive context windows, pasting thousands of documents into a prompt is incredibly slow and expensive. Also, models tend to lose track of information when you overwhelm them with too much text.
The high cost of retraining: You could theoretically fine-tune an LLM on your private data. But fine-tuning is complicated and expensive. More importantly, knowledge changes constantly. If you update a company policy, you would have to fine-tune the model all over again to teach it the new rule.

RAG solves all of these problems. It gives the LLM access to real-time, private data without needing to retrain the model.

How RAG Works Internally

To make RAG work, we need a specific pipeline of technologies. Let's explore every major concept in the RAG architecture.

Documents

Everything starts with your raw data. These are your PDFs, database records, text files, or scraped websites. In the AI world, we refer to all of these source materials generally as "documents".

Chunking

You can't feed a 500-page book into an AI all at once for a simple question. It's inefficient. Instead, we break the documents down into smaller, manageable pieces called "chunks". A chunk might be a single paragraph or a few sentences.

This matters because when a user asks a question, we only want to retrieve the specific paragraphs that contain the answer, not the entire book. If we skipped chunking, the system would retrieve massive walls of text, which would crash the LLM's context window.

Embeddings

This is the most intimidating term for beginners, but the concept is brilliant. Computers don't understand words, but they're great at math. Embeddings are a way to translate human language into lists of numbers (vectors) that capture the actual meaning of the text.

Imagine a 2D map. We can plot the word "Dog" at coordinates [2, 3] and the word "Puppy" at [2.1, 3.1]. Even though they're different words, the computer knows they mean similar things because their coordinates are physically close together on the map. The word "Car" might be way over at [10, 10].

In a real AI system, an embedding model doesn't use just 2 dimensions. It maps sentences across thousands of dimensions to capture deep semantic meaning.

Vector Databases

Once we convert all of our text chunks into number coordinates (embeddings), we need a place to store them. Traditional SQL databases are great at finding exact keyword matches. But they're terrible at finding "similar meanings".

A vector database is specifically designed to store lists of numbers and quickly calculate the distance between them. Popular vector databases include ChromaDB, Pinecone, Weaviate, FAISS, and Milvus.

Semantic Search and Similarity Matching

When a user types a question into our chatbot, we run the question through the exact same embedding model. The question becomes a list of numbers.

We then ask the vector database to perform a similarity search. The database looks at the coordinates of the user's question and finds the stored chunks that are located closest to it in mathematical space. Because distance equals meaning, the closest chunks will contain the most relevant information to answer the question.

Prompt Augmentation

Now we have the user's original question and the text chunks we retrieved from the database. We "augment" (add to) the prompt. We create a hidden template behind the scenes that looks like this:

"You are a helpful assistant. Use ONLY the following context to answer the user's question.

Context:

[Insert retrieved chunks here]

Question:

[Insert user question here]"

Final LLM Response

We send this giant, augmented prompt to the LLM. The LLM reads the context, processes the question, and generates a factual response based entirely on the provided data.

Quick Recap

A RAG pipeline usually looks like this:

How to Build a Real RAG Project

Let's build a real-world RAG application. We'll build an AI chatbot that reads and understands a PDF document.

To make this completely free to build, we'll use Python, LangChain (a popular AI framework), Google's Gemini API (which has a generous free tier for developers), and ChromaDB (a local vector database).

Note: We'll be using the free Gemini tier here for illustration purposes so you can learn without spending money. Because LangChain is modular, you can easily swap this out for any other production-grade model later just by changing one line (or a few lines) of code.

Project Setup

First, open your terminal or command prompt, create a new directory for your project, and navigate into it:

mkdir my-rag-project
cd my-rag-project

Next, it's a best practice to create an isolated virtual environment. This ensures that the packages we install for this project don't conflict with other Python projects on your computer.

To create and activate a virtual environment, run the commands for your specific operating system:

For macOS and Linux:

python3 -m venv venv
source venv/bin/activate

For Windows (Command Prompt):

python -m venv venv
venv\Scripts\activate

For Windows (PowerShell):

python -m venv venv
.\venv\Scripts\Activate.ps1

Once activated, you'll see (venv) appear at the beginning of your terminal line. Now, go ahead and install the required libraries inside your fresh environment:

python -m pip install --upgrade pip
pip install langchain langchain-google-genai langchain-community chromadb python-dotenv pypdf

You'll also need a Google Gemini API key. You can get one for free from Google AI Studio.

Instead of running messy terminal configuration commands for different operating systems, create a new file named .env in the root of your project folder and add your key like this:

GOOGLE_API_KEY=your_actual_api_key_here

Preparing the PDF

Since this is a "Chat with PDF" project, you’ll need a sample PDF document to work with. To keep things simple, download this ready-made sample document below and place it inside your project folder.

You can then use this PDF throughout the tutorial for testing uploads, parsing, embeddings, and chat functionality.

Writing the RAG Code Step-by-Step

Create a Python file named rag_app.py in your project folder. Instead of copying a massive block of code, we'll build this application block by block so we can understand exactly how data flows through our pipeline.

Step 1: Imports and Environment Setup

At the very top of your file, add the necessary library imports and initialize your environment configuration:

import os
from dotenv import load_dotenv
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_google_genai import GoogleGenerativeAIEmbeddings, ChatGoogleGenerativeAI
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import PromptTemplate
from langchain_core.runnables import RunnablePassthrough

# Load environment variables from the .env file
load_dotenv()

We're bringing in LangChain modules to handle loading, splitting, embedding, storing, and prompting. The load_dotenv() function is mandatory because it scans our .env file and loads the GOOGLE_API_KEY into our system's background environment variables, ensuring our AI models can authenticate seamlessly without hardcoding passwords.

Step 2: Loading the PDF Document

Next, let's point our script to the PDF document we downloaded earlier:

print("Loading PDF document...")
loader = PyPDFLoader("TechCorp_Official_Employee_Handbook.pdf")
document = loader.load()

print(document[0].page_content)

Computers can't read a PDF like a standard text file because PDFs contain complex layout streams. PyPDFLoader handles the heavy lifting of opening the file, stripping away visual layout formatting, and extracting the raw text characters into a clean format that LangChain can work with.

At this point, when you run the script, you should see the text content from the first page of the PDF printed in the terminal. This is a quick way to verify that the PDF was loaded successfully and that PyPDFLoader was able to extract readable text from the document correctly.

Step 3: Chunking the Text

Now that the raw text is in memory, we need to chop it up into smaller pieces:

print("Chunking text...")
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = text_splitter.split_documents(document)

print(chunks[0].page_content)

If a user asks a simple question, sending an entire 100-page document to the LLM is incredibly slow and expensive. RecursiveCharacterTextSplitter cuts the text into segments of roughly 500 characters.

The chunk_overlap=50 parameter tells the text splitter to repeat the last 50 characters of one chunk at the beginning of the next. This helps preserve context between chunks so that sentences or ideas are not abruptly cut off.

Without overlap, important information near chunk boundaries could be separated, making retrieval less accurate. By maintaining a small shared section between neighboring chunks, the model can better understand continuity in the document, resulting in more reliable search results and higher-quality responses.

When you run the script, you should now see the contents of the first text chunk printed in the terminal.

Step 4: Creating Embeddings and Initializing the Vector DB

With our chunks ready, we'll convert them into vector coordinates and save them locally:

print("Creating vector database...")
embeddings = GoogleGenerativeAIEmbeddings(model="models/embedding-001")
vector_db = Chroma.from_documents(
    documents=chunks, 
    embedding=embeddings, 
    persist_directory="./chroma_db"
)

This is the mathematical core of RAG. GoogleGenerativeAIEmbeddings takes a raw text chunk and turns it into a list of numbers representing its conceptual meaning. We then hand those chunks and numbers to Chroma, which maps them into a local database directory named chroma_db on your hard drive, allowing for lightning-fast mathematical lookups later.

Step 5: Setting Up the Retriever and Prompt Template

Now we need a mechanism to query our database and a structure to house our instructions:

# Configure the database to act as a document retriever
retriever = vector_db.as_retriever(search_kwargs={"k": 2})

# Define the hidden prompt structure for the LLM
template = """
Use the following pieces of retrieved context to answer the question. 
If you don't know the answer, just say that you don't know. 
Use three sentences maximum and keep the answer concise.

Context: {context}

Question: {question}

Answer:
"""
prompt = PromptTemplate.from_template(template)

vector_db.as_retriever() converts the vector database into a retriever object that can search through stored document embeddings and return the most relevant chunks for a user’s question. Setting k=2 on our retriever tells the database to only pull the top two most relevant chunks for any given question, which keeps things clean and efficient.

The prompt template acts as hidden instructions for the model. When a user asks a question, LangChain automatically replaces {context} with the retrieved document chunks and {question} with the user’s actual query. The template also acts as a safety guardrail. By explicitly telling the model to say "I don't know" if the context lacks information, we heavily suppress the model's tendency to hallucinate fake answers.

Step 6: Initializing the LLM and Constructing the RAG Chain

Next, we hook up our language model and construct our execution pipeline:

# Initialize the free Gemini model tier
llm = ChatGoogleGenerativeAI(model="gemini-1.5-flash", temperature=0)

# Helper function to stitch retrieved chunks into a single text block
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

# Connect everything together using LangChain Expression Language (LCEL)
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
)

We use gemini-3.5-flash with a temperature=0 setting to force the model to be completely factual and analytical rather than creative.

The retriever returns multiple document chunks as structured objects. The format_docs function converts those chunks into a single continuous text block by joining their page_content. This step is necessary because the prompt expects a clean, readable context string rather than a list of document objects.

Finally, we connect everything using LangChain Expression Language (LCEL). When a question comes in, it passes it to the retriever, formats the resulting text documents, passes the filled template to the prompt handler, and pushes the final product straight to the LLM.

Step 7: Invoking the Chain with a Question

Finally, let's execute the pipeline and print the result out to the console:

user_question = "What days can I work from home?"
print(f"\nQuestion: {user_question}")

response = rag_chain.invoke(user_question)
print(f"Answer: {response.content}")

This is where the magic happens. The invoke command sets off the entire chain reaction we just built. When you run this, the console will output:

Loading PDF document...
Chunking text...
Creating vector database...

Question: What days can I work from home?
Answer: [{'type': 'text', 'text': 'You are permitted to work from home on Tuesdays and Thursdays. Additional remote flexibility may also be approved by your department manager.', 'extras': {'signature': 'Eo0JCooJAQw51seue7vZT7Vby90GMDLhtOBWLKm5UjfEro7f8dRoKC0KAIHxSqQSLXq0s3kf6yfzTsgaUMFiNd0fnwtNSNoApzcZ7huRD8iq+f+xomoXGhmFYClnLApHUKtOLykICluJnM1j6DfYGaVHKLqU0MF4+Fng9CdqXVqPgN9HcfJEvSpeMAc9vTYENj07s8N6MidlMvMt1w0fl4GCjxAZXyEngdU4kGfjUqaKyjjCQ9yLFeoXrV55pqZdkElLxXEK4ZWNnMGh5NDqGmt2b0kMG4KoCdunUltBr1ctV15rZ+724T0qnjDvI+pIgp/ZtKa423gaVXSkSmdvSePEog38blJ2dgjtZg72XF5xlh45Yv06fZVu7e60ZB1sTn4W8iWuYGQ61i/xCN6xCX/e3SuitjwQoHSlEe/iuoaNf5BXhdp87TUyQTawiY+qIZjgWz2AMLUbMcOvns/0iFt6jpUkXr/dO4eYF39UCosrbWC5TZQp2gllNQ6mlrczTAKqe8mPZwmBVuTJ3kx3q+SsVROln584EdD94IxXrgLXhuLkbR9ub0qyvjBfAmIfvUEK5pcaBCGydQvheH9wsIvAOG1kspMb/wqjAv/mpmii8J9vztSvM9PR9v7L3YLu8vcANol80w2PfeHhyWUJWit8R58kKd7HHor5GJhA436x+tCukIlBq2oTcob+ydxVJydA12pRsiuw4kYkEIU8nr5yCiIwjYCDtVm6Ws0RUnhyk5u+dRONPZ6g+mfBShKCnahcIMzzJpXznmPXvmP2C96uD64SGTI6L86EMlLEz06/cTJTabgqAYqe2AhERgnYc/4d0XabQOkzvDmBKMr5/LOAt3ZZg7X4PIuefEwxx0eB60gLROefcbbu8k+KPazqFsDP/YA/aPyAxyss/6V43EID0amJcDA81LKJzazL9KnclefQZrN9viIwteMaV04IIlx+Ynk1vZi/LVgWiFuDVWF3Ql2luY4KwFpfFDxQ728gkrhvUdTBrfUeKRSLV1W4ox6I7ogo0e9i7db2lkOQljctGs3Km3hWu4JOkH+YzLNmcDHMF3imfgQH5Ml99H9PXh1ScBjq47MXKzJPdHijkY5ZRSjceEIlKEGv8afQO60NB8lk1MQAGwd+CxqIwVg11N8q9EFSwdJmVVmoyM1nINGJERSKhKOrkqBsOELfpKDjv14tuNgDUy4wdtuxn8C4tJBKvN8t/hrW/Z65VoBGdMwA08sRSV6Fp5l/gSdYeB9yA/Lx/VGkgVqaP5tU73XrE/XO8ysJ/kgRDXiTvsg+2uayU1Q9PfKFAawopslwybCHtdOwaVgsRdA5R4f1NIkPoP/sX+iBxyR0kKg6v4RRAj851WifM2fQ8Vsw5dtFSeh/4TfYg1GCCCDNT4JwrtI8fqcF+qMQqUb+oUqoyzjzFqqSRxXcyqHXOLV9V9C6yWYmZ3TSY043WL9L4kGGJGxFHD5VWG77Quiy+rHWGO13LOc5EBKIO05sg1xnI88QQTUgkxwJeuntytIy3f3pfMVrFYFkvi8w5LzL4RK68+4HMg=='}}]

Modern LLMs like Google's Gemini are multimodal. This means they're designed to read and generate not just plain text, but images, video, and audio simultaneously. Because of this, the LangChain Google integration doesn't always return a simple text string. Instead, it returns a list of content blocks.

In your output, the AI successfully returned your text, but it also included an extras dictionary containing a signature. This signature is a behind-the-scenes data point used by Google for AI safety tracking, grounding metadata, and thought-process verification.

To get a clean, human-readable string, you simply need to extract the text value from that list. You can update your final print statement to check if the response is a list and extract the text automatically:

# Clean up the output if Gemini returns a list of content blocks
if isinstance(response.content, list):
    clean_answer = response.content[0]['text']
else:
    clean_answer = response.content

print(f"Answer: {clean_answer}")

Now, your output will look like this:

Question: What days can I work from home?
Answer: You are permitted to work from home on Tuesdays and Thursdays. Additional remote flexibility may also be approved by your department manager.

Step 8: Making it Conversational

Right now, our script hardcodes a single question, prints the answer, and immediately exits. In the real world, you want to chat with your documents naturally. Let's upgrade our script to run continuously in your terminal so you can ask as many questions as you want without restarting the program.

Replace the bottom section of your code with a simple while loop:

# Chat with your PDF in a continuous loop
print("\n--- PDF Chatbot Initialized ---")
print("Type 'exit' or 'quit' to stop.")

while True:
    # 1. Wait for the user to type a question
    user_question = input("\nYour Question: ")

    # 2. Allow the user to break the loop and close the program
    if user_question.lower() in ['exit', 'quit']:
        print("Shutting down chatbot. Goodbye!")
        break

    # 3. Send the question through our RAG chain
    response = rag_chain.invoke(user_question)

    # 4. Clean up the output format
    if isinstance(response.content, list):
        clean_answer = response.content[0]['text']
    else:
        clean_answer = response.content

    # 5. Print the final answer to the console
    print(f"Answer: {clean_answer}")

By using Python's input() function wrapped inside an infinite while True loop, we keep the Python script alive. The PDF chunks and vector database stay loaded in your computer's memory, allowing you to fire off consecutive questions instantly. This transforms your script from a static demonstration into a fully interactive AI tool!

Here's a sample run:

Full Code

import os
from dotenv import load_dotenv
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_google_genai import GoogleGenerativeAIEmbeddings, ChatGoogleGenerativeAI
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import PromptTemplate
from langchain_core.runnables import RunnablePassthrough

# Load environment variables from the .env file
load_dotenv()

print("Loading PDF document...")
loader = PyPDFLoader("TechCorp_Official_Employee_Handbook.pdf")
document = loader.load()
# print(document[0].page_content)

print("Chunking text...")
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = text_splitter.split_documents(document)
# print(chunks[0].page_content)

print("Creating vector database...")
embeddings = GoogleGenerativeAIEmbeddings(model="gemini-embedding-001")
vector_db = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)

# Configure the database to act as a document retriever
retriever = vector_db.as_retriever(search_kwargs={"k": 2})

# Define the hidden prompt structure for the LLM
template = """
Use the following pieces of retrieved context to answer the question. 
If you don't know the answer, just say that you don't know. 
Use three sentences maximum and keep the answer concise.

Context: {context}

Question: {question}

Answer:
"""
prompt = PromptTemplate.from_template(template)

# Initialize the free Gemini model tier
llm = ChatGoogleGenerativeAI(model="gemini-3.5-flash", temperature=0)

# Helper function to stitch retrieved chunks into a single text block
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)


# Connect everything together using LangChain Expression Language (LCEL)
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
)
"""
user_question = "What days can I work from home?"
print(f"\nQuestion: {user_question}")

response = rag_chain.invoke(user_question)
# print(f"Answer: {response.content}")

# Clean up the output if Gemini returns a list of content blocks
if isinstance(response.content, list):
    clean_answer = response.content[0]['text']
else:
    clean_answer = response.content

print(f"Answer: {clean_answer}")
"""

# Chat with your PDF in a continuous loop
print("\n--- PDF Chatbot Initialized ---")
print("Type 'exit' or 'quit' to stop.")

while True:
    # 1. Wait for the user to type a question
    user_question = input("\nYour Question: ")

    # 2. Allow the user to break the loop and close the program
    if user_question.lower() in ['exit', 'quit']:
        print("Shutting down chatbot. Goodbye!")
        break

    # 3. Send the question through our RAG chain
    response = rag_chain.invoke(user_question)

    # 4. Clean up the output format
    if isinstance(response.content, list):
        clean_answer = response.content[0]['text']
    else:
        clean_answer = response.content

    # 5. Print the final answer to the console
    print(f"Answer: {clean_answer}")

Taking it out of the terminal

Once you have your terminal chatbot working, you probably want to give it a proper visual interface. The easiest way to do this in Python is using an open-source library called Gradio. Gradio has a built-in ChatInterface feature that can wrap your existing RAG code and automatically generate a beautiful, ChatGPT-style web UI in your browser with just three extra lines of code. It's highly recommended as your next mini-project.

The Full Data Flow

To truly solidify your understanding, let's map out the exact lifecycle of a single user question in our system:

Breaking Down the Execution Timeline

The request begins: The user interfaces with our console and asks a text-based question: "How much vacation do I get?" At this exact moment, our application code takes control of the program flow.
The text-to-vector translation: Computers can't compute similarity using raw text characters. Our app makes a fast network call to the Google Embedding Model, handing over the raw question. The model converts the text into a massive array of numbers that mathematically represents the user's intent.
The database distance calculation: Our application script takes those coordinate numbers and passes them directly to ChromaDB. ChromaDB scans the local hard drive, running a similarity math function against the numbers stored for each of our PDF chunks. It locates the text chunk mentioning "20 days of paid time off" because its coordinates are physically closest to the query coordinates.
The prompt augmentation: ChromaDB hands the raw text strings of those relevant pieces back to our script. The code automatically unrolls our prompt template, plugging the raw chunks into the {context} slot and the user's original text into the {question} slot.
The final generation: Our application drops this combined package into the final network call, pushing it directly to the Gemini LLM. Because temperature=0 is configured, the model acts strictly as a reading comprehension engine. It reads the custom context, formats a clean sentence, and sends it back to our terminal to be printed out beautifully for the user.

Common RAG Problems

Building a simple RAG app is easy. Building a RAG app that works perfectly in production is very difficult. Here are the most common problems engineers face and how they fix them.

1. Bad Chunking

If your chunks are too large, they include irrelevant information that confuses the LLM. If they're too small, they lose vital context. Engineers can solve this by experimenting with different chunk sizes or using semantic chunking (splitting by whole sentences or paragraphs rather than strict character counts).

2. Irrelevant Retrieval

Sometimes semantic search fails. If a user searches for "Apple" expecting information about fruit, but the database only has data about the tech company, the system will confidently return tech company documents. Engineers can fix this by adjusting the embedding models or adding keyword search rules.

3. Hallucinations

Even with RAG, an LLM might ignore the retrieved context and rely on its training memory. Engineers mitigate this by heavily engineering the prompt template with strict rules like "ONLY use the provided text."

4. Latency

RAG requires an embedding network call, a database search, and an LLM network call. This takes time. Engineers can optimize this by using faster, locally hosted embedding models or caching common questions.

5. Stale Data

If HR updates the company policy PDF, the vector database still holds the old numbers. The AI will give outdated answers. Engineers build update pipelines that automatically delete old vectors and embed new ones whenever a source file changes.

Advanced RAG Concepts

Once you master basic RAG, the AI engineering world opens up to highly advanced techniques.

Hybrid Search

Vector databases are great at understanding meaning, but bad at finding exact ID numbers or specific names. Hybrid search combines traditional keyword search (like searching a SQL database) with semantic vector search to get the best of both worlds.

Reranking

Sometimes the vector database returns 10 chunks, but the best answer is accidentally placed at the bottom of the list. Reranking uses a second, specialized AI model to read the retrieved chunks and sort them strictly by relevance before sending them to the LLM.

Agentic RAG

Instead of forcing the system to retrieve documents every single time, Agentic RAG uses an AI "Agent" to decide if it even needs to search. If you say "Hello", the agent skips the database and just says "Hi". If you ask a hard question, it decides to query the database.

Graph RAG

Instead of breaking text into isolated chunks, Graph RAG extracts entities (people, places, concepts) and maps how they relate to each other in a Knowledge Graph. This is incredibly powerful for complex datasets with deep relationships.

Traditional RAG only reads text. Multi-modal RAG processes images, charts, and audio files, allowing users to ask questions like, "What does the graph on page 4 indicate?"

Final Thoughts

Retrieval-Augmented Generation is the bridge between incredible reasoning engines (LLMs) and reliable factual knowledge (your data).

Understanding RAG is no longer optional for software engineers. Nearly every enterprise software product being built today involves some form of it. By learning how chunking, embeddings, vector databases, and prompt augmentation work together, you have demystified the magic behind modern AI.

Your next step is to build on the code we wrote today. Try pointing the PDF loader to your résumé, a school textbook, or a financial report. Once you experience your own code answering questions about your personal data, you'll start to truly understand the power of AI engineering.

llm - freeCodeCamp.org

How to Build Your Own MCP Server and Publish Your ChatGPT App with Supabase Auth and DigitalOcean

What We'll Cover:

What is an MCP Server?

What Can You Do with an MCP Server?

Level 1: How to Build Your Own MCP Server

Step 0: Prepare your project

Step 1: Create a Node.js Server

Step 2: Setting Up MCP Server SDK

Step 3: Add MCP Server Tools – Create and Add a Todo

Step 4: List Todos from MCP Server

Step 5: Add Todo Complete Functions

Step 6: Connect Your MCP Server with the Node.js Server

How to Test Your MCP Server

Level 2: How to Build the UI

Step 1: Create the HTML File to Show the UI

Todo list

Step 2: Add a JavaScript Module to Handle MCP Server Data.

Step 3: Styling your UI

Step 4: Add the UI to your MCP Server:

Step 5: Update Your MCP Server to Handle the UI

How to Test Your ChatGPT App

Level 3: How to Add Supabase (Auth and Database) to the MCP Server

Step 1: Create the Todos Table

Step 2: Enabling the MCP Server to Connect with Supabase Auth

Step 3: Create a Proxy Server for the MCP Server to Handle the Auth.

Step 4: Implementing the Consent and Login Page

Authentication

Authorization Required

Step 5: Testing the OAuth Implementation with MCP Server Inspector

Step 6: Adding OAuth Security to Your MCP Server Tools

Step 7: Updating the MCP Server Function to Handle the Authentication

Step 8: Testing the Server with Supabase:

How to Deploy your MCP Server to DigitalOcean

How to Publish Your ChatGPT App

What to Do Next

Acknowledgments:

References:

AI Paper Review: Self-Consistency Improves Chain of Thought Reasoning in Language Models

Paper Overview

Table of Contents:

Prerequisites

Abstract

Introduction

Self-Consistency over Diverse Reasoning Paths

Experiments

Main Results

Common Sense and Symbolic Reasoning

Self-Consistency Helps When Chain-of-Thought Hurts Performance

Comparison to Other Existing Approaches

Sample-and-Rank

Beam Search

Ensemble-Based Approaches

Additional Studies

Review of Related Work

Discussion

Conclusion

Resources:

How to Export a Claude Conversation as a PDF

Table of Contents

How to Save Claude Conversations as a PDF Using the Browser Print Option

How Browsers Generate PDFs From Web Pages:

Step 1: Open the Browser's Print Dialog

Step 2: Save the Conversation as a PDF

Step 3: Adjust the Print Settings

Margins

Scale

Background graphics

Headers and footers

How to Copy Claude Responses into Google Docs and Save Them as PDFs

How to Convert Claude Share Links into PDFs

How to Export Claude Artifacts as PDFs

How to Download All Claude Conversations from Settings

How to Choose the Best Export Method

Video Tutorial: How to Export a Claude Conversation as PDF

Conclusion

Codex vs Claude Code: Which AI Coding Assistant to Choose

What We'll Cover Here:

Understanding Codex

Understanding Claude Code

What Exactly is `val_bpb`?

1. `prepare.py`

2. `train.py`

3. `program.md`