It doesn't learn.

You ingest 500 documents. You ask a question. The system retrieves the three most similar chunks and hands them to the LLM. Repeat for the next query.

The system knows exactly as much as it did on day one. It's a library that never builds a card catalog, never cross-references its own shelves, never notices that three of its books are saying contradictory things.

That's what I set out to fix with a knowledge reflection layer. After every ingest, the system finds semantically related documents already in the index and asks an LLM to synthesise what's new, how it connects, and what gap remains. That synthesis gets embedded, stored, and boosted in search results.

The knowledge base gets smarter as you add more documents — not just bigger.

This tutorial shows you exactly how to build it.

Table of Contents

1. What You Will Build

2. Prerequisites

3. How to Set Up the Base System

4. Why Standard RAG Has a Memory Problem

5. Step 1: Schema Update

6. Step 2: The Reflection Engine

7. Step 3: Consolidation

8. Step 4: Wire It Into Your Ingest Handler

9. Step 5: Boost Reflections in Search

10. Step 6: Filtering by doc_type

11. What Changes After You Build This

12. Deploying

13. What to Build Next

What You Will Build

In this tutorial, you'll build a post-ingest reflection pipeline that:

1. Fires automatically after every document ingest

2. Finds the most semantically related documents already in the index

3. Asks Kimi K2.5 to synthesise a three-sentence insight linking the new document to existing knowledge

4. Stores that reflection with doc_type=reflection and a 1.5× ranking boost in search results

5. Consolidates reflections into summaries every three ingests

By the end, searching your knowledge base will surface both raw document chunks and reflection artifacts the system wrote on ingest.

Prerequisites

You will need:

• A Cloudflare account — free tier works

• Node.js v18+ and Wrangler CLI installed (npm install -g wrangler)

• Basic TypeScript familiarity

No external API keys. Everything runs on Cloudflare's infrastructure.

How to Set Up the Base System

If you have already built the RAG system from my freeCodeCamp handbook, skip this section — your system is ready for the reflection layer.

If you're starting fresh, this section gets you to a working base in about 15 minutes.

Scaffold the Project

npm create cloudflare@latest rag-reflection-system
cd rag-reflection-system

Choose: Hello World example → TypeScript → No deploy yet.

Create the Vectorize Index and D1 Database

npx wrangler vectorize create rag-index --dimensions=384 --metric=cosine
npx wrangler d1 create rag-db

Configure wrangler.toml

name = "rag-reflection-system"
main = "src/index.ts"
compatibility_date = "2026-01-01"

[[vectorize]]
binding = "VECTORIZE"
index_name = "rag-index"

[[d1_databases]]
binding = "DB"
database_name = "rag-db"
database_id = "YOUR_DB_ID"

[ai]
binding = "AI"

Create the documents Table

-- migrations/001_init.sql
CREATE TABLE IF NOT EXISTS documents (
  id TEXT PRIMARY KEY,
  content TEXT NOT NULL,
  source TEXT,
  date_created TEXT DEFAULT (datetime('now'))
);

npx wrangler d1 execute rag-db --remote --file=./migrations/001_init.sql

Add the ingest and search endpoints

Replace src/index.ts with this minimal working system:

export interface Env {
  VECTORIZE: VectorizeIndex;
  DB: D1Database;
  AI: Ai;
}

export default {
  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
    const url = new URL(request.url);

    if (url.pathname === '/ingest' && request.method === 'POST') {
      const { id, content, source } = await request.json() as any;

      const embResult = await env.AI.run('@cf/baai/bge-small-en-v1.5', {
        text: [content.slice(0, 512)],
      }) as any;
      const vector = embResult.data[0];

      await env.VECTORIZE.upsert([{
        id,
        values: vector,
        metadata: { content: content.slice(0, 1000), source, doc_type: 'raw' },
      }]);

      await env.DB.prepare(
        'INSERT OR REPLACE INTO documents (id, content, source) VALUES (?, ?, ?)'
      ).bind(id, content, source ?? '').run();

      return Response.json({ success: true, id });
    }

    if (url.pathname === '/search' && request.method === 'POST') {
      const { query } = await request.json() as any;

      const embResult = await env.AI.run('@cf/baai/bge-small-en-v1.5', {
        text: [query],
      }) as any;
      const vector = embResult.data[0];

      const results = await env.VECTORIZE.query(vector, {
        topK: 5,
        returnMetadata: 'all',
      });

      const context = results.matches
        .map(m => m.metadata?.content as string)
        .filter(Boolean)
        .join('\n\n');

      const answer = await env.AI.run('@cf/moonshotai/kimi-k2.5', {
        messages: [
          { role: 'system', content: 'Answer using only the context provided.' },
          { role: 'user', content: `Context:\n\({context}\n\nQuestion: \){query}` },
        ],
        max_tokens: 256,
      }) as any;

      return Response.json({ answer: answer.response, sources: results.matches.map(m => m.id) });
    }

    return new Response('RAG system running', { status: 200 });
  },
};

Deploy and Verify

npx wrangler deploy

Test it:

# Ingest a document
curl -X POST https://your-worker.workers.dev/ingest \
  -H "Content-Type: application/json" \
  -d '{"id": "doc-001", "content": "Cursor pagination beats offset pagination for live-updating datasets because offset becomes unreliable when rows are inserted or deleted during pagination."}'

# Search
curl -X POST https://your-worker.workers.dev/search \
  -H "Content-Type: application/json" \
  -d '{"query": "what pagination approach should I use?"}'

If you get a grounded answer back, the base system is working. The next sections add the reflection layer on top of this foundation.

Why Standard RAG Has a Memory Problem

Standard RAG retrieval is stateless. Every query goes in cold. The system has no memory of what it found before, no synthesis of what it learned across documents, and no growing understanding of what questions remain unanswered.

Imagine you've ingested 200 documents about your product. Twelve of them touch on a pricing decision made last year. No single one has the full picture — it's distributed across quarterly reports, meeting notes, an internal Slack export, a few Notion pages.

A user asks: "Why did we change our pricing structure?"

Standard RAG retrieves the three most similar chunks. If those three chunks collectively have the answer, great. If they don't — if the real answer requires synthesising across those twelve documents — the system has no mechanism for that. It returns fragments. The LLM makes its best guess.

The reflection layer addresses this directly. When the twelfth pricing document gets ingested, the system finds the eleven related documents, synthesises what connects them, and stores that synthesis as a retrievable artifact. The answer to "why did we change our pricing structure" exists in the index before anyone asks the question.

Not smarter retrieval — smarter indexing.

Step 1: Schema Update

The reflection layer needs two new fields in your D1 documents table. Run this migration:

-- migrations/003_add_reflection_fields.sql
ALTER TABLE documents ADD COLUMN doc_type TEXT DEFAULT 'raw';
ALTER TABLE documents ADD COLUMN reflection_score REAL DEFAULT 0;
ALTER TABLE documents ADD COLUMN parent_reflection_id TEXT;

Apply it:

wrangler d1 execute mcp-knowledge-db --remote --file=./migrations/003_add_reflection_fields.sql

doc_type distinguishes raw documents (raw), single-document reflections (reflection), and consolidated multi-reflection summaries (summary). You'll use this field to filter — exposing only reflections to users who want the distilled view, or excluding them for users who want raw source chunks.

Step 2: The Reflection Engine

Create src/engines/reflection.ts. This is the core of the layer.

import { Env } from '../types/env';
import { resolveEmbeddingModel, resolveReflectionModel } from '../config/models';

const REFLECTION_BOOST = 1.5;
const CONSOLIDATION_THRESHOLD = 3; // consolidate every N new reflections

export async function reflect(
  newDocId: string,
  newDocContent: string,
  env: Env
): Promise<void> {
  // 1. Find semantically related documents already in the index
  const embModel = resolveEmbeddingModel(env.EMBEDDING_MODEL);
  const embResult = await env.AI.run(embModel.id as any, {
    text: [newDocContent.slice(0, 512)],
  });
  const queryVector = (embResult as any).data?.[0];
  if (!queryVector) return;

  const related = await env.VECTORIZE.query(queryVector, {
    topK: 5,
    filter: { doc_type: { $eq: 'raw' } },
    returnMetadata: 'all',
  });

  const relatedDocs = (related.matches ?? []).filter(
    m => m.id !== newDocId && (m.score ?? 0) > 0.65
  );

  if (relatedDocs.length === 0) return; // nothing related yet — skip

  // 2. Build synthesis prompt
  const relatedSummaries = relatedDocs
    .slice(0, 3)
    .map((m, i) => `Document \({i + 1}: \){String(m.metadata?.content ?? '').slice(0, 300)}`)
    .join('\n\n');

  const prompt = `You are synthesising knowledge across documents in a knowledge base.

New document:
${newDocContent.slice(0, 600)}

Related existing documents:
${relatedSummaries}

Write exactly three sentences:
1. What the new document adds that the existing documents don't already cover
2. How the new document connects to or extends the existing documents
3. What gap or question remains unanswered across all these documents

Be specific. Reference actual content. Do not summarise — synthesise.`;

  // 3. Call the reflection model
  const reflModel = resolveReflectionModel(env.REFLECTION_MODEL);
  const llmResp = await env.AI.run(reflModel.id as any, {
    messages: [{ role: 'user', content: prompt }],
    max_tokens: 180,
  });

  const reflectionText = (llmResp as any)?.response?.trim();
  if (!reflectionText || reflectionText.length < 40) return;

  // 4. Embed and store the reflection
  const reflEmbResult = await env.AI.run(embModel.id as any, {
    text: [reflectionText],
  });
  const reflVector = (reflEmbResult as any).data?.[0];
  if (!reflVector) return;

  const reflectionId = `refl_\({newDocId}_\){Date.now()}`;

  await env.VECTORIZE.upsert([
    {
      id: reflectionId,
      values: reflVector,
      metadata: {
        content: reflectionText,
        doc_type: 'reflection',
        parent_id: newDocId,
        reflection_score: REFLECTION_BOOST,
        source_doc_ids: relatedDocs.map(m => m.id).join(','),
        date_created: new Date().toISOString(),
      },
    },
  ]);

  await env.DB.prepare(
    `INSERT INTO documents
     (id, content, doc_type, reflection_score, parent_id, date_created)
     VALUES (?, ?, 'reflection', ?, ?, ?)`
  )
    .bind(reflectionId, reflectionText, REFLECTION_BOOST, newDocId, new Date().toISOString())
    .run();

  // 5. Check if consolidation is due
  const recentCount = await env.DB
    .prepare(`SELECT COUNT(*) as cnt FROM documents WHERE doc_type = 'reflection' AND date_created > datetime('now', '-1 hour')`)
    .first<{ cnt: number }>();

  if ((recentCount?.cnt ?? 0) >= CONSOLIDATION_THRESHOLD) {
    await consolidate(env);
  }
}

Two things worth noting here.

First, the semantic threshold (score > 0.65) matters. Too low and you're synthesising unrelated documents. Too high and you're rarely finding connections. 0.65 works well with bge-small. You can bump it to 0.72 with qwen3-0.6b (1024d) where scores cluster higher.

The prompt structure is deliberate. Three sentences, each doing a specific job: what's new, how it connects, what remains. This keeps reflections useful for retrieval. A freeform synthesis prompt produces beautiful prose that doesn't retrieve well. This structure produces retrievable artifacts.

Step 3: Consolidation

As reflections accumulate, they need their own synthesis layer — otherwise you're adding noise at a higher abstraction level.

Add this to src/engines/reflection.ts:

export async function consolidate(env: Env): Promise<void> {
  // Fetch recent reflections not yet consolidated
  const recent = await env.DB
    .prepare(
      `SELECT id, content FROM documents
       WHERE doc_type = 'reflection'
       AND id NOT IN (
         SELECT DISTINCT parent_id FROM documents
         WHERE doc_type = 'summary' AND parent_id IS NOT NULL
       )
       ORDER BY date_created DESC
       LIMIT 6`
    )
    .all<{ id: string; content: string }>();

  if (!recent.results || recent.results.length < CONSOLIDATION_THRESHOLD) return;

  const reflectionTexts = recent.results.map((r, i) => `Reflection \({i + 1}: \){r.content}`).join('\n\n');

  const prompt = `You are consolidating multiple knowledge reflections into a single compressed insight.

${reflectionTexts}

Write two to three sentences that capture the most important cross-cutting pattern or tension across these reflections. What does the knowledge base now understand that it didn't before these documents were added? What's the most important open question?

Be precise. No preamble.`;

  const reflModel = resolveReflectionModel(env.REFLECTION_MODEL);
  const llmResp = await env.AI.run(reflModel.id as any, {
    messages: [{ role: 'user', content: prompt }],
    max_tokens: 320,
  });

  const summaryText = (llmResp as any)?.response?.trim();
  if (!summaryText || summaryText.length < 40) return;

  const embModel = resolveEmbeddingModel(env.EMBEDDING_MODEL);
  const embResult = await env.AI.run(embModel.id as any, { text: [summaryText] });
  const summaryVector = (embResult as any).data?.[0];
  if (!summaryVector) return;

  const summaryId = `summary_${Date.now()}`;

  await env.VECTORIZE.upsert([
    {
      id: summaryId,
      values: summaryVector,
      metadata: {
        content: summaryText,
        doc_type: 'summary',
        reflection_score: REFLECTION_BOOST * 1.2,
        source_reflection_ids: recent.results.map(r => r.id).join(','),
        date_created: new Date().toISOString(),
      },
    },
  ]);

  await env.DB.prepare(
    `INSERT INTO documents (id, content, doc_type, reflection_score, date_created)
     VALUES (?, ?, 'summary', ?, ?)`
  )
    .bind(summaryId, summaryText, REFLECTION_BOOST * 1.2, new Date().toISOString())
    .run();
}

Summaries get a 1.2× multiplier on top of the base reflection boost. In search results, a summary synthesising twelve related documents should rank above any single document chunk on broad conceptual queries. On specific factual queries, the raw chunks will score higher. The ranking sorts itself.

Step 4: Wire It Into Your Ingest Handler

The reflection runs as a background job. It doesn't block the ingest response — that would add 2–3 seconds to every ingest call.

In your src/handlers/ingest.ts, after you've stored the document:

import { reflect } from '../engines/reflection';

// ... existing ingest logic ...

// After VECTORIZE.upsert() and DB insert succeed:
ctx.waitUntil(
  reflect(documentId, content, env).catch(err => {
    console.warn('[reflection] failed for', documentId, err.message);
  })
);

return new Response(JSON.stringify({
  success: true,
  documentId,
  chunks: chunkCount,
  // ... rest of response
}), { headers: { 'Content-Type': 'application/json' } });

ctx.waitUntil() is the Cloudflare Workers primitive for background work. The response returns immediately. The reflection runs after. The ingest API stays fast.

The .catch() is important. A failed reflection should never fail an ingest. Raw documents are the source of truth. Reflections are derived value — useful, but not critical path.

Step 5: Boost Reflections in Search

Add the reflection boost to your ranking logic in src/engines/hybrid.ts. After RRF fusion and before returning results:

// Apply reflection boost
const boosted = results.map(r => ({
  ...r,
  score: r.doc_type === 'reflection' || r.doc_type === 'summary'
    ? r.score * (r.reflection_score ?? 1.5)
    : r.score,
}));

return boosted.sort((a, b) => b.score - a.score);

This is a post-fusion boost, not a pre-fusion rerank. The reasoning: apply RRF across all results first, so reflections earn their place on raw relevance before getting boosted. A reflection that would not rank in the top 20 on raw similarity shouldn't appear just because it has a boost multiplier.

Step 6: Filtering by doc_type

Your search endpoint should accept a doc_type filter so callers can control what they see:

// In your search request handler:
const docTypeFilter = body.filters?.doc_type;

// Pass to Vectorize query:
const vectorFilter: Record<string, unknown> = {};
if (docTypeFilter) {
  vectorFilter.doc_type = docTypeFilter;
}

This gives callers three modes:

# Only reflections and summaries
POST /search
{ "query": "pricing decisions", "filters": { "doc_type": { "$in": ["reflection", "summary"] } } }

# Only source documents
POST /search
{ "query": "pricing decisions", "filters": { "doc_type": { "$eq": "raw" } } }

# Default: all types, reflections boosted
POST /search
{ "query": "pricing decisions" }

The default (no filter) is the most useful. Let the boost do its job. Restrict to raw when you need citations. Restrict to reflections when you want the synthesised view.

What Changes After You Build This

At 200 documents, the difference becomes noticeable. Queries that previously returned five fragmented chunks now surface a reflection that already synthesised those chunks. Broad conceptual queries — "what do we know about X?" — start returning genuinely useful summaries instead of just the most-similar individual paragraph.

At 2,000 documents, the reflection layer is the most valuable part of the system. The raw chunks answer specific factual questions. The reflections and summaries answer conceptual questions that could not be answered from any single document. The system has learned something no individual document contains.

One failure mode worth knowing: if your embedding model has poor semantic clustering — old bge-small at 384d with mixed-domain documents — the related-documents retrieval step will surface weak connections and produce shallow reflections. The 0.65 threshold filters most of this out, but if you're seeing reflections that seem off-topic, your embeddings are the first thing to check.

Deploying

wrangler d1 execute mcp-knowledge-db --remote --file=./migrations/003_add_reflection_fields.sql
wrangler deploy

Then ingest a few documents and watch what happens:

# Ingest document 1
curl -X POST https://your-worker.workers.dev/ingest \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"id": "doc-001", "content": "Your document text here..."}'

# After a few seconds, check if a reflection was created
curl "https://your-worker.workers.dev/search" \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "your topic", "filters": {"doc_type": {"$eq": "reflection"}}}'

Reflections won't appear until there are related documents to synthesise. Ingest at least three documents on similar topics before expecting to see them.

What to Build Next

The reflection layer as described here fires after every ingest. That's expensive at high ingest volume: if you're batch-importing 10,000 documents, you don't want 10,000 individual reflection calls.

For bulk ingestion, gate it: call reflect() only when a document's similarity search returns a match above 0.8, or batch-run reflection after the bulk import completes. The POST /ingest/batch endpoint in the full repo does this.

The second thing worth building: surfacing reflections in your UI with a visual distinction. A search result that's a reflection should look different from a raw chunk. In the dashboard included in the repo, reflections render with a 💡 badge and a "synthesised from N documents" note.

Full source at github.com/dannwaneri/vectorize-mcp-worker — reflection engine, consolidation, batch ingest, dashboard, OpenAPI spec.

The codebase is TypeScript, deploys with a single wrangler deploy, runs for roughly $1–5/month at 10,000 queries/day.

Standard RAG retrieves. This learns.

This tutorial is different. I run a production RAG system (vectorize-mcp-worker) that handles real traffic at a total cost of \(5/month. The alternatives I evaluated ranged from \)100–$200/month. The difference isn't magic. It's architecture.

Here, you'll build rag-tutorial-simple: a clean, minimal RAG chatbot deployed on Cloudflare Workers. No external API keys. No paid vector database subscriptions. No servers to manage. Just Cloudflare's free tier – Workers, Vectorize, and Workers AI – doing the heavy lifting at the edge.

Table of Contents

1. What You Will Build

2. Prerequisites

3. How RAG Works

4. How to Set Up Your Project

5. How to Build the Data Pipeline

6. How to Build the Query Pipeline

7. How to Add Error Handling and Security

8. Performance and Cost Analysis

9. Conclusion

What You Will Build

By the end of this tutorial, you'll have a globally deployed RAG API that:

• Accepts a natural language question via HTTP

• Converts it to a vector embedding using Workers AI

• Searches a knowledge base stored in Cloudflare Vectorize

• Passes the retrieved context to an LLM (also on Workers AI) to generate an answer

• Returns a grounded, accurate response (not a hallucination)

The complete source code is available at github.com/dannwaneri/rag-tutorial-simple.

Prerequisites

This is an intermediate-level tutorial. You should be comfortable with:

• JavaScript/TypeScript: async/await, promises, basic types

• HTTP APIs: REST, request/response, JSON

• Command line basics: running npm commands, navigating directories

You will need:

• Node.js 18 or higher: check with node --version

• A Cloudflare account: free tier is fine, sign up at cloudflare.com

• A code editor: VS Code recommended for TypeScript support

That's it. No OpenAI key. No credit card for embeddings. Let's build.

How RAG Works

Before you write any code, you'll need a clear mental model of what you're building. This section explains the three core components of a RAG system, how data flows between them, and why this architecture works at scale.

The Mental Model

Think of a traditional LLM like a doctor who studied medicine for years but has been in a remote cabin with no internet since their graduation day. They are brilliant, but they only know what they knew when they left. Ask them about a drug approved last year and they'll either say they don't know or – worse – confidently give you wrong information.

RAG gives that doctor access to an up-to-date medical library. Before answering your question, they can look up the relevant pages, read them, and use that information to give you an accurate answer. Their training still matters (that is, they know how to read and interpret the information), but they're no longer limited to what they memorized years ago.

In technical terms, RAG works in three steps on every request:

1. Retrieve: find the most relevant documents from your knowledge base

2. Augment: add those documents to the LLM prompt as context

3. Generate: let the LLM produce an answer using both its training and the retrieved context

The Three Components

Every RAG system has three moving parts. Understanding each one will help you debug problems and make better architectural decisions as you build.

The Embedding Model

An embedding model converts text into a vector – an array of numbers that represents the meaning of that text. The model you will use in this tutorial, @cf/baai/bge-base-en-v1.5, outputs 768 numbers for any piece of text you give it.

The critical property of embeddings is that semantically similar text produces numerically similar vectors. "How do I install Node.js?" and "What's the process for setting up Node?" will produce vectors that are close together. "How do I install Node.js?" and "What is the capital of France?" will produce vectors that are far apart.

This is what makes semantic search possible. You aren't matching keywords, you're matching meaning.

One rule you must never break: your documents and your queries must be embedded with the same model. If you embed your documents with bge-base-en-v1.5 and your queries with a different model, the vectors won't be comparable and your searches will return garbage.

The Vector Database

The vector database stores your embeddings and lets you search them by similarity. In this tutorial, you'll use Cloudflare Vectorize.

When you run a similarity search, you pass in a query vector and Vectorize returns the K most similar vectors it has stored, along with their metadata and similarity scores. This is called approximate nearest neighbor search, and Vectorize is optimized to do it fast even across millions of vectors.

The key advantage of using Vectorize over an external vector database like Pinecone is co-location. Vectorize runs in the same Cloudflare network as your Worker. There's no external API call, no authentication roundtrip, and no network latency between your application and your database.

The Language Model

The LLM is responsible for one thing: reading the retrieved context and generating a natural language answer. It doesn't search anything. It doesn't decide what's relevant. It just reads what you give it and writes a response.

This separation of concerns is intentional. The LLM is good at language: understanding questions, synthesizing information, writing clearly. The vector database is good at retrieval: finding relevant documents fast. RAG combines their strengths without asking either component to do something it is not designed for.

In this tutorial you'll use @cf/meta/llama-3.3-70b-instruct-fp8-fast through Workers AI. No API key required.

A Note on Visual Embeddings

If you plan to extend this system to search images, you may be tempted to use a vision-language model like CLIP to generate visual embeddings (vectors that represent the image itself rather than a text description of it). This sounds clever but works worse for RAG in practice.

Visual embeddings match pixel similarity. They are good for "find images that look like this one." They are poor for "find the login screen" or "find dashboards showing error rates" because those queries are about meaning, not pixels.

The better approach – used in production – is to pass the image through a multimodal model like Llama 4 Scout, which generates a detailed text description and extracts visible text via OCR. You then embed that description using the same BGE model as your other documents.

The result lives in one unified index, works with your existing query pipeline, and produces better search results than visual embeddings for RAG use cases.

Cloudflare Workers AI does not support CLIP anyway. But even if it did, descriptions would outperform it for semantic search.

How a Query Flows Through the System

Here is exactly what happens when a user sends the question "What is RAG?" to your finished Worker:

1. Step 1 – Embed the question (20-30ms): Your Worker calls Workers AI with the question text. The embedding model returns a 768-dimensional vector representing the meaning of the question.

2. Step 2 – Search Vectorize (30-50ms): Your Worker passes that vector to Vectorize, which searches your knowledge base and returns the 3 most similar documents with their similarity scores.

3. Step 3 – Filter and build context (< 1ms): Documents with a similarity score below 0.5 are discarded. The remaining document texts are joined into a context string.

4. Step 4 – Generate the answer (500-1500ms): Your Worker sends the context and the question to the LLM. The LLM reads the context and generates a grounded answer.

5. Step 5 – Return to the user: The answer and source metadata are returned as JSON.

Total time: typically 600-1600ms end to end. The LLM generation step dominates. Everything else is fast.

Why This Works at Scale

A common objection to Cloudflare RAG is that it cannot meet sub-200ms retrieval requirements. That objection comes from a specific architectural mistake: trying to run the entire RAG pipeline, including heavy embedding generation and reranking, inside a single synchronous request. That's the wrong architecture.

The architecture you're building in this tutorial separates the loading step (which is slow and runs once) from the query step (which is fast and runs on every request). By the time a user asks a question, your documents are already embedded and stored. The query pipeline only needs to embed the question, run one vector search, and call the LLM. Those three steps are fast.

My production system (vectorize-mcp-worker) runs this architecture and handles real traffic at $5/month. The full performance breakdown is here. Cloudflare RAG works. You just have to build it correctly.

How to Set Up Your Project

In this section, you'll scaffold a Cloudflare Worker, create a Vectorize index to store your embeddings, and configure the bindings that connect them together.

How to Create the Project

Open your terminal and create a new directory for the project.

On Mac/Linux:

mkdir rag-tutorial-simple && cd rag-tutorial-simple

On Windows PowerShell:

mkdir rag-tutorial-simple
cd rag-tutorial-simple

Then run the Cloudflare scaffolding tool:

npm create cloudflare@latest

Answer the prompts like this:

• Directory/app name: rag-tutorial-simple

• What would you like to start with? Hello World example

• TypeScript? Yes

• Deploy? No

When it finishes, you'll have a working TypeScript Worker with Wrangler already configured.

How to Create the Vectorize Index

Vectorize is Cloudflare's vector database. It lives in the same network as your Worker, which means no external API call and no added latency when you search it.

npx wrangler vectorize create rag-tutorial-index --dimensions=768 --metric=cosine

Two things to note here.

--dimensions=768 tells Vectorize how many numbers make up each embedding. This must match the output of the embedding model you use. The model you will use (@cf/baai/bge-base-en-v1.5) outputs 768 dimensions. If this number doesn't match, your searches will fail.

--metric=cosine is how Vectorize measures similarity between vectors. Cosine similarity measures the angle between two vectors rather than the distance between them. For text embeddings, this captures semantic meaning more accurately than other metrics.

How to Configure wrangler.toml

Open wrangler.toml and replace its contents with the following:

name = "rag-tutorial-simple"
main = "src/index.ts"
compatibility_date = "2026-02-25"

[[vectorize]]
binding = "VECTORIZE"
index_name = "rag-tutorial-index"

[ai]
binding = "AI"

The [[vectorize]] block connects your Worker to the index you just created. The [ai] block gives your Worker access to Workers AI – both for generating embeddings and for running the language model that produces answers.

Notice that there are no API keys anywhere. Cloudflare handles authentication internally because everything – your Worker, Vectorize, and Workers AI – runs under the same account.

How to Update src/index.ts

Open src/index.ts and replace the generated code with this:

export interface Env {
  VECTORIZE: VectorizeIndex;
  AI: Ai;
  LOAD_SECRET: string;
}

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    return new Response("RAG tutorial worker is running", { status: 200 });
  },
};

The Env interface tells TypeScript what bindings are available inside your Worker. VectorizeIndex and Ai are types provided by Cloudflare's type definitions.

How to Verify Your Setup

Start the local development server:

npx wrangler dev

Open your browser and visit http://localhost:8787. You should see:

RAG tutorial worker is running

You will see two warnings in your terminal. Both are expected.

The first warning says that Vectorize doesn't support local mode. This means Vectorize queries won't work during local development unless you run with the --remote flag. You'll do this later when testing the full pipeline.

The second warning says the AI binding always accesses remote resources. This means that embedding generation and LLM calls always hit Cloudflare's servers, even in local development. This is fine: usage within the free tier limits costs nothing.

Your project structure at this point:

rag-tutorial-simple/
├── scripts/
│   └── knowledge-base.ts
├── src/
│   └── index.ts
├── wrangler.toml
├── package.json
└── tsconfig.json

How to Build the Data Pipeline

The data pipeline is responsible for two things: generating embeddings for each document in your knowledge base, and storing those embeddings in Vectorize. You'll handle both steps inside the Worker itself using a /load endpoint.

This approach has a key advantage: you don't need an API token, an Account ID, or any external tooling. Everything uses the bindings you already configured in wrangler.toml.

How to Create the Knowledge Base

Create a scripts/ folder in your project and add a file called knowledge-base.ts:

mkdir scripts

Add your documents to scripts/knowledge-base.ts:

export const documents = [
  {
    id: "1",
    text: "Cloudflare Workers run JavaScript at the edge, in over 300 data centers worldwide. Requests are handled close to the user, reducing latency significantly compared to a single-region server.",
    metadata: { source: "cloudflare-docs", category: "workers" },
  },
  {
    id: "2",
    text: "Vectorize is Cloudflare's vector database. It stores embeddings and lets you search them by semantic similarity. It runs in the same network as your Worker, so there is no external API call needed.",
    metadata: { source: "cloudflare-docs", category: "vectorize" },
  },
  {
    id: "3",
    text: "Workers AI lets you run machine learning models directly on Cloudflare's infrastructure. You can generate embeddings and run LLM inference without leaving the Cloudflare network.",
    metadata: { source: "cloudflare-docs", category: "workers-ai" },
  },
  {
    id: "4",
    text: "RAG stands for Retrieval Augmented Generation. Instead of relying only on what the LLM was trained on, RAG retrieves relevant context from a knowledge base and adds it to the prompt before generating an answer.",
    metadata: { source: "ai-concepts", category: "rag" },
  },
  {
    id: "5",
    text: "An embedding is a numerical representation of text. Similar pieces of text produce similar embeddings. This is what makes semantic search possible — you search by meaning, not exact keywords.",
    metadata: { source: "ai-concepts", category: "embeddings" },
  },
  {
    id: "6",
    text: "The BGE model (bge-base-en-v1.5) is available through Workers AI. It generates 768-dimensional embeddings and works well for English semantic search tasks.",
    metadata: { source: "cloudflare-docs", category: "workers-ai" },
  },
  {
    id: "7",
    text: "Cosine similarity measures the angle between two vectors. For text embeddings, it captures semantic similarity regardless of text length, which makes it more reliable than Euclidean distance.",
    metadata: { source: "ai-concepts", category: "embeddings" },
  },
  {
    id: "8",
    text: "Cloudflare Workers have a free tier that includes 100,000 requests per day. Vectorize is available on both the Workers Free and Paid plans. The free tier lets you prototype and experiment. The Workers Paid plan starts at $5/month and includes higher usage allocations for production workloads.",
    metadata: { source: "cloudflare-docs", category: "pricing" },
  },
];

Each document has three fields. The id is a unique string that Vectorize uses to identify the vector. The text is what gets converted into an embedding. The metadata is stored alongside the vector and returned in search results. You'll use it later to display the source of each answer.

Understanding Embeddings

Before writing the loading code, it helps to understand what you're actually generating.

An embedding is an array of 768 numbers that represents the meaning of a piece of text. The model reads a sentence and outputs those 768 numbers in a way where similar sentences produce similar arrays of numbers.

When a user asks a question, you convert that question into an embedding using the same model, then ask Vectorize to find the stored embeddings that are closest to it. The documents those embeddings came from are your most relevant context.

This is why the model choice matters: your documents and your queries must be embedded with the same model, or the similarity scores will be meaningless.

How to Build the Load Endpoint

Open src/index.ts and update it with a /load route. Here is the complete file at this stage:

import { documents } from "../scripts/knowledge-base";

export interface Env {
  VECTORIZE: VectorizeIndex;
  AI: Ai;
  LOAD_SECRET: string;
}

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const url = new URL(request.url);

    if (url.pathname === "/load" && request.method === "POST") {
      return handleLoad(env, request);
    }

    return new Response("RAG tutorial worker is running", { status: 200 });
  },
};

async function handleLoad(env: Env, request: Request): Promise<Response> {
  const authHeader = request.headers.get("X-Load-Secret");
  if (authHeader !== env.LOAD_SECRET) {
    return Response.json({ error: "Unauthorized" }, { status: 401 });
  }

  const results: { id: string; status: string }[] = [];

  for (const doc of documents) {
    const response = await env.AI.run("@cf/baai/bge-base-en-v1.5", {
      text: [doc.text],
    }) as { data: number[][] };

    await env.VECTORIZE.upsert([
      {
        id: doc.id,
        values: response.data[0],
        metadata: {
          ...doc.metadata,
          text: doc.text,
        },
      },
    ]);

    results.push({ id: doc.id, status: "loaded" });
  }

  return Response.json({ success: true, loaded: results });
}

Notice that env.AI.run() and env.VECTORIZE.upsert() require no credentials. The bindings handle authentication because the Worker runs inside your Cloudflare account. There are no secrets to manage for internal service communication.

The text: doc.text field inside metadata is important. Vectorize stores the vector values and whatever metadata you provide, but it doesn't store the original text separately. By including the text in metadata, you can retrieve and display it in search results later.

The as { data: number[][] } cast is necessary because the TypeScript type definitions for Workers AI do not yet reflect the exact return shape of every model. The actual response always contains a data array, and the cast tells TypeScript to trust that.

How to Deploy and Load Your Knowledge Base

First, set the secret that will protect your load endpoint:

npx wrangler secret put LOAD_SECRET

Type a strong value when prompted. Then deploy:

npx wrangler deploy

Trigger the load endpoint. You only need to do this once, or any time you update your knowledge base:

curl -X POST https://rag-tutorial-simple.<your-subdomain>.workers.dev/load \
  -H "X-Load-Secret: your-secret-value"

On Windows PowerShell:

Note: PowerShell uses backtick (`) for line continuation, not backslash.

Invoke-WebRequest `
  -Uri "https://rag-tutorial-simple.<your-subdomain>.workers.dev/load" `
  -Method POST `
  -Headers @{"X-Load-Secret"="your-secret-value"} `
  -UseBasicParsing

You should see:

{
  "success": true,
  "loaded": [
    { "id": "1", "status": "loaded" },
    { "id": "2", "status": "loaded" },
    { "id": "3", "status": "loaded" },
    { "id": "4", "status": "loaded" },
    { "id": "5", "status": "loaded" },
    { "id": "6", "status": "loaded" },
    { "id": "7", "status": "loaded" },
    { "id": "8", "status": "loaded" }
  ]
}

Your knowledge base is now stored in Vectorize as vectors. In the next section, you'll build the query pipeline that searches those vectors and generates answers.

How to Build the Query Pipeline

The query pipeline is the core of your RAG system. When a user sends a question, the pipeline runs four steps in sequence: embed the question, search Vectorize, build context from the results, and generate an answer with the LLM.

Add a /query route to your fetch handler and the complete handleQuery function. Here is the full updated src/index.ts:

import { documents } from "../scripts/knowledge-base";

export interface Env {
  VECTORIZE: VectorizeIndex;
  AI: Ai;
  LOAD_SECRET: string;
}

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const url = new URL(request.url);

    if (url.pathname === "/load" && request.method === "POST") {
      return handleLoad(env, request);
    }

    if (url.pathname === "/query" && request.method === "POST") {
      return handleQuery(request, env);
    }

    return new Response("RAG tutorial worker is running", { status: 200 });
  },
};

async function handleLoad(env: Env, request: Request): Promise<Response> {
  const authHeader = request.headers.get("X-Load-Secret");
  if (authHeader !== env.LOAD_SECRET) {
    return Response.json({ error: "Unauthorized" }, { status: 401 });
  }

  const results: { id: string; status: string }[] = [];

  for (const doc of documents) {
    const response = await env.AI.run("@cf/baai/bge-base-en-v1.5", {
      text: [doc.text],
    }) as { data: number[][] };

    await env.VECTORIZE.upsert([
      {
        id: doc.id,
        values: response.data[0],
        metadata: {
          ...doc.metadata,
          text: doc.text,
        },
      },
    ]);

    results.push({ id: doc.id, status: "loaded" });
  }

  return Response.json({ success: true, loaded: results });
}

async function handleQuery(request: Request, env: Env): Promise<Response> {
  const body = await request.json() as { question: string };

  if (!body.question) {
    return Response.json({ error: "question is required" }, { status: 400 });
  }

  // Step 1: Embed the question using the same model as your documents
  const embeddingResponse = await env.AI.run("@cf/baai/bge-base-en-v1.5", {
    text: [body.question],
  }) as { data: number[][] };

  // Step 2: Search Vectorize for the 3 most similar documents
  const searchResults = await env.VECTORIZE.query(
    embeddingResponse.data[0],
    {
      topK: 3,
      returnMetadata: "all",
    }
  );

  // Step 3: Build context from results above the similarity threshold
  const context = searchResults.matches
    .filter((match) => match.score > 0.5)
    .map((match) => match.metadata?.text as string)
    .filter(Boolean)
    .join("\n\n");

  if (!context) {
    return Response.json({
      answer: "I could not find relevant information to answer that question.",
      sources: [],
    });
  }

  // Step 4: Generate an answer using the retrieved context
  const aiResponse = await env.AI.run("@cf/meta/llama-3.3-70b-instruct-fp8-fast", {
    messages: [
      {
        role: "system",
        content: "You are a helpful assistant. Answer the question using only the context provided. If the context does not contain enough information, say so.",
      },
      {
        role: "user",
        content: `Context:\n\({context}\n\nQuestion: \){body.question}`,
      },
    ],
    max_tokens: 256,
  }) as { response: string };

  // Step 5: Return the answer with its sources
  const sources = searchResults.matches
    .filter((match) => match.score > 0.5)
    .map((match) => match.metadata?.source as string)
    .filter(Boolean);

  return Response.json({
    answer: aiResponse.response,
    sources: [...new Set(sources)],
  });
}

What each step does:

1. Step 1 – Embed the question: You convert the user's question into a 768-dimensional vector using the same model you used when loading your documents. This is critical: the question and the documents must be embedded with the same model or the similarity scores will be meaningless.

2. Step 2 – Search Vectorize: You pass the question embedding to Vectorize, which returns the three most similar documents. returnMetadata: "all" tells Vectorize to include the metadata you stored alongside each vector — including the original text.

3. Step 3 – Build context: You filter out any results with a similarity score below 0.5 and join the remaining document texts into a single context string. The 0.5 threshold prevents the LLM from receiving irrelevant documents just because nothing better matched.

4. Step 4 – Generate the answer: You pass the context and the question to the LLM using the chat format with messages. The system prompt explicitly instructs the model to answer using only the provided context. This is what keeps the LLM grounded. Without this instruction, it will ignore your context and answer from its training data instead.

5. Step 5 – Return sources: You include the source metadata in the response so callers know which documents the answer came from. The Set deduplicates sources in case multiple chunks came from the same document.

How to Test the Query Pipeline

Deploy your Worker:

npx wrangler deploy

Send a question:

curl -X POST https://rag-tutorial-simple.<your-subdomain>.workers.dev/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What is RAG?"}'

On Windows PowerShell:

Invoke-WebRequest `
  -Uri "https://rag-tutorial-simple.<your-subdomain>.workers.dev/query" `
  -Method POST `
  -ContentType "application/json" `
  -Body '{"question": "What is RAG?"}' `
  -UseBasicParsing

You should receive a response like this:

{
  "answer": "RAG stands for Retrieval Augmented Generation. It's a method that enhances generation by retrieving relevant context from a knowledge base and adding it to the prompt before generating an answer.",
  "sources": ["ai-concepts"]
}

The answer came from your knowledge base, not from the LLM's training data. That's the entire point of RAG: grounded, verifiable answers with traceable sources.

How to Add Error Handling and Security

A tutorial that only shows the happy path is not production-ready. In this section, you'll add error handling to every step of the query pipeline and protect the /load endpoint from unauthorized access.

How to Secure the Load Endpoint

The /load endpoint generates embeddings and writes to your Vectorize index. Without protection, anyone who discovers your Worker URL can trigger it repeatedly, consuming your Workers AI quota and overwriting your data.

The LOAD_SECRET binding you added to Env and the wrangler secret put command you ran earlier handle this. The check at the top of handleLoad rejects any request that doesn't include the correct secret header:

const authHeader = request.headers.get("X-Load-Secret");
if (authHeader !== env.LOAD_SECRET) {
  return Response.json({ error: "Unauthorized" }, { status: 401 });
}

A request without the header returns {"error":"Unauthorized"} with a 401 status. The secret itself is stored as an encrypted environment variable in your Worker. It never appears in your code or wrangler.toml.

To trigger the load endpoint, you must include the secret in the request header:

curl -X POST https://rag-tutorial-simple.<your-subdomain>.workers.dev/load \
  -H "X-Load-Secret: your-secret-value"

How to Handle Query Errors

Replace your handleQuery function with this hardened version:

async function handleQuery(request: Request, env: Env): Promise<Response> {
  // Guard against malformed request body
  let body: { question: string };
  try {
    body = await request.json() as { question: string };
  } catch {
    return Response.json({ error: "Invalid JSON in request body" }, { status: 400 });
  }

  if (!body.question || typeof body.question !== "string" || body.question.trim() === "") {
    return Response.json({ error: "question must be a non-empty string" }, { status: 400 });
  }

  // Sanitize: trim whitespace and cap length
  const question = body.question.trim().slice(0, 500);

  // Step 1: Embed the question
  let embeddingResponse: { data: number[][] };
  try {
    embeddingResponse = await env.AI.run("@cf/baai/bge-base-en-v1.5", {
      text: [question],
    }) as { data: number[][] };
  } catch (err) {
    console.error("Embedding generation failed:", err);
    return Response.json({ error: "Failed to process your question" }, { status: 503 });
  }

  // Step 2: Search Vectorize
  let searchResults: Awaited<ReturnType<typeof env.VECTORIZE.query>>;
  try {
    searchResults = await env.VECTORIZE.query(
      embeddingResponse.data[0],
      { topK: 3, returnMetadata: "all" }
    );
  } catch (err) {
    console.error("Vectorize query failed:", err);
    return Response.json({ error: "Failed to search knowledge base" }, { status: 503 });
  }

  // Step 3: Build context
  const context = searchResults.matches
    .filter((match) => match.score > 0.5)
    .map((match) => match.metadata?.text as string)
    .filter(Boolean)
    .join("\n\n");

  if (!context) {
    return Response.json({
      answer: "I could not find relevant information to answer that question. Try rephrasing or asking something else.",
      sources: [],
    });
  }

  // Step 4: Generate answer
  let aiResponse: { response: string };
  try {
    aiResponse = await env.AI.run("@cf/meta/llama-3.3-70b-instruct-fp8-fast", {
      messages: [
        {
          role: "system",
          content: "You are a helpful assistant. Answer the question using only the context provided. If the context does not contain enough information, say so.",
        },
        {
          role: "user",
          content: `Context:\n\({context}\n\nQuestion: \){question}`,
        },
      ],
      max_tokens: 256,
    }) as { response: string };
  } catch (err) {
    console.error("LLM generation failed:", err);
    return Response.json({ error: "Failed to generate an answer" }, { status: 503 });
  }

  // Step 5: Return answer with sources
  const sources = searchResults.matches
    .filter((match) => match.score > 0.5)
    .map((match) => match.metadata?.source as string)
    .filter(Boolean);

  return Response.json({
    answer: aiResponse.response,
    sources: [...new Set(sources)],
  });
}

What each error handling decision means:

• try/catch around request.json(): request.json() throws if the body is not valid JSON. Without this catch, a malformed request crashes your Worker with an unhandled 500 error. With it, the caller gets a clear 400 explaining what went wrong.

• Input validation before processing: You check that question exists, is a string, and is not empty before calling any external service. This prevents wasted AI calls on invalid input.

• .slice(0, 500) on the question: This caps the input length before it reaches the embedding model. Without it, a malicious caller could send a very long string designed to inflate your AI usage or hit Workers CPU limits.

• 503 for AI and Vectorize failures: HTTP 503 means "service temporarily unavailable." It signals to callers that the error is on the server side and the request can be retried.

• .filter(Boolean) on context: After mapping match.metadata?.text, some results may be undefined if metadata was stored without a text field. This filters them out before joining, preventing "undefined" from appearing in the context string you send to the LLM.

How to Test Error Handling

Deploy your updated Worker:

npx wrangler deploy

Test each error case:

# Missing secret on load endpoint — should return 401
curl -X POST https://rag-tutorial-simple.<your-subdomain>.workers.dev/load

# Invalid JSON — should return 400
curl -X POST https://rag-tutorial-simple.<your-subdomain>.workers.dev/query \
  -H "Content-Type: application/json" \
  -d 'not json'

# Empty question — should return 400
curl -X POST https://rag-tutorial-simple.<your-subdomain>.workers.dev/query \
  -H "Content-Type: application/json" \
  -d '{"question": ""}'

Performance and Cost Analysis

This section uses real production data from my vectorize-mcp-worker deployment. It uses the same architecture you just built, measured from Port Harcourt, Nigeria to Cloudflare's edge.

Real Performance Numbers

Here is what the pipeline actually costs in time on every request:

This covers embedding generation and vector search only – the retrieval layer. LLM generation adds 500-1500ms on top, which is why end-to-end response time typically runs 600-1600ms.

The embedding step and vector search dominate. Everything else is negligible. For context, a comparable setup using OpenAI embeddings and Pinecone would add two external API roundtrips on top of this, easily pushing total latency past 1 second.

These numbers come from a single-region measurement. Your actual latency will vary based on your location and Cloudflare's load at the time of the request. The architectural point holds regardless: co-locating everything on the edge eliminates inter-service network hops, which is where most latency in traditional RAG stacks comes from.

Real Cost Breakdown

For 10,000 searches per day (300,000 per month) with 10,000 stored vectors:

This stack:

Traditional alternatives for the same volume:

That is an 85-95% cost reduction depending on which alternative you compare against. For a bootstrapped startup adding semantic search, that difference is $1,500-2,000 per year.

Why the Cost Difference Is So Large

Traditional RAG stacks have three cost problems that compound each other.

The first is idle compute. A dedicated server or container running your embedding service costs money even when no searches are happening. Cloudflare Workers charge only for actual execution time.

The second is inter-service data transfer. Every time your application calls an external service for an embedding, then calls a separate service for a search, you're paying for two external API calls with metered pricing. In this stack, both operations happen inside Cloudflare's network at no additional transfer cost.

The third is minimum plan pricing. Pinecone's Standard plan costs \(50/month as a floor, regardless of how little you use it. Cloudflare's pricing scales from the \)5/month Workers Paid plan base.

When the Included Allocation Is Enough

For smaller usage levels, you may not pay beyond the $5/month Workers Paid base price:

• Workers: 10 million requests per month included

• Workers AI: generous daily neuron allocation included

• Vectorize: available on both Free and Paid plans, with a free allocation included

A side project, internal tool, or small business with under 3,000 searches per day will likely stay within the included allocations entirely.

The Trade-off to Know About

This cost advantage comes with one operational constraint worth understanding before you build: Vectorize does not work in local development mode.

When you run wrangler dev, your Worker runs locally but Vectorize calls fail. You have to deploy to Cloudflare to test your vector search. For most development workflows this means testing your query logic locally with mocked responses, then deploying to a staging environment for full integration tests.

This is a real friction point. It's the honest trade-off for having a managed vector database with no infrastructure to operate.

Conclusion

In this tutorial, you have built and deployed a production-ready RAG system on Cloudflare's edge network. Let's look at what you actually built and what it costs to run.

What You Built

Your completed system has three endpoints:

• GET /: health check confirming the Worker is running

• POST /load: loads your knowledge base into Vectorize, protected by a secret header

• POST /query: accepts a question, retrieves relevant context, and returns a grounded answer with sources

The full query pipeline runs in four steps on every request:

1. The question is converted to a 768-dimensional embedding using @cf/baai/bge-base-en-v1.5

2. Vectorize finds the three most semantically similar documents

3. Documents above the 0.5 similarity threshold are assembled into context

4. Llama 3.3 generates an answer using only that context

Everything runs on Cloudflare's infrastructure. No external API keys. No separate vector database subscription. No servers to manage.

What to Build Next

This tutorial covered the core RAG pattern. Here are four directions to take it further.

Add more documents

The knowledge base in this tutorial has 8 documents. A real system might have thousands. The loading pattern is identical: add documents to knowledge-base.ts, hit /load with your secret, and Vectorize handles the rest.

For very large knowledge bases, update handleLoad to batch documents in groups of 20-50 rather than upserting one at a time.

Improve chunking

Each document in this tutorial is a single short paragraph. Real-world documents like PDFs, articles, documentation pages need to be split into chunks before embedding. Chunk at natural boundaries like paragraphs and sentences, aim for 200-400 tokens per chunk, and include 50-token overlaps between chunks to preserve context across boundaries.

Add conversation history

The current system treats every query as independent. To support follow-up questions, store previous messages in a Cloudflare KV namespace and include the last 2-3 exchanges in the LLM messages array alongside the retrieved context.

Stream the response

For long answers, users stare at a blank screen until generation completes. Cloudflare Workers support streaming responses via TransformStream. Switching to streaming means the first tokens appear in under 100ms while the rest generates.

Consider dimensions vs reranking trade-offs

This tutorial uses bge-base-en-v1.5 at 768 dimensions. My production system uses bge-small-en-v1.5 at 384 dimensions. Testing showed upgrading from 384 to 768 dims only improved accuracy by about 2%, but doubled cost and latency.

Adding a reranker (@cf/baai/bge-reranker-base) gave a larger accuracy improvement than the dimension upgrade for a fraction of the cost. The exact improvement will vary by domain and query distribution — test both on your actual data before deciding. If you're optimizing for production, add a reranker before you increase dimensions.

The Complete Project

Clone and deploy in five commands:

git clone https://github.com/dannwaneri/rag-tutorial-simple
cd rag-tutorial-simple
npm install
npx wrangler vectorize create rag-tutorial-index --dimensions=768 --metric=cosine
npx wrangler secret put LOAD_SECRET
npx wrangler deploy

Then load your knowledge base:

curl -X POST https://<your-worker>.workers.dev/load \
  -H "X-Load-Secret: your-secret"

If you found this useful, the production system this tutorial is based on is open source at github.com/dannwaneri/vectorize-mcp-worker. It extends this foundation with hybrid search combining vector and BM25, multimodal support for searching images with AI vision, a reranker for more accurate results, and a live dashboard. It runs on the same Cloudflare stack you just built – Workers, Vectorize, Workers AI – plus D1 for document storage.

One difference you'll notice: the production system uses bge-small-en-v1.5 at 384 dimensions rather than the 768 dimensions in this tutorial. That is an intentional trade-off: the reranker adds more accuracy than the extra dimensions at lower cost. The jump from what you built today to that system is smaller than it looks.

They answer questions they should not, they break when document retrieval is weak, they time out due to network latency, and nobody can tell exactly what happened because there are no logs and no tests.

In this tutorial, you’ll build a beginner-friendly Retrieval Augmented Generation (RAG) application designed to survive production realities. This isn’t just a script that calls an API. It’s a system featuring a FastAPI backend, a persisted FAISS vector store, and essential safety guardrails (including a retrieval gate and fallbacks).

Table of Contents

1. Why RAG Alone Does Not Equal Production-Ready

2. The Architecture You Are Building

3. Project Setup and Structure

4. How to Build the RAG Layer with FAISS

5. How to Add the LLM Call with Structured Output

6. How to Add Guardrails: Retrieval Gate and Fallbacks

7. FastAPI App: Creating the /answer Endpoint

8. How to Add Beginner-Friendly Evals

9. What to Improve Next: Realistic Upgrades

Why RAG Alone Does Not Equal Production-Ready

Retrieval Augmented Generation (RAG) is often hailed as the hallucination killer. By grounding the model in retrieved text, we provide it with the facts it needs to be accurate. But simply connecting a vector database to an LLM isn’t enough for a production environment.

Production issues usually arise from the silent failures in the system surrounding the model:

• Weak retrieval: If the app retrieves irrelevant chunks of text, the model tries to bridge the gap by inventing an answer anyway. Without a designated “I do not know” path, the model is essentially forced to hallucinate.

• Lack of visibility: Without structured outputs and basic logging, you can’t tell if bad retrieval, a confusing prompt, or a model update caused a wrong answer.

• Fragility: A simple API timeout or malformed provider response becomes a user-facing outage if you don’t implement fallbacks.

• No regression testing: In traditional software, we have unit tests. In AI, we need evals. Without them, a small tweak to your prompt might fix one issue but break ten others without you realising it.

We’ll solve each of these issues systematically in this guide.

Prerequisites

This tutorial is beginner-friendly, but it assumes you have a few basics in place so you can focus on building a robust RAG system instead of getting stuck on setup issues.

Knowledge

You should be comfortable with:

• Python fundamentals (functions, modules, virtual environments)

• Basic HTTP + JSON (requests, response payloads)

• APIs with FastAPI (what an endpoint is and how to run a server)

• High-level LLM concepts (prompting, temperature, structured outputs)

Tools + Accounts

You’ll need:

• Python 3.10+

• A working OpenAI-compatible API key (OpenAI or any provider that supports the same request/response shape)

• A local environment where you can run a FastAPI app (Mac/Linux/Windows)

What This Tutorial Covers (and What It Doesn’t)

We’ll build a production-minded baseline:

• A FAISS-backed retriever with a persisted index + metadata

• A retrieval gate to prevent “forced hallucination”

• Structured JSON outputs so your backend is stable

• Fallback behavior for timeouts and provider errors

• A small eval harness to prevent regressions

We won’t implement advanced upgrades such as rerankers, semantic chunking, auth, background jobs beyond a roadmap at the end.

The Architecture You Are Building

The flow of our application follows a disciplined path so every answer is grounded in evidence:

1. User query: The user submits a question via a FastAPI endpoint.

2. Retrieval: The system embeds the question and retrieves the top-k most similar document chunks.

3. The retrieval gate: We evaluate the similarity score. If the context is not relevant enough, we stop immediately and refuse the query.

4. Augmentation and generation: If the gate passes, we send a context-augmented prompt to the LLM.

5. Structured response: The model returns a JSON object containing the answer, sources used, and a confidence level.

Project Setup and Structure

To keep things organized and maintainable, we’ll use a modular structure. This allows you to swap out your LLM provider or your vector database without rewriting your entire core application.

Project Structure

.
├── app.py              # FastAPI entry point and API logic
├── rag.py              # FAISS index, persistence, and document retrieval
├── llm.py              # LLM API interface and JSON parsing
├── prompts.py          # Centralized prompt templates
├── data/               # Source .txt documents
├── index/              # Persisted FAISS index and metadata
└── evals/              # Evaluation dataset and runner script
    ├── eval_set.json
    └── run_evals.py

Install Dependencies

First, create a virtual environment to isolate your project:

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install fastapi uvicorn faiss-cpu numpy pydantic requests python-dotenv

Configure the Environment

Create a .env file in the root directory. We are targeting OpenAI-compatible providers:

OPENAI_API_KEY=your_actual_api_key_here
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_MODEL=gpt-4o-mini

Important note on compatibility: The code below assumes an OpenAI-style API. If you use a provider that is not compatible, you must change the URL, headers (for example X-API-Key), and the way you extract embeddings and final message content in embed_texts() and call_llm().

How to Build the RAG Layer with FAISS

In rag.py, we handle the “Retriever” part of RAG. This involves turning raw text into mathematical vectors that the computer can compare.

What is FAISS (and What Does It Do)?

FAISS (Facebook AI Similarity Search) is a fast library for vector similarity search. In a RAG system, each chunk of text becomes an embedding vector (a list of floats). FAISS stores those vectors in an index so you can quickly ask:

“Given this question embedding, which document chunks are closest to it?”

In this tutorial, we use IndexFlatIP inner product and normalise vectors with faiss.normalize_L2(...). With normalised vectors, the inner product behaves like cosine similarity, giving us a stable score we can use for a retrieval gate.

Chunking Strategy With Overlap

We’ll use chunking with overlap. If we split a document at exactly 1,000 characters, we might cut a sentence in half, losing its meaning. By using an overlap, for example, 200 characters, we ensure that the end of one chunk and the beginning of the next share context.

Implementation of rag.py

import os
import faiss
import numpy as np
import requests
import json
from typing import List, Dict
from dotenv import load_dotenv

load_dotenv()

INDEX_PATH = "index/faiss.index"
META_PATH = "index/meta.json"

def chunk_text(text: str, size: int = 1000, overlap: int = 200) -> List[str]:
    chunks = []
    step = max(1, size - overlap)
    for i in range(0, len(text), step):
        chunk = text[i : i + size].strip()
        if chunk:
            chunks.append(chunk)
    return chunks

def embed_texts(texts: List[str]) -> np.ndarray:
    # Note: If your provider is not OpenAI-compatible, change this URL and headers
    url = f"{os.getenv('OPENAI_BASE_URL')}/embeddings"
    headers = {"Authorization": f"Bearer {os.getenv('OPENAI_API_KEY')}"}
    payload = {"input": texts, "model": "text-embedding-3-small"}

    resp = requests.post(url, headers=headers, json=payload, timeout=30)
    resp.raise_for_status()
    # If your provider uses a different response format, change the line below
    vectors = np.array([item["embedding"] for item in resp.json()["data"]], dtype="float32")
    return vectors

def build_index() -> None:
    all_chunks: List[str] = []
    metadata: List[Dict] = []

    if not os.path.exists("data"):
        os.makedirs("data")
        return

    for file in os.listdir("data"):
        if not file.endswith(".txt"):
            continue

        with open(f"data/{file}", "r", encoding="utf-8") as f:
            text = f.read()

        chunks = chunk_text(text)
        all_chunks.extend(chunks)
        for c in chunks:
            metadata.append({"source": file, "text": c})

    if not all_chunks:
        return

    embeddings = embed_texts(all_chunks)
    faiss.normalize_L2(embeddings)

    dim = embeddings.shape[1]
    index = faiss.IndexFlatIP(dim)
    index.add(embeddings)

    os.makedirs("index", exist_ok=True)
    faiss.write_index(index, INDEX_PATH)

    with open(META_PATH, "w", encoding="utf-8") as f:
        json.dump(metadata, f, ensure_ascii=False)

def load_index():
    if not (os.path.exists(INDEX_PATH) and os.path.exists(META_PATH)):
        raise FileNotFoundError(
            "FAISS index not found. Add .txt files to data/ and run build_index()."
        )

    index = faiss.read_index(INDEX_PATH)
    with open(META_PATH, "r", encoding="utf-8") as f:
        metadata = json.load(f)
    return index, metadata

def retrieve(query: str, k: int = 5) -> List[Dict]:
    index, metadata = load_index()

    q_emb = embed_texts([query])
    faiss.normalize_L2(q_emb)

    scores, ids = index.search(q_emb, k)
    results = []
    for score, idx in zip(scores[0], ids[0]):
        if idx == -1:
            continue
        m = metadata[idx]
        results.append(
            {"score": float(score), "source": m["source"], "text": m["text"], "id": int(idx)}
        )
    return results

How to Add the LLM Call with Structured Output

A major failure point in AI apps is the “chatty” nature of LLMs. If your backend expects a list of sources but the LLM returns conversational filler, your code will crash.

We solve this with structured output: instruct the model to return a strict JSON object, then parse it safely.

Implementation of llm.py

import json
import requests
import os
from typing import Dict, Any

def call_llm(system_prompt: str, user_prompt: str) -> Dict[str, Any]:
    # Note: Change URL/Headers if using a non-OpenAI compatible provider
    url = f"{os.getenv('OPENAI_BASE_URL')}/chat/completions"
    headers = {
        "Authorization": f"Bearer {os.getenv('OPENAI_API_KEY')}",
        "Content-Type": "application/json",
    }

    payload = {
        "model": os.getenv("OPENAI_MODEL"),
        "messages": [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        "response_format": {"type": "json_object"},
        "temperature": 0,
    }

    try:
        resp = requests.post(url, headers=headers, json=payload, timeout=30)
        resp.raise_for_status()
        content = resp.json()["choices"][0]["message"]["content"]

        parsed = json.loads(content)
        parsed.setdefault("answer", "")
        parsed.setdefault("refusal", False)
        parsed.setdefault("confidence", "medium")
        parsed.setdefault("sources", [])
        return parsed

    except (requests.Timeout, requests.ConnectionError):
        return {
            "answer": "The system is temporarily unavailable (network issue). Please try again.",
            "refusal": True,
            "confidence": "low",
            "sources": [],
            "error_type": "network_error",
        }
    except Exception:
        return {
            "answer": "A system error occurred while generating the answer.",
            "refusal": True,
            "confidence": "low",
            "sources": [],
            "error_type": "unknown_error",
        }

How to Add Guardrails: Retrieval Gate and Fallbacks

Guardrails are interceptors. They sit between the user and the model to prevent predictable failures.

The Retrieval Gate: How It Works and How to Add It

In a standard RAG pipeline, the system always calls the LLM. If the user asks an irrelevant question, the retriever will still return the “closest” (but wrong) chunks.

The solution is the retrieval gate:

1. Retrieve top-k chunks and get the top similarity score

2. If the score is below a threshold (for example 0.30), refuse immediately

3. Only call the LLM when retrieval is strong enough to ground the answer

A threshold of 0.30 is a reasonable starting point when using normalised cosine similarity, but you should tune it using evals (next section).

Fallbacks and Why They Matter

Fallbacks ensure that if an API fails or times out, the user gets a helpful message instead of a crash. They also keep your API response shape consistent, which prevents frontend errors and makes logging meaningful.

In this tutorial, fallbacks are implemented inside call_llm() so your FastAPI layer stays simple.

FastAPI App: Creating the /answer Endpoint

The app.py file is the conductor. It ties retrieval, guardrails, prompting, and generation together.

Implementation of app.py

from fastapi import FastAPI
from pydantic import BaseModel
from rag import retrieve
from llm import call_llm
import prompts
import time
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("rag_app")

app = FastAPI(title="Production-Ready RAG")

class QueryRequest(BaseModel):
    question: str

@app.post("/answer")
async def get_answer(req: QueryRequest):
    start_time = time.time()
    question = (req.question or "").strip()

    if not question:
        return {
            "answer": "Please provide a non-empty question.",
            "refusal": True,
            "confidence": "low",
            "sources": [],
            "latency_sec": round(time.time() - start_time, 2),
        }

    # 1) Retrieval
    results = retrieve(question, k=5)
    top_score = results[0]["score"] if results else 0.0

    logger.info("query=%r top_score=%.3f num_results=%d", question, top_score, len(results))

    # 2) Retrieval Gate (Guardrail)
    if top_score < 0.30:
        return {
            "answer": "I do not have documents to answer that question.",
            "refusal": True,
            "confidence": "low",
            "sources": [],
            "latency_sec": round(time.time() - start_time, 2),
            "retrieval": {"top_score": top_score, "k": 5},
        }

    # 3) Augment
    context_text = "\n\n".join([f"Source {r['source']}: {r['text']}" for r in results])
    user_prompt = f"Context:\n{context_text}\n\nQuestion: {question}"

    # 4) Generation with Fallback
    response = call_llm(prompts.SYSTEM_PROMPT, user_prompt)

    # 5) Attach debug metadata
    response["latency_sec"] = round(time.time() - start_time, 2)
    response["retrieval"] = {"top_score": top_score, "k": 5}
    return response

Centralized Prompt – Template: prompts.py

A small but important habit: keep prompts centralised so they’re versionable and easy to evaluate.

Example prompts.py

SYSTEM_PROMPT = """You are a RAG assistant. Use ONLY the provided Context to answer.
If the context does not contain the answer, respond with refusal=true.

Return a valid JSON object with exactly these keys:
- answer: string
- refusal: boolean
- confidence: "low" | "medium" | "high"
- sources: array of strings (source filenames you used)

Do not include any extra keys. Do not include markdown. Do not include commentary."""

How to Add Beginner-Friendly Evals

In AI systems, outputs are probabilistic. This makes testing harder than traditional software. Evals (evaluations) are a set of “golden questions” and “expected behaviours” you run repeatedly to detect regressions.

Instead of “does it output exactly this string,” you test:

• Should the app refuse when the retrieval is weak?

• When it answers, does it include sources?

• Is the behaviour stable across prompt tweaks and model changes?

Step 1: Create evals/eval_set.json

This should contain both positive and negative cases.

[
  {
    "id": "in_scope_01",
    "question": "What is a retrieval gate and why is it important?",
    "expect_refusal": false,
    "notes": "Should explain gating and relate it to hallucination prevention."
  },
  {
    "id": "out_of_scope_01",
    "question": "What is the capital of France?",
    "expect_refusal": true,
    "notes": "If the knowledge base only includes our docs, the app should refuse."
  },
  {
    "id": "edge_01",
    "question": "",
    "expect_refusal": true,
    "notes": "Empty input should not call the LLM."
  }
]

Step 2: Create evals/run_evals.py

This runner calls your API endpoint (end-to-end) and checks expected behaviours.

import json
import requests

API_URL = "http://127.0.0.1:8000/answer"

def run():
    with open("evals/eval_set.json", "r", encoding="utf-8") as f:
        cases = json.load(f)

    passed = 0
    failed = 0

    for case in cases:
        resp = requests.post(API_URL, json={"question": case["question"]}, timeout=60)
        resp.raise_for_status()
        out = resp.json()

        got_refusal = bool(out.get("refusal", False))
        expect_refusal = bool(case["expect_refusal"])

        ok = (got_refusal == expect_refusal)

        # Beginner-friendly: if it answers, sources should exist and be a list
        if not got_refusal:
            ok = ok and isinstance(out.get("sources"), list)

        if ok:
            passed += 1
            print(f"PASS {case['id']}")
        else:
            failed += 1
            print(f"FAIL {case['id']} expected_refusal={expect_refusal} got_refusal={got_refusal}")
            print("Output:", json.dumps(out, indent=2))

    print(f"\nDone. Passed={passed} Failed={failed}")
    if failed:
        raise SystemExit(1)

if __name__ == "__main__":
    run()

How to Use Evals in Practice

Run your server:

uvicorn app:app --reload

In another terminal, run evals:

python evals/run_evals.py

If an eval fails, you have a concrete signal that something changed in retrieval, gating, prompting, or provider behaviour.

What to Improve Next: Realistic Upgrades

Building a reliable RAG app is iterative. Here are realistic next steps:

• Semantic chunking: Break text based on meaning instead of character count.

• Reranking: Use a cross-encoder reranker to reorder the top-k chunks for higher precision.

• Metadata filtering: Filter results by category, date, or department to reduce false positives.

• Better citations: Store chunk IDs and show exactly which chunk(s) the answer came from.

• Observability: Add request IDs, structured logs, and traces so “what happened?” is answerable.

• Async + background indexing: Move index building to a background job and keep the API responsive.

Final Thoughts: Production-Ready Is a Set of Habits

Building an AI application that survives in the real world is about building a system that is predictable, measurable, and safe.

• Retrieval quality is measurable: Use similarity scores to gate your LLM.

• Refusal is a feature: It is better to say “I do not know” than to lie.

• Fallbacks are mandatory: Design for the moment the API goes down.

• Evals prevent regressions: Never deploy a change without running your tests.

About Me

I am Chidozie Managwu, an award-winning AI Product Architect and founder focused on helping global tech talent build real, production-ready skills. I contribute to global AI initiatives as a GAFAI Delegate and lead AI Titans Network, a community for developers learning how to ship AI products.

My work has been recognized with the Global Tech Hero award and featured on platforms like HackerNoon.

But it doesn't have to work that way. In this tutorial, you'll deploy a fully serverless RAG pipeline that processes documents, images, video, and audio, then scales to zero when nobody's using it.

Everything runs in your AWS account, your data never leaves your infrastructure, and your ongoing monthly cost for a modest knowledge base will be closer to 2-3 USD than 300 USD.

We'll use RAGStack-Lambda, an open-source project I built on AWS. By the end, you'll have a deployed pipeline with a dashboard, an AI chat interface with source citations, a drop-in web component you can embed in any app, and an MCP server you can use to feed your assistant context.

Here's what we'll cover:

• What This Actually Costs

• What You're Building

• Prerequisites

• Deploying from AWS Marketplace

• Deploying from Source

• Uploading Your First Documents

• Chatting With Your Knowledge Base

• Embedding the Web Component in Your App

• Using the MCP Server

• What You Can Build From Here

• Wrapping Up

What This Actually Costs

Before we build anything, let's talk money, because the cost story is the whole point.

RAG pipelines have two cost phases: ingestion (processing your documents once) and operation (querying them over time).

Most platforms charge you a flat monthly rate regardless of which phase you're in. A serverless architecture flips that: ingestion costs something, and then everything scales to zero.

Ingestion: The One-Time Hit

When you upload documents, several things happen: text extraction (OCR for PDFs and images), embedding generation, metadata extraction, and storage. Here's what that actually costs per service:

Textract (OCR): This is the most expensive part of ingestion, and it only applies to scanned PDFs and images that need text extraction. Plain text, HTML, CSV, and other text-based formats skip this entirely.

Textract charges about 1.50 USD per 1,000 pages for standard text detection. If you're uploading 500 pages of scanned PDFs, that's about 0.75 USD. A heavy initial load of several thousand scanned pages might run 5-10 USD. But once your documents are processed, you never pay this again unless you add new ones.

Bedrock Embeddings (Nova Multimodal): This is where your content gets converted into vectors for semantic search. The pricing is almost comically cheap:

• Text: 0.00002 USD per 1,000 input tokens

• Images: 0.00115 USD per image

• Video/Audio: 0.00200 USD per minute

To put that in perspective: if you have 1,500 text documents averaging 2,500 tokens each after chunking, your total embedding cost is about 0.08 USD. A knowledge base with 500 images runs 0.58 USD. Even a mixed corpus of text, images, and a few hours of video stays well under 2 USD for the entire embedding pass. This is a one-time cost – you only re-embed if you add or update documents.

Bedrock LLM (Metadata Extraction): RAGStack uses an LLM to analyze each document and extract structured metadata automatically. This is a few inference calls per document using Nova Lite or a similar model. At 0.06 USD/0.24 USD per million input/output tokens, processing 1,500 documents costs well under 1 USD.

S3 Vectors (Storage): Storing your embeddings. At 0.06 USD per GB/month, a knowledge base of 1,500 documents with 1,024-dimension vectors takes up a trivially small amount of space. We're talking pennies per month.

S3 (Document Storage): Your source documents in standard S3. Even cheaper, 0.023 USD per GB/month.

DynamoDB: Stores document metadata and processing state. The on-demand pricing model means you pay per request during ingestion, then essentially nothing at rest. A few cents for the initial load.

To put real numbers on it: if you upload 200 text documents (PDFs, HTML, markdown), your total ingestion cost is likely under 1 USD. If you upload 1,000 scanned PDFs that need OCR, you might see 5-8 USD as a one-time hit. That 7-10 USD figure you might see referenced? That's the upper end for a heavy initial load with lots of OCR work.

Operation: Where Scale-to-Zero Shines

Once your documents are ingested, the pipeline is waiting. Not running. Waiting. Here's what each query costs:

Lambda: Invocations are billed per request and duration. The free tier covers 1 million requests/month. For a personal or small-team knowledge base, you may never leave the free tier.

S3 Vectors (Queries): 2.50 USD per million query API calls, plus a per-TB data processing charge. For a small index queried a few hundred times a month, this rounds to effectively zero.

Bedrock (Chat Inference): This is your main operating cost. Each chat response requires an LLM call. Using Nova Lite at 0.06 USD per million input tokens and 0.24 USD per million output tokens, a typical RAG query (retrieval context + user question + response) might cost 0.001-0.003 USD per query. A hundred queries a month is 0.10-0.30 USD.

Step Functions: Orchestrates the document processing pipeline. Standard workflows charge 0.025 USD per 1,000 state transitions. Minimal during operation since it's only active during ingestion.

Cognito: User authentication. Free for the first 10,000 monthly active users.

CloudFront: Serves the dashboard UI. Free tier covers 1 TB of data transfer per month.

API Gateway: Handles GraphQL API requests. Free tier covers 1 million API calls per month.

Add it all up for a knowledge base with 500 documents getting a few hundred queries per month, and your monthly operating cost is somewhere between 0.50 USD and 3.00 USD. Most of that is the LLM inference for chat responses.

The Comparison That Matters

Here's the same pipeline on a traditional always-on stack:

The LLM inference cost per query is roughly the same everywhere – that's Bedrock's on-demand pricing regardless of your architecture. The difference is everything else. Traditional stacks pay a floor cost whether anyone's using them or not. A serverless stack pays for what it uses, and idle costs essentially nothing.

What About Transcribe?

If you're uploading video or audio, AWS Transcribe adds cost for speech-to-text conversion. Standard transcription runs about 0.024 USD per minute of audio. A 10-minute video costs 0.24 USD to transcribe. This is a one-time ingestion cost, once transcribed and embedded, the resulting text chunks are queried like any other document.

What You're Building

By the end of this tutorial, you'll have a deployed pipeline that does the following:

1. You upload a document (PDF, image, video, audio, HTML, CSV, the full list is extensive) through a web dashboard.

2. The pipeline detects the file type and routes it to the right processor. Scanned PDFs go through OCR via Textract. Video and audio go through Transcribe for speech-to-text, split into 30-second searchable chunks with speaker identification. Images get visual embeddings and any caption text you provide.

3. An LLM analyzes each document and extracts structured metadata, topic, document type, date range, people mentioned, whatever's relevant. This happens automatically.

4. Everything gets embedded using Amazon Nova Multimodal Embeddings and stored in a Bedrock Knowledge Base backed by S3 Vectors.

5. You (or your users) ask questions through an AI chat interface. The pipeline retrieves relevant documents, passes them as context to a Bedrock LLM, and returns an answer with collapsible source citations, including timestamp links for video and audio that jump to the exact position.

All of this runs in your AWS account. No external control plane, no third-party services beyond AWS itself.

The Architecture

A few things to note about this architecture:

Step Functions orchestrate everything. When a document is uploaded, a state machine manages the entire processing flow, detecting the file type, routing to the right processor, waiting for async operations like Transcribe jobs, then triggering embedding and metadata extraction.

This is what makes the pipeline reliable without a running server. If a step fails, it retries. You can see exactly where every document is in the processing pipeline.

Lambda does the compute. Every processing step is a Lambda function. They spin up when needed, run for a few seconds to a few minutes, and shut down. There's no EC2 instance idling at 3 AM.

S3 Vectors is the vector store. Your embeddings live in S3's purpose-built vector storage rather than in a dedicated vector database like Pinecone or OpenSearch.

This is what makes the "scale to zero" cost possible: you're paying object storage rates for vector data instead of keeping a database cluster warm. It also means your vectors are sitting in your own S3 bucket, not in a third-party managed service that holds your data on their terms.

Cognito handles auth. The dashboard and API are protected with Cognito user pools. When you deploy, you get a temporary password via email. The web component uses IAM-based authentication, and server-side integrations use API key auth.

CloudFront serves the UI. The dashboard is a static React app served through CloudFront, so there's no web server to maintain.

Two Ways to Deploy

You have two deployment paths depending on what you want:

AWS Marketplace (the fast path), click deploy, fill in two fields (stack name and email), and wait about 10 minutes. No local tooling required. This is the path we'll walk through first.

From Source (the developer path), Clone the repo, run publish.py, and deploy via SAM CLI. This is the path for when you want to customize the processing pipeline, modify the UI, or contribute to the project. We'll cover this after the Marketplace walkthrough.

Both paths produce the same stack. The Marketplace version just wraps the CloudFormation template in a one-click deployment.

Prerequisites

Before you deploy, you'll need:

• An AWS account with permissions to create CloudFormation stacks, Lambda functions, S3 buckets, DynamoDB tables, and Cognito user pools. If you're using an admin account, you're covered.

• Bedrock model access: RAGStack defaults to us-east-1 because that's where Nova Multimodal Embeddings is available. Amazon's own models (including Nova) are available by default in Bedrock, no manual enablement required. Just make sure your IAM role has the necessary bedrock:InvokeModel permissions.

• For the Marketplace path: just a web browser.

• For the source path: Python 3.13+, Node.js 24+, AWS CLI and SAM CLI configured, and Docker (for building Lambda layers).

Deploying from AWS Marketplace

This is the fastest path – no local tools, no CLI, no Docker. You'll launch a CloudFormation stack and have a working pipeline in about 10 minutes.

Step 1: Launch the Stack

Click the direct deploy link to open CloudFormation's "Quick create stack" page with the template pre-loaded.

Step 2: Fill In Two Fields

The page has a lot of options, but you only need two:

• Stack name: Must be lowercase. This becomes the prefix for all your AWS resources (for example, my-docs, team-kb, project-notes). Keep it short.

• Admin Email: Under Required Settings. Cognito will send your temporary login credentials here. Use an email you can access right now.

Everything else – Build Options, Advanced Settings, OCR Backend, model selections – can stay at the defaults. They're there for customization later, but the defaults work out of the box.

Step 3: Deploy

Scroll to the bottom, check the three acknowledgment boxes under "Capabilities and transforms," and click Create stack.

Deployment takes roughly 10 minutes. You can watch the progress in the CloudFormation Events tab if you're curious, but there's nothing to do until the stack status flips to CREATE_COMPLETE.

Step 4: Log In

Once the stack finishes, check your email. Cognito sends you the dashboard URL and a temporary password. Log in, set a new password, and you're looking at an empty dashboard ready for documents.

Deploying from Source

If you want to customize the pipeline, modify the UI, or contribute to the project, deploy from source instead.

Step 1: Clone and Set Up

git clone https://github.com/HatmanStack/RAGStack-Lambda.git
cd RAGStack-Lambda

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt

Step 2: Deploy

The publish.py script handles everything: building the frontend, packaging Lambda functions, and deploying via SAM CLI.

python publish.py \
  --project-name my-docs \
  --admin-email admin@example.com

This defaults to us-east-1 for Nova Multimodal Embeddings. The script will build the React dashboard, build the web component, package all Lambda layers with Docker, and deploy the CloudFormation stack through SAM.

First deploy takes longer (15-20 minutes) because it's building everything from scratch. Subsequent deploys are faster since SAM caches unchanged resources.

If you only want to iterate on the backend and skip UI builds:

# Skip dashboard build (still builds web component)
python publish.py --project-name my-docs --admin-email admin@example.com --skip-ui

# Skip ALL UI builds
python publish.py --project-name my-docs --admin-email admin@example.com --skip-ui-all

Once it finishes, you'll get the same Cognito email and dashboard URL as the Marketplace path.

Uploading Your First Documents

The dashboard has tabs for different content types. We'll start with the Documents tab since that's the most common use case.

Documents

Click the Documents tab and upload a file. RAGStack accepts a wide range of formats: PDF, DOCX, XLSX, HTML, CSV, JSON, XML, EML, EPUB, TXT, and Markdown. Drag and drop or use the file picker.

Once uploaded, the document enters the processing pipeline. You'll see the status update in real time:

1. UPLOADED: File received and stored in S3.

2. PROCESSING: Step Functions has picked it up and routed it to the right processor. Text-based files (HTML, CSV, Markdown) go through direct extraction. Scanned PDFs and images go through Textract OCR. The LLM analyzes the content and extracts structured metadata, topic, document type, people mentioned, date ranges, whatever's relevant to the content.

3. INDEXED: Embeddings generated, vectors stored, document is searchable.

Text documents typically process in 1-5 minutes. OCR-heavy documents (scanned PDFs, images with text) can take 2-15 minutes depending on page count.

Images

The Images tab works differently. Upload a JPG, PNG, GIF, or WebP and you can add a caption. Both the visual content and caption text get embedded using Nova Multimodal Embeddings, so you can search by what's in the image or by your description of it.

This is where multimodal embeddings earn their keep. A traditional text-only RAG pipeline would need you to describe every image manually. Here, the image itself becomes searchable, and since everything stays in your AWS account, you're not sending personal photos or sensitive visual content to an external service to get there.

What About Video and Audio?

Upload video or audio files and RAGStack routes them through AWS Transcribe for speech-to-text conversion. The transcript gets split into 30-second chunks with speaker identification, then embedded like any other document. When chat results reference a video source, you get timestamp links that jump to the exact position in the recording.

Web Scraping

The Scrape tab lets you pull websites directly into your knowledge base. Enter a URL and RAGStack crawls the page, extracts the content, and processes it through the same pipeline as uploaded documents, metadata extraction, embedding, indexing.

This is useful for building a knowledge base from existing web content without manually saving and uploading pages. Documentation sites, blog archives, reference material, anything publicly accessible.

Chatting With Your Knowledge Base

This is the payoff. Go to the Chat tab, type a question, and RAGStack retrieves relevant documents from your knowledge base, passes them as context to a Bedrock LLM, and returns an answer with source citations.

The citations are collapsible, so click to expand and see which documents informed the answer, with the option to download the source file. For video and audio sources, you get clickable timestamps that jump to the relevant moment.

Metadata Filtering

If you've uploaded enough documents to have meaningful metadata categories, the chat interface lets you filter search results by metadata before querying. RAGStack auto-discovers the metadata structure from your documents, so you don't configure this manually, it just appears as your knowledge base grows.

This is useful when you have a large mixed corpus. Instead of hoping the vector search picks the right context from thousands of documents, you can narrow it down: "only search documents about project X" or "only search content from Q4 2024."

Embedding the Web Component in Your App

The dashboard is useful for managing your knowledge base, but the real power is embedding RAGStack's chat in your own application. The web component works with any framework, React, Vue, Angular, Svelte, plain HTML.

Load the script once from your CloudFront distribution:

<script src="https://your-cloudfront-url/ragstack-chat.js"></script>

Then drop the component wherever you want a chat interface:

<ragstack-chat
  conversation-id="my-app"
  header-text="Ask About Documents"
></ragstack-chat>

That's it. The component handles authentication (via IAM), manages conversation state, and renders source citations, all self-contained. Your CloudFront URL is in the stack outputs.

For server-side integrations that don't need a UI, the GraphQL API is available with API key authentication. You can find your endpoint and API key in the dashboard under Settings.

Using the MCP Server

RAGStack includes an MCP server that connects your knowledge base to AI assistants like Claude Desktop, Cursor, VS Code, and Amazon Q CLI. Instead of switching to the dashboard to search your documents, you ask your assistant directly.

Install it:

pip install ragstack-mcp

Then add it to your AI assistant's MCP configuration:

{
  "ragstack": {
    "command": "uvx",
    "args": ["ragstack-mcp"],
    "env": {
      "RAGSTACK_GRAPHQL_ENDPOINT": "YOUR_ENDPOINT",
      "RAGSTACK_API_KEY": "YOUR_API_KEY"
    }
  }
}

Your endpoint and API key are in the dashboard under Settings. Once configured, type @ragstack in your assistant's chat to invoke the MCP server, then ask things like "search my knowledge base for authentication docs" and it queries RAGStack directly.

See the MCP Server docs for the full list of available tools and setup details.

What You Can Build From Here

You've got a deployed RAG pipeline that costs almost nothing to run and handles text, images, video, and audio. A few directions you might take it:

A searchable personal archive. Every conference talk you've saved, every PDF textbook, every tutorial video that's sitting in a folder somewhere. Upload it all, and now you have one search interface across years of accumulated material. The multimodal embeddings mean your screenshots and diagrams are searchable too, not just the text.

I built a family archive app this way, scanned letters, old photos, home videos, with RAGStack deployed as a nested CloudFormation stack so the whole family can search across decades of memories using the chat widget.

A second brain for a client project. Scrape the client's existing docs, upload the SOW and meeting notes, drop in the codebase documentation. Now you've got a searchable knowledge base scoped to that engagement. Spin it up at the start, tear it down when the contract ends. At these costs, it's disposable infrastructure.

AI chat over a niche dataset. Recipe collections, legal filings, research papers, local government meeting minutes, any corpus that's too specialized for general-purpose LLMs to know well. The web component means you can ship it as a standalone tool without building a frontend from scratch.

RAG for your MCP workflow. If you're already using Claude Desktop or Cursor, the MCP server turns your knowledge base into another tool your assistant can reach for. Upload your team's runbooks and architecture docs, and now @ragstack in your editor gives you instant context without tab-switching.

Wrapping Up

The serverless RAG pipeline you just deployed handles document processing, multimodal embeddings, metadata extraction, and AI chat with source citations, all scaling to zero when idle, all running in your AWS account. Your documents, your vectors, your infrastructure. The traditional approach to this stack costs 120-500 USD/month in baseline infrastructure. This one costs pocket change.

The full source is at github.com/HatmanStack/RAGStack-Lambda. File issues, open PRs, or just poke around the architecture. If you want to go deeper on the technical tradeoffs, particularly how filtered vector search behaves on cost-optimized backends like S3 Vectors, that's a story for the next post.

By the end of this guide, you'll have a fully functional application that can:

• Upload and process PDF, DOCX, and TXT files

• Store documents in Supabase Storage

• Generate embeddings using OpenAI

• Perform semantic search across document chunks

• Provide AI-generated answers based on document content

• View and manage uploaded documents

This is a production-ready solution that you can deploy and use immediately.

Table of Contents

• What You'll Learn

• Prerequisites

• Understanding the Technologies

• Project Overview

• Step 1: Create Your Next.js Project

• Step 2: Install Required Dependencies

• Step 3: Set Up Your Supabase Project

• Step 4: Configure Environment Variables

• Step 5: Create the Upload API Route

• Step 6: Create the RAG Search API Route

• Step 7: Create the Documents API Route

• Step 8: Create the Upload Modal Component

• Step 9: Create the PDF Viewer Modal Component

• Step 10: Create the Navigation Component

• Step 11: Create the Home Page (Search Interface)

• Step 12: Create the Documents Page

• Step 13: Test Your Application

• Step 14: Deploy Your Application

• How RAG Search Works

• Troubleshooting Common Issues

• Next Steps

• Conclusion

What You'll Learn

In this handbook, you'll learn how to:

• Set up a Next.js application with TypeScript

• Configure Supabase for database and file storage

• Integrate OpenAI embeddings and chat completions

• Implement document text extraction and chunking

• Build a vector search system using PostgreSQL

• Create a modern UI with React components

• Handle file uploads and storage

• Implement RAG (Retrieval-Augmented Generation) search

Prerequisites

Before you begin, make sure you have:

• Node.js 18 or higher installed on your computer

• A Supabase account (free tier works fine)

• An OpenAI API key

• Basic knowledge of React and TypeScript

• Familiarity with Next.js (helpful but not required)

Understanding the Technologies

Before we dive into building the application, you should understand the key technologies and concepts you'll be working with:

What is RAG (Retrieval-Augmented Generation)?

RAG is an AI pattern that combines information retrieval with text generation. Instead of relying solely on an AI model's training data, RAG retrieves relevant information from your own documents. It then uses that information as context to generate accurate, up-to-date answers. This approach gives you:

• Accuracy: Answers are based on your actual documents, not just the AI's training data

• Transparency: You can see which document sections were used to generate the answer

• Efficiency: Only relevant document chunks are used, reducing token costs

What are Embeddings and Vector Database?

Embeddings are numerical representations of text that capture semantic meaning. When you convert text to an embedding, similar meanings are represented by similar numbers. For example, "dog" and "puppy" would have similar embeddings. Meanwhile, "dog" and "airplane" would have very different ones.

OpenAI's embedding models convert text into vectors. These are arrays of numbers that can be compared mathematically. This allows you to find documents that are semantically similar to a search query. You can find matches even if they don't contain the exact same words.

A vector database is a specialized database designed to store and search through embeddings efficiently. Instead of searching for exact text matches, vector databases use mathematical operations. They use operations like cosine similarity to find the most semantically similar content.

In this tutorial, you'll use Supabase's PostgreSQL database with the pgvector extension. This extension adds vector storage and similarity search capabilities to PostgreSQL. This lets you store embeddings alongside your regular database data. You can also perform fast similarity searches.

What is Text Chunking?

Text chunking is the process of breaking large documents into smaller, manageable pieces. This is necessary for several reasons.

First, AI models have token limits. These are maximum input sizes. Second, smaller chunks allow for more precise retrieval. Third, overlapping chunks ensure context isn't lost at boundaries.

You'll use LangChain's RecursiveCharacterTextSplitter. This tool intelligently splits text while trying to preserve sentence and paragraph boundaries.

What is Supabase?

Supabase is an open-source Firebase alternative. It provides several key features.

You get a PostgreSQL database, which is a powerful, open-source relational database. You also get storage, which is file storage similar to AWS S3. There are real-time features that provide real-time subscriptions to database changes. Finally, there's built-in user authentication.

For this project, you'll use Supabase's database to store document chunks and embeddings. You'll also use Supabase Storage to store the original uploaded files.

What is Tailwind CSS?

Tailwind CSS is a utility-first CSS framework that lets you style your application by applying pre-built utility classes directly in your HTML/JSX. Instead of writing custom CSS, you use classes like bg-blue-600, text-white, and rounded-lg to style elements.

You'll use Tailwind CSS in this project because it speeds up development by providing ready-made styling utilities. It also ensures consistent design across the application. Plus, it makes it easy to create responsive, modern UIs. Finally, it works seamlessly with Next.js.

Now that you understand the core concepts and tools we’ll be using, let's start building the application.

Project Overview

Your RAG search application will consist of:

1. Frontend: Next.js application with React components for uploading documents and searching

2. Backend API Routes: Next.js API routes for handling uploads, searches, and document management

3. Database: Supabase PostgreSQL with vector extension for storing embeddings

4. Storage: Supabase Storage for storing original files

5. AI Integration: OpenAI for generating embeddings and chat completions

The application will have two main pages:

• Search Page: Where users can ask questions about their uploaded documents and get AI-generated answers

• Documents Page: Where users can view all uploaded documents, upload new ones, preview files, and manage their document library

Let's start building!

If you ever get stuck on the source code, you can view it on GitHub here:

https://github.com/mayur9210/rag-search-app

Step 1: Create Your Next.js Project

Start by creating a new Next.js project with TypeScript. Open your terminal and run:

npx create-next-app@latest rag-search-app --typescript --tailwind --app

When prompted, choose the following options:

• TypeScript: Yes

• ESLint: Yes

• Tailwind CSS: Yes

• App Router: Yes (default)

• Customize import alias: No

Navigate into your project directory:

cd rag-search-app

Now that your project is set up, you'll need to install the additional packages required for document processing, AI integration, and database operations.

Step 2: Install Required Dependencies

You'll need several packages for this project. You can install them using npm:

npm install @supabase/supabase-js @langchain/openai @langchain/textsplitters langchain openai mammoth pdf2json

Here's what each package does:

• @supabase/supabase-js: Client library for interacting with Supabase (database and storage)

• @langchain/openai: LangChain integration for OpenAI (helps with text processing)

• @langchain/textsplitters: Text splitting utilities for chunking documents into smaller pieces

• langchain: Core LangChain library (provides AI workflow tools)

• openai: Official OpenAI SDK (for generating embeddings and chat completions)

• mammoth: Converts DOCX files to plain text

• pdf2json: Extracts text from PDF files

Install the TypeScript types for pdf2json:

npm install --save-dev @types/pdf-parse

With all dependencies installed, you're ready to set up your Supabase project, which will handle your database and file storage needs.

Step 3: Set Up Your Supabase Project

Create a Supabase Project

First, you’ll need to create a new Supabase project, which you can do by following these steps:

1. Go to supabase.com and sign in or create an account

2. Click "New Project"

3. Fill in your project details:

   • Name: rag-search-app (or any name you prefer)

   • Database Password: Choose a strong password (save this – you'll need it)

   • Region: Select the region closest to you

4. Click "Create new project" and wait for it to be ready (this takes a few minutes)

Get Your Supabase Credentials

Once your project is ready, go to Settings and then API.

Copy the following values:

• Project URL (this is your NEXT_PUBLIC_SUPABASE_URL)

• anon public key (this is your NEXT_PUBLIC_SUPABASE_PUBLISHABLE_DEFAULT_KEY)

• service_role key (this is your SUPABASE_SERVICE_ROLE_KEY)

Important: Keep your service role key secret. Never expose it in client-side code. It bypasses Row-Level Security (RLS) policies, which is necessary for server-side file uploads but should never be used in browser code.

Set Up the Database Schema

Now you'll set up the database structure to store your documents and embeddings. Go to SQL Editor in your Supabase dashboard and run the following SQL:

-- Enable the vector extension for embeddings
-- This extension allows PostgreSQL to store and search vector data efficiently
CREATE EXTENSION IF NOT EXISTS vector;

-- Create the documents table
-- This table stores document chunks, their metadata, and embeddings
CREATE TABLE documents (
  id BIGSERIAL PRIMARY KEY,
  content TEXT NOT NULL,
  metadata JSONB,
  embedding vector(1536)  -- OpenAI's text-embedding-3-small produces 1536-dimensional vectors
  file_path text null,
  file_url text null,
);

-- Create an index on the embedding column for faster similarity search
-- The ivfflat index speeds up vector similarity queries
CREATE INDEX ON documents USING ivfflat (embedding vector_cosine_ops);

-- Create a function for matching documents based on similarity
-- This function finds the most similar document chunks to a query embedding
CREATE OR REPLACE FUNCTION match_documents(
  query_embedding vector(1536),
  match_threshold float,
  match_count int
)
RETURNS TABLE (
  id bigint,
  content text,
  metadata jsonb,
  similarity float
)
LANGUAGE plpgsql
AS $$
BEGIN
  RETURN QUERY
  SELECT
    documents.id,
    documents.content,
    documents.metadata,
    1 - (documents.embedding <=> query_embedding) AS similarity
  FROM documents
  WHERE 1 - (documents.embedding <=> query_embedding) > match_threshold
  ORDER BY documents.embedding <=> query_embedding
  LIMIT match_count;
END;
$$;

This SQL does the following:

• Enables the vector extension: This adds vector storage and similarity search capabilities to PostgreSQL

• Creates the documents table: Stores document chunks, metadata (file name, type, and so on), and their embeddings

• Creates an index: Speeds up similarity searches on the embedding column

• Creates a match function: Finds the most similar document chunks to a query embedding using cosine similarity

The <=> operator calculates cosine distance between vectors. A smaller distance means more similar content.

Set Up Supabase Storage

You’ll need a storage bucket to store uploaded files. This is separate from the database and holds the original PDF, DOCX, and TXT files.

To set up your storage bucket:

1. Go to Storage in your Supabase dashboard

2. Click New bucket

3. Name it documents

4. Set it to Public (this allows file downloads)

5. Click Create bucket

If you prefer a private bucket, you can use the service role key for server-side operations, which bypasses Row-Level Security policies. For this tutorial, a public bucket is simpler and works well.

Now that your Supabase project is configured, you'll set up your environment variables to connect your Next.js application to Supabase and OpenAI.

Step 4: Configure Environment Variables

Create a .env.local file in your project root:

NEXT_PUBLIC_SUPABASE_URL=your_supabase_project_url
NEXT_PUBLIC_SUPABASE_PUBLISHABLE_DEFAULT_KEY=your_supabase_anon_key
SUPABASE_SERVICE_ROLE_KEY=your_supabase_service_role_key
OPENAI_API_KEY=your_openai_api_key

Replace the placeholder values with your actual credentials:

• Get Supabase values from Settings → API in your Supabase dashboard

• Get your OpenAI API key from platform.openai.com/api-keys

Security Note: Never commit .env.local to version control. It's already in .gitignore by default, but double-check to ensure your secrets stay secure.

With your environment configured, you're ready to start building the API routes that will handle file uploads, searches, and document management.

Step 5: Create the Upload API Route

Now you'll create the API route that handles file uploads. This route will process uploaded files, extract their text, split them into chunks, generate embeddings, and store everything in your database and storage.

Create src/app/api/upload/route.ts:

import { createClient } from '@supabase/supabase-js';
import OpenAI from 'openai';
import { NextResponse } from 'next/server';
import { RecursiveCharacterTextSplitter } from '@langchain/textsplitters';
import mammoth from 'mammoth';

const url = process.env.NEXT_PUBLIC_SUPABASE_URL!;
const anonKey = process.env.NEXT_PUBLIC_SUPABASE_PUBLISHABLE_DEFAULT_KEY!;
const serviceKey = process.env.SUPABASE_SERVICE_ROLE_KEY;
const supabaseStorage = createClient(url, serviceKey || anonKey);
const supabase = createClient(url, anonKey);
const openai = new OpenAI();

function safeDecodeURIComponent(str: string): string {
  try { 
    return decodeURIComponent(str); 
  } catch { 
    try { 
      return decodeURIComponent(str.replace(/%/g, '%25')); 
    } catch { 
      return str; 
    } 
  }
}

async function extractTextFromFile(file: File): Promise<string> {
  const buffer = Buffer.from(await file.arrayBuffer());
  const fileName = file.name.toLowerCase();

  if (fileName.endsWith('.pdf')) {
    const PDFParser = (await import('pdf2json')).default;
    return new Promise((resolve, reject) => {
      const pdfParser = new (PDFParser as any)(null, true);
      pdfParser.on('pdfParser_dataError', (err: any) => 
        reject(new Error(`PDF parsing error: ${err.parserError}`))
      );
      pdfParser.on('pdfParser_dataReady', (pdfData: any) => {
        try {
          let fullText = '';
          pdfData.Pages?.forEach((page: any) => 
            page.Texts?.forEach((text: any) => 
              text.R?.forEach((r: any) => 
                r.T && (fullText += safeDecodeURIComponent(r.T) + ' ')
              )
            )
          );
          resolve(fullText.trim());
        } catch (error: any) {
          reject(new Error(`Error extracting text: ${error.message}`));
        }
      });
      pdfParser.parseBuffer(buffer);
    });
  } else if (fileName.endsWith('.docx')) {
    const result = await mammoth.extractRawText({ buffer });
    return result.value;
  } else if (fileName.endsWith('.txt')) {
    return buffer.toString('utf-8');
  } else {
    throw new Error('Unsupported file type. Please upload PDF, DOCX, or TXT files.');
  }
}

export async function POST(req: Request) {
  try {
    const file = (await req.formData()).get('file') as File;
    if (!file) {
      return NextResponse.json({ error: 'No file provided' }, { status: 400 });
    }

    const documentId = crypto.randomUUID();
    const uploadDate = new Date().toISOString();
    const filePath = `${documentId}.${file.name.split('.').pop() || 'bin'}`;

    // Upload file to Supabase Storage
    const fileBuffer = Buffer.from(await file.arrayBuffer());
    const { error: storageError } = await supabaseStorage.storage
      .from('documents')
      .upload(filePath, fileBuffer, {
        contentType: file.type || 'application/octet-stream',
        upsert: false,
      });

    if (storageError) {
      const msg = storageError.message || 'Unknown storage error';
      if (msg.includes('row-level security') || msg.includes('RLS')) {
        return NextResponse.json({ 
          success: false, 
          error: `Storage RLS error: ${msg}. Ensure SUPABASE_SERVICE_ROLE_KEY is set.` 
        }, { status: 500 });
      }
      return NextResponse.json({ 
        success: false, 
        error: `Failed to store file: ${msg}` 
      }, { status: 500 });
    }

    // Get public URL for the file
    const { data: urlData } = supabaseStorage.storage
      .from('documents')
      .getPublicUrl(filePath);

    // Extract text from file
    const text = await extractTextFromFile(file);
    if (!text || text.trim().length === 0) {
      return NextResponse.json({ 
        error: 'Could not extract text from file' 
      }, { status: 400 });
    }

    // Split text into chunks
    // Chunk size of 800 characters with 100-character overlap ensures
    // we don't lose context at chunk boundaries
    const textSplitter = new RecursiveCharacterTextSplitter({
      chunkSize: 800,
      chunkOverlap: 100,
    });
    const chunks = await textSplitter.splitText(text);

    // Process each chunk: generate embedding and store in database
    for (let i = 0; i < chunks.length; i++) {
      const chunk = chunks[i];

      // Generate embedding using OpenAI
      // This converts the text chunk into a 1536-dimensional vector
      const emb = await openai.embeddings.create({
        model: 'text-embedding-3-small',
        input: chunk,
      });

      // Store chunk with embedding in database
      const { error } = await supabase.from('documents').insert({
        content: chunk,
        metadata: { 
          source: file.name,
          document_id: documentId,
          file_name: file.name,
          file_type: file.type || file.name.split('.').pop(),
          file_size: file.size,
          upload_date: uploadDate,
          chunk_index: i,
          total_chunks: chunks.length,
          file_path: filePath,
          file_url: urlData.publicUrl,
        },
        embedding: JSON.stringify(emb.data[0].embedding),
      });

      if (error) {
        return NextResponse.json({ 
          success: false, 
          error: error.message 
        }, { status: 500 });
      }
    }

    return NextResponse.json({ 
      success: true, 
      documentId, 
      fileName: file.name, 
      chunks: chunks.length, 
      textLength: text.length, 
      fileUrl: urlData.publicUrl 
    });
  } catch (error: any) {
    return NextResponse.json({ 
      success: false, 
      error: error.message || 'Failed to process file' 
    }, { status: 500 });
  }
}

This route handles the complete upload workflow:

1. Receives the file from the client via FormData

2. Generates a unique document ID using crypto.randomUUID()

3. Uploads the file to Supabase Storage for safekeeping

4. Extracts text based on file type (PDF, DOCX, or TXT)

5. Splits the text into chunks of 800 characters with 100-character overlap

6. Generates embeddings for each chunk using OpenAI's embedding model

7. Stores each chunk with its embedding and metadata in the database

The overlap between chunks ensures that if a sentence or concept spans a chunk boundary, it won't be lost. Now that you can upload and process documents, let's create the search functionality.

Step 6: Create the RAG Search API Route

This route implements the core RAG functionality: it takes a user's query, finds the most relevant document chunks, and uses them to generate an accurate answer.

Create src/app/api/search/route.ts:

import { createClient } from '@supabase/supabase-js';
import OpenAI from 'openai';
import { NextResponse } from 'next/server';

const supabase = createClient(
  process.env.NEXT_PUBLIC_SUPABASE_URL!,
  process.env.NEXT_PUBLIC_SUPABASE_PUBLISHABLE_DEFAULT_KEY!
);
const openai = new OpenAI();

export async function POST(req: Request) {
  try {
    const { query } = await req.json();

    // Generate embedding for the user's query
    // This converts the search query into the same vector space as document chunks
    const emb = await openai.embeddings.create({ 
      model: 'text-embedding-3-small', 
      input: query 
    });

    // Find similar documents using vector similarity search
    // The match_documents function finds the 5 most similar chunks
    const { data: results, error } = await supabase.rpc('match_documents', {
      query_embedding: JSON.stringify(emb.data[0].embedding),
      match_threshold: 0.0,  // Accept any similarity (you can increase this for stricter matching)
      match_count: 5,        // Return top 5 most similar chunks
    });

    if (error) {
      return NextResponse.json({ error: error.message }, { status: 500 });
    }

    // Combine retrieved chunks into context
    // These chunks will be used as context for the AI to generate an answer
    const context = results?.map((r: any) => r.content).join('\n---\n') || '';

    // Generate answer using OpenAI with retrieved context
    // This is the "Generation" part of RAG
    const completion = await openai.chat.completions.create({
      model: 'gpt-4o-mini',
      messages: [
        { 
          role: 'system', 
          content: 'You are a helpful assistant. Use the provided context to answer questions. If the answer is not in the context, say you do not know.' 
        },
        { 
          role: 'user', 
          content: `Context: ${context}\n\nQuestion: ${query}` 
        }
      ],
    });

    return NextResponse.json({ 
      answer: completion.choices[0].message.content, 
      sources: results 
    });
  } catch (error: any) {
    return NextResponse.json({ error: error.message }, { status: 500 });
  }
}

This route implements the RAG pattern. Here's how the complete RAG workflow works:

1. Converts the query to an embedding: The user's question is transformed into the same vector space as your document chunks. This uses the same embedding model (text-embedding-3-small) that processed the documents, ensuring they're in the same "vector space."

2. Searches for similar chunks: Uses the match_documents function to find the 5 most semantically similar document chunks. This uses cosine similarity on the embeddings. Cosine similarity measures the angle between vectors - smaller angles mean more similar content, even if the exact words differ.

3. Uses chunks as context: The retrieved chunks are passed to GPT-4o-mini as context. These chunks contain the most relevant information from your documents.

4. Generates an answer: The AI model generates an answer based on the provided context. The system prompt instructs the AI to only answer based on the provided context, ensuring accuracy and preventing hallucinations.

5. Returns results: Both the answer and source chunks are returned so users can verify the information.

This RAG approach gives you several benefits. First, you get accuracy because answers are based on your actual documents, not just the AI's training data. Second, you get transparency because you can see which document chunks were used to generate each answer. Third, you get efficiency because only relevant chunks are used, which reduces token usage and costs. Finally, you get up-to-date information because you can update your knowledge base by uploading new documents without retraining the AI.

Now let's create the API route for managing documents.

Step 7: Create the Documents API Route

This route handles listing, viewing, downloading, and deleting documents. It serves multiple purposes depending on the query parameters.

Create src/app/api/documents/route.ts:

import { createClient } from '@supabase/supabase-js';
import { NextResponse } from 'next/server';

const url = process.env.NEXT_PUBLIC_SUPABASE_URL!;
const anonKey = process.env.NEXT_PUBLIC_SUPABASE_PUBLISHABLE_DEFAULT_KEY!;
const serviceKey = process.env.SUPABASE_SERVICE_ROLE_KEY || anonKey;
const supabase = createClient(url, anonKey);
const supabaseStorage = createClient(url, serviceKey);

export async function GET(req: Request) {
  try {
    const reqUrl = new URL(req.url);
    const id = reqUrl.searchParams.get('id');
    const file = reqUrl.searchParams.get('file') === 'true';
    const view = reqUrl.searchParams.get('view') === 'true';

    // Handle file download/view
    if (id && file) {
      const { data: documents } = await supabase
        .from('documents')
        .select('metadata')
        .eq('metadata->>document_id', id)
        .limit(1);

      if (!documents || documents.length === 0) {
        return NextResponse.json({ error: 'Document not found' }, { status: 404 });
      }

      const meta = documents[0].metadata;
      const fileName = meta?.file_name || 'document';
      const fileType = meta?.file_type || 'application/octet-stream';
      const filePath = meta?.file_path || `${id}.${fileName.split('.').pop() || 'pdf'}`;

      const { data: fileData, error: downloadError } = await supabaseStorage.storage
        .from('documents')
        .download(filePath);

      if (downloadError || !fileData) {
        return NextResponse.json({ 
          error: downloadError?.message || 'File not stored' 
        }, { status: 404 });
      }

      const buffer = Buffer.from(await fileData.arrayBuffer());
      if (buffer.length === 0) {
        return NextResponse.json({ error: 'File is empty' }, { status: 500 });
      }

      const isPDF = fileType === 'application/pdf' || fileName.toLowerCase().endsWith('.pdf');
      return new NextResponse(new Uint8Array(buffer), {
        headers: {
          'Content-Type': fileType,
          'Content-Disposition': (view && isPDF) 
            ? `inline; filename="${fileName}"` 
            : `attachment; filename="${fileName}"`,
          'Content-Length': buffer.length.toString(),
          ...(view && isPDF ? { 'X-Content-Type-Options': 'nosniff' } : {}),
        },
      });
    }

    // Get single document with text content
    if (id) {
      const { data: chunks, error } = await supabase
        .from('documents')
        .select('content, metadata')
        .eq('metadata->>document_id', id)
        .order('metadata->>chunk_index', { ascending: true });

      if (error || !chunks || chunks.length === 0) {
        return NextResponse.json({ error: 'Document not found' }, { status: 404 });
      }

      const m = chunks[0].metadata || {};
      return NextResponse.json({
        id,
        file_name: m.file_name || 'Unknown',
        file_type: m.file_type || 'unknown',
        file_size: m.file_size || 0,
        upload_date: m.upload_date || new Date().toISOString(),
        total_chunks: chunks.length,
        fullText: chunks.map((c: any) => c.content).join('\n\n'),
        file_url: m.file_url,
        file_path: m.file_path
      });
    }

    // List all documents
    const { data: documents, error } = await supabase
      .from('documents')
      .select('metadata');

    if (error) {
      return NextResponse.json({ error: error.message }, { status: 500 });
    }

    // Deduplicate documents by document_id
    // Since each document is split into multiple chunks, we need to group them
    const map = new Map();
    documents?.forEach((doc: any) => {
      const m = doc.metadata;
      if (m?.document_id && !map.has(m.document_id)) {
        map.set(m.document_id, {
          id: m.document_id,
          file_name: m.file_name || 'Unknown',
          file_type: m.file_type || 'unknown',
          file_size: m.file_size || 0,
          upload_date: m.upload_date || new Date().toISOString(),
          total_chunks: m.total_chunks || 0,
          file_url: m.file_url,
          file_path: m.file_path,
        });
      }
    });

    return NextResponse.json({ documents: Array.from(map.values()) });
  } catch (error: any) {
    return NextResponse.json({ error: error.message }, { status: 500 });
  }
}

export async function DELETE(req: Request) {
  try {
    const id = new URL(req.url).searchParams.get('id');
    if (!id) {
      return NextResponse.json({ error: 'Document ID required' }, { status: 400 });
    }

    // Get file path from metadata
    const { data: docs } = await supabase
      .from('documents')
      .select('metadata')
      .eq('metadata->>document_id', id)
      .limit(1);

    const filePath = docs?.[0]?.metadata?.file_path;

    // Delete file from storage
    if (filePath) {
      await supabaseStorage.storage.from('documents').remove([filePath]);
    }

    // Delete all chunks from database
    const { error } = await supabase
      .from('documents')
      .delete()
      .eq('metadata->>document_id', id);

    if (error) {
      return NextResponse.json({ error: error.message }, { status: 500 });
    }

    return NextResponse.json({ success: true, fileDeleted: !!filePath });
  } catch (error: any) {
    return NextResponse.json({ error: error.message }, { status: 500 });
  }
}

This route handles:

• GET without ID: Lists all documents (deduplicated since each document has multiple chunks)

• GET with ID: Returns document details and full text (all chunks combined)

• GET with ID and file=true: Downloads the original file from storage

• DELETE with ID: Deletes the document and its file from both storage and database

Now that your API routes are complete, let's build the user interface components, starting with the upload modal.

Step 8: Create the Upload Modal Component

The upload modal provides a user-friendly interface for selecting and uploading documents. It handles file selection, upload progress, and displays success or error messages.

Create src/app/components/UploadModal.tsx:

'use client';
import { useState, useEffect } from 'react';

interface UploadModalProps {
  isOpen: boolean;
  onClose: () => void;
  onUploadSuccess?: () => void;
}

export default function UploadModal({ isOpen, onClose, onUploadSuccess }: UploadModalProps) {
  const [file, setFile] = useState<File | null>(null);
  const [uploading, setUploading] = useState(false);
  const [message, setMessage] = useState<{ type: 'success' | 'error'; text: string } | null>(null);

  useEffect(() => {
    document.body.style.overflow = isOpen ? 'hidden' : 'unset';
    if (!isOpen) { 
      setFile(null); 
      setMessage(null); 
    }
    return () => { 
      document.body.style.overflow = 'unset'; 
    };
  }, [isOpen]);

  const handleFileChange = (e: React.ChangeEvent<HTMLInputElement>) => {
    if (e.target.files && e.target.files[0]) {
      setFile(e.target.files[0]);
      setMessage(null);
    }
  };

  const handleUpload = async () => {
    if (!file) {
      setMessage({ type: 'error', text: 'Please select a file' });
      return;
    }

    setUploading(true);
    setMessage(null);

    try {
      const formData = new FormData();
      formData.append('file', file);

      const res = await fetch('/api/upload', {
        method: 'POST',
        body: formData,
      });

      const data = await res.json();

      if (data.success) {
        setMessage({
          type: 'success',
          text: `File "${data.fileName}" uploaded successfully! Processed ${data.chunks} chunks.`,
        });
        setFile(null);
        (document.getElementById('upload-file-input') as HTMLInputElement)?.setAttribute('value', '');
        setTimeout(() => { 
          onUploadSuccess?.(); 
          onClose(); 
        }, 1500);
      } else {
        setMessage({ type: 'error', text: data.error || 'Upload failed' });
      }
    } catch (error: any) {
      setMessage({ type: 'error', text: error.message || 'Upload failed' });
    } finally {
      setUploading(false);
    }
  };

  if (!isOpen) return null;

  return (
    <div
      className="fixed inset-0 z-50 flex items-center justify-center bg-black bg-opacity-75 p-4"
      onClick={onClose}
    >
      <div
        className="relative bg-white dark:bg-gray-900 rounded-lg shadow-xl w-full max-w-2xl max-h-[90vh] overflow-y-auto"
        onClick={(e) => e.stopPropagation()}
      >
        <div className="flex items-center justify-between p-6 border-b border-gray-200 dark:border-gray-800">
          <h2 className="text-2xl font-semibold text-gray-900 dark:text-gray-100">
            Upload Document
          </h2>
          <button
            onClick={onClose}
            className="p-2 text-gray-500 hover:text-gray-700 dark:text-gray-400 dark:hover:text-gray-200 rounded-lg hover:bg-gray-100 dark:hover:bg-gray-800"
            aria-label="Close"
          >
            <svg className="w-6 h-6" fill="none" stroke="currentColor" viewBox="0 0 24 24">
              <path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M6 18L18 6M6 6l12 12" />
            </svg>
          </button>
        </div>

        <div className="p-6">
          <div className="mb-6">
            <label htmlFor="upload-file-input" className="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
              Select a file (PDF, DOCX, or TXT)
            </label>
            <input
              id="upload-file-input"
              type="file"
              accept=".pdf,.docx,.txt"
              onChange={handleFileChange}
              className="block w-full text-sm text-gray-500
                file:mr-4 file:py-2 file:px-4
                file:rounded-lg file:border-0
                file:text-sm file:font-semibold
                file:bg-blue-50 file:text-blue-700
                hover:file:bg-blue-100
                dark:file:bg-blue-900 dark:file:text-blue-300
                dark:hover:file:bg-blue-800"
            />
          </div>

          {file && (
            <div className="mb-6 p-4 bg-gray-50 dark:bg-gray-800 rounded-lg text-sm text-gray-600 dark:text-gray-400 space-y-1">
              <p><span className="font-medium">Selected:</span> {file.name}</p>
              <p><span className="font-medium">Size:</span> {(file.size / 1024).toFixed(2)} KB</p>
              <p><span className="font-medium">Type:</span> {file.type || file.name.split('.').pop()}</p>
            </div>
          )}

          <button
            onClick={handleUpload}
            disabled={!file || uploading}
            className="w-full bg-blue-600 text-white px-6 py-3 rounded-lg hover:bg-blue-700 disabled:bg-gray-400 disabled:cursor-not-allowed font-medium"
          >
            {uploading ? 'Uploading and Processing...' : 'Upload Document'}
          </button>

          {message && (
            <div
              className={`mt-6 p-4 rounded-lg ${
                message.type === 'success'
                  ? 'bg-green-50 text-green-800 dark:bg-green-900 dark:text-green-200'
                  : 'bg-red-50 text-red-800 dark:bg-red-900 dark:text-red-200'
              }`}
            >
              {message.text}
            </div>
          )}

          <div className="mt-8 p-4 bg-blue-50 dark:bg-blue-900/20 rounded-lg text-sm">
            <p className="font-medium text-blue-900 dark:text-blue-200 mb-2">Supported: PDF, DOCX, TXT</p>
            <p className="text-blue-700 dark:text-blue-400">Files will be processed and embedded for RAG search.</p>
          </div>
        </div>
      </div>
    </div>
  );
}

This component provides a clean interface for file uploads with proper error handling and user feedback. Next, let's create the PDF viewer component for previewing documents.

Step 9: Create the PDF Viewer Modal Component

The PDF viewer modal allows users to preview PDFs and view extracted text from any document. It's particularly useful for verifying that documents were processed correctly.

Create src/app/components/PDFViewerModal.tsx:

'use client';
import { useEffect, useState } from 'react';

interface PDFViewerModalProps {
  isOpen: boolean;
  onClose: () => void;
  fileUrl: string;
  fileName: string;
  documentId?: string;
  isPDF?: boolean;
}

export default function PDFViewerModal({ 
  isOpen, 
  onClose, 
  fileUrl, 
  fileName, 
  documentId, 
  isPDF = true 
}: PDFViewerModalProps) {
  const [error, setError] = useState<string | null>(null);
  const [loading, setLoading] = useState(true);
  const [activeTab, setActiveTab] = useState<'preview' | 'content'>('preview');
  const [text, setText] = useState<string>('');
  const [textLoading, setTextLoading] = useState(false);
  const [textError, setTextError] = useState<string | null>(null);

  useEffect(() => {
    document.body.style.overflow = isOpen ? 'hidden' : 'unset';
    if (isOpen) { 
      setError(null); 
      setLoading(true); 
      setActiveTab(isPDF ? 'preview' : 'content'); 
      setText(''); 
      setTextError(null); 
    }
    return () => { 
      document.body.style.overflow = 'unset'; 
    };
  }, [isOpen, isPDF]);

  useEffect(() => {
    if (isOpen && documentId && activeTab === 'content' && !text && !textLoading && !textError) {
      fetchDocumentText();
    }
  }, [isOpen, documentId, activeTab, text, textLoading, textError]);

  useEffect(() => {
    if (isOpen && fileUrl && isPDF) {
      fetch(fileUrl, { method: 'GET', headers: { 'Accept': 'application/json' } })
        .then(async res => {
          if (res.headers.get('content-type')?.includes('application/json')) {
            const data = await res.json();
            throw new Error(data.error || 'File not available');
          }
          if (!res.ok) throw new Error(`Failed to load: ${res.status}`);
          setLoading(false);
        })
        .catch(err => {
          setError(err.message || 'Failed to load PDF');
          setLoading(false);
        });
    } else if (isOpen && !isPDF) {
      setLoading(false);
    }
  }, [isOpen, fileUrl, isPDF]);

  const fetchDocumentText = async () => {
    if (!documentId) return;
    setTextLoading(true); 
    setTextError(null);
    try {
      const res = await fetch(`/api/documents?id=${documentId}`);
      const data = await res.json();
      if (data.error) {
        setTextError(data.error);
      } else {
        setText(data.fullText || 'No text content available');
      }
    } catch (err) {
      setTextError(err instanceof Error ? err.message : 'Failed to fetch document text');
    } finally {
      setTextLoading(false);
    }
  };

  if (!isOpen) return null;

  return (
    <div
      className="fixed inset-0 z-50 flex items-center justify-center bg-black bg-opacity-75 p-4"
      onClick={onClose}
    >
      <div
        className="relative bg-white dark:bg-gray-900 rounded-lg shadow-xl w-full max-w-6xl h-[90vh] flex flex-col"
        onClick={(e) => e.stopPropagation()}
      >
        <div className="flex flex-col border-b border-gray-200 dark:border-gray-800">
          <div className="flex items-center justify-between p-4">
            <h2 className="text-xl font-semibold text-gray-900 dark:text-gray-100 truncate flex-1 mr-4">
              {fileName}
            </h2>
            <div className="flex items-center gap-2">
              <button
                onClick={onClose}
                className="p-2 text-gray-500 hover:text-gray-700 dark:text-gray-400 dark:hover:text-gray-200 rounded-lg hover:bg-gray-100 dark:hover:bg-gray-800"
                aria-label="Close"
              >
                <svg className="w-6 h-6" fill="none" stroke="currentColor" viewBox="0 0 24 24">
                  <path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M6 18L18 6M6 6l12 12" />
                </svg>
              </button>
            </div>
          </div>

          {isPDF && (
            <div className="flex border-t border-gray-200 dark:border-gray-800">
              {(['preview', 'content'] as const).map(tab => (
                <button 
                  key={tab} 
                  onClick={() => setActiveTab(tab)} 
                  className={`flex-1 px-4 py-3 text-sm font-medium transition-colors ${
                    activeTab === tab 
                      ? 'text-blue-600 dark:text-blue-400 border-b-2 border-blue-600 dark:border-blue-400 bg-blue-50 dark:bg-blue-900/20' 
                      : 'text-gray-500 dark:text-gray-400 hover:text-gray-700 dark:hover:text-gray-300 hover:bg-gray-50 dark:hover:bg-gray-800'
                  }`}
                >
                  {tab.charAt(0).toUpperCase() + tab.slice(1)}
                </button>
              ))}
            </div>
          )}
        </div>

        <div className="flex-1 overflow-hidden">
          {isPDF && activeTab === 'preview' && (
            <div className="h-full overflow-hidden">
              {error ? (
                <div className="flex flex-col items-center justify-center h-full p-8">
                  <div className="bg-yellow-50 dark:bg-yellow-900/20 border border-yellow-200 dark:border-yellow-800 rounded-lg p-6 max-w-md">
                    <h3 className="text-lg font-semibold text-yellow-800 dark:text-yellow-200 mb-2">
                      PDF File Not Available
                    </h3>
                    <p className="text-yellow-700 dark:text-yellow-300 mb-4">{error}</p>
                    {documentId && (
                      <button 
                        onClick={() => setActiveTab('content')} 
                        className="px-4 py-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700 font-medium"
                      >
                        View Extracted Text Instead
                      </button>
                    )}
                  </div>
                </div>
              ) : loading ? (
                <div className="flex items-center justify-center h-full">
                  <p className="text-gray-500 dark:text-gray-400">Loading PDF...</p>
                </div>
              ) : (
                <iframe
                  src={`${fileUrl}${fileUrl.includes('?') ? '&' : '?'}view=true#toolbar=1&navpanes=0&scrollbar=1`}
                  className="w-full h-full border-0"
                  title={fileName}
                  allow="fullscreen"
                  onError={() => setError('Failed to load PDF')}
                />
              )}
            </div>
          )}

          {(!isPDF || activeTab === 'content') && (
            <div className="h-full overflow-auto p-6">
              {textLoading ? (
                <div className="flex items-center justify-center h-full">
                  <p className="text-gray-500 dark:text-gray-400">Loading...</p>
                </div>
              ) : textError ? (
                <div className="bg-red-50 dark:bg-red-900/20 border border-red-200 dark:border-red-800 rounded-lg p-4">
                  <p className="text-red-800 dark:text-red-200">Error: {textError}</p>
                </div>
              ) : (
                <div className="space-y-4">
                  <p className="text-sm text-gray-500 dark:text-gray-400">
                    Formatting may be inconsistent from source.
                  </p>
                  <pre className="whitespace-pre-wrap text-sm text-gray-800 dark:text-gray-200 font-mono bg-gray-50 dark:bg-gray-800 p-4 rounded-lg">
                    {text || 'No text content available'}
                  </pre>
                </div>
              )}
            </div>
          )}
        </div>
      </div>
    </div>
  );
}

This component provides a full-screen modal for viewing PDFs and extracted text, with tabs to switch between preview and text content. Now let's create a simple navigation component to tie everything together.

Step 10: Create the Navigation Component

The navigation component provides easy access to the Search and Documents pages. It highlights the current page and provides a clean, consistent navigation experience.

Create src/app/components/Navigation.tsx:

'use client';
import Link from 'next/link';
import { usePathname } from 'next/navigation';

export default function Navigation() {
  const pathname = usePathname();

  const navItems = [
    { href: '/', label: 'Search' },
    { href: '/documents', label: 'Documents' },
  ];

  return (
    <nav className="border-b border-gray-200 dark:border-gray-800 mb-8">
      <div className="max-w-7xl mx-auto px-4 sm:px-6 lg:px-8">
        <div className="flex space-x-8">
          {navItems.map((item) => (
            <Link
              key={item.href}
              href={item.href}
              className={`py-4 px-1 border-b-2 font-medium text-sm ${
                pathname === item.href
                  ? 'border-blue-500 text-blue-600 dark:text-blue-400'
                  : 'border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300 dark:text-gray-400 dark:hover:text-gray-300'
              }`}
            >
              {item.label}
            </Link>
          ))}
        </div>
      </div>
    </nav>
  );
}

With navigation in place, let's create the main search page where users can query their documents.

Step 11: Create the Home Page (Search Interface)

The search page is the main interface where users ask questions about their uploaded documents. It displays the AI-generated answers along with source citations, allowing users to verify the information.

Update src/app/page.tsx:

'use client';
import { useState } from 'react';
import Navigation from './components/Navigation';

export default function Home() {
  const [query, setQuery] = useState('');
  const [answer, setAnswer] = useState('');
  const [loading, setLoading] = useState(false);
  const [sources, setSources] = useState<any[]>([]);

  const handleSearch = async () => {
    if (!query.trim()) return;
    setLoading(true); 
    setAnswer(''); 
    setSources([]);
    try {
      const res = await fetch('/api/search', { 
        method: 'POST', 
        headers: { 'Content-Type': 'application/json' }, 
        body: JSON.stringify({ query }) 
      });
      const data = await res.json();
      if (data.error) {
        setAnswer(`Error: ${data.error}`);
      } else { 
        setAnswer(data.answer || 'No answer generated'); 
        setSources(data.sources || []); 
      }
    } catch (error: any) {
      setAnswer(`Error: ${error.message}`);
    } finally {
      setLoading(false);
    }
  };

  const handleKeyPress = (e: React.KeyboardEvent) => {
    if (e.key === 'Enter' && (e.metaKey || e.ctrlKey)) {
      handleSearch();
    }
  };

  return (
    <div className="min-h-screen">
      <Navigation />
      <main className="max-w-4xl mx-auto p-8">
        <h1 className="text-3xl font-bold mb-6">RAG Search</h1>

        <div className="bg-white dark:bg-gray-900 border border-gray-200 dark:border-gray-800 rounded-lg p-6 shadow-sm mb-6">
          <textarea 
            className="w-full p-4 border border-gray-300 dark:border-gray-700 rounded-lg shadow-sm bg-white dark:bg-gray-800 text-gray-900 dark:text-gray-100 resize-none focus:ring-2 focus:ring-blue-500 focus:border-transparent"
            placeholder="Ask a question about your uploaded documents..."
            value={query}
            onChange={(e) => setQuery(e.target.value)}
            onKeyDown={handleKeyPress}
            rows={4}
          />
          <button 
            onClick={handleSearch}
            className="mt-4 bg-blue-600 text-white px-8 py-3 rounded-lg hover:bg-blue-700 disabled:bg-gray-400 disabled:cursor-not-allowed font-medium"
            disabled={loading || !query.trim()}
          >
            {loading ? 'Searching...' : 'Search'}
          </button>
          <p className="mt-2 text-sm text-gray-500 dark:text-gray-400">
            Press Cmd/Ctrl + Enter to search
          </p>
        </div>

        {answer && (
          <div className="bg-white dark:bg-gray-900 border border-gray-200 dark:border-gray-800 rounded-lg p-6 shadow-sm mb-6">
            <h2 className="text-xl font-semibold mb-3">Answer:</h2>
            <p className="text-gray-800 dark:text-gray-200 leading-relaxed whitespace-pre-wrap">
              {answer}
            </p>
          </div>
        )}

        {sources && sources.length > 0 && (
          <div className="bg-white dark:bg-gray-900 border border-gray-200 dark:border-gray-800 rounded-lg p-6 shadow-sm">
            <h2 className="text-xl font-semibold mb-3">Sources ({sources.length}):</h2>
            <div className="space-y-3">
              {sources.map((source, index) => (
                <div
                  key={index}
                  className="p-4 bg-gray-50 dark:bg-gray-800 rounded-lg border border-gray-200 dark:border-gray-700"
                >
                  <p className="text-sm text-gray-600 dark:text-gray-400 mb-1">
                    <span className="font-medium">Source:</span>{' '}
                    {source.metadata?.source || source.metadata?.file_name || 'Unknown'}
                  </p>
                  <p className="text-sm text-gray-800 dark:text-gray-200 line-clamp-3">
                    {source.content}
                  </p>
                </div>
              ))}
            </div>
          </div>
        )}
      </main>
    </div>
  );
}

This page provides a clean search interface with a textarea for queries, a search button, and sections to display answers and source citations. The sources section helps users verify where the information came from, which is crucial for trust and accuracy. Now let's create the documents management page.

Step 12: Create the Documents Page

The documents page serves as your document library. It displays all uploaded documents in a table format, shows metadata like file size and chunk count, and provides actions to preview, download, or delete documents. This page is essential for managing your document collection and verifying uploads.

Create src/app/documents/page.tsx:

'use client';
import { useState, useEffect } from 'react';
import Navigation from '../components/Navigation';
import PDFViewerModal from '../components/PDFViewerModal';
import UploadModal from '../components/UploadModal';

interface Document {
  id: string;
  file_name: string;
  file_type: string;
  file_size: number;
  upload_date: string;
  total_chunks: number;
  file_url?: string;
  file_path?: string;
}

export default function DocumentsPage() {
  const [documents, setDocuments] = useState<Document[]>([]);
  const [loading, setLoading] = useState(true);
  const [error, setError] = useState<string | null>(null);
  const [showPDFModal, setShowPDFModal] = useState(false);
  const [selectedPDF, setSelectedPDF] = useState<{ url: string; name: string; id?: string; isPDF?: boolean } | null>(null);
  const [deletingId, setDeletingId] = useState<string | null>(null);
  const [showUploadModal, setShowUploadModal] = useState(false);

  useEffect(() => {
    fetchDocuments();
  }, []);

  const fetchDocuments = async () => {
    try {
      setLoading(true);
      const res = await fetch('/api/documents');
      const data = await res.json();
      if (data.error) {
        setError(data.error);
      } else {
        setDocuments(data.documents || []);
      }
    } catch (err) {
      setError(err instanceof Error ? err.message : 'Failed to fetch documents');
    } finally {
      setLoading(false);
    }
  };

  const formatDate = (s: string) => {
    try {
      const d = new Date(s);
      return isNaN(d.getTime()) 
        ? s 
        : d.toLocaleString('en-US', { 
            year: 'numeric', 
            month: 'short', 
            day: 'numeric', 
            hour: '2-digit', 
            minute: '2-digit', 
            hour12: true 
          });
    } catch { 
      return s; 
    }
  };

  const formatFileSize = (b: number) => 
    b < 1024 
      ? `${b} B` 
      : b < 1024 * 1024 
        ? `${(b / 1024).toFixed(2)} KB` 
        : `${(b / (1024 * 1024)).toFixed(2)} MB`;

  const handleDelete = async (id: string, name: string) => {
    if (!confirm(`Delete "${name}"? This will permanently delete the document, embeddings, and file.`)) {
      return;
    }
    setDeletingId(id);
    try {
      const res = await fetch(`/api/documents?id=${id}`, { method: 'DELETE' });
      const data = await res.json();
      if (data.error) {
        alert(`Error: ${data.error}`);
      } else {
        setDocuments(documents.filter(doc => doc.id !== id));
      }
    } catch (err) {
      alert(err instanceof Error ? err.message : 'Failed to delete');
    } finally {
      setDeletingId(null);
    }
  };

  return (
    <div className="min-h-screen">
      <Navigation />
      <main className="max-w-7xl mx-auto p-8">
        <div className="flex items-center justify-between mb-6">
          <h1 className="text-3xl font-bold">Documents</h1>
          <button
            onClick={() => setShowUploadModal(true)}
            className="px-4 py-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700 font-medium"
          >
            Upload Document
          </button>
        </div>

        {loading ? (
          <div className="text-center py-12">
            <p className="text-gray-500 dark:text-gray-400">Loading documents...</p>
          </div>
        ) : error ? (
          <div className="bg-red-50 dark:bg-red-900/20 border border-red-200 dark:border-red-800 rounded-lg p-4">
            <p className="text-red-800 dark:text-red-200">Error: {error}</p>
          </div>
        ) : documents.length === 0 ? (
          <div className="bg-gray-50 dark:bg-gray-800 border border-gray-200 dark:border-gray-700 rounded-lg p-12 text-center">
            <p className="text-gray-500 dark:text-gray-400 mb-4">No documents uploaded yet.</p>
            <button
              onClick={() => setShowUploadModal(true)}
              className="text-blue-600 dark:text-blue-400 hover:underline font-medium"
            >
              Upload your first document
            </button>
          </div>
        ) : (
          <div className="bg-white dark:bg-gray-900 border border-gray-200 dark:border-gray-800 rounded-lg shadow-sm overflow-hidden">
            <div className="overflow-x-auto">
              <table className="min-w-full divide-y divide-gray-200 dark:divide-gray-800">
                <thead className="bg-gray-50 dark:bg-gray-800">
                  <tr>
                    <th className="px-6 py-3 text-left text-xs font-medium text-gray-500 dark:text-gray-400 uppercase tracking-wider">
                      File Name
                    </th>
                    <th className="px-6 py-3 text-left text-xs font-medium text-gray-500 dark:text-gray-400 uppercase tracking-wider">
                      Type
                    </th>
                    <th className="px-6 py-3 text-left text-xs font-medium text-gray-500 dark:text-gray-400 uppercase tracking-wider">
                      Size
                    </th>
                    <th className="px-6 py-3 text-left text-xs font-medium text-gray-500 dark:text-gray-400 uppercase tracking-wider">
                      Chunks
                    </th>
                    <th className="px-6 py-3 text-left text-xs font-medium text-gray-500 dark:text-gray-400 uppercase tracking-wider">
                      Upload Date
                    </th>
                    <th className="px-6 py-3 text-left text-xs font-medium text-gray-500 dark:text-gray-400 uppercase tracking-wider">
                      Actions
                    </th>
                  </tr>
                </thead>
                <tbody className="bg-white dark:bg-gray-900 divide-y divide-gray-200 dark:divide-gray-800">
                  {documents.map((doc) => (
                    <tr key={doc.id} className="hover:bg-gray-50 dark:hover:bg-gray-800">
                      <td className="px-6 py-4 whitespace-nowrap">
                        <div className="text-sm font-medium text-gray-900 dark:text-gray-100">
                          {doc.file_name}
                        </div>
                      </td>
                      <td className="px-6 py-4 whitespace-nowrap">
                        <span className="px-2 inline-flex text-xs leading-5 font-semibold rounded-full bg-blue-100 text-blue-800 dark:bg-blue-900 dark:text-blue-200">
                          {doc.file_type || 'unknown'}
                        </span>
                      </td>
                      <td className="px-6 py-4 whitespace-nowrap text-sm text-gray-500 dark:text-gray-400">
                        {formatFileSize(doc.file_size)}
                      </td>
                      <td className="px-6 py-4 whitespace-nowrap text-sm text-gray-500 dark:text-gray-400">
                        {doc.total_chunks}
                      </td>
                      <td className="px-6 py-4 whitespace-nowrap text-sm text-gray-500 dark:text-gray-400">
                        {formatDate(doc.upload_date)}
                      </td>
                      <td className="px-6 py-4 whitespace-nowrap text-sm font-medium">
                        <div className="flex gap-3 items-center">
                          {doc.file_name.toLowerCase().endsWith('.pdf') ? (
                            <button 
                              onClick={() => {
                                const pdfUrl = doc.file_url 
                                  ? `${doc.file_url}?view=true` 
                                  : `/api/documents?id=${doc.id}&file=true&view=true`;
                                setSelectedPDF({ url: pdfUrl, name: doc.file_name, id: doc.id });
                                setShowPDFModal(true);
                              }} 
                              className="text-blue-600 hover:text-blue-900 dark:text-blue-400 dark:hover:text-blue-300"
                            >
                              Preview
                            </button>
                          ) : (
                            <>
                              <button 
                                onClick={() => {
                                  setSelectedPDF({ 
                                    url: doc.file_url || `/api/documents?id=${doc.id}&file=true`, 
                                    name: doc.file_name, 
                                    id: doc.id, 
                                    isPDF: false 
                                  });
                                  setShowPDFModal(true);
                                }} 
                                className="text-blue-600 hover:text-blue-900 dark:text-blue-400 dark:hover:text-blue-300"
                              >
                                View
                              </button>
                              {(doc.file_url || doc.file_path) && (
                                <a 
                                  href={doc.file_url || `/api/documents?id=${doc.id}&file=true`} 
                                  download={doc.file_name}
                                  className="text-green-600 hover:text-green-900 dark:text-green-400 dark:hover:text-green-300" 
                                  target="_blank" 
                                  rel="noopener noreferrer"
                                >
                                  Download
                                </a>
                              )}
                            </>
                          )}
                          <button 
                            onClick={() => handleDelete(doc.id, doc.file_name)} 
                            disabled={deletingId === doc.id}
                            className="text-red-600 hover:text-red-900 dark:text-red-400 dark:hover:text-red-300 disabled:opacity-50 disabled:cursor-not-allowed"
                          >
                            {deletingId === doc.id ? 'Deleting...' : 'Delete'}
                          </button>
                        </div>
                      </td>
                    </tr>
                  ))}
                </tbody>
              </table>
            </div>
          </div>
        )}

        {selectedPDF && (
          <PDFViewerModal 
            isOpen={showPDFModal} 
            onClose={() => { 
              setShowPDFModal(false); 
              setSelectedPDF(null); 
            }}
            fileUrl={selectedPDF.url} 
            fileName={selectedPDF.name} 
            documentId={selectedPDF.id} 
            isPDF={selectedPDF.isPDF !== false} 
          />
        )}
        <UploadModal 
          isOpen={showUploadModal} 
          onClose={() => setShowUploadModal(false)} 
          onUploadSuccess={fetchDocuments} 
        />
      </main>
    </div>
  );
}

This page provides a comprehensive document management interface with a table showing all documents, their metadata, and action buttons for preview, download, and deletion. The page automatically refreshes after uploads and handles loading and error states gracefully.

Now that all your components and pages are built, let's test the complete application.

Step 13: Test Your Application

Start your development server:

npm run dev

Open http://localhost:3000 in your browser.

Test the Upload Flow

1. Navigate to the Documents page

2. Click "Upload Document"

3. Select a PDF, DOCX, or TXT file

4. Wait for the upload and processing to complete (this may take a moment as embeddings are generated)

5. You should see your document in the list with its metadata:

Test the Search Flow

1. Navigate to the Search page (or click "Search" in the navigation)

2. Make sure you've uploaded at least one document first

3. Type a question about your uploaded document (for example, "What is this document about?" or ask about specific content)

4. Click "Search" or press Cmd/Ctrl + Enter

5. You should see an AI-generated answer with source citations showing which document chunks were used

Once the embedding is done, you can navigate to search and look for the sample test command based on the documents you have uploaded. You can also check the source from which the search results were pulled.

Test Document Management

1. On the Documents page, click "Preview" or "View" on a document

2. Try downloading a document

3. Test deleting a document (be careful - this is permanent)

If everything works correctly, you're ready to deploy your application!

Step 14: Deploy Your Application

Deploy to Vercel

Vercel is the easiest way to deploy Next.js applications and is made by the creators of Next.js:

To get started, you’ll need to push your code to GitHub. So go ahead and create a repository and push your code.

Then go to vercel.com and sign in with your GitHub account. Click "New Project" and import your GitHub repository.

Add your environment variables in the project settings:

• NEXT_PUBLIC_SUPABASE_URL

• NEXT_PUBLIC_SUPABASE_PUBLISHABLE_DEFAULT_KEY

• SUPABASE_SERVICE_ROLE_KEY

• OPENAI_API_KEY

Then click "Deploy", and your application will be live in minutes! Vercel automatically builds and deploys your Next.js application, and you'll get a URL like your-app.vercel.app.

Important Deployment Notes

• Make sure all environment variables are set in your Vercel project settings

• The service role key is required for file uploads to work

• Supabase Storage bucket should be accessible (public or with proper RLS policies)

• Your OpenAI API key should have sufficient credits

How RAG Search Works

Your application uses the RAG (Retrieval-Augmented Generation) pattern. This combines information retrieval with AI text generation. Here's how it works step by step:

1. Document processing: When you upload a document, it's split into chunks. These are typically 800 characters each with 100-character overlap. Each chunk gets an embedding. This is a 1536-dimensional vector that represents its semantic meaning.

2. Storage: Embeddings are stored in a vector database. This is PostgreSQL with the pgvector extension. They're stored alongside the original text chunks. The original files are stored in Supabase Storage.

3. Query processing: When you search, your query is converted into an embedding. It uses the same model that processed the documents. This ensures the query and documents are in the same "vector space."

4. Similarity search: The system finds the most similar document chunks. It uses cosine similarity on the embeddings. Cosine similarity measures the angle between vectors. Smaller angles mean more similar content, even if the exact words differ.

5. Answer generation: The retrieved chunks are used as context for an AI model. This model is GPT-4o-mini. It generates an accurate answer. The system prompt instructs the AI to only answer based on the provided context. This ensures accuracy.

This approach gives you several benefits.

First, you get accuracy. Answers are based on your actual documents, not just the AI's training data. Second, you get transparency. You can see which document chunks were used to generate each answer. Third, you get efficiency. Only relevant chunks are used, which reduces token usage and costs. Finally, you get up-to-date information. You can update your knowledge base by uploading new documents without retraining the AI.

Troubleshooting Common Issues

"Storage RLS error" when uploading

This means your SUPABASE_SERVICE_ROLE_KEY is not set or incorrect. Make sure the key is in your .env.local file for local development. Also make sure you're using the service role key, not the anon key. Finally, make sure the key is correctly set in your deployment environment, such as Vercel.

"Failed to extract text from file"

Make sure your file is a valid PDF, DOCX, or TXT file. Check that the file isn't corrupted. For PDFs, ensure they contain extractable text. Scanned PDFs with only images won't work without OCR.

"No answer generated"

Make sure you've uploaded at least one document. Try a different query that's more likely to match your documents. Check that embeddings were successfully created. You can verify this in your Supabase database.

Vector similarity search not working

Ensure the vector extension is enabled in Supabase. You can do this by running CREATE EXTENSION IF NOT EXISTS vector;. Verify the match_documents function exists in your database. You can check this in the SQL Editor. Check that embeddings are being stored correctly. They should be JSON strings in the embedding column.

Slow search or upload times

Large documents take longer to process. This is because more chunks mean more embedding API calls. Consider reducing chunk size or processing documents in batches. Also check your OpenAI API rate limits.

Next Steps

Now that you have a working RAG search application, you can extend it with additional features. Here are some examples of useful features you could add:

• You can add more file types by extending the text extraction to support Markdown, HTML, or other formats.

• You can improve chunking by experimenting with different chunk sizes, overlap strategies, or semantic chunking.

• You can add authentication to protect your documents with user authentication using Supabase Auth.

• You can enhance the UI by adding features like search history, document tags, or advanced filters.

• You can optimize performance by adding caching, pagination, or streaming responses.

• You can add filters to allow users to search within specific documents or date ranges.

• Finally, you can improve search by adding hybrid search, which combines keyword and semantic search, or reranking.

Conclusion

You've built a complete RAG search application from scratch. This application demonstrates modern web development with Next.js and TypeScript. It shows vector database operations with Supabase and pgvector. It demonstrates AI integration with OpenAI embeddings and chat completions. It includes file handling and storage with Supabase Storage. Finally, it features a production-ready user interface with Tailwind CSS.

The RAG pattern you've implemented is used by many production applications. These include chatbots, knowledge bases, document search systems, and AI assistants. You now have the foundation to build more advanced features on top of this.

The skills you've learned are highly valuable in today's AI-driven development landscape. You've learned to work with embeddings, vector databases, and the RAG pattern. You can apply these concepts to build intelligent search systems, document Q&A applications, or AI-powered knowledge bases.

If you upload a PDF like a company policy, research paper, or contract, the model cannot magically read it unless you give it that content.

This is where Retrieval Augmented Generation, or RAG, becomes useful.

RAG lets you combine a language model with your own data. Instead of asking the model to guess, you first retrieve the right parts of the document and then ask the model to answer using that information.

In this article, you will learn how to chat with your own PDF using RAG. You will build the backend using LangChain and create a simple React user interface to ask questions and see answers.

You should be comfortable with basic Python and JavaScript, and have a working knowledge of React and REST APIs. Familiarity with language models and a basic understanding of embeddings or vector search will be helpful but not mandatory.

What We’ll Cover

1. What Problem Are We Solving?

2. What Is Retrieval Augmented Generation?

3. Setting Up the Backend with LangChain

   • Installing Dependencies

   • Loading and Splitting the PDF

   • Creating Embeddings and Vector Store

   • Creating the Retrieval Chain

   • Exposing an API with FastAPI

   • Building a Simple React Chat UI

4. How the Full Flow Works

5. Why This Approach Works Well

6. Common Improvements You Can Add

7. Final Thoughts

What Problem Are We Solving?

Imagine you have a long PDF with hundreds of pages. Searching manually is slow. Copying text into ChatGPT is not practical.

You want to ask simple questions like “What is the leave policy?” or “What does this contract say about termination?”

A normal language model cannot answer these questions correctly because it has never seen your PDF. RAG solves this by adding a retrieval step before generation.

The system first finds relevant parts of the PDF and then uses those parts as context for the answer.

What Is Retrieval Augmented Generation?

Retrieval Augmented Generation is a pattern with three main steps.

First, your document is split into small chunks. Each chunk is converted into a vector embedding. These embeddings are stored in a vector database.

Second, when a user asks a question, that question is also converted into an embedding. The system searches the vector database to find the most similar chunks.

Third, those chunks are sent to the language model along with the question. The model uses only that context to generate an answer.

This approach keeps answers grounded in your document and reduces hallucinations.

The system has four main parts:

• A PDF loader reads the document.

• A text splitter breaks it into chunks.

• An embedding model converts text into vectors and stores them in a vector store.

• A language model answers questions using retrieved chunks.

The frontend is a simple chat interface built in React. It sends the user’s question to a backend API and displays the response.

This type of custom RAG development helps companies build internal tools that work with their own private data instead of sending it to large language models.

Setting Up the Backend with LangChain

We’ll use Python and LangChain for the backend. The backend will load the PDF, build the vector store, and expose an API to answer questions.

Installing Dependencies

Start by installing the required libraries.

pip install langchain langchain-community langchain-openai faiss-cpu pypdf fastapi uvicorn

This setup uses FAISS as a local vector store and OpenAI for embeddings and chat. You can swap these later for other models.

Loading and Splitting the PDF

The first step is to load the PDF and split it into chunks that are small enough for embeddings.

from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

loader = PyPDFLoader("document.pdf")
documents = loader.load()
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
chunks = text_splitter.split_documents(documents)

Chunking is important. If chunks are too large, embeddings become less accurate. If they are too small, context is lost.

Creating Embeddings and Vector Store

Next, convert the chunks into embeddings and store them in FAISS.

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS

embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(chunks, embeddings)

This step is usually done once. In a real app, you would persist the vector store to disk.

Creating the Retrieval Chain

Now create a retrieval-based question answering chain.

from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA

llm = ChatOpenAI(
    temperature=0,
    model="gpt-4o-mini"
)
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
    return_source_documents=False
)

The retriever finds the top matching chunks. The language model answers using only those chunks.

Exposing an API with FastAPI

Now wrap this logic in an API so the React app can use it.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()
class QuestionRequest(BaseModel):
    question: str
@app.post("/ask")
def ask_question(req: QuestionRequest):
    result = qa_chain.run(req.question)
    return {"answer": result}

Run the server using this command:

uvicorn main:app --reload

Your backend is now ready.

Building a Simple React Chat UI

Next, build a simple React interface that sends questions to the backend and shows answers.

You can use any React setup. A simple Vite or Create React App project works fine.

Inside your main component, manage the question input and answer state.

import { useState } from "react";

function App() {
  const [question, setQuestion] = useState("");
  const [answer, setAnswer] = useState("");
  const [loading, setLoading] = useState(false);
  const askQuestion = async () => {
    setLoading(true);
    const res = await fetch("http://localhost:8000/ask", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ question })
    });
    const data = await res.json();
    setAnswer(data.answer);
    setLoading(false);
  };
  return (
    <div style={{ padding: "2rem", maxWidth: "600px", margin: "auto" }}>
      <h2>Chat with your PDF</h2>
      <textarea
        value={question}
        onChange={(e) => setQuestion(e.target.value)}
        rows={4}
        style={{ width: "100%" }}
        placeholder="Ask a question about the PDF"
      />
      <button onClick={askQuestion} disabled={loading}>
        {loading ? "Thinking..." : "Ask"}
      </button>
      <div style={{ marginTop: "1rem" }}>
        <strong>Answer</strong>
        <p>{answer}</p>
      </div>
    </div>
  );
}
export default App;

This UI is simple but effective. It lets users type a question, sends it to the backend, and shows the answer. Make sure to use the latest version of React to avoid the growing React vulnerabilities.

How the Full Flow Works

When the app starts, the backend has already processed the PDF and built the vector store. When a user types a question, the React app sends it to the API.

The backend converts the question into an embedding. It searches the vector store for similar chunks. Those chunks are passed to the language model as context. The model generates an answer based only on that context.

The answer is sent back to the frontend and displayed to the user.

Why This Approach Works Well

RAG works well because it keeps answers grounded in real data. The model is not guessing – it’s reading from your document.

This approach also scales well. You can add more PDFs, reindex them, and reuse the same chat interface. You can also swap FAISS for a hosted vector database if needed.

Another benefit is control. You decide what data the model can see. This is important for private or sensitive documents.

Common Improvements You Can Add

You can improve this setup in many ways. You can persist the vector store so it doesn’t rebuild on every restart. You can also add document citations to the answer. And you can stream responses for a better chat experience.

You can also add authentication, upload new PDFs from the UI, or support multiple documents per user.

Final Thoughts

Chatting with PDFs using Retrieval Augmented Generation is one of the most practical uses of language models today. It turns static documents into interactive knowledge sources.

With LangChain handling retrieval and a simple React UI for interaction, you can build a useful system with very little code. The same pattern can be used for HR policies, legal documents, technical manuals, or research papers.

Once you understand this flow, you can adapt it to many real world problems where answers must come from trusted documents rather than from the model’s memory alone.

Mastering RAG (Retrieval Augmented Generation)

Everyone is talking about RAG, but many people struggle to understand how it works under the hood. This course starts by breaking down how to connect a model to your own private information. You will learn how to turn documents into embeddings (mathematical representations of meaning) and store them in vector databases like Chroma.

The course also covers the "precision problem." You will learn why just uploading a massive PDF doesn't work and how to use chunking strategies to ensure the AI finds exactly the right paragraph to answer a user's question.

Coordination with MCP

While RAG gives an AI knowledge, the Model Context Protocol (MCP) gives it the ability to coordinate actions. MCP allows AI agents to interact with third-party software, databases, and local files. Instead of writing custom code for every single API, MCP provides a standardized way for agents to discover what a server can do and then execute tasks.

You will learn how to build your own MCP server and client using the Python SDK, giving your AI the "hands" it needs to perform real-world tasks.

Watch the full course on the freeCodeCamp.org YouTube channel (2-hour watch).

The system is great at answering "What is X?" but falls apart when you ask, "How does X relate to Y, and what happened after Z?".

The problem is that standard RAG, by its very nature, breaks context. It chops documents into isolated chunks, finds them based on semantic similarity, and hopes the LLM can piece the puzzle back together. This approach is blind to the relational context—the web of timelines, causes, and connections—that gives facts their meaning.

When queries require synthesizing information across multiple documents or complex, multi-step reasoning, standard RAG fails.

In this article, I’ll give you a practical, code-first guide to solving this problem. We'll move beyond simple vector search by implementing a robust, graph-based pattern to build more reliable, knowledge-aware systems.

Table of Contents:

• Prerequisites

• The Brittle Baseline: Our Standard RAG Setup

• A More Robust Implementation: The KnowledgeGraph

  • What is a Knowledge Graph?

  • Why is this More Effective?

• 5 RAG Failures and Their Graph-Based Solutions

  • Pattern 1: The Multi-Hop Failure

  • Pattern 2: The Causal Synthesis Failure

  • Pattern 3: The Entity Ambiguity Trap

  • Pattern 4: The Contradictory Information Failure

  • Pattern 5: The Implicit Relationship Hallucination

• Final Thoughts

Prerequisites

This is a practical, code-first guide intended for developers and engineers who have some experience with RAG. To follow along, you should have the following:

Conceptual Knowledge

• A solid understanding of what Retrieval-Augmented Generation (RAG) is and its basic components (like vector stores and LLMs).

• Familiarity with basic graph concepts (nodes, edges, and relationships) is also helpful.

Technical Setup

• A Python environment.

• An active Google API Key to use the Gemini API.

• The Python libraries langchain, langchain_google_genai, faiss-cpu, and networkx installed.

The Brittle Baseline: Our Standard RAG Setup

First, let's establish our baseline. This is a standard, "naïve" RAG pipeline using LangChain and the Gemini API. It ingests a list of Document objects, embeds them, and uses a FAISS vector store to retrieve the top-k chunks to answer a question.

This create_rag_chain function will serve as our point of comparison.

# Install necessary libraries
# !pip install -q -U langchain langchain_google_genai faiss-cpu networkx

import os
import networkx as nx
from collections import defaultdict
from langchain_google_genai import GoogleGenerativeAI, GoogleGenerativeAIEmbeddings
from langchain.vectorstores import FAISS
from langchain.schema.document import Document
from langchain.prompts import PromptTemplate
from langchain.schema.runnable import RunnablePassthrough
from langchain.schema.output_parser import StrOutputParser

# --- Configure API Key (example) ---
# from google.colab import userdata
# GOOGLE_API_KEY = userdata.get('GOOGLE_API_KEY') 
# os.environ['GOOGLE_API_KEY'] = GOOGLE_API_KEY 

# --- Initialize Models ---
# Make sure your API key is set in your environment
llm = GoogleGenerativeAI(model="gemini-1.5-pro-latest")
embeddings = GoogleGenerativeAIEmbeddings(model="models/embedding-001")

def create_rag_chain(docs):
    """Creates a simple RAG chain using FAISS as the vector store.""" 

    # Create vector store from documents
    vectorstore = FAISS.from_documents(docs, embeddings)
    # K=3 means it will retrieve the top 3 most relevant chunks
    retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

    template = """
    Answer the following question based ONLY on the context provided.
    If the context doesn't contain the answer, say "I don't have enough information from the context."

    CONTEXT:
    {context}

    QUESTION:
    {question}
    """

    prompt = PromptTemplate.from_template(template)

    # Build the chain
    rag_chain = (
        {"context": retriever, "question": RunnablePassthrough()} 
        | prompt
        | llm 
        | StrOutputParser() 
    )

    return rag_chain

A More Robust Implementation: The KnowledgeGraph

What is a Knowledge Graph?

At its core, a knowledge graph (KG) is a way of storing data as a network of nodes and edges.

• Nodes represent entities: people, companies, concepts, or events.

• Edges represent the explicit, labeled relationships between them: ceo_of, attended, or partners_with.

Instead of storing a document like "Jim Farley is the CEO of Ford," you store two nodes (Jim Farley, Ford) connected by a directed edge (ceo_of).

Why is this More Effective?

This structure is more effective because it preserves and makes relationships a first-class citizen.

Standard RAG relies on "semantic similarity". It's good at finding text chunks that sound like your query. But it’s "blind to the relational context" – the very thing you need for complex questions.

The graph-based approach solves this. When a query requires multi-step reasoning, you don't just search for similar text. You traverse a structured, explicit path in the graph. This allows the system to:

1. Follow chains of logic: It can answer multi-hop questions by finding a literal path from one node to another (for example, F-150 → made_by → Ford → ceo → Jim Farley).

2. Disambiguate entities: It can use node attributes (like type: "company") to distinguish between two entities with the same name.

3. Resolve contradictions: It can store metadata (like dates) directly on the edge to programmatically determine the most current fact.

You move from "guessing from a cloud of semantically similar text" to querying a "global memory" of how facts are explicitly connected.

Here is the practical implementation of our KnowledgeGraph. This class uses networkx to store the nodes and edges we just discussed, and includes specific methods to run the structured query patterns needed to solve our RAG failures.

class KnowledgeGraph:
    """
    A wrapper around networkx.DiGraph to store and query
    explicit entities and their relationships.
    """
    def __init__(self):
        self.graph = nx.DiGraph() 

    def add_data(self, nodes=None, edges=None):
        """Populates the graph with nodes and edges."""
        if nodes:
            for node, attrs in nodes:
                self.graph.add_node(node, **attrs) 
        if edges:
            for u, v, attrs in edges:
                self.graph.add_edge(u, v, **attrs) 

    # --- Query Patterns ---

    def query_multi_hop_path(self, source, target):
        """
        Pattern 1: Solves multi-hop queries by finding a path.
        """
        try:
            path = nx.shortest_path(self.graph, source=source, target=target) 
            # Format the answer based on the discovered path
            return f"{path[-2]} attended {path[-1]}." 
        except nx.NetworkXNoPath:
            return "Could not find a connection."

    def query_with_conflict_resolution(self, entity, relation, time_attr="year"):
        """
        Pattern 4: Resolves contradictions using metadata (like timestamps)
        stored on the edges.
        """
        candidates = []
        for neighbor in self.graph.neighbors(entity):
            edge_data = self.graph.get_edge_data(entity, neighbor) 
            if edge_data.get("label") == relation: 
                candidates.append((neighbor, edge_data.get(time_attr, 0))) 

        if not candidates: 
            return "No information found." 

        # Sort by the time attribute, descending, and take the latest
        latest = sorted(candidates, key=lambda item: item[1], reverse=True)[0] 
        return f"{latest[0]} (as of {latest[1]})" 

    def query_disambiguated(self, entity_name, entity_type, attribute_key):
        """
        Pattern 3: Uses node 'type' attributes to disambiguate
        entities with the same name.
        """
        for node, attrs in self.graph.nodes(data=True): 
            # Find the node that matches both name and type
            if entity_name in node and attrs.get("type") == entity_type: 
                # Return the requested attribute
                year = attrs['year']
                product = attrs[attribute_key]
                return f"{node}'s first product was the {product} in {year}." 
        return "Cannot disambiguate entity."

    def query_explicit_relation(self, source_node, relation_label):
        """
        Pattern 5: Finds partners based on an explicit edge label,
        preventing semantic 'bleed-over' from unrelated entities.
        """
        partners = [
            v for u, v, data in self.graph.edges(data=True) 
            if u == source_node and data.get('label') == relation_label
        ] 

        if partners:
            return f"{source_node} partnered with {', '.join(partners)}." 
        return f"No partners found for {source_node}."

# A helper function for Pattern 2 (Causal Rules)
# This logic is more rule-based but can be backed by a graph
def query_causal_chain(facts):
    """
    Pattern 2: Synthesizes a direct conclusion by following a
    chain of causal rules.
    """
    try:
        if facts["John"]["takes"] == "aspirin": 
            if facts["aspirin"]["is_a"] == "blood thinner": 
                if facts["blood thinner"]["risk_for"] == "surgery":
                    return "John is NOT safe due to increased bleeding risk from aspirin, a blood thinner."
    except KeyError:
        pass # Fall through to default
    return "Insufficient information to determine risk."

5 RAG Failures and Their Graph-Based Solutions

Let's run five scenarios to see how our standard RAG chain performs against our new KnowledgeGraph.

Pattern 1: The Multi-Hop Failure

The multi-hop failure occurs when an answer requires connecting multiple, separate facts – a chain of reasoning that RAG often breaks.

• Query: "Which university did the CEO of the company that makes the F-150 attend?"

• Problem: A standard retriever might get chunks for F-150 -> Ford and Jim Farley -> CEO, but miss the Jim Farley -> Georgetown chunk. The chain is broken.

Why the Naïve RAG Fails

The retriever's job is to find the top-k=3 chunks that are semantically similar to the entire query. When the user asks, "Which university did the CEO of the company that makes the F-150 attend?", the retriever will search our 6-document list and will likely retrieve:

1. The chunk about the University of Michigan (because of the words "university" and "car companies").

2. The chunk about Jim Farley (because of "CEO," "Ford," and "F-150 line").

3. The chunk about the F-150 engine options (because of "F-150").

The top-k=3 context handed to the LLM is now full of irrelevant facts. The one chunk that contains the actual answer ("...Mr Farley... from Georgetown University") is semantically too far from the main query and is never retrieved. The LLM fails not because it's unintelligent, but because it was never given the correct piece of the puzzle.

Why the GraphRAG Succeeds

The knowledge graph doesn't care about semantic similarity. It performs a deterministic traversal of explicit, verified relationships.

We ask for the path from the F-150 node to the Georgetown University node. The graph follows the chain we defined: F-150 → made_by → Ford Motor Company → ceo → Jim Farley → attended → Georgetown University. It can't fail or be distracted by the "noise" documents because it's not searching – it's navigating a pre-built map.

# --Naive RAG
docs_s1 = [
    # --- The 3 "Answer" Chunks ---
    Document(page_content="The Ford F-150 is a full-size pickup truck made by Ford Motor Company."),
    Document(page_content="Jim Farley is the current CEO of Ford Motor Company."),
    Document(page_content="Mr. Farley received his undergraduate degree from Georgetown University."),

    # --- The 3 "Noise" Chunks (to distract the retriever) ---
    Document(page_content="The University of Michigan is renowned for its automotive engineering program, which partners with many car companies."),
    Document(page_content="The F-150 comes with several engine options, including a powerful 3.5L EcoBoost V6."),
    Document(page_content="Mary Barra, the CEO of General Motors, is a major competitor to Ford and its F-150 line.")
]
query_s1 = "Which university did the CEO of the company that makes the F-150 attend?"
rag_chain_s1 = create_rag_chain(docs_s1) # This uses top_k=3
print(f"Naive RAG Answer: {rag_chain_s1.invoke(query_s1)}")
#
# GraphRAG Pattern
graph_s1 = KnowledgeGraph()
edges_s1 = [
    ("F-150", "Ford Motor Company", {"label": "made_by"}),
    ("Ford Motor Company", "Jim Farley", {"label": "ceo"}),
    ("Jim Farley", "Georgetown University", {"label": "attended"}),
]
graph_s1.add_data(edges=edges_s1)
print(f"GraphRAG Answer: {graph_s1.query_multi_hop_path('F-150', 'Georgetown University')}")

Output:

Naive RAG Answer: I don't have enough information from the context.
GraphRAG Answer: Jim Farley attended Georgetown University.

Pattern 2: The Causal Synthesis Failure

This is the failure to move from retrieval to synthesis. RAG lists facts but can't combine them to form a new conclusion.

• Query: "Is John safe to undergo surgery while on aspirin?"

• Problem: RAG will retrieve "John takes aspirin," "Aspirin is a blood thinner," and "Blood thinners increase surgery risk." But it will fail to synthesize these into a direct "No, it's not safe" answer.

Why the Naïve RAG Fails

The retriever searches for chunks that are semantically similar to the query: "John," "safe," "surgery," and "aspirin." In a real document base, it's highly likely to retrieve distracting, topically-related "noise" chunks.

In our example, the top-k=3 chunks it retrieves might be:

1. "John is currently taking daily low-dose aspirin." (Relevant: "John," "aspirin")

2. "Pre-surgery safety checks are standard procedure..." (Relevant: "surgery safety")

3. "John is otherwise in good health and is cleared for the procedure..." (Relevant: "John," "safe," "procedure")

The key causal link ("Aspirin... is considered a blood thinner") is semantically less similar to the full query and gets pushed out of the top-k=3 context. The LLM is then given incomplete information. It sees "John takes aspirin" and "John is cleared," so it provides a weak, hedged answer and cannot make the correct logical leap.

Why the GraphRAG Succeeds

This approach doesn't use semantic search. It uses explicit logical rules (which could be backed by a causal graph). The query_causal_chain function is not searching for text – it's executing a pre-defined chain of logic:

1. Fact: Does John take aspirin? Yes.

2. Fact: Is aspirin a blood thinner? Yes.

3. Fact: Is a blood thinner a risk for surgery? Yes.

4. Conclusion: Therefore, John is not safe.

This deterministic, rule-based reasoning is immune to the "semantic noise" that distracts the naive RAG.

# Naive RAG
docs_s2 = [
    # --- The 3 "Answer" Chunks ---
    Document(page_content="Aspirin reduces blood clotting and is considered a blood thinner."),
    Document(page_content="Patients on blood thinners have increased bleeding risk during surgery."),
    Document(page_content="John is currently taking daily low-dose aspirin."),

    # --- The 3 "Noise" Chunks (to distract the retriever) ---
    Document(page_content="John is otherwise in good health and is cleared for the procedure by his cardiologist."),
    Document(page_content="Pre-surgery safety checks are standard procedure and usually focus on anesthesia allergies."),
    Document(page_content="Aspirin is also commonly used to relieve minor aches and pains, but this is not why John takes it.")
]
query_s2 = "Is John safe to undergo surgery while on aspirin?"
rag_chain_s2 = create_rag_chain (docs_s2)
print(f"Naive RAG Answer: {rag_chain_s2.invoke(query_s2)}")

# GraphRAG Pattern
facts_s2 = {
    "aspirin": {"is_a": "blood thinner"},
    "blood thinner": {"risk_for": "surgery"},
    "John": {"takes": "aspirin"},
}
print(f"GraphRAG Answer: {query_causal_chain(facts_s2)}")

Output:

Naive RAG Answer: Based on the context, John is currently taking daily low-dose aspirin...
GraphRAG Answer: John is NOT safe due to increased bleeding risk from aspirin, a blood thinner.

Pattern 3: The Entity Ambiguity Trap

Vector search struggles with polysemy (words with multiple meanings). It relies on local semantic context, which can easily be confused.

• Query: "When did Apple release its first product?"

• Problem: The query "Apple" might retrieve documents for both Apple (company) and Apple (fruit), confusing the LLM.

Why the Naïve RAG Fails

The query "When did Apple release its first product?" is semantically ambiguous. The vector retriever, which looks for semantic closeness, will be strongly attracted to the "noise" chunks we added about the fruit.

The top-k=3 chunks it retrieves will likely be:

1. "The 'Cosmic Crisp' is a new apple product... first released..." (Extremely high semantic similarity to "Apple releases its first product").

2. "The Granny Smith apple... is a popular product..."

3. "Many apple orchards release their new harvest..."

The correct chunk ("The Apple I was introduced by Apple Inc...") is about a "company" and a specific "product" name. It might be semantically less similar to the general query than the "Cosmic Crisp" chunk. The LLM is then handed a context exclusively about fruits and confidently (but incorrectly) answers about the "Cosmic Crisp" apple.

Why the GraphRAG Succeeds

The graph approach is immune to this ambiguity. The query_disambiguated function is not just searching for "Apple." It is explicitly looking for a node that matches two criteria: name='Apple' AND type='company'.

This query structurally guarantees that it finds the Apple Inc. node and ignores the apple (fruit) node, regardless of semantic similarity. It then reliably retrieves the first_product attribute from the correct node.

# Naive RAG
docs_s3 = [
    # --- The "Answer" Chunks ---
    Document(page_content="The Apple was introduced by Apple Inc. in 1976."),
    Document(page_content="Apple Inc. is a technology company based in Cupertino."),

    # --- "Noise" Chunks (to create ambiguity) ---
    Document(page_content="The 'Cosmic Crisp' is a new apple product developed by Washington State University, first released to consumers in 2019."),
    Document(page_content="Apples (the fruit) were first cultivated in Central Asia thousands of years ago."),
    Document(page_content="The Granny Smith apple, first discovered in Australia, is a popular product for baking."),
    Document(page_content="Many apple orchards release their new harvest in the fall.")
]
query_s3 = "When did Apple release its first product?"
rag_chain_s3 = create_rag_chain(docs_s3)
print(f"Naive RAG Answer: {rag_chain_s3.invoke(query_s3)}")

# GraphRAG Pattern
graph_s3 = KnowledgeGraph()
nodes_s3 = [
    ("Apple Inc.", {"type": "company", "first_product": "Apple I", "year": 1976}),
    ("apple", {"type": "fruit", "origin": "Central Asia"}),
]
graph_s3.add_data(nodes=nodes_s3)
print(f"GraphRAG Answer: {graph_s3.query_disambiguated('Apple', 'company', 'first_product')}")

Output:

Naive RAG Answer: The 'Cosmic Crisp', a new apple product, was first released to consumers in 2019.
GraphRAG Answer: Apple Inc.'s first product was the Apple I in 1976.

Pattern 4: The Contradictory Information Failure

RAG is blind to knowledge conflicts. If it retrieves two or more contradictory facts, it can't resolve them using metadata like dates or source credibility. It will hedge, merge them into a false statement, or present all of them.

• Query: "Who is the CEO of Twitter?"

• Problem: The retriever finds one chunk saying "Parag Agrawal (2022)" and another saying "Elon Musk (2023)". It may also find other related, confusing information. The LLM has no way to know which fact is the most current and authoritative.

Why the Naïve RAG Fails

The query "Who is the CEO of Twitter?" is semantically similar to all documents containing the words "CEO" and "Twitter." In a real-world, evolving knowledge base, this is a recipe for disaster.

The top-k=3 chunks our retriever finds will be a mess of contradictions:

1. "In 2023, Elon Musk became the CEO of Twitter." (Correct, but old)

2. "In 2022, Parag Agrawal was the CEO of Twitter." (Old)

3. "Linda Yaccarino is the current CEO of X (formerly Twitter)..." (Also correct, but a different person/role).

The LLM is handed three different, conflicting names for "CEO of Twitter" from different time periods. Because it is instructed to answer only from the context and has no mechanism to identify which fact is the most recent, it cannot give a single, confident answer. It’s forced to list the conflicts it found.

Why the GraphRAG Succeeds

The knowledge graph is built for this. We've stored the "CEO" relationship as an edge with metadata, specifically a year attribute.

Our query_with_conflict_resolution function doesn't just find all CEO-related edges. It programmatically:

1. Finds all nodes connected to "Twitter" by a ceo label.

2. Extracts the year from each of those edges.

3. Sorts the candidates by year in descending order.

4. Returns only the top result.

This provides a deterministic, programmatic way to resolve conflicts and always provide the most current fact based on the explicit timestamps in our graph.

# Naive RAG
docs_s4 = [
    # --- The "Answer" Chunks (conflicting) ---
    Document(page_content="In 2022, Parag Agrawal was the CEO of Twitter."),
    Document(page_content="In 2023, Elon Musk became the CEO of Twitter."),

    # --- "Noise" Chunks (to add more conflict/confusion) ---
    Document(page_content="Linda Yaccarino is the current CEO of X (formerly Twitter), overseeing business operations."),
    Document(page_content="Jack Dorsey, a co-founder and former CEO of Twitter, is now focused on his company Block."),
    Document(page_content="CEOs of major tech companies, including Twitter's, have recently testified before Congress.")
]
query_s4 = "Who is the CEO of Twitter?"
rag_chain_s4 = create_rag_chain(docs_s4)
print(f"Naive RAG Answer: {rag_chain_s4.invoke(query_s4)}")

#GraphRAG Pattern
graph_s4 = KnowledgeGraph()
edges_s4 = [
    ("Twitter", "Parag Agrawal", {"label": "ceo", "year": 2022}),
    ("Twitter", "Elon Musk", {"label": "ceo", "year": 2023}),
]
graph_s4.add_data(edges=edges_s4)
print(f"GraphRAG Answer: {graph_s4.query_with_conflict_resolution('Twitter', 'ceo', 'year')}")

Output:

Naive RAG Answer: According to the context, in 2022, Parag Agrawal was the CEO of Twitter. In 2023, Elon Musk became the CEO... Linda Yaccarino is the current CEO of X (formerly Twitter)...
GraphRAG Answer: Elon Musk (as of 2023)

Pattern 5: The Implicit Relationship Hallucination

RAG relies on implicit semantic closeness, which can be dangerous. If "Tesla," "Toyota," and "Panasonic" all appear near the word "battery" in the vector space, the LLM might hallucinate a relationship that doesn't exist.

• Query: "Who did Tesla partner with on batteries?"

• Problem: The query is semantically "close" to any document mentioning "Tesla," "partner," and "batteries." The retriever will fetch chunks based on this closeness, even if they don't explicitly state a partnership, leading the LLM to infer one.

Why the Naïve RAG Fails

The vector retriever will look for chunks that "sound" like the query. In our expanded document list, it's highly likely to retrieve a confusing context for the LLM.

The top-k=3 chunks it finds will likely be:

1. "Panasonic has a long-standing partnership to manufacture batteries..." (Relevant: "Panasonic," "partnership," "batteries")

2. "Tesla develops electric vehicles and relies on advanced battery tech..." (Relevant: "Tesla," "battery")

3. "Toyota also manufactures batteries and has discussed battery technology..." (Relevant: "Toyota," "manufactures batteries")

When the LLM receives this context, it has "Panasonic," "Tesla," and "Toyota" all in a "battery" context. The chunk for Panasonic doesn't explicitly link it to Tesla. The chunk for Toyota also mentions batteries. The LLM, forced to synthesize an answer, may incorrectly infer a partnership that doesn't exist (like with Toyota) or state the facts without confirming the relationship.

Why the GraphRAG Succeeds

The knowledge graph isn’t vulnerable to this kind of "semantic bleed-over." It doesn’t care if nodes are "semantically near" each other.

Our query_explicit_relation function asks a very specific, structural question: "Start at the node 'Tesla' and return only the nodes connected to it by an edge with the exact label 'partners_with'".

The graph then traverses its edges and finds only one: ("Tesla", "Panasonic", {"label": "partners_with"}). It is structurally impossible for it to hallucinate a partnership with "Toyota" because no such partners_with edge exists for Tesla in the graph.

# Naive RAG
docs_s5 = [
    # --- The "Answer" Chunks (ambiguous) ---
    Document(page_content="Tesla develops electric vehicles and relies on advanced battery tech."),
    Document(page_content="Panasonic has a long-standing partnership to manufacture batteries for electric vehicles."),

    # --- "Noise" Chunks (to create a false signal) ---
    Document(page_content="Toyota also manufactures batteries and hybrid powertrains for its own vehicle lineup."),
    Document(page_content="Tesla, Panasonic, and Toyota are all major players in the EV and battery supply chain."),
    Document(page_content="A new partnership for solid-state batteries was announced, but it did not involve Tesla.")
]
query_s5 = "Who did Tesla partner with on batteries?"
rag_chain_s5 = create_rag_chain(docs_s5)
print(f"Naive RAG Answer: {rag_chain_s5.invoke(query_s5)}")
#
# GraphRAG Pattern
graph_s5 = KnowledgeGraph()
edges_s5 = [
    ("Tesla", "Panasonic", {"label": "partners_with"}),
    ("Toyota", "Toyota", {"label": "partners_with"}),
]
graph_s5.add_data(edges=edges_s5)
print(f"GraphRAG Answer: {graph_s5.query_explicit_relation('Tesla', 'partners_with')}")

Output:

Naive RAG Answer: Based on the context, Panasonic has a partnership to manufacture batteries, and Tesla relies on advanced battery tech. Toyota also manufactures batteries.
GraphRAG Answer: Tesla partnered with Panasonic.

Final Thoughts

Standard RAG is an essential tool, but its strength is retrieval, not reasoning. It falters when true synthesis is required.

You may find that a powerful LLM like Gemini can still correctly answer some of the simple scenarios in this article. The five patterns shown here are meant to build intuition. They demonstrate what can and does go wrong as your knowledge base grows larger and more complex.

The real failure of naive RAG emerges as you feed it more and more conflicting, ambiguous, or incomplete information. This "noisy" context forces the LLM to either hallucinate connections or fail to reason altogether.

By moving from a "bag of chunks" to a structured Knowledge Graph, you build a more reliable and intelligent system. You give your system a "global memory" of how facts explicitly connect, allowing it to answer complex questions by traversing a verified path rather than just guessing from a cloud of semantically similar text.

In this tutorial, I’ll walk you through the process step-by-step. You don’t need prior experience with machine learning models, as we’ll build up the system gradually and test each part as we go. By the end, you will have a fully local mobile voice assistant powered by:

• Whisper for Automatic Speech Recognition (ASR)

• Machine Learning Compiler (MLC) LLM for on-device reasoning

• System Text-to-Speech (TTS) using built-in Android TTS

Your assistant will be able to:

• Understand your voice commands offline

• Respond to you with synthesized speech

• Perform tool calling actions (such as controlling smart devices)

• Store personal memories and preferences

• Use Retrieval-Augmented Generation (RAG) to answer questions from your own notes

• Perform multi-step agentic workflows such as generating a morning briefing and optionally sending the summary to a contact

This tutorial focuses on Android using Termux (the terminal environment for Android) for a fully local workflow.

Table of Contents

• System Overview

• Requirements

• Step 1: Test Microphone and Audio Playback on Android

• Step 2: Install and Run Whisper for ASR

• Step 3: Install a Local LLM with MLC

• Step 4: Local Text-to-Speech (TTS)

• Step 5: The Core Voice Loop

• Step 6: Tool Calling (Make It Act)

• Step 7: Memory and Personalization

• Step 8: Retrieval-Augmented Generation (RAG)

• Step 9: Multi-Step Agentic Workflow

• Conclusion and Next Steps

System Overview

This diagram shows how your voice moves through the assistant: speech in → transcription → reasoning → action → spoken reply.

This pipeline describes the core flow:

• You speak into the microphone.

• Whisper converts audio into text.

• The local LLM interprets your request.

• The assistant may call tools (for example, send notifications or create events).

• The response is spoken aloud using the device’s Text-to-Speech system.

Key Concepts Used in This Tutorial

• Automatic Speech Recognition (ASR): Converts your speech into text. We use Whisper or Faster‑Whisper.

• Local Large Language Model (LLM): A reasoning model running on your phone using the MLC engine.

• Text‑to‑Speech (TTS): Converts text back to speech. We use Android’s built‑in system TTS.

• Tool Calling: Allows the assistant to perform actions (for example, sending a notification or creating an event).

• Memory: Stores personalized facts the assistant learns during conversation.

• Retrieval‑Augmented Generation (RAG): Lets the assistant reference your documents or notes.

• Agent Workflow: A multi‑step chain where the assistant uses multiple abilities together.

Requirements

What you should already be familiar with:

• Basic command line usage (running commands, navigating directories)

• Very basic Python (calling a function, editing a .py script)

You do not need to have:

• Machine learning experience

• A deep understanding of neural networks

• Prior experience with speech or audio models

Here are the tools and technologies you’ll need to follow along:

• An Android phone with Snapdragon 8+ Gen 1 or newer recommended (older devices will still work, but responses may be slower)

• Termux

• Python 3.9+ inside Termux

• Enough free storage (at least 4–6 GB) to store the model and audio files

Why these requirements matter:

Whisper and Llama models run on-device, so the phone must handle real‑time compute. MLC optimizes models for your device's GPU / NPU, so newer processors will run faster and cooler. And system TTS and Termux APIs let the assistant speak and interact with the phone locally.

If your phone is older or mid‑range, switch the model in Step 3 to Phi-3.5-Mini which is smaller and faster.

We’ll start by setting up your Android environment with Termux, Python, media access, and storage permissions so later steps can record audio, run models, and speak.

Run it now:

# In Termux
pkg update && pkg upgrade -y
pkg install -y python git ffmpeg termux-api
termux-setup-storage  # grant storage permission

Step 1: Test Microphone and Audio Playback on Android

What this step does: Verifies that your device microphone and speakers work correctly through Termux before connecting them to the voice assistant.

On-device assistants need reliable access to the microphone and speakers. On Android, Termux provides utilities to record audio and play media. This avoids complex audio dependencies and works on more devices.

These commands let you quickly test your microphone and audio playback without writing any code. This is useful to verify that your device permissions and audio paths are working before introducing Whisper or TTS.

• termux-microphone-record records from the device microphone to a .wav file

• termux-media-player plays audio files

• termux-tts-speak speaks text using the system TTS voice (fast fallback)

Run it now:

# Start a 4 second recording
termux-microphone-record -f in.wav -l 4 && termux-microphone-record -q

# Play back the captured audio
termux-media-player play in.wav

# Speak text via system TTS (fallback if you do not install a Python TTS)
termux-tts-speak "Hello, this is your on-device assistant running locally."

Step 2: Install and Run Whisper for ASR

What this step does: Converts recorded speech into text so the language model can understand what you said.

Whisper listens to your audio recording and converts it into text. Smaller versions like tiny or base run faster on most phones and are good enough for everyday commands.

Install Whisper:

pip install openai-whisper

If you run into installation issues, you can use Faster‑Whisper instead:

pip install faster-whisper

Below is a small Python script that takes the recorded audio file and turns it into text. It tries Whisper first, and if that isn’t available, it will automatically fall back to Faster‑Whisper.

# Convert recorded speech to text (asr_transcribe.py)
import sys

# Try Whisper, fallback to Faster-Whisper if needed
try:
    import whisper
    use_faster = False
except Exception:
    use_faster = True

if use_faster:
    from faster_whisper import WhisperModel
    model = WhisperModel("tiny.en")
    segments, info = model.transcribe(sys.argv[1])
    text = " ".join(s.text for s in segments)
    print(text.strip())
else:
    model = whisper.load_model("tiny.en")
    result = model.transcribe(sys.argv[1], fp16=False)
    print(result["text"].strip())

Run it now:

# Record 4 seconds and transcribe
termux-microphone-record -f in.wav -l 4 && termux-microphone-record -q
python asr_transcribe.py in.wav

Step 3: Install a Local LLM with MLC

What this step does: Installs and tests the on-device reasoning model that will generate responses to transcribed speech.

MLC compiles transformer models to mobile GPUs and Neural Processing Units, enabling on-device inference. You will run an instruction-tuned model with 4-bit or 8-bit weights for speed.

Install the command-line interface like this:

# Clone and install Python bindings (for scripting) and CLI
git clone https://github.com/mlc-ai/mlc-llm.git
cd mlc-llm
pip install -r requirements.txt
pip install -e python

We will use Llama 3 8B Instruct q4 because it offers strong reasoning while still running on many recent Android devices. If your phone has less memory or you want faster responses, you can swap in Phi-3.5 Mini (about 3.8B) without changing any code.

Download a mobile-optimized model:

mlc_llm download Llama-3-8B-Instruct-q4f16_1

We will use a short Python script to send text to the model and print the response. This lets us verify that the model is installed correctly before we connect it to audio.

# Local LLM text generation (local_llm.py)
from mlc_llm import MLCEngine
import sys

engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")
prompt = sys.argv[1] if len(sys.argv) > 1 else "Hello"
resp = engine.chat([{"role": "user", "content": prompt}])
# The engine may return different structures across versions
reply_text = resp.get("message", resp) if isinstance(resp, dict) else str(resp)
print(reply_text)

Run it now:

python local_llm.py "Summarize this in one sentence: building a local voice assistant on Android"

Step 4: Local Text-to-Speech (TTS)

What this step does: Turns the model’s text responses into spoken audio so the assistant can talk back.

This step converts the text returned by the model into spoken audio so the assistant can talk back. It uses the built-in Android Text-to-Speech voice and requires no additional Python packages.

termux-tts-speak "Hello, I am running entirely on your device."

This is the voice output method we will use throughout the tutorial.

Step 5: The Core Voice Loop

What this step does: Connects speech recognition, language model reasoning, and speech synthesis into a single interactive conversation loop.

This loop ties together recording, transcription, response generation, and playback.

# Core voice loop tying ASR + LLM + TTS (voice_loop.py)
import subprocess, os

def run(cmd): return subprocess.check_output(cmd).decode().strip()

print("Listening...")
subprocess.run(["termux-microphone-record", "-f", "in.wav", "-l", "4"]) ; subprocess.run(["termux-microphone-record", "-q"])
text = run(["python", "asr_transcribe.py", "in.wav"])
reply = run(["python", "local_llm.py", text])
try:
    subprocess.run(["python", "speak_xtts.py", reply]); subprocess.run(["termux-media-player", "play", "out.wav"])
except:
    subprocess.run(["termux-tts-speak", reply])

Run:

python voice_loop.py

Step 6: Tool Calling (Make It Act)

What this step does: Enables the assistant to perform actions – not just reply – by calling real functions on your device.

Tool calling lets the assistant perform actions, not just answer. When the model recognizes an action request, it outputs a small JSON instruction, and your code runs the corresponding function. You show the model which tools exist and how to call them. The program intercepts calls and runs the corresponding code.

Example use case:

You say: "Schedule a meeting tomorrow at 3 PM with John."

The assistant:

1. Transcribes what you said.

2. Detects that this is not a question, but an action request.

3. Calls the add_event() function with the correct parameters.

4. Confirms: "Okay, I scheduled that."

Here’s the structure of how tool calls will work:

• Define Python functions such as add_event, control_light

• Provide a schema for the model to output when it wants to call a tool

• Detect that schema in the LLM output and execute the function

# Tool calling functions (tools.py)
import json

def add_event(title: str, date: str) -> dict:
    # Replace with actual calendar integration
    return {"status": "ok", "title": title, "date": date}

TOOLS = {
    "add_event": add_event,
}

def run_tool(call_json: str) -> str:
    """call_json: '{"tool":"add_event","args":{"title":"Dentist","date":"2025-11-10 10:00"}}'"""
    data = json.loads(call_json)
    name = data["tool"]
    args = data.get("args", {})
    if name in TOOLS:
        result = TOOLS[name](**args)
        return json.dumps({"tool_result": result})
    return json.dumps({"error": "unknown tool"})

Prompt the model to use tools:

# LLM wrapper enabling tool use (llm_with_tools.py)
from mlc_llm import MLCEngine
import json, sys

SYSTEM = (
    "You can call tools by emitting a single JSON object with keys 'tool' and 'args'. "
    "Available tools: add_event(title:str, date:str). "
    "If no tool is needed, answer directly."
)

engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")
user = sys.argv[1]
resp = engine.chat([
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": user},
])
print(resp.get("message", resp) if isinstance(resp, dict) else str(resp))

And then glue it together:

# Run LLM with tool call detection (run_with_tools.py)
import subprocess, json
from tools import run_tool

user = "Add a dentist appointment next Thursday at 10"
raw = subprocess.check_output(["python", "llm_with_tools.py", user]).decode().strip()

# If the model returned a JSON tool call, run it
try:
    data = json.loads(raw)
    if isinstance(data, dict) and "tool" in data:
        print("Tool call:", data)
        print(run_tool(raw))
    else:
        print("Assistant:", raw)
except Exception:
    print("Assistant:", raw)

Run it now:

python run_with_tools.py

Step 7: Memory and Personalization

What this step does: Allows the assistant to remember personal information you share so conversations feel continuous and adaptive.

A helpful assistant should feel like it learns alongside you. Memory allows the system to keep track of small details you mention naturally in conversation.

Without memory, every conversation starts from scratch. With memory, your assistant can remember personal facts (for example, birthdays, favorite music), your routines, device settings, or notes you mention in conversation. This unlocks more natural interactions and enables personalization over time.

You can start with a simple key-value store and expand over time. Your program reads memory before inference and writes back new facts after.

# Simple key-value memory store (memory.py)
import json
from pathlib import Path

MEM_PATH = Path("memory.json")

def mem_load():
    return json.loads(MEM_PATH.read_text()) if MEM_PATH.exists() else {}

def mem_save(mem):
    MEM_PATH.write_text(json.dumps(mem, indent=2))

def remember(key: str, value: str):
    mem = mem_load()
    mem[key] = value
    mem_save(mem)

Use memory in the loop:

# Voice loop with memory loading and updating (voice_loop_with_memory.py)
import subprocess, json
from memory import mem_load, remember

# 1) Record and transcribe
subprocess.run(["termux-microphone-record", "-f", "in.wav", "-l", "4"]) 
subprocess.run(["termux-microphone-record", "-q"]) 
user_text = subprocess.check_output(["python", "asr_transcribe.py", "in.wav"]).decode().strip()

# 2) Load memory and add as system context
mem = mem_load()
SYSTEM = "Known facts: " + json.dumps(mem)

# 3) Ask the model
from mlc_llm import MLCEngine
engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")
resp = engine.chat([
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": user_text},
])
reply = resp.get("message", resp) if isinstance(resp, dict) else str(resp)
print("Assistant:", reply)

# 4) Very simple pattern: if the user said "remember X is Y", store it
if user_text.lower().startswith("remember ") and " is " in user_text:
    k, v = user_text[9:].split(" is ", 1)
    remember(k.strip(), v.strip())

Run it now:

python voice_loop_with_memory.py

Step 8: Retrieval-Augmented Generation (RAG)

What this step does: Lets the assistant search your offline notes or documents at answer time, improving accuracy for personal tasks.

To use RAG, we first install a lightweight vector database, then add documents to it, and later query it when answering questions.

A language model cannot magically know details about your life, your work, or your files unless you give it a way to look things up.

Retrieval-Augmented Generation (RAG) bridges that gap. RAG allows the assistant to search your own stored data at query time. This means the assistant can answer questions about your projects, home details, travel plans, studies, or any personal documents you store completely offline.

RAG allows the assistant to reference your actual notes when answering, instead of relying only on the model's internal training.

Install the vector store:

pip install chromadb

Add and search your notes:

# Local vector DB indexing and querying (rag.py)
from chromadb import Client

client = Client()
notes = client.create_collection("notes")

# Add your documents (repeat as needed)
notes.add(documents=["Contractor quote was 42000 United States Dollars for the extension."], ids=["q1"]) 

# Query the local vector database
results = notes.query(query_texts=["extension quote"], n_results=1)
context = results["documents"][0][0]
print(context)

Use retrieved context in responses:

# LLM answering using retrieved context (llm_with_rag.py)
from mlc_llm import MLCEngine
from chromadb import Client

engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")
client = Client()
notes = client.get_or_create_collection("notes")

question = "What was the quoted amount for the home extension?"
res = notes.query(query_texts=[question], n_results=2)
ctx = "\n".join([d[0] for d in res["documents"]])

SYSTEM = "Use the provided context to answer accurately. If missing, say you do not know.\nContext:\n" + ctx
ans = engine.chat([
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": question},
])
print(ans.get("message", ans) if isinstance(ans, dict) else str(ans))

Run it now:

python rag.py
python llm_with_rag.py

Step 9: Multi-Step Agentic Workflow

What this step does: Combines listening, reasoning, memory, and tool usage into a multi-step routine that runs automatically.

Now that the assistant can listen, respond, remember facts, and call tools, we can combine those abilities into a small routine that performs several steps automatically.

Practical example: "Morning Briefing" on your phone

Goal: when you say "Give me my morning briefing and text it to my partner", the assistant will:

1. Read today's agenda from a local file,

2. summarize it,

3. speak it aloud, and

4. send the summary via SMS using Termux.

Diagram: Multi-step morning briefing workflow with retrieval, summary, speech output, and SMS action.

Prepare your agenda file

This file stores your events for the day. You can edit it manually, generate it, or sync it later if you want.

Create agenda.json in the same folder:

{
  "2025-11-03": [
    {"time": "09:30", "title": "Standup meeting"},
    {"time": "13:00", "title": "Lunch with Priya"},
    {"time": "16:30", "title": "Gym"}
  ]
}

Phone-integrated tools for this workflow:

# Phone-integrated agent tools (tools_phone.py)
import json, subprocess, datetime
from pathlib import Path

AGENDA_PATH = Path("agenda.json")

def load_today_agenda():
    today = datetime.date.today().isoformat()
    if not AGENDA_PATH.exists():
        return []
    data = json.loads(AGENDA_PATH.read_text())
    return data.get(today, [])

def send_sms(number: str, text: str) -> dict:
    # Requires Termux:API and SMS permission
    subprocess.run(["termux-sms-send", "-n", number, text])
    return {"status": "sent", "to": number}

def notify(title: str, content: str) -> dict:
    subprocess.run(["termux-notification", "--title", title, "--content", content])
    return {"status": "notified"}

Create the agent routine:

# Multi-step morning briefing agent (agent_morning.py)
import json, subprocess, os
from mlc_llm import MLCEngine
from tools_phone import load_today_agenda, send_sms, notify

PARTNER_PHONE = os.environ.get("PARTNER_PHONE", "+15551234567")

TOOLS = {
    "send_sms": send_sms,
    "notify": notify,
}

SYSTEM = (
  "You assist on a phone. You may emit a single-line JSON when an action is needed "
  "with keys 'tool' and 'args'. Available tools: send_sms(number:str, text:str), "
  "notify(title:str, content:str). Keep messages concise. If no tool is needed, answer in plain text."
)

engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")

agenda = load_today_agenda()
agenda = load_today_agenda()
agenda_text = "
".join(f"{e['time']} - {e['title']}" for e in agenda) or "No events for today."

user_request = "Give me my morning briefing and text it to my partner." "Give me my morning briefing and text it to my partner."

# 1) Ask LLM for a 2-3 sentence summary to speak
summary = engine.chat([
  {"role": "system", "content": "Summarize this agenda in 2-3 sentences for a morning briefing:"},
  {"role": "user", "content": agenda_text},
])
summary_text = summary.get("message", summary) if isinstance(summary, dict) else str(summary)
print("Briefing:
", summary_text)

# 2) Speak locally (prefer XTTS, fallback to system TTS)
try:
    subprocess.run(["python", "speak_xtts.py", summary_text], check=True)
    subprocess.run(["termux-media-player", "play", "out.wav"]) 
except Exception:
    subprocess.run(["termux-tts-speak", summary_text])

# 3) Ask LLM whether to send SMS and with what text, using tool schema
resp = engine.chat([
  {"role": "system", "content": SYSTEM},
  {"role": "user", "content": f"User said: '{user_request}'. Partner phone is {PARTNER_PHONE}. Summary: {summary_text}"},
])
msg = resp.get("message", resp) if isinstance(resp, dict) else str(resp)

# 4) If the model requested a tool, execute it
try:
    data = json.loads(msg)
    if isinstance(data, dict) and data.get("tool") in TOOLS:
        # Auto-fill phone number if missing
        if data["tool"] == "send_sms" and "number" not in data.get("args", {}):
            data.setdefault("args", {})["number"] = PARTNER_PHONE
        result = TOOLS[data["tool"]](**data.get("args", {}))
        print("Tool result:", result)
    else:
        print("Assistant:", msg)
except Exception:
    print("Assistant:", msg)

Run it now:

export PARTNER_PHONE=+15551234567
python agent_morning.py

This example is realistic on Android because it uses Termux utilities you already installed: local TTS for speech output, termux-sms-send for messaging, and termux-notification for a quick on-device confirmation. You can extend it with a Home Assistant tool later if you have a local server (for example, to toggle lights or set thermostat scenes).

Conclusion and Next Steps

Building a fully local voice assistant is an incremental process. Each step you added – speech recognition, text generation, memory, retrieval, and tool execution – unlocked new capabilities and moved the system closer to behaving like a real assistant.

You built a fully local voice assistant on your phone with:

• On-device Automatic Speech Recognition with Whisper (with Faster-Whisper fallback)

• On-device reasoning with MLC Large Language Model

• Local Text-to-Speech using the built-in system TTS

• Tool calling for real actions

• Memory and personalization

• Retrieval-Augmented Generation for document-based knowledge

• A simple agent loop for multi-step work

From here you can add:

• Wake word detection (for example, Porcupine or open wake word models)

• Device-specific integrations (for example, Home Assistant, smart lighting)

• Better memory schemas and calendars or contacts adapters

Your data never leaves your device, and you control every part of the stack. This is a private, customizable assistant you can expand however you like.

In this guide, you'll build a RAG-based AI agent in TypeScript using Langbase SDK. You'll plug in your own data as memory, use any embedding model, retrieve relevant context, and call an LLM to generate a precise response.

By the end of this tutorial, you'll have a working RAG system that:

• Stores and retrieves documents with semantic memory

• Uses custom embeddings for vector search

• Handles user queries with relevant context

• Generates responses via OpenAI, Anthropic, or any LLM

Table of Contents

• Prerequisites

• What is agentic RAG?

• Langbase SDK

• Step 1: Setup your project

• Step 2: Get Langbase API Key

• Step 3: Add LLM API keys

• Step 4: Create an agentic AI memory

• Step 5: Add documents to AI memory

• Step 6: Perform RAG retrieval

• Step 7: Create support pipe agent

• Step 8: Generate RAG responses

• The result

Prerequisites

Before we begin creating a RAG-based AI agent, you’ll need to have some tools ready to go.

In this tutorial, I’ll be using the following tech stack:

• Langbase – the platform to build and deploy your serverless AI agents.

• Langbase SDK – a TypeScript AI SDK, designed to work with JavaScript, TypeScript, Node.js, Next.js, React, and the like.

• OpenAI – to get the LLM key for the preferred model.

You’ll also need to:

• Sign up on Langbase to get access to the API key.

• Sign up on OpenAI to generate the LLM key for the model you want to use (for this demo, I’ll be using the openai:text-embedding-3-large model). You can generate the key here.

What is Agentic RAG?

Retrieval augmented generation (RAG) is an architecture for optimizing the performance of an artificial intelligence (AI) model by connecting it with external knowledge bases. RAG helps large language models (LLMs) deliver more relevant responses at a higher quality.

When we use AI agents to facilitate RAG, it becomes Agentic RAG. Agentic RAG systems add AI agents to the RAG pipeline to increase adaptability and accuracy. Compared to traditional RAG systems, agentic RAG allows LLMs to conduct information retrieval from multiple sources and handle more complex workflows.

Here’s the tabular comparison of RAG vs Agentic RAG:

Langbase SDK

The Langbase SDK makes it easy to build powerful AI tools using TypeScript. It gives you everything you need to work with any LLM, connect your own embedding models, manage document memory, and build AI agents that can reason and respond.

The SDK is designed to work with Node.js, Next.js, React, or any modern JavaScript stack. You can use it to upload documents, create semantic memory, and run AI workflows (called Pipe agents) with just a few lines of code.

Langbase is an API-first AI platform. Its TypeScript SDK smooths out the experience, making it easy to get started without dealing with infrastructure. Just drop in your API key, write your logic, and you're good to go.

Now that you know about Langbase SDK, let’s start building the RAG agent.

Step 1: Setup Your Project

We’ll be building a basic Node.js app in TypeScript that uses the Langbase SDK to create an agentic RAG system. For that, create a new directory for your project and navigate to it.

mkdir agentic-rag && cd agentic-rag

Then initialize a Node.js project and create different TypeScript files by running this command in your terminal:

npm init -y && touch index.ts agents.ts create-memory.ts upload-docs.ts create-pipe.ts

Here’s a breakdown of what each file will do in the project:

• index.ts: This is typically the entry point of a TypeScript project. It orchestrates agent creation, memory setup, and document upload.

• agents.ts: This file handles AI agent creation and configuration.

• create-memory.ts: This sets up Langbase Memory (RAG) for storing and retrieving context.

• upload-docs.ts: This file will upload documents to Memory so agents can access and use them.

• create-pipe.ts: This file sets up a Langbase Pipe agent which is a serverless AI agent with unified APIs for every LLM.

After this, we will be using the Langbase SDK to create RAG agents and dotenv to manage environment variables. So, let's install these dependencies.

npm i langbase dotenv

Step 2: Get Langbase API Key

Every request you send to Langbase needs an API key. You can generate API keys from the Langbase studio by following these steps:

1. Switch to your user or org account.

2. From the sidebar, click on the Settings menu.

3. In the developer settings section, click on the Langbase API keys link.

4. From here you can create a new API key or manage existing ones.

For more details, check out the Langbase API keys documentation.

After generating the API key, create an .env file in the root of your project and add your Langbase API key in it:

LANGBASE_API_KEY=xxxxxxxxx

Replace xxxxxxxxx with your Langbase API key.

Step 3: Add LLM API Keys

Once you have the Langbase API key, you’ll need the LLM key as well to run the RAG agent. If you have set up LLM API keys in your profile, the AI memory and agent pipe will automatically use them. Otherwise, navigate to the LLM API keys page and add keys for different providers like OpenAI, Anthropic, and so on.

Follow these steps to add the LLM keys:

1. Add LLM API keys in your account using Langbase studio

2. Switch to your user or org account.

3. From the sidebar, click on the Settings menu.

4. In the developer settings section, click on the LLM API keys link.

5. From here you can add LLM API keys for different providers like OpenAI, TogetherAI, Anthropic, and so on.

Step 4: Create an Agentic AI Memory

Let’s now use the Langbase SDK to create an AI memory (Langbase memory agent) where your agent can store and retrieve context.

Langbase serverless memory agents (long-term memory solution) are designed to acquire, process, retain, and retrieve information seamlessly. They dynamically attach private data to any LLM, enabling context-aware responses in real time and reducing hallucinations.

These agents combine vector storage, RAG, and internet access to create a powerful managed context search API. You can use them to build smarter, more capable AI applications.

In a RAG setup, memory – when connected directly to a Langbase Pipe Agent – becomes a memory agent. This pairing gives the LLM the ability to fetch relevant data and deliver precise, contextually accurate answers – addressing the limitations of LLMs when it comes to handling private data.

To create it, add the following code to create-memory.ts file you created in Step 1:

import 'dotenv/config';
import {Langbase} from 'langbase';

const langbase = new Langbase({
    apiKey: process.env.LANGBASE_API_KEY!,
});

async function main() {
    const memory = await langbase.memories.create({
        name: 'knowledge-base',
        description: 'An AI memory for agentic memory workshop',
        embedding_model: 'openai:text-embedding-3-large'
    });

    console.log('AI Memory:', memory);
}

main();

Here’s what’s happening in the above code:

• Import the dotenv package to load environment variables.

• Import the Langbase class from the langbase package.

• Create a new instance of the Langbase class with your API key.

• Use the memories.create method to create a new AI memory.

• Set the name and description of the memory.

• Use the openai:text-embedding-3-large model for embedding.

• Log the created memory to the console.

After this, let's create the agentic memory by running the create-memory.ts file.

npx tsx create-memory.ts

This will create an AI memory and log the memory details to the console.

Step 5: Add Documents to AI Memory

Now that you’ve created an AI memory agent, the next step is to add documents in it. These documents will serve as the context your agent can reference during interactions.

First, create a docs directory in your project root, and add two sample text files:

• agent-architectures.txt

• langbase-faq.txt

Next, open the upload-docs.ts file created in Step 1 and paste in the following code:

import 'dotenv/config';
import { Langbase } from 'langbase';
import { readFile } from 'fs/promises';
import path from 'path';

const langbase = new Langbase({
    apiKey: process.env.LANGBASE_API_KEY!,
});

async function main() {
    const cwd = process.cwd();
    const memoryName = 'knowledge-base';

    // Upload agent architecture document
    const agentArchitecture = await readFile(path.join(cwd, 'docs', 'agent-architectures.txt'));
    const agentResult = await langbase.memories.documents.upload({
        memoryName,
        contentType: 'text/plain',
        documentName: 'agent-architectures.txt',
        document: agentArchitecture,
        meta: { category: 'Examples', topic: 'Agent architecture' },
    });

    console.log(agentResult.ok ? '✓ Agent doc uploaded' : '✗ Agent doc failed');

    // Upload FAQ document
    const langbaseFaq = await readFile(path.join(cwd, 'docs', 'langbase-faq.txt'));
    const faqResult = await langbase.memories.documents.upload({
        memoryName,
        contentType: 'text/plain',
        documentName: 'langbase-faq.txt',
        document: langbaseFaq,
        meta: { category: 'Support', topic: 'Langbase FAQs' },
    });

    console.log(faqResult.ok ? '✓ FAQ doc uploaded' : '✗ FAQ doc failed');
}

main();

Let’s break down what’s happening in this code:

• dotenv/config is used to load environment variables from your .env file.

• Langbase is imported from the SDK to interact with the API.

• readFile from the fs/promises module reads each document file asynchronously.

• path.join() ensures file paths work across different operating systems.

• A Langbase client instance is created using your API key.

• memories.documents.upload is used to upload each .txt file to the AI memory.

• Each upload includes metadata like category and topic to help organize the content.

• Upload success or failure is logged to the console.

This step ensures your AI agent will have actual content to pull from – FAQs, architecture docs, or anything else you upload into memory.

Then, run the upload-docs.ts file to upload the documents to the AI memory by this command in your terminal. This will upload the documents to the AI memory:

npx tsx upload-docs.ts

Step 6: Perform RAG Retrieval

In this step, we’ll perform RAG against a query using the documents we uploaded to AI memory.

Add the following code to your agents.ts file created in Step 1:

import 'dotenv/config';
import { Langbase } from 'langbase';

const langbase = new Langbase({
    apiKey: process.env.LANGBASE_API_KEY!,
});

export async function runMemoryAgent(query: string) {
    const chunks = await langbase.memories.retrieve({
        query,
        topK: 4,
        memory: [
            {
                name: 'knowledge-base',
            },
        ],
    });

    return chunks;
}

Let’s break down what this does:

• Import the Langbase class from the Langbase SDK.

• Initialize the Langbase client using your API key from environment variables.

• Define an async function runMemoryAgent that takes a query string as input.

• Use memories.retrieve to query the memory for the most relevant chunks, retrieving the top 4 results (topK: 4) from the memory named "knowledge-base".

• Return the retrieved memory chunks.

Now let's add the following code to the index.ts file created in Step 1 to run the memory agent:

import { runMemoryAgent } from './agents';

async function main() {
    const chunks = await runMemoryAgent('What is agent parallelization?');
    console.log('Memory chunk:', chunks);
}

main();

This code runs a Langbase memory query for “What is agent parallelization?” It uses runMemoryAgent to retrieve the top matching chunks from your AI memory and logs the results. It’s how you fetch relevant knowledge with RAG.

After this, run the index.ts file to perform RAG retrieval against the query by this command in terminal:

npx tsx index.ts

You will see the memory agent output as retrieved memory chunks in the console as follows:

[
  {
    text: '---\n' +
      '\n' +
      '## Agent Parallelization\n' +
      '\n' +
      'Parallelization runs multiple LLM tasks at the same time to improve speed or accuracy. It works by splitting a task into independent parts (sectioning) or generating multiple responses for comparison (voting).\n' +
      '\n' +
      'Voting is a parallelization method where multiple LLM calls generate different responses for the same task. The best result is selected based on agreement, predefined rules, or quality evaluation, improving accuracy and reliability.\n' +
      '\n' +
      "`This code implements an email analysis system that processes incoming emails through multiple parallel AI agents to determine if and how they should be handled. Here's the breakdown:",
    similarity: 0.7146744132041931,
    meta: {
      docName: 'agent-architectures.txt',
      documentName: 'agent-architectures.txt',
      category: 'Examples',
      topic: 'Agent architecture'
    }
  },
  {
    text: 'async function main(inputText: string) {\n' +
      '\ttry {\n' +
      '\t\t// Create pipes first\n' +
      '\t\tawait createPipes();\n' +
      '\n' +
      '\t\t// Step A: Determine which agent to route to\n' +
      '\t\tconst route = await routerAgent(inputText);\n' +
      "\t\tconsole.log('Router decision:', route);\n" +
      '\n' +
      '\t\t// Step B: Call the appropriate agent\n' +
      '\t\tconst agent = agentConfigs[route.agent];\n' +
      '\n' +
      '\t\tconst response = await langbase.pipes.run({\n' +
      '\t\t\tstream: false,\n' +
      '\t\t\tname: agent.name,\n' +
      '\t\t\tmessages: [\n' +
      "\t\t\t\t{ role: 'user', content: `${agent.prompt} ${inputText}` }\n" +
      '\t\t\t]\n' +
      '\t\t});\n' +
      '\n' +
      '\t\t// Final output\n' +
      '\t\tconsole.log(\n' +
      '\t\t\t`Agent: ${agent.name} \\n\\n Response: ${response.completion}`\n' +
      '\t\t);\n' +
      '\t} catch (error) {\n' +
      "\t\tconsole.error('Error in main workflow:', error);\n" +
      '\t}\n' +
      '}\n' +
      '\n' +
      '// Example usage:\n' +
      "const inputText = 'Why days are shorter in winter?';\n" +
      '\n' +
      'main(inputText);\n' +
      '```\n' +
      '\n' +
      '\n' +
      '---\n' +
      '\n' +
      '## Agent Parallelization\n' +
      '\n' +
      'Parallelization runs multiple LLM tasks at the same time to improve speed or accuracy. It works by splitting a task into independent parts (sectioning) or generating multiple responses for comparison (voting).',
    similarity: 0.5911030173301697,
    meta: {
      docName: 'agent-architectures.txt',
      documentName: 'agent-architectures.txt',
      category: 'Examples',
      topic: 'Agent architecture'
    }
  },
  {
    text: "`This code implements a sophisticated task orchestration system with dynamic subtask generation and parallel processing. Here's how it works:\n" +
      '\n' +
      '1. Orchestrator Agent (Planning Phase):\n' +
      '   - Takes a complex task as input\n' +
      '   - Analyzes the task and breaks it down into smaller, manageable subtasks\n' +
      '   - Returns both an analysis and a list of subtasks in JSON format\n' +
      '\n' +
      '2. Worker Agents (Execution Phase):\n' +
      '   - Multiple workers run in parallel using Promise.all()\n' +
      '   - Each worker gets:\n' +
      '     - The original task for context\n' +
      '     - Their specific subtask to complete\n' +
      '   - All workers use Gemini 2.0 Flash model\n' +
      '\n' +
      '3. Synthesizer Agent (Integration Phase):\n' +
      '   - Takes all the worker outputs\n' +
      '   - Combines them into a cohesive final result\n' +
      '   - Ensures the pieces flow together naturally',
    similarity: 0.5393730401992798,
    meta: {
      docName: 'agent-architectures.txt',
      documentName: 'agent-architectures.txt',
      category: 'Examples',
      topic: 'Agent architecture'
    }
  },
  {
    text: "`This code implements an email analysis system that processes incoming emails through multiple parallel AI agents to determine if and how they should be handled. Here's the breakdown:\n" +
      '\n' +
      '1. Three Specialized Agents running in parallel:\n' +
      '   - Sentiment Analysis Agent: Determines if the email tone is positive, negative, or neutral\n' +
      '   - Summary Agent: Creates a concise summary of the email content\n' +
      '   - Decision Maker Agent: Takes the outputs from the other agents and decides:\n' +
      '     - If the email needs a response\n' +
      "     - Whether it's spam\n" +
      '     - Priority level (low, medium, high, urgent)\n' +
      '\n' +
      '2. The workflow:\n' +
      '   - Takes an email input\n' +
      '   - Runs sentiment analysis and summary generation in parallel using Promise.all()\n' +
      '   - Feeds those results to the decision maker agent\n' +
      '   - Outputs a final decision object with response requirements\n' +
      '\n' +
      '3. All agents use Gemini 2.0 Flash model and are structured to return parsed JSON responses',
    similarity: 0.49115753173828125,
    meta: {
      docName: 'agent-architectures.txt',
      documentName: 'agent-architectures.txt',
      category: 'Examples',
      topic: 'Agent architecture'
    }
  }
]

Step 7: Create Support Pipe Agent

In this step, we will create a support agent using the Langbase SDK. Go ahead and add the following code to the create-pipe.ts file created in Step 1:

import 'dotenv/config';
import { Langbase } from 'langbase';

const langbase = new Langbase({
    apiKey: process.env.LANGBASE_API_KEY!,
});

async function main() {
    const supportAgent = await langbase.pipes.create({
        name: `ai-support-agent`,
        description: `An AI agent to support users with their queries.`,
        messages: [
            {
                role: `system`,
                content: `You're a helpful AI assistant.
                You will assist users with their queries.
                Always ensure that you provide accurate and to the point information.`,
            },
        ],
    });

    console.log('Support agent:', supportAgent);
}

main();

Let's go through the above code:

• Initialize the Langbase SDK with your API key.

• Use the pipes.create method to create a new pipe agent.

• Log the created pipe agent to the console.

Now run the create-pipe.ts file to create the pipe agent by this command in your terminal:

npx tsx create-pipe.ts

This will create a support agent and log the agent details to the console.

Step 8: Generate RAG Responses

Up until now, we’ve created a Langbase memory agent, added documents in it, performed RAG retrieval against a query, and created a support agent using Langbase Pipe agent. The only thing left in creating this complete RAG agent is generating comprehensive responses using LLMs.

To do this, add the following code to the agents.ts file created in Step 1:

import 'dotenv/config';
import { Langbase, MemoryRetrieveResponse } from 'langbase';

const langbase = new Langbase({
    apiKey: process.env.LANGBASE_API_KEY!,
});

export async function runAiSupportAgent({
    chunks,
    query,
}: {
    chunks: MemoryRetrieveResponse[];
    query: string;
}) {
    const systemPrompt = await getSystemPrompt(chunks);

    const { completion } = await langbase.pipes.run({
        stream: false,
        name: 'ai-support-agent',
        messages: [
            {
                role: 'system',
                content: systemPrompt,
            },
            {
                role: 'user',
                content: query,
            },
        ],
    });

    return completion;
}

async function getSystemPrompt(chunks: MemoryRetrieveResponse[]) {
    let chunksText = '';
    for (const chunk of chunks) {
        chunksText += chunk.text + '\n';
    }

    const systemPrompt = `
    You're a helpful AI assistant.
    You will assist users with their queries.

    Always ensure that you provide accurate and to the point information.
    Below is some CONTEXT for you to answer the questions. ONLY answer from the CONTEXT. CONTEXT consists of multiple information chunks. Each chunk has a source mentioned at the end.

For each piece of response you provide, cite the source in brackets like so: [1].

At the end of the answer, always list each source with its corresponding number and provide the document name. like so [1] Filename.doc. If there is a URL, make it hyperlink on the name.

 If you don't know the answer, say so. Ask for more context if needed.
    ${chunksText}`;

    return systemPrompt;
}

export async function runMemoryAgent(query: string) {
    const chunks = await langbase.memories.retrieve({
        query,
        topK: 4,
        memory: [
            {
                name: 'knowledge-base',
            },
        ],
    });

    return chunks;
}

The above code:

• Creates a function runAiSupportAgent that takes chunks and query as input.

• Uses the pipes.run method to generate responses using the LLM.

• Creates a function getSystemPrompt to generate a system prompt for the LLM.

• Combines the retrieved chunks to create a system prompt.

• Returns the generated completion.

Now, let's run the support agent with AI memory chunks. Add the following code to the index.ts file:

import { runMemoryAgent, runAiSupportAgent } from './agents';

async function main() {
    const query = 'What is agent parallelization?';
    const chunks = await runMemoryAgent(query);

    const completion = await runAiSupportAgent({
        chunks,
        query,
    });

    console.log('Completion:', completion);
}

main();

This code runs two agents: one to retrieve memory chunks relevant to a query (runMemoryAgent), and another (runAiSupportAgent) to generate a final answer using those chunks.

Then, run the index.ts file to generate responses using the LLM.

npx tsx index.ts

The Result

After running the support agent, you’ll see the following output generated in your console:

Completion: Agent parallelization is a process that runs multiple LLM (Language Model) tasks simultaneously to enhance speed or accuracy. This technique can be implemented in two main ways:

1. **Sectioning**: A task is divided into independent parts that can be processed concurrently.
2. **Voting**: Multiple LLM calls generate different responses for the same task, and the best result is selected based on agreement, predefined rules, or quality evaluation. This approach improves accuracy and reliability by comparing various outputs.

In practice, agent parallelization involves orchestrating multiple specialized agents to handle different aspects of a task, allowing for efficient processing and improved outcomes.

If you need more detailed examples or further clarification, feel free to ask!

This is how you can build an agentic RAG system with TypeScript using the Langbase SDK.

Thank you for reading!

Connect with me by 🙌:

• Subscribing to my YouTube Channel if you want to learn about AI and agents.

• Subscribing to my free newsletter The Agentic Engineer where I share all the latest AI and agents news/trends/jobs and much more.

• Follow me on X (Twitter).

You can enhance the performance of an LLM by providing context — structured or unstructured data, such as documents, articles, or knowledge bases — tailored to the domain or information you want the model to specialize in. Using techniques like prompt engineering and context injection, you can build an intelligent chatbot capable of navigating extensive datasets, retrieving relevant information, and delivering responses.

Whether it's storing recipes, code documentation, research articles, or answering domain-specific queries, an LLM-based chatbot can adapt to your needs with customization and privacy. You can deploy it locally to create a highly specialized conversational assistant that respects your data.

In this article, you will learn how to build a local Retrieval-Augmented Generation (RAG) application using Ollama and ChromaDB in R. By the end, you'll have a custom conversational assistant with a Shiny interface that efficiently retrieves information while maintaining privacy and customization.

Table of Contents

• What is RAG?

• Project Overview

• Project Setup

• Ollama Installation

• Data Collection and Cleaning

• How to Create Chunks

• How to Generate Sentence Embeddings

• How to Set Up the Vector Database for Embedding Storage

• How to Write the User Input Query Embedding Function

• Tool Calling

• How to Initialize the Chat System, Design Prompts, and Integrate Tools

• How to Interact with Your Chatbot Using a Shiny App

• Complete Code

• Conclusion

What is RAG?

Retrieval-Augmented Generation (RAG) is a method that integrates retrieval systems with generative AI, enabling chatbots to access recent and specific information from external sources.

By using a retrieval pipeline, the chatbot can fetch up-to-date, relevant data and combine it with the generative model’s language capabilities, producing responses that are both accurate and contextually enriched. This makes RAG particularly useful for applications requiring fact-based, real-time knowledge delivery.

Project Overview

Project Setup

Prerequisites

Before you begin, ensure you have installed the latest version of the items listed here:

1. RStudio: The IDE – RStudio is the primary workspace where you'll write and test your R code. Its user-friendly interface, debugging tools, and integrated environment make it ideal for data analysis and chatbot development.

2. R: The Programming Language – R is the backbone of your project. You'll use it to handle data manipulation, apply statistical models, and integrate your recipe chatbot components seamlessly.

3. Python – Some libraries, like the embedding library you'll use for text vectorization, are built on Python. It’s vital to have Python installed to enable these functionalities alongside your R code.

4. Java – Java serves as a foundational element for certain embedding libraries. It ensures efficient processing and compatibility for text embedding tasks required to train your chatbot.

5. Docker Desktop – Docker Desktop allows you to run ChromaDB, the vector database, locally on your machine. This enables fast and reliable storage of embeddings, ensuring your chatbot retrieves relevant information quickly.

6. Ollama – Ollama brings powerful Large Language Models (LLMs) directly to your local computer, removing the need for cloud resources. It lets you access multiple models, customize outputs, and integrate them into your chatbot effortlessly.

Ollama Installation

Ollama is an open-sourced tool you can use to run and manage LLMs on your computer. Once installed, you can access various LLMs as per your needs. You will be using llama3.2:3b-instruct-q4_K_M model to build this chatbot.

A quantized model is a version of a machine learning model that has been optimized to use less memory and computational power by reducing the precision of the numbers it uses. This enables you to use an LLM locally, especially when you don’t have access to a GPU (Graphics Processing Unit – a specialized processor that perform complex computations).

To start, you can download and install the Ollama software here.

Then you can confirm installation by running this command:

ollama --version

Run the following command to start Ollama:

ollama serve

Next, run the following command to pull the Q4_K_M quantization of llama3.2:3b-instruct:

ollama pull llama3.2:3b-instruct-q4_K_M

Then confirm that the model was extracted with this:

ollama list

If the model extraction was successful, a list containing the model’s name, ID, and size will be returned, like so:

Now you can chat with the model:

ollama run llama3.2:3b-instruct-q4_K_M

If successful, you should receive a prompt that you can test by asking a question and getting an answer. For example:

Then you can exit the console by typing /bye or ctrl + D

Data Collection and Cleaning

The chatbot you are building will be a cooking assistant that suggests recipes given your available ingredients, what you want to eat, and how much food a recipe yields.

You first have to get the data to train the model. You will be using a dataset that contains recipes from Kaggle.

To start, load the necessary libraries:

# loading required libraries
library(xml2) #read, parse, and manipulate XML,HTML documents
library(jsonlite) #manipulate JSON objects

library(RKaggle) # download datasets from Kaggle 
library(dplyr)   # data manipulation

Then download and save recipe dataset:

# Download and read the "recipe" dataset from Kaggle
recipes_list <- RKaggle::get_dataset("thedevastator/better-recipes-for-a-better-life")

Inspect the dataframe and extract the first element like this:

# inspect the dataset
class(recipes_list)
str(recipes_list)
head(recipes_list)
# extract the first tibble
recipes_df <- recipes_list[[1]]

A quick inspection of the recipes_list object shows that it contains two objects of type tibble. You will be using only the first element for this project. A tibble is a type of data structure used for storing and manipulating data. It’s similar to a traditional dataframe, but it’s designed to enforce stricter rules and perform fewer automatic actions compared to traditional dataframes.

We’ll use a regular dataframe in this project because more people are likely familiar with it. It can also efficiently handle row indexing, which is crucial for accessing and manipulating specific rows in our recipe dataset.

In the code block below, you’ll convert the tibble to a dataframe and then drop the first column, which is the index column. Then you’ll inspect the newly converted dataframe and drop unnecessary columns.

Unnecessary columns are best removed to streamline the dataset and focus on relevant features. In this project, we’ll drop certain columns that aren’t particularly useful for training the chatbot. This ensures that the model concentrates on meaningful data to improve its accuracy and functionality.

# convert to dataframe and drop the first column
recipes_df <- as.data.frame(recipes_df[, -1])
# inspect the converted dataframe
head(recipes_df)
class(recipes_df)
colnames(recipes_df)
# drop unnecessary columns
cleaned_recipes_df <- subset(recipes_df, select = -c(yield,rating,url,cuisine_path,nutrition,timing,img_src))

Now you need to identify rows with NA (missing) values, which you can do like this:

# Identify rows and columns with NA values
which(is.na(cleaned_recipes_df), arr.ind = TRUE)

# a quick inspection reveals columns [2:4] have missing values
subset_column_names <- colnames(cleaned_recipes_df)[2:4]
subset_column_names

It is important to handle NA values to ensure that your data is complete, to prevent errors, and to preserve context.

Now, replace the NA values and confirm that there are no missing values:

# Replace NA values dynamically based on conditions
cols_to_modify <- c("prep_time", "cook_time", "total_time")
cleaned_recipes_df[cols_to_modify] <- lapply(
  cleaned_recipes_df[cols_to_modify],
  function(x, df) {
    # Replace NA in prep_time and cook_time where both are NA
    replace(x, is.na(df$prep_time) & is.na(df$cook_time), "unknown")
  },
  df = cleaned_recipes_df  # Pass the whole dataframe for conditions
)
cleaned_recipes_df <- cleaned_recipes_df %>%
  mutate(
    prep_time = case_when(
      # If cooktime is present but preptime is NA, replace with "no preparation required"
      !is.na(cook_time) & is.na(prep_time) ~ "no preparation required",
      # Otherwise, retain original value
      TRUE ~ as.character(prep_time)
    ),
    cook_time = case_when(
      # If prep_time is present but cook_time is NA, replace with "no cooking required"
      !is.na(prep_time) & is.na(cook_time) ~ "no cooking required",
      # Otherwise, retain original value
      TRUE ~ as.character(cook_time)
    )
  )
# confirm there are no missing values
any(is.na(cleaned_recipes_df))
)

# confirm the replacing NA logic works by inspecting specific rows
cleaned_recipes_df[1081,]
cleaned_recipes_df[1,]
cleaned_recipes_df[405,]

For this tutorial, we’ll subset the dataframe to the first 250 rows for demo purposes. This saves on time when it comes to generating embeddings.

# recommended for demo/learning purposes
cleaned_recipes_df <- head(cleaned_recipes_df,250)

How to Create Chunks

To understand why chunking is important before embedding, you need to understand what an embedding is.

An embedding is a vectoral representation of a word or a sentence. Machines don’t understand human text – they understand numbers. LLMs work by transforming human text to numerical representations in order to give answers. The process of generating embeddings requires a lot of computation, and breaking down the data to be embedded optimizes the embedding process.

So now we’re going to split the dataframe into smaller chunks of a specified size to enable efficient batch processing and iteration.

# Define the size of each chunk (number of rows per chunk)
chunk_size <- 1

# Get the total number of rows in the dataframe
n <- nrow(cleaned_recipes_df)

# Create a vector of group numbers for chunking
# Each group number repeats for 'chunk_size' rows
# Ensure the vector matches the total number of rows
r <- rep(1:ceiling(n/chunk_size), each = chunk_size)[1:n]

# Split the dataframe into smaller chunks (subsets) based on the group numbers
chunks <- split(cleaned_recipes_df, r)

How to Generate Sentence Embeddings

As previously mentioned, embeddings are vector representations of words or sentences. Embeddings can be generated from both words and sentences. How you choose to generate embeddings depends on your intended application of the LLM.

Word embeddings are numerical representations of individual words in a continuous vector space. They capture semantic relationships between words, allowing similar words to have vectors close to each other.

Word embeddings can be used in search engines as they support word-level queries by matching embeddings to retrieve relevant documents. They can also be used in text classification to classify documents, emails, or tweets based on word-level features (for example, detecting spam emails or sentiment analysis).

Sentence embeddings are numerical representations of entire sentences in a vector space, designed to capture the overall meaning and context of the sentence. They are used in settings where sentences provide better context like question answering systems where user queries are matched to relevant sentences or documents for more precise retrieval.

For our recipe chatbot, sentence embedding is the best choice.

First, create an empty dataframe that has three columns.

#empty dataframe
recipe_sentence_embeddings <-  data.frame(
  recipe = character(),
  recipe_vec_embeddings = I(list()),
  recipe_id = character()
)

The first column will hold the actual recipe in text form, the recipe_vec_embeddings column will hold the generated sentence embeddings, and the recipe_id holds a unique id for each recipe. This will help in indexing and retrieval from the vector database.

Next, it’s helpful to define a progress bar, which you can do like this:

# create a progress bar
pb <- txtProgressBar(min = 1, max = length(chunks), style = 3)

Embedding can take a while, so it’s important to keep track of the progress of the process.

Now it’s time to generate embeddings and populate the dataframe.

Write a for loop that executes the code block as long as the length of the chunks.

for (i in 1:length(chunks)) {}

The recipe field is the text at the chunk that is currently being executed and the unique chunk id is generated by pasting the index of the chunk and the text “chunk”.

for (i in 1:length(chunks)) {
    recipe <- as.character(chunks[i])
    recipe_id <- paste0("recipe",i)
}

The text embed function from the text library generates either sentence or word embeddings. It takes in a character variable or a dataframe and produces a tibble of embeddings. You can read loading instructions here for smooth running of the text library.

The batch_size defines how many rows are embedded at a time from the input. Setting the keep_token_embeddings discards the embeddings for individual tokens after processing, and aggregation_from_layers_to_tokens “concatenates” or combines embeddings from specified layers to create detailed embeddings for each token. A token is the smallest unit of text that a model can process.

for (i in 1:length(chunks)) {
    recipe <- as.character(chunks[i])
    recipe_id <- paste0("recipe",i)
    recipe_embeddings <- textEmbed(as.character(recipe),
                                layers = 10:11,
                                aggregation_from_layers_to_tokens = "concatenate",
                                aggregation_from_tokens_to_texts = "mean",
                                keep_token_embeddings = FALSE,
                                batch_size = 1
  )
}

In order to specify sentence embeddings, you need to set the argument to the aggregation_from_tokens_to_texts parameter as "mean".

aggregation_from_tokens_to_texts = "mean"

The "mean" operation averages the embeddings of all tokens in a sentence to generate a single vector that represents the entire sentence. This sentence-level embedding captures the overall meaning and semantics of the text, regardless of its token length.

# convert tibble to vector
  recipe_vec_embeddings <- unlist(recipe_embeddings, use.names = FALSE)
  recipe_vec_embeddings <- list(recipe_vec_embeddings)

The embedding function returns a tibble object. In order to obtain a vector embedding, you need to first unlist the tibble and drop the row names and then list the result to form a simple vector.

  # Append the current chunk's data to the dataframe
  recipe_sentence_embeddings <- recipe_sentence_embeddings %>%
    add_row(
      recipe = recipe,
      recipe_vec_embeddings = recipe_vec_embeddings,
      recipe_id = recipe_id
    )

Finally, update the empty dataframe after each iteration with the newly generated data.

  # track embedding progress
  setTxtProgressBar(pb, i)

In order to keep track of the embedding progress, you can use the earlier defined progress bar inside the loop. It will update at the end of every iteration.

Complete Code Block:

# load required library
library(text)
# # ensure to read loading instructions here for smooth running of the 'text' library
# # https://www.r-text.org/
# embedding data
for (i in 1:length(chunks)) {
  recipe <- as.character(chunks[i])
  recipe_id <- paste0("recipe",i)
  recipe_embeddings <- textEmbed(as.character(recipe),
                                layers = 10:11,
                                aggregation_from_layers_to_tokens = "concatenate",
                                aggregation_from_tokens_to_texts = "mean",
                                keep_token_embeddings = FALSE,
                                batch_size = 1
  )

  # convert tibble to vector
  recipe_vec_embeddings <- unlist(recipe_embeddings, use.names = FALSE)
  recipe_vec_embeddings <- list(recipe_vec_embeddings)

  # Append the current chunk's data to the dataframe
  recipe_sentence_embeddings <- recipe_sentence_embeddings %>%
    add_row(
      recipe = recipe,
      recipe_vec_embeddings = recipe_vec_embeddings,
      recipe_id = recipe_id
    )

  # track embedding progress
  setTxtProgressBar(pb, i)

}

How to Set Up the Vector Database for Embedding Storage

A vector database is a special type of database that stores embeddings and allows you to query and retrieve relevant information. There are numerous vector databases available, but for this project, you will use ChromaDB, an open-source option that integrates with the R environment through the rchroma library.

ChromaDB runs locally in a Docker container. Just make sure you have Docker installed and running on your device.

Then load the rchroma library and run your ChromaDB instance:

# load rchroma library
library(rchroma)
# run ChromaDB instance.
chroma_docker_run()

If it was successful, you should see this in the console:

Next, connect to a local ChromaDB instance and check the connection:

# Connect to a local ChromaDB instance
client <- chroma_connect()

# Check the connection
heartbeat(client)
version(client)

Now you’ll need to create a collection and confirm that it was created. Collections in ChromaDB function similarly to tables in conventional databases.

# Create a new collection
create_collection(client, "recipes_collection")

# List all collections
list_collections(client)

Now, add embeddings to the collection. To add embeddings to the recipes_collection, use the add_documents function.

# Add documents to the collection
add_documents(
  client,
  "recipes_collection",
  documents = recipe_sentence_embeddings$recipe,
  ids = recipe_sentence_embeddings$recipe_id,
  embeddings = recipe_sentence_embeddings$recipe_vec_embeddings
)

The add_documents() function is used to add recipe data to the recipes_collection. Here's a breakdown of its arguments and how the corresponding data is accessed:

1. documents: This argument represents the recipe text. It is sourced from the recipe column of the recipe_sentence_embeddings dataframe.

2. ids: This is the unique identifier for each recipe. It is extracted from the recipe_id column of the same dataframe.

3. embeddings: This contains the sentence embeddings, which were previously generated for each recipe. These embeddings are accessed from the recipe_vec_embeddings column of the dataframe.

All three arguments—documents, ids, and embeddings—are obtained by subsetting their respective columns from the recipe_sentence_embeddings dataframe.

How to Write the User Input Query Embedding Function

In order to retrieve information from a vector database, you must first embed your query text. The database compares your query's embedding with its stored embeddings to find and retrieve the most relevant document.

It's important to ensure that the dimensions (rows × columns) of your query embedding match those of the database embeddings. This alignment is achieved by using the same embedding model to generate your query.

Matching embeddings involves calculating the similarity (for example, cosine similarity) between the query and stored embeddings, identifying the closest match for effective retrieval.

Let’s write a function that allows us to embed a query which then queries similar documents using the generated embeddings. Wrapping it in a function makes it reusable.

  #sentence embeddings function and query
  question <- function(sentence){
    sentence_embeddings <- textEmbed(sentence,
                                     layers = 10:11,
                                     aggregation_from_layers_to_tokens = "concatenate",
                                     aggregation_from_tokens_to_texts = "mean",
                                     keep_token_embeddings = FALSE
    )

    # convert tibble to vector
    sentence_vec_embeddings <- unlist(sentence_embeddings, use.names = FALSE)
    sentence_vec_embeddings <- list(sentence_vec_embeddings)

    # Query similar documents using embeddings
    results <- query(
      client,
      "recipes_collection",
      query_embeddings = sentence_vec_embeddings ,
      n_results = 2
    )
    results

  }

This chunk of code is similar to how we have previously used the text_embed() function. The query() function is added to enable querying the vector database, particularly the recipes' collection, and returns the top two documents that closely match a user’s query.

Our function thus takes in a sentence as an argument and embeds the sentence to generate sentence embeddings. It then queries the database and returns two documents that match the query most.

Tool Calling

To interact with Ollama in R, you will utilize the ellmer library. This library streamlines the use of large language models (LLMs) by offering an interface that enables seamless access to and interaction with a variety of LLM providers.

To enhance the LLM’s usage, we need to provide context to it. You can do this by tool calling. Tool calling allows an LLM to access external resources in order to enhance its functionality.

For this project, we are implementing Retrieval-Augmented Generation (RAG), which combines retrieving relevant information from a vector database and generating responses using an LLM. This approach improves the chatbot's ability to provide accurate and contextually relevant answers.

Now, define a function that links to the LLM to provide context using the tool() function from the ellmer library.

# load ellmer library
library(ellmer)

# function that links to llm to provide context
  tool_context  <- tool(
    question,
    "obtains the right context for a given question",
    sentence = type_string()

  )

The tool() function takes the question function that returns the relevant documents that we’ll use as context as the first argument. We’ll use the documents to help the LLM answer questions accordingly.

The text, "obtains the right context for a given question", is a description of what the tool will be doing.

Finally, the sentence = type_string() defines what type of object the question() function expects.

How to Initialize the Chat System, Design Prompts, and Integrate Tools

Next, you’ll set up a conversational AI system by defining its role and functionality. Using system prompt design, you will shape the assistant’s behavior, tone, and focus as a culinary assistant. You’ll also integrate external tools to extend the chatbot’s capabilities by registering tools. Let’s dive in.

First, you need to initialize a Chat Object:

#  Initialize the chat system with propmpt instructions.
  chat <- chat_ollama(system_prompt = "You are a knowledgeable culinary assistant specializing in recipe recommendations. 
                      You provide tailored meal suggestions based on the user's available ingredients and the desired amount of food or servings.
                      Ensure the recipes align closely with the user's inputs and yield the expected quantity.",
                      model = "llama3.2:3b-instruct-q4_K_M")

You can do that using the chat_ollama() function. This sets up a conversational agent with the specified system prompt and model.

The system prompt defines the conversational behavior, tone, and focus of the LLM while the model argument specifies the language model (llama3.2:3b-instruct-q4_K_M) that the chat system will use to generate responses.

Next, you need to register a tool.

 #register tool
  chat$register_tool(tool_context)

We need to tell our chat object about our tool_context() function. Do this by registering a tool using the register_tool() function.

How to Interact with Your Chatbot Using a Shiny App

To interact with the chatbot you’ve just created, we’ll use Shiny, a framework for building interactive web applications in R. Shiny provides a user-friendly graphical interface that allows seamless interaction with the chatbot.

For this purpose, we’ll use the shinychat library, which simplifies the process of building a chat interface within a Shiny app. This involves defining two key components:

1. User Interface (UI):

   • Responsible for the visual layout and what the user sees.

   • In this case, chat_ui("chat") is used to create the interactive chat interface.

2. Server Function:

   • Handles the functionality and logic of the application.

   • It connects the chatbot to external tools and manages processes like embedding queries, retrieving relevant responses, and handling user inputs.

# load the required library
library(shinychat)

# wrap the chat code in a Shiny App
ui <- bslib::page_fluid(
  chat_ui("chat")
)

server <- function(input, output, session) {
  # Connect to a local ChromaDB instance running on docker with embeddings loaded
  client <- chroma_connect()

  #sentence embeddings function and query
  question <- function(sentence){
    sentence_embeddings <- textEmbed(sentence,
                                     layers = 10:11,
                                     aggregation_from_layers_to_tokens = "concatenate",
                                     aggregation_from_tokens_to_texts = "mean",
                                     keep_token_embeddings = FALSE
    )

    # convert tibble to vector
    sentence_vec_embeddings <- unlist(sentence_embeddings, use.names = FALSE)
    sentence_vec_embeddings <- list(sentence_vec_embeddings)

    # Query similar documents using embeddings
    results <- query(
      client,
      "recipes_collection",
      query_embeddings = sentence_vec_embeddings ,
      n_results = 2
    )
    results

  }


  # function that provides context
  tool_context  <- tool(
    question,
    "obtains the right context for a given question",
    sentence = type_string()

  )

  #  Initialize the chat system with the first chunk
  chat <- chat_ollama(system_prompt = "You are a knowledgeable culinary assistant specializing in recipe recommendations. 
                      You provide tailored meal suggestions based on the user's available ingredients and the desired amount of food or servings.
                      Ensure the recipes align closely with the user's inputs and yield the expected quantity.",
                      model = "llama3.2:3b-instruct-q4_K_M")
  #register tool
  chat$register_tool(tool_context)

  observeEvent(input$chat_user_input, {
    stream <- chat$stream_async(input$chat_user_input)
    chat_append("chat", stream)
  })
}

shinyApp(ui, server)

Alright, let’s understand how this is working:

1. User input monitoring with observeEvent(): The observeEvent() block monitors user inputs from the chat interface (input$chat_user_input). When a user sends a message, the chatbot processes it, retrieves relevant context using the embeddings, and streams the response dynamically to the chat interface.

2. Tool calling for context: The chatbot employs tool calling to interact with external resources (like the vector database) and enhance its functionality. In this project, Retrieval-Augmented Generation (RAG) ensures the chatbot provides accurate and context-rich responses by integrating retrieval and generation seamlessly.

This approach brings the chatbot to life, enabling users to interact with it dynamically through a responsive Shiny app.

Complete Code

The R scripts have been split in two, with data.R containing code that handles data gathering and cleaning, text chunking, sentence embeddings generation, creating a vector database, and loading documents to it.

The chat.R script contains code that handles user input querying, context retrieval, chat initialization, system prompt design, tool integration, and a chat Shiny app.

data.R

# install and load required packages
# install devtools from CRAN
install.packages('devtools')
devtools::install_github("benyamindsmith/RKaggle")

library(text)
library(rchroma)
library(RKaggle)
library(dplyr)

# run ChromaDB instance.
chroma_docker_run()

# Connect to a local ChromaDB instance
client <- chroma_connect()

# Check the connection
heartbeat(client)
version(client)


# Create a new collection
create_collection(client, "recipes_collection")

# List all collections
list_collections(client)

# Download and read the "recipe" dataset from Kaggle
recipes_list <- RKaggle::get_dataset("thedevastator/better-recipes-for-a-better-life")

# extract the first tibble
recipes_df <- recipes_list[[1]]

# convert to dataframe and drop the first column
recipes_df <- as.data.frame(recipes_df[, -1])

# drop unnecessary columns
cleaned_recipes_df <- subset(recipes_df, select = -c(yield,rating,url,cuisine_path,nutrition,timing,img_src))

## Replace NA values dynamically based on conditions
# Replace NA when all columns have NA values
cols_to_modify <- c("prep_time", "cook_time", "total_time")
cleaned_recipes_df[cols_to_modify] <- lapply(
  cleaned_recipes_df[cols_to_modify],
  function(x, df) {
    # Replace NA in prep_time and cook_time where both are NA
    replace(x, is.na(df$prep_time) & is.na(df$cook_time), "unknown")
  },
  df = cleaned_recipes_df  
)

# Replace NA when either or columns have NA values
cleaned_recipes_df <- cleaned_recipes_df %>%
  mutate(
    prep_time = case_when(
      # If cook_time is present but prep_time is NA, replace with "no preparation required"
      !is.na(cook_time) & is.na(prep_time) ~ "no preparation required",
      # Otherwise, retain original value
      TRUE ~ as.character(prep_time)
    ),
    cook_time = case_when(
      # If prep_time is present but cook_time is NA, replace with "no cooking required"
      !is.na(prep_time) & is.na(cook_time) ~ "no cooking required",
      # Otherwise, retain original value
      TRUE ~ as.character(cook_time)
    )
  )

# chunk the dataset
chunk_size <- 1
n <- nrow(cleaned_recipes_df)
r <- rep(1:ceiling(n/chunk_size),each = chunk_size)[1:n]
chunks <- split(cleaned_recipes_df,r)

#empty dataframe
recipe_sentence_embeddings <-  data.frame(
  recipe = character(),
  recipe_vec_embeddings = I(list()),
  recipe_id = character()
)

# create a progress bar
pb <- txtProgressBar(min = 1, max = length(chunks), style = 3)

# embedding data
for (i in 1:length(chunks)) {
  recipe <- as.character(chunks[i])
  recipe_id <- paste0("recipe",i)
  recipe_embeddings <- textEmbed(as.character(recipe),
                                layers = 10:11,
                                aggregation_from_layers_to_tokens = "concatenate",
                                aggregation_from_tokens_to_texts = "mean",
                                keep_token_embeddings = FALSE,
                                batch_size = 1
  )

  # convert tibble to vector
  recipe_vec_embeddings <- unlist(recipe_embeddings, use.names = FALSE)
  recipe_vec_embeddings <- list(recipe_vec_embeddings)

  # Append the current chunk's data to the dataframe
  recipe_sentence_embeddings <- recipe_sentence_embeddings %>%
    add_row(
      recipe = recipe,
      recipe_vec_embeddings = recipe_vec_embeddings,
      recipe_id = recipe_id
    )

  # track embedding progress
  setTxtProgressBar(pb, i)

}

# Add documents to the collection
add_documents(
  client,
  "recipes_collection",
  documents = recipe_sentence_embeddings$recipe,
  ids = recipe_sentence_embeddings$recipe_id,
  embeddings = recipe_sentence_embeddings$recipe_vec_embeddings
)

chat.R

# Load required packages
library(ellmer)
library(text)
library(rchroma)
library(shinychat)

ui <- bslib::page_fluid(
  chat_ui("chat")
)

server <- function(input, output, session) {
  # Connect to a local ChromaDB instance running on docker with embeddings loaded 
  client <- chroma_connect()

  # sentence embeddings function and query
  question <- function(sentence){
    sentence_embeddings <- textEmbed(sentence,
                                     layers = 10:11,
                                     aggregation_from_layers_to_tokens = "concatenate",
                                     aggregation_from_tokens_to_texts = "mean",
                                     keep_token_embeddings = FALSE
    )

    # convert tibble to vector
    sentence_vec_embeddings <- unlist(sentence_embeddings, use.names = FALSE)
    sentence_vec_embeddings <- list(sentence_vec_embeddings)

    # Query similar documents
    results <- query(
      client,
      "recipes_collection",
      query_embeddings = sentence_vec_embeddings ,
      n_results = 2
    )
    results

  }


  # function that provides context
  tool_context  <- tool(
    question,
    "obtains the right context for a given question",
    sentence = type_string()

  )

  #  Initialize the chat system 
  chat <- chat_ollama(system_prompt = "You are a knowledgeable culinary assistant specializing in recipe recommendations. 
                      You provide tailored meal suggestions based on the user's available ingredients and the desired amount of food or servings.
                      Ensure the recipes align closely with the user's inputs and yield the expected quantity.",
                      model = "llama3.2:3b-instruct-q4_K_M")
  #register tool
  chat$register_tool(tool_context)

  observeEvent(input$chat_user_input, {
    stream <- chat$stream_async(input$chat_user_input)
    chat_append("chat", stream)
  })
}

shinyApp(ui, server)

You can find the complete code here.

Conclusion

Building a local Retrieval-Augmented Generation (RAG) application using Ollama and ChromaDB in R programming offers a powerful way to create a specialized conversational assistant.

By leveraging the capabilities of large language models and vector databases, you can efficiently manage and retrieve relevant information from extensive datasets.

This approach not only enhances the performance of language models but also ensures customization and privacy by running the application locally.

Whether you're developing a cooking assistant or any other domain-specific chatbot, this method provides a robust framework for delivering intelligent and contextually aware responses.

We just published a new course on the freeCodeCamp.org YouTube channel that will teach you how to build and deploy a RAG chatbot using your own data. This hands-on 90-minute tutorial, led by popular creator Ania Kubow, will teach you how to create a Retrieval-Augmented Generation (RAG) chatbot with JavaScript using tools like LangChain.js, Next.js, and OpenAI. You’ll also deploy it to Vercel and integrate a vector database with DataStax.

By the end of this course, you’ll have built a custom Formula 1 chatbot capable of answering real-time questions about the sport. And you’ll learn how to apply this concept to any dataset, whether it’s FAQs from your website, private business documents, or any other data source you choose.

What You’ll Learn

This course introduces essential AI concepts while equipping you with practical skills to build, train, and deploy a chatbot. Here's what’s covered:

1. What is RAG?
   Learn the fundamentals of Retrieval-Augmented Generation (RAG), an approach that extends the capabilities of large language models (LLMs) by combining them with external data sources to generate more accurate and relevant answers.

2. Prerequisites and Setup

   • Learn the tools you’ll need, including a Datastax API key for vector database management.

   • Explore the basics of integrating LangChain.js with your development environment.

3. Vector Embeddings and Databases
   Understand how vector embeddings convert textual data into numerical formats that AI models can process. Then, learn to manage these embeddings using a Datastax vector database.

4. OpenAI Integration
   Discover how to enhance chatbot responses by leveraging OpenAI’s powerful language models to add conversational flair.

5. Building the F1 RAG Chatbot
   Follow step-by-step instructions to scrape real-time Formula 1 data, store it as vector embeddings, and create a chatbot capable of answering the latest questions about F1.

6. Deployment on Vercel
   Learn to deploy your chatbot with Next.js on Vercel, making it live and accessible to anyone.

Real-Life Use Cases

This course doesn’t just stop at Formula 1. The skills you’ll learn can be applied to countless other scenarios:

• Business Applications: Create a chatbot to answer customer FAQs or navigate complex company policies.

• Education: Build tools that provide up-to-date resources for students and educators.

• Personal Use: Develop chatbots that assist with scheduling, document retrieval, or even journaling.

What is RAG and Why Does It Matter?

Retrieval-Augmented Generation (RAG) combines the text generation capabilities of LLMs with custom or up-to-date data retrieval. This approach avoids the high costs and complexity of retraining large models while addressing limitations like outdated training data. Whether you’re topping up an LLM’s knowledge with real-time internet data or private documents, RAG ensures your chatbot delivers relevant, context-aware answers.

Ready to Build Your First RAG Chatbot?

This course is packed with everything you need to create and deploy a powerful, real-time chatbot. Watch the full tutorial on the freeCodeCamp.org YouTube channel and unlock the potential of generative AI for JavaScript developers (2-hour watch).

One of the biggest challenges faced by LLMs is hallucination. This occurs when the model generates text that is factually incorrect or misleading, often based on patterns it has learned from its training data. So how can Retrieval-Augmented Generation, or RAG, help mitigate this issue?

By retrieving relevant information from a more vast, wider knowledge base, RAG ensures that the LLM's responses are grounded in real-world facts. This significantly reduces the likelihood of hallucinations and improves the overall accuracy and reliability of the generated content.

Table of Contents:

1. What is Retrieval Augmented Generation (RAG)?

2. Understanding the Components of a RAG Pipeline

3. Prerequisites

4. Let's Get Started!

5. How to Fine-Tune the Pipeline

6. Real-World Applications of RAG

7. RAG Best Practices and Considerations

8. Conclusion

What is Retrieval Augmented Generation (RAG)?

RAG is a technique that combines information retrieval with language generation. Think of it as a two-step process:

1. Retrieval: The model first retrieves relevant information from a large corpus of documents based on the user's query.

2. Generation: Using this retrieved information, the model then generates a comprehensive and informative response.

Why use LlamaIndex for RAG?

LlamaIndex is a powerful framework that simplifies the process of building RAG pipelines. It provides a flexible and efficient way to connect retrieval components (like vector databases and embedding models) with generation components (like LLMs).

Some of the key benefits of using Llama-Index include:

• Modularity: It allows you to easily customize and experiment with different components.

• Scalability: It can handle large datasets and complex queries.

• Ease of use: It provides a high-level API that abstracts away much of the underlying complexity.

What You'll Learn Here:

In this article, we will delve deeper into the components of a RAG pipeline and explore how you can use LlamaIndex to build these systems.

We will cover topics such as vector databases, embedding models, language models, and the role of LlamaIndex in connecting these components.

Understanding the Components of a RAG Pipeline

Here's a diagram that'll help familiarize you with the basics of RAG architecture:

This diagram is inspired by this article. Let's go through the key pieces.

Components of RAG

Retrieval Component:

• Vector Databases: These databases are optimized for storing and searching high-dimensional vectors. They are crucial for efficiently finding relevant information from a vast corpus of documents.

• Embedding Models: These models convert text into numerical representations or embeddings. These embeddings capture the semantic meaning of the text, allowing for efficient comparison and retrieval in vector databases.

A vector is a mathematical object that represents a quantity with both magnitude (size) and direction. In the context of RAG, embeddings are high-dimensional vectors that capture the semantic meaning of text. Each dimension of the vector represents a different aspect of the text's meaning, allowing for efficient comparison and retrieval.

Generation Component:

• Language Models: These models are trained on massive amounts of text data, enabling them to generate human-quality text. They are capable of understanding and responding to prompts in a coherent and informative manner.

The RAG Flow

1. Query Submission: A user submits a query or question.

2. Embedding Creation: The query is converted into an embedding using the same embedding model used for the corpus.

3. Retrieval: The embedding is searched against the vector database to find the most relevant documents.

4. Contextualization: The retrieved documents are combined with the original query to form a context.

5. Generation: The language model generates a response based on the provided context.

LamaIndex

LlamaIndex plays a crucial role in connecting the retrieval and generation components. It acts as an index that maps queries to relevant documents. By efficiently managing the index, LlamaIndex ensures that the retrieval process is fast and accurate.

Prerequisites

We will be using Python and IBM watsonx via LlamaIndex in this article. You should have the following on your system before getting started:

• Python 3.9+

• IBM watsonx project and API key

• Curiosity to learn

Let's Get Started!

In this article, we will be using LlamaIndex to make a simple RAG Pipeline.

Let's create a virtual environment for Python using the following command in your terminal: python -m venv venv . This will create a virtual environment (venv) for your project. If you are a Windows user you can activate it using .\venv\Scripts\activate, and Mac users can activate it with source venv/bin/activate.

Now let's install the packages:

pip install wikipedia llama-index-llms-ibm llama-index-embeddings-huggingface

Once these packages are installed, you will need watsonx.ai's API key as well. This in turn will help you use LLMs via LlamaIndex.

To learn about how to get your watsonx.ai API keys, click here. You need the project ID and API Key to be able to work on the "Generation" aspect of RAG. Having them will help you make LLM calls through watsonx.ai.

import wikipedia

# Search for a specific page
page = wikipedia.page("Artificial Intelligence")

# Access the content
print(page.content)

Now let's save the page content to a text document. We are doing it so that we can access it later. You can do this using the below code:

import os

# Create the 'Document' directory if it doesn't exist
if not os.path.exists('Document'):
    os.mkdir('Document')

# Open the file 'AI.txt' in write mode with UTF-8 encoding
with open('Document/AI.txt', 'w', encoding='utf-8') as f:
    # Write the content of the 'page' object to the file
    f.write(page.content)

Now we'll be using watsonx.ai via LlamaIndex. It will help us generate responses based on the user's query.

Note: Make sure to replace the parameters WATSONX_APIKEY and project_id with your values in the below code:

import os
from llama_index.llms.ibm import WatsonxLLM
from llama_index.core import SimpleDirectoryReader, Document


# Define a function to generate responses using the WatsonxLLM instance
def generate_response(prompt):
    """
    Generates a response to the given prompt using the WatsonxLLM instance.

    Args:
        prompt (str): The prompt to provide to the large language model.

    Returns:
        str: The generated response from the WatsonxLLM.
    """

    response = watsonx_llm.complete(prompt)
    return response

# Set the WATSONX_APIKEY environment variable (replace with your actual key)
os.environ["WATSONX_APIKEY"] = 'YOUR_WATSONX_APIKEY'  # Replace with your API key

# Define model parameters (adjust as needed)
temperature = 0
max_new_tokens = 1500
additional_params = {
    "decoding_method": "sample",
    "min_new_tokens": 1,
    "top_k": 50,
    "top_p": 1,
}

# Create a WatsonxLLM instance with the specified model, URL, project ID, and parameters
watsonx_llm = WatsonxLLM(
    model_id="meta-llama/llama-3-1-70b-instruct",
    url="https://us-south.ml.cloud.ibm.com",
    project_id="YOUR_PROJECT_ID",
    temperature=temperature,
    max_new_tokens=max_new_tokens,
    additional_params=additional_params,
)

# Load documents from the specified directory
documents = SimpleDirectoryReader(
    input_files=["Document/AI.txt"]
).load_data()

# Combine the text content of all documents into a single Document object
combined_documents = Document(text="\n\n".join([doc.text for doc in documents]))

# Print the combined document
print(combined_documents)

Here's a breakdown of the parameters:

• temperature = 0: This setting makes the model generate the most likely text sequence, leading to a more deterministic and predictable output. It's like telling the model to stick to the most common words and phrases.

• max_new_tokens = 1500: This limits the generated text to a maximum of 1500 new tokens (words or parts of words).

• additional_params:

  • decoding_method = "sample": This means the model will generate text randomly based on the probability distribution of each token.

  • min_new_tokens = 1: Ensures that at least one new token is generated, preventing the model from repeating itself.

  • top_k = 50: This limits the model's choices to the 50 most likely tokens at each step, making the output more focused and less random.

  • top_p = 1: This sets the nucleus sampling probability to 1, meaning all tokens with a probability greater than or equal to the top_p value will be considered.

You can tweak these parameters for experimentation and see how they affect your response. Now we'll be building and loading a vector store index from the given document. But first, let's understand what it is.

Understanding Vector Store Indexes

A vector store index is a specialized data structure designed to efficiently store and retrieve high-dimensional vectors. In the context of the Llama Index, these vectors represent the semantic embeddings of documents.

Key characteristics of vector store indexes:

• High-dimensional vectors: Each document is represented as a high-dimensional vector, capturing its semantic meaning.

• Efficient retrieval: Vector store indexes are optimized for fast similarity search, allowing you to quickly find documents that are semantically similar to a given query.

• Scalability: They can handle large datasets and scale efficiently as the number of documents grows.

How Llama Index uses vector store indexes:

1. Document Embedding: Documents are first converted into high-dimensional vectors using a language model like Llama.

2. Index Creation: The embeddings are stored in a vector store index.

3. Query Processing: When a user submits a query, it is also converted into a vector. The vector store index is then used to find the most similar documents based on their embeddings.

4. Response Generation: The retrieved documents are used to generate a relevant response.

In the below code, you'll come across the word "chunk". A chunk is a smaller, manageable unit of text extracted from a larger document. It's typically a paragraph or a few sentences long. They are used to make the retrieval and processing of information more efficient, especially when dealing with large documents.

By breaking down documents into chunks, RAG systems can focus on the most relevant parts and generate more accurate and concise responses.

from llama_index.core.node_parser import SentenceSplitter
from llama_index.core import VectorStoreIndex, load_index_from_storage
from llama_index.core import Settings
from llama_index.core import StorageContext

def get_build_index(documents, embed_model="local:BAAI/bge-small-en-v1.5", save_dir="./vector_store/index"):
    """
    Builds or loads a vector store index from the given documents.

    Args:
        documents (list[Document]): A list of Document objects.
        embed_model (str, optional): The embedding model to use. Defaults to "local:BAAI/bge-small-en-v1.5".
        save_dir (str, optional): The directory to save or load the index from. Defaults to "./vector_store/index".

    Returns:
        VectorStoreIndex: The built or loaded index.
    """

    # Set index settings
    Settings.llm = watsonx_llm
    Settings.embed_model = embed_model
    Settings.node_parser = SentenceSplitter(chunk_size=1000, chunk_overlap=200)
    Settings.num_output = 512
    Settings.context_window = 3900

    # Check if the save directory exists
    if not os.path.exists(save_dir):
        # Create and load the index
        index = VectorStoreIndex.from_documents(
            [documents], service_context=Settings
        )
        index.storage_context.persist(persist_dir=save_dir)
    else:
        # Load the existing index
        index = load_index_from_storage(
            StorageContext.from_defaults(persist_dir=save_dir),
            service_context=Settings,
        )
    return index

# Get the Vector Index
vector_index = get_build_index(documents=documents, embed_model="local:BAAI/bge-small-en-v1.5", save_dir="./vector_store/index")

This is the last part of RAG: we create a query engine with metadata replacement and sentence transformer reranking. Bruh! What is a re-ranker now?

A re-ranker is a component that reorders the retrieved documents based on their relevance to the query. It uses additional information, such as semantic similarity or context-specific factors, to refine the initial ranking provided by the retrieval system. This helps ensure that the most relevant documents are presented to the user, leading to more accurate and informative responses.

from llama_index.core.postprocessor import MetadataReplacementPostProcessor, SentenceTransformerRerank

def get_query_engine(sentence_index, similarity_top_k=6, rerank_top_n=2):
    """
    Creates a query engine with metadata replacement and sentence transformer reranking.

    Args:
        sentence_index (VectorStoreIndex): The sentence index to use.
        similarity_top_k (int, optional): The number of similar nodes to consider. Defaults to 6.
        rerank_top_n (int, optional): The number of nodes to rerank. Defaults to 2.

    Returns:
        QueryEngine: The query engine.
    """

    postproc = MetadataReplacementPostProcessor(target_metadata_key="window")
    rerank = SentenceTransformerRerank(
        top_n=rerank_top_n, model="BAAI/bge-reranker-base"
    )
    engine = sentence_index.as_query_engine(
        similarity_top_k=similarity_top_k, node_postprocessors=[postproc, rerank]
    )
    return engine

# Create a query engine with the specified parameters
query_engine = get_query_engine(sentence_index=vector_index, similarity_top_k=8, rerank_top_n=5)

# Query the engine with a question
query = 'What is Deep learning?'
response = query_engine.query(query)
prompt = f'''Generate a detailed response for the query asked based only on the context fetched:
            Query: {query}
            Context: {response}

            Instructions:
            1. Show query and your generated response based on context.
            2. Your response should be detailed and should cover every aspect of the context.
            3. Be crisp and concise.
            4. Don't include anything else in your response - no header/footer/code etc
            '''
response = generate_response(prompt)
print(response.text)

'''
OUTPUT - 
Query: What is Deep learning? 

Deep learning is a subset of artificial intelligence that utilizes multiple layers of neurons between the network's inputs and outputs to progressively extract higher-level features from raw input data. 
This technique allows for improved performance in various subfields of AI, such as computer vision, speech recognition, natural language processing, and image classification. 
The multiple layers in deep learning networks are able to identify complex concepts and patterns, including edges, faces, digits, and letters.
The reason behind deep learning's success is not attributed to a recent theoretical breakthrough, but rather the significant increase in computer power, particularly the shift to using graphics processing units (GPUs), which provided a hundred-fold increase in speed. 
Additionally, the availability of vast amounts of training data, including large curated datasets, has also contributed to the success of deep learning.
Overall, deep learning's ability to analyze and extract insights from raw data has led to its widespread application in various fields, and its performance continues to improve with advancements in technology and data availability. '''

How to Fine-Tune the Pipeline

Once you've built a basic RAG pipeline, the next step is to fine-tune it for optimal performance. This involves iteratively adjusting various components and parameters to improve the quality of the generated responses.

How to Evaluate the Pipeline's Performance

To assess the pipeline's effectiveness, you can use metrics like:

• Accuracy: How often does the pipeline generate correct and relevant responses?

• Relevance: How well do the retrieved documents match the query?

• Coherence: Is the generated text well-structured and easy to understand?

• Factuality: Are the generated responses accurate and consistent with known facts?

Iterate on the Index Structure, Embedding Model, and Language Model

You can experiment with different index structures (for example flat index, hierarchical index) to find the one that best suits your data and query patterns. Consider using different embedding models to capture different semantic nuances. Fine-tuning the language model can also improve its ability to generate high-quality responses.

Experiment with Different Hyperparameters

Hyperparameters are settings that control the behaviour of the pipeline components. By experimenting with different values, you can optimize the pipeline's performance. Some examples of hyperparameters include:

• Embedding dimension: The size of the embedding vectors

• Index size: The maximum number of documents to store in the index

• Retrieval threshold: The minimum similarity score for a document to be considered relevant

Real-World Applications of RAG

RAG pipelines have a wide range of applications, including:

• Customer support chatbots: Providing informative and helpful responses to customer inquiries

• Knowledge base search: Efficiently retrieving relevant information from large document collections

• Summarization of large documents: Condensing lengthy documents into concise summaries

• Question answering systems: Answering complex questions based on a given corpus of knowledge

RAG Best Practices and Considerations

To build effective RAG pipelines, consider these best practices:

• Data quality and preprocessing: Ensure your data is clean, consistent, and relevant to your use case. Preprocess the data to remove noise and improve its quality.

• Embedding model selection: Choose an embedding model that is appropriate for your specific domain and task. Consider factors like accuracy, computational efficiency, and interpretability.

• Index optimization: Optimize the index structure and parameters to improve retrieval efficiency and accuracy.

• Ethical considerations and biases: Be aware of potential biases in your data and models. Take steps to mitigate bias and ensure fairness in your RAG pipeline.

Conclusion

RAG pipelines offer a powerful approach to leveraging large language models for a variety of tasks. By carefully selecting and fine-tuning the components of an RAG pipeline, you can build systems that provide informative, accurate, and relevant responses.

Key points to remember:

• RAG combines information retrieval and language generation.

• Llama-Index simplifies the process of building RAG pipelines.

• Fine-tuning is essential for optimizing pipeline performance.

• RAG has a wide range of real-world applications.

• Ethical considerations are crucial in building responsible RAG systems.

As RAG technology continues to evolve, we can expect to see even more innovative and powerful applications in the future. Till then, let's wait for the future to unfold!

We just published a course on the freeCodeCamp.org YouTube channel that will teach you all about Retrieval-Augmented Generation (RAG). Created by Paulo Dichone, this course starts with the basics and progressively covers more advanced aspects, ensuring a comprehensive understanding of RAG and its practical applications. You'll learn the fundamental concepts and components of RAG, build a system for chatting with documents, explore advanced techniques, and understand the drawbacks of naive RAG approaches.

The course begins with an introduction, outlining the course content and objectives, setting the stage for your journey into RAG. You’ll learn the essential principles of Retrieval-Augmented Generation, including how it integrates retrieval mechanisms with generative models. This foundation will help you grasp the various components that make up a RAG system and understand their roles and interactions.

As the course progresses, you'll dive deeper into RAG, examining its inner workings and intricacies. This deep dive will prepare you for a hands-on project where you'll build a RAG system designed to interact with documents, enhancing your practical skills. This practical application is crucial for solidifying your understanding and ability to implement RAG in real-world scenarios.

Following this, the course introduces more sophisticated RAG techniques that can improve system performance and accuracy. You’ll gain insights into the limitations and common issues associated with simple, naive RAG implementations. This understanding is critical, as it highlights the importance of advanced techniques in overcoming these challenges.

One of the advanced techniques you'll learn about is query expansion, which involves generating more relevant answers through expanded queries. This section includes both theoretical explanations and hands-on projects, allowing you to apply query expansion techniques practically. The course also explores using multiple queries to further enhance the effectiveness of a RAG system, providing another hands-on project to deepen your understanding.

To test your skills and knowledge, the course presents a challenge that lets you implement what you've learned. Finally, the course concludes with a look at potential next steps and further learning opportunities in the field of RAG.

By the end of this course, you'll have a solid understanding of Retrieval-Augmented Generation, equipped with the knowledge and skills to build and enhance your own RAG systems. Watch the full course on the freeCodeCamp.org YouTube channel (2-hour watch).

We just posted an in-depth course on the freeCodeCamp.org YouTube channel that will teach you how to implement RAG from scratch. Lance Martin created this course. He is a software engineer at LangChain with a PhD in applied machine learning from Stanford.

What is RAG?

Retrieval-Augmented Generation (RAG) is a powerful framework that integrates retrieval into the sequence generation process. Essentially, RAG operates by fetching relevant documents or data snippets based on a query and then using this retrieved information to generate a coherent and contextually appropriate response. This method is particularly valuable in fields like chatbot development, where the ability to provide precise answers derived from extensive databases of knowledge is crucial.

RAG fundamentally enhances the natural language understanding and generation capabilities of models by allowing them to access and leverage a vast amount of external knowledge. The approach is built upon the synergy between two main components: a retrieval system and a generative model. The retrieval system first identifies relevant information from a knowledge base, which the generative model then uses to craft responses that are not only accurate but also rich in detail and scope.

Course Breakdown

Lance Martin’s course meticulously covers all aspects of RAG, beginning with an overview that sets the stage for deeper exploration. The course is structured to walk students through the entire process of implementing a RAG system from the ground up:

• Indexing: Learners will start by understanding how to create efficient indexing systems to store and retrieve data, which is fundamental for any retrieval-based model.

• Retrieval: This section dives into the mechanics of retrieving the most relevant documents in response to a query.

• Generation: After retrieval, the focus shifts to generating coherent text from the retrieved data, using advanced natural language processing techniques.

• Query Translation: Multiple strategies for translating and refining queries are discussed, including Multi-Query techniques, RAG Fusion, Decomposition, Step Back, and HyDE approaches, each offering unique benefits depending on the application.

• Routing, Query Construction, and Advanced Indexing Techniques: These segments explore more sophisticated elements of RAG systems, such as routing queries to appropriate models, constructing effective queries, and advanced indexing techniques like RAPTOR and ColBERT.

• CRAG and Adaptive RAG: The course also introduces CRAG (Conditional RAG) and Adaptive RAG, enhancements that provide even more flexibility and power to the standard RAG framework.

• Is RAG Really Dead?: Finally, a discussion on the current and future relevance of RAG in research and practical applications, stimulating critical thinking and exploration beyond the course.

Each section is packed with practical exercises, real-life examples, and detailed explanations that ensure students not only learn the theory but also apply the concepts in practical settings.

This course is ideal for software engineers, data scientists, and researchers with a solid foundation in machine learning and natural language processing who are looking to expand their expertise in advanced AI techniques.

Watch the full course on the freeCodeCamp.org YouTube channel (2.5-hour watch).