This article is intentionally vendor-neutral. You can implement these patterns using any AI voice platform that supports WebRTC (directly or via an SFU, selective forwarding unit) and server-side token minting. The goal is to help you ship a voice agent architecture that is secure, observable, and operable in production.

Disclosure: This article reflects my personal views and experience. It does not represent the views of my employer or any vendor mentioned.

Table of Contents

• What You'll Build

• How to Avoid Common Production Failures in Voice Agents

• How to Design a Latency Budget for a Real-Time Voice Agent

• Production Voice Agent Architecture (Vendor-Neutral)

  • Step 0: Set Up the Project

  • Step 1: Keep Credentials Server-side

  • Step 2: Build a Backend Token Endpoint

  • Step 3: Connect from the Web Client (WebRTC + SFU)

  • Step 4: Add Client Actions (Agent Suggests, App Executes)

  • Step 5: Add Tool Integrations Safely

  • Step 6: Add post-call processing (where durable value appears)

• Production readiness checklist

• Closing

What You'll Build

By the end, you'll have:

• A web client that streams microphone audio and plays agent audio.

• A backend token endpoint that keeps credentials server-side.

• A safe coordination channel between the agent and the application.

• Structured messages between the application and the agent.

• A production checklist for security, reliability, observability, and cost control.

Prerequisites

You should be comfortable with:

• JavaScript or TypeScript

• Node.js 18+ (so fetch works server-side) and an HTTP framework (Express in examples)

• Browser microphone permissions

• Basic WebRTC concepts (high level is fine)

TL;DR

A production-ready voice agent needs:

• A server-side token service (no secrets in the browser)

• A real-time media plane (WebRTC) for low-latency audio

• A data channel for structured messages between your app and the agent

• Tool guardrails (allowlists, confirmations, timeouts, audit logs)

• Post-call processing (summary, actions, CRM (Customer Relationship Management), tickets)

• Observability-first implementation (state transitions + metrics)

How to Avoid Common Production Failures in Voice Agents

If you've operated distributed systems, you've seen most failures happen at boundaries:

• timeouts and partial connectivity

• retries that amplify load

• unclear ownership between components

• missing observability

• “helpful automation” that becomes unsafe

Voice agents amplify those risks because:

Latency is User Experience: A slow agent feels broken. Conversational UX is less forgiving than web UX.

Audio + UI + Tools is a Distributed System: You coordinate browser audio capture, WebRTC transport, STT (speech-to-text), model reasoning, tool calls, TTS (text-to-speech), and playback buffering. Each stage has different clocks and failure modes.

Security Boundaries are Non-negotiable: A leaked API key is catastrophic. A tool misfire can trigger real-world side effects.

Debuggability determines whether you can ship: If you don't log state transitions and capture post-call artifacts, you can't operate or improve the system safely.

How to Design a Latency Budget for a Real-Time Voice Agent

Conversations have a “feel.” That feel is mostly latency.

A practical guideline:

• Under ~200ms feels instant

• 300–500ms feels responsive

• Over ~700ms feels broken

Your end-to-end latency is the sum of mic capture, network RTT (round-trip time), STT, reasoning, tool execution, TTS, and playback buffering. Budget for it explicitly or you’ll ship a technically correct system that users perceive as unintelligent.

How to Design a Production Voice Agent Architecture (Vendor-Neutral)

A scalable voice agent architecture typically has these layers:

1. Web client: mic capture, audio playback, UI state

2. Token service: short-lived session tokens (secrets stay server-side)

3. Real-time plane: WebRTC media + a data channel

4. Agent runtime: STT → reasoning → TTS, plus tool orchestration

5. Tool layer: external actions behind safety controls

6. Post-call processor: summary + structured outputs after the session ends

This separation makes failure domains and trust boundaries explicit.

Step 0: Set Up the Project

Create a new project directory:

mkdir voice-agent-app
cd voice-agent-app
npm init -y
npm pkg set type=module
npm pkg set scripts.start="node server.js"

Install dependencies:

npm install express dotenv

Create this folder structure:

voice-agent-app/
├── server.js
├── .env
└── public/
    ├── index.html
    └── client.js

Add a .env file:

VOICE_PLATFORM_URL=https://your-provider.example
VOICE_PLATFORM_API_KEY=your_api_key_here

Now you’re ready to implement each part of the system.

Step 1: Keep Credentials Server-side

Treat every API key like production credentials:

• store it in environment variables or a secrets manager

• rotate it if exposed

• never embed it in browser or mobile apps

• avoid logging secrets (log only a short suffix if necessary)

Even if a vendor supports CORS, the browser is not a safe place for long-lived credentials.

Step 2: Build a Backend Token Endpoint

Your backend should:

• authenticate the user

• mint a short-lived session token using your platform API

• return only what the client needs (URL + token + expiry)

Create server.js (Node.js + Express)

import express from "express";
import dotenv from "dotenv";
import path from "path";
import { fileURLToPath } from "url";

dotenv.config();

const app = express();
app.use(express.json());

// Serve the web client from /public
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
app.use(express.static(path.join(__dirname, "public")));

const VOICE_PLATFORM_URL = process.env.VOICE_PLATFORM_URL;
const VOICE_PLATFORM_API_KEY = process.env.VOICE_PLATFORM_API_KEY;

app.post("/api/voice-token", async (req, res) => {
  res.setHeader("Cache-Control", "no-store");

  try {
    if (!VOICE_PLATFORM_URL || !VOICE_PLATFORM_API_KEY) {
      return res.status(500).json({
        error: "Missing VOICE_PLATFORM_URL or VOICE_PLATFORM_API_KEY in .env",
      });
    }

    // TODO: Authenticate the caller before minting tokens.

    const r = await fetch(`${VOICE_PLATFORM_URL}/api/v1/token`, {
      method: "POST",
      headers: {
        "X-API-Key": VOICE_PLATFORM_API_KEY,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ participant_name: "Web User" }),
    });

    if (!r.ok) {
      const detail = await r.text().catch(() => "");
      return res.status(r.status).json({ error: "Token request failed", detail });
    }

    const data = await r.json();

    res.json({
      rtc_url: data.rtc_url || data.livekit_url,
      token: data.token,
      expires_in: data.expires_in,
    });
  } catch (err) {
    res.status(500).json({ error: "Failed to mint token" });
  }
});

app.listen(3000, () => console.log("Open http://localhost:3000"));

Run the server

npm start

Then open: http://localhost:3000

How this code works

• You load credentials from environment variables so secrets never enter the browser.

• The /api/voice-token endpoint calls the voice platform’s token API.

• You return only the rtc_url, token, and expiration time.

• The browser never sees the API key.

• If the provider returns an error, you forward a structured error response.

Production Notes

• rate-limit /api/voice-token (cost + abuse control)

• instrument token mint latency and error rate

• keep TTL short and handle refresh/reconnect

• return minimal fields

Step 3: Connect from the Web Client (WebRTC + SFU)

In this step, you'll build a minimal web UI that:

• Requests a short-lived token from your backend

• Connects to a real-time WebRTC room (often via an SFU)

• Plays the agent's audio track

• Captures and publishes microphone audio

Create public/index.html

<!doctype html>
<html>
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width,initial-scale=1" />
    <title>Voice Agent Demo</title>
  </head>
  <body>
    <h1>Voice Agent Demo</h1>

    <button id="startBtn">Start Call</button>
    <button id="endBtn" disabled>End Call</button>

    <p id="status">Idle</p>

    <script type="module" src="/client.js"></script>
  </body>
</html>

Create public/client.js

Note: This uses a LiveKit-style client SDK to demonstrate the pattern. If you're using a different provider, swap this import and the connect/publish calls for your provider's WebRTC client.

import { Room, RoomEvent, Track } from "https://unpkg.com/livekit-client@2.10.1/dist/livekit-client.esm.mjs";

const startBtn = document.getElementById("startBtn");
const endBtn = document.getElementById("endBtn");
const statusEl = document.getElementById("status");

let room = null;
let intentionallyDisconnected = false;
let audioEls = [];

function setStatus(text) {
  statusEl.textContent = text;
}

function detachAllAudio() {
  for (const el of audioEls) {
    try { el.pause?.(); } catch {}
    el.remove();
  }
  audioEls = [];
}

async function mintToken() {
  const res = await fetch("/api/voice-token", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ participant_name: "Web User" }),
    cache: "no-store",
  });

  if (!res.ok) {
    const detail = await res.text().catch(() => "");
    throw new Error(`Token request failed: ${detail || res.status}`);
  }

  const { rtc_url, token } = await res.json();
  if (!rtc_url || !token) throw new Error("Token response missing rtc_url or token");
  return { rtc_url, token };
}

function wireRoomEvents(r) {
  // 1) Play the agent audio track when subscribed
  r.on(RoomEvent.TrackSubscribed, (track) => {
    if (track.kind !== Track.Kind.Audio) return;

    const el = track.attach();
    audioEls.push(el);
    document.body.appendChild(el);

    // Autoplay restrictions vary by browser/device.
    el.play?.().catch(() => {
      setStatus("Connected (audio may be blocked — click the page to enable)");
    });
  });

  // 2) Reconnect on disconnect (token expiry often shows up this way)
  r.on(RoomEvent.Disconnected, async () => {
    if (intentionallyDisconnected) return;
    setStatus("Disconnected (reconnecting...)");
    await attemptReconnect();
  });
}

async function connectOnce() {
  const { rtc_url, token } = await mintToken();

  const r = new Room();
  wireRoomEvents(r);

  await r.connect(rtc_url, token);

  // Mic permission + publish mic
  try {
    await r.localParticipant.setMicrophoneEnabled(true);
  } catch {
    try { r.disconnect(); } catch {}
    throw new Error("Microphone access denied. Allow mic permission and try again.");
  }

  return r;
}

async function startCall() {
  if (room) return;

  intentionallyDisconnected = false;
  setStatus("Connecting...");

  room = await connectOnce();

  setStatus("Connected");
  startBtn.disabled = true;
  endBtn.disabled = false;
}

async function stopCall() {
  intentionallyDisconnected = true;

  try {
    await room?.localParticipant?.setMicrophoneEnabled(false);
  } catch {}

  try {
    room?.disconnect();
  } catch {}

  room = null;
  detachAllAudio();

  setStatus("Disconnected");
  startBtn.disabled = false;
  endBtn.disabled = true;
}

async function attemptReconnect() {
  // Simplified exponential backoff reconnect.
  // In production, add jitter, max attempts, and better error classification.
  const delaysMs = [250, 500, 1000, 2000];

  for (const delay of delaysMs) {
    if (intentionallyDisconnected) return;

    try {
      // Tear down current state before reconnecting
      try { room?.disconnect(); } catch {}
      room = null;
      detachAllAudio();

      await new Promise((r) => setTimeout(r, delay));

      room = await connectOnce();
      setStatus("Reconnected");
      startBtn.disabled = true;
      endBtn.disabled = false;
      return;
    } catch {
      // keep retrying
    }
  }

  setStatus("Disconnected (reconnect failed)");
  startBtn.disabled = false;
  endBtn.disabled = true;
}

startBtn.addEventListener("click", async () => {
  try {
    await startCall();
  } catch (err) {
    setStatus(err?.message || "Connection failed");
    startBtn.disabled = false;
    endBtn.disabled = true;
    room = null;
    detachAllAudio();
  }
});

endBtn.addEventListener("click", async () => {
  await stopCall();
});

How this Step works (and why these details matter)

• The Start button gives you a user gesture so browsers are more likely to allow audio playback.

• Mic permission is handled explicitly: if the user denies access, you show a clear error and avoid a half-connected session.

• Disconnect cleanup removes audio elements so you don't leak resources across retries.

• The reconnect loop demonstrates the production pattern: if a disconnect happens (often due to token expiry or network churn), the client re-mints a token and reconnects.

In the next step, you'll add a structured data-channel handler to safely process agent-suggested “client actions”.

Handle These Explicitly

Autoplay Restriction Example

Add this to index.html:

<button id="startBtn">Start Call</button>
<button id="endBtn" disabled>End Call</button>
<div id="status"></div>

In client.js:

const startBtn = document.getElementById("startBtn");
const endBtn = document.getElementById("endBtn");
const statusEl = document.getElementById("status");

let room;

startBtn.addEventListener("click", async () => {
  try {
    room = await connectVoice();
    statusEl.textContent = "Connected";
    startBtn.disabled = true;
    endBtn.disabled = false;
  } catch (err) {
    statusEl.textContent = "Connection failed";
  }
});

Microphone denial

try {
  await navigator.mediaDevices.getUserMedia({ audio: true });
} catch (err) {
  statusEl.textContent = "Microphone access denied";
  throw err;
}

Disconnect cleanup

endBtn.addEventListener("click", () => {
  if (room) {
    room.disconnect();
    statusEl.textContent = "Disconnected";
    startBtn.disabled = false;
    endBtn.disabled = true;
  }
});

Token refresh (simplified)

room.on(RoomEvent.Disconnected, async () => {
  const res = await fetch("/api/voice-token");
  const { rtc_url, token } = await res.json();
  await room.connect(rtc_url, token);
});

Step 4: Add Client Actions (Agent Suggests, App Executes)

A production voice agent often needs to:

• open a runbook/dashboard URL

• show a checklist in the UI

• request confirmation for an irreversible action

• receive structured context (account, region, incident ID)

The key safety rule:

The agent suggests actions. The application validates and executes them.

Use structured messages over the data channel:

{
  "type": "client_action",
  "action": "open_url",
  "payload": { "url": "https://internal.example.com/runbook" },
  "id": "action_123"
}

Add guardrails:

• allowlist permitted actions

• validate payload shape

• confirmation gates for irreversible actions

• idempotency via id

• audit logs for every request and outcome

This boundary limits damage from hallucinations or prompt injection.

// Guardrails: allowlist + validation + idempotency + confirmation

const ALLOWED_ACTIONS = new Set(["open_url", "request_confirm"]);
const EXECUTED_ACTION_IDS = new Set();
const ALLOWED_HOSTS = new Set(["internal.example.com"]);

function parseClientAction(text) {
  let msg;
  try {
    msg = JSON.parse(text);
  } catch {
    return null;
  }

  if (msg?.type !== "client_action") return null;
  if (typeof msg.id !== "string") return null;
  if (!ALLOWED_ACTIONS.has(msg.action)) return null;

  return msg;
}

async function handleClientAction(msg, room) {
  if (EXECUTED_ACTION_IDS.has(msg.id)) return; // idempotency
  EXECUTED_ACTION_IDS.add(msg.id);

  console.log("[client_action]", msg); // audit log (demo)

  if (msg.action === "open_url") {
    const url = msg.payload?.url;
    if (typeof url !== "string") return;

    const u = new URL(url);
    if (!ALLOWED_HOSTS.has(u.host)) {
      console.warn("Blocked navigation to:", u.host);
      return;
    }

    window.open(url, "_blank", "noopener,noreferrer");
    return;
  }

  if (msg.action === "request_confirm") {
    const prompt = msg.payload?.prompt || "Confirm this action?";
    const ok = window.confirm(prompt);

    // Send confirmation back to agent/app
    room.localParticipant.publishData(
  new TextEncoder().encode(
    JSON.stringify({ type: "user_confirmed", id: msg.id, ok })
  ),
  { topic: "client_events", reliable: true }
);
  }
}

room.on(RoomEvent.DataReceived, (payload, participant, kind, topic) => {
  if (topic !== "client_actions") return;

  const text = new TextDecoder().decode(payload);
  const msg = parseClientAction(text);
  if (!msg) return;

  handleClientAction(msg, room);
});

Step 5: Add Tool Integrations Safely

Tools turn a voice agent into automation. Regardless of vendor, enforce these rules:

• timeouts on every tool call

• circuit breakers for flaky dependencies

• audit logs (inputs, outputs, duration, trace IDs)

• explicit confirmation for destructive actions

• credentials stored server-side (never in prompts or clients)

If tools fail, degrade gracefully (“I can’t access that system right now, here’s the manual fallback.”). Silence reads as failure.

Create a server-side tool runner (example)

Paste this into server.js:

const TOOL_ALLOWLIST = {
  get_status: { destructive: false },
  create_ticket: { destructive: true },
};

let failures = 0;
let circuitOpenUntil = 0;

function circuitOpen() {
  return Date.now() < circuitOpenUntil;
}

async function withTimeout(promise, ms) {
  return Promise.race([
    promise,
    new Promise((_, reject) => setTimeout(() => reject(new Error("timeout")), ms)),
  ]);
}

async function runToolSafely(tool, args) {
  if (circuitOpen()) throw new Error("circuit_open");

  try {
    const result = await withTimeout(Promise.resolve({ ok: true, tool, args }), 2000);
    failures = 0;
    return result;
  } catch (err) {
    failures++;
    if (failures >= 3) circuitOpenUntil = Date.now() + 10_000;
    throw err;
  }
}

app.post("/api/tools/run", async (req, res) => {
  const { tool, args, user_confirmed } = req.body || {};

  if (!TOOL_ALLOWLIST[tool]) return res.status(400).json({ error: "Tool not allowed" });

  if (TOOL_ALLOWLIST[tool].destructive && user_confirmed !== true) {
    return res.status(400).json({ error: "Confirmation required" });
  }

  try {
    const started = Date.now();
    const result = await runToolSafely(tool, args);
    console.log("[tool_call]", { tool, ms: Date.now() - started }); // audit log
    res.json({ ok: true, result });
  } catch (err) {
    console.log("[tool_error]", { tool, err: String(err) });
    res.status(500).json({ ok: false, error: "Tool call failed" });
  }
});

Step 6: Add post-call processing (where durable value appears)

After a call ends, generate structured artifacts:

• summary

• action items

• follow-up email draft

• CRM entry or ticket creation

A production pattern:

• store transcript + metadata

• enqueue a background job (queue/worker)

• produce outputs as JSON + a human-readable report

• apply integrations with retries + idempotency

• store a “call report” for audits and incident reviews

Create a post-call webhook endpoint (example)

Paste into server.js:

app.post("/webhooks/call-ended", async (req, res) => {
  const payload = req.body;

  console.log("[call_ended]", {
    call_id: payload.call_id,
    ended_at: payload.ended_at,
  });

  setImmediate(() => processPostCall(payload));
  res.json({ ok: true });
});

function processPostCall(payload) {
  const transcript = payload.transcript || [];
  const summary = transcript.slice(0, 3).map(t => `- \({t.speaker}: \){t.text}`).join("\n");

  const report = {
    call_id: payload.call_id,
    summary,
    action_items: payload.action_items || [],
    created_at: new Date().toISOString(),
  };

  console.log("[call_report]", report);
}

Test it locally

curl -X POST http://localhost:3000/webhooks/call-ended \
  -H "Content-Type: application/json" \
  -d '{
    "call_id": "call_123",
    "ended_at": "2026-02-26T00:10:00Z",
    "transcript": [
      {"speaker": "user", "text": "I need help resetting my password."},
      {"speaker": "agent", "text": "Sure — I can help with that."}
    ],
    "action_items": ["Send password reset link", "Verify account email"]
  }'

Production readiness checklist

Security

• no API keys in the browser

• strict allowlist for client actions

• confirmation gates for destructive actions

• schema validation on all inbound messages

• audit logging for actions and tool calls

Reliability

• reconnect strategy for expired tokens

• timeouts + circuit breakers for tools

• graceful degradation when dependencies fail

• idempotent side effects

Observability

Log state transitions (for example):
listening → thinking → speaking → ended

Track:

• connect failure rate

• end-to-end latency (STT + reasoning + TTS)

• tool error rate

• reconnect frequency

Cost control

• rate-limit token minting and sessions

• cap max call duration

• bound context growth (summarize or truncate)

• track per-call usage drivers (STT/TTS minutes, tool calls)

Optional resources

How to Try a Managed Voice Platform Quickly

If you want a managed provider to test quickly, you can sign up for a Vocal Bridge account and implement these steps using their token minting + real-time session APIs.

But the core production voice agent architecture in this article is vendor-agnostic. You can replace any component (SFU, STT/TTS, agent runtime, tool layer) as long as you preserve the boundaries: secure token service, real-time media, safe tool execution, and strong observability.

Watch a full demo and explore a complete reference repo

If you'd like to see these patterns working together in a realistic scenario (incident triage), here are two optional resources:

- Demo video: Voice-First Incident Triage (end-to-end run)
This is a hackathon run-through showing client actions, decision boundaries for irreversible actions, and a structured post-call summary.

- GitHub repo (architecture + design + working code): https://github.com/natarajsundar/voice-first-incident-triage

These links are optional, you can follow the tutorial end-to-end without them.

Closing

Production-ready voice agents work when you treat them like real-time distributed systems.

Start with the baseline:

• token service + web client + real-time audio

Then layer in:

• controlled client actions

• safe tools

• post-call automation

• observability and cost controls

That’s how you ship a voice agent architecture you can operate. You now have a vendor-neutral reference architecture you can adapt to your stack, with clear trust boundaries, safe tool execution, and operational visibility.

If you’re shipping real-time AI systems, what’s been your biggest production bottleneck so far: latency, reliability, or tool safety? I’d love to hear what you’re seeing in the wild. Connect with me on LinkedIn.

In this tutorial, I’ll walk you through the process step-by-step. You don’t need prior experience with machine learning models, as we’ll build up the system gradually and test each part as we go. By the end, you will have a fully local mobile voice assistant powered by:

• Whisper for Automatic Speech Recognition (ASR)

• Machine Learning Compiler (MLC) LLM for on-device reasoning

• System Text-to-Speech (TTS) using built-in Android TTS

Your assistant will be able to:

• Understand your voice commands offline

• Respond to you with synthesized speech

• Perform tool calling actions (such as controlling smart devices)

• Store personal memories and preferences

• Use Retrieval-Augmented Generation (RAG) to answer questions from your own notes

• Perform multi-step agentic workflows such as generating a morning briefing and optionally sending the summary to a contact

This tutorial focuses on Android using Termux (the terminal environment for Android) for a fully local workflow.

Table of Contents

• System Overview

• Requirements

• Step 1: Test Microphone and Audio Playback on Android

• Step 2: Install and Run Whisper for ASR

• Step 3: Install a Local LLM with MLC

• Step 4: Local Text-to-Speech (TTS)

• Step 5: The Core Voice Loop

• Step 6: Tool Calling (Make It Act)

• Step 7: Memory and Personalization

• Step 8: Retrieval-Augmented Generation (RAG)

• Step 9: Multi-Step Agentic Workflow

• Conclusion and Next Steps

System Overview

This diagram shows how your voice moves through the assistant: speech in → transcription → reasoning → action → spoken reply.

This pipeline describes the core flow:

• You speak into the microphone.

• Whisper converts audio into text.

• The local LLM interprets your request.

• The assistant may call tools (for example, send notifications or create events).

• The response is spoken aloud using the device’s Text-to-Speech system.

Key Concepts Used in This Tutorial

• Automatic Speech Recognition (ASR): Converts your speech into text. We use Whisper or Faster‑Whisper.

• Local Large Language Model (LLM): A reasoning model running on your phone using the MLC engine.

• Text‑to‑Speech (TTS): Converts text back to speech. We use Android’s built‑in system TTS.

• Tool Calling: Allows the assistant to perform actions (for example, sending a notification or creating an event).

• Memory: Stores personalized facts the assistant learns during conversation.

• Retrieval‑Augmented Generation (RAG): Lets the assistant reference your documents or notes.

• Agent Workflow: A multi‑step chain where the assistant uses multiple abilities together.

Requirements

What you should already be familiar with:

• Basic command line usage (running commands, navigating directories)

• Very basic Python (calling a function, editing a .py script)

You do not need to have:

• Machine learning experience

• A deep understanding of neural networks

• Prior experience with speech or audio models

Here are the tools and technologies you’ll need to follow along:

• An Android phone with Snapdragon 8+ Gen 1 or newer recommended (older devices will still work, but responses may be slower)

• Termux

• Python 3.9+ inside Termux

• Enough free storage (at least 4–6 GB) to store the model and audio files

Why these requirements matter:

Whisper and Llama models run on-device, so the phone must handle real‑time compute. MLC optimizes models for your device's GPU / NPU, so newer processors will run faster and cooler. And system TTS and Termux APIs let the assistant speak and interact with the phone locally.

If your phone is older or mid‑range, switch the model in Step 3 to Phi-3.5-Mini which is smaller and faster.

We’ll start by setting up your Android environment with Termux, Python, media access, and storage permissions so later steps can record audio, run models, and speak.

Run it now:

# In Termux
pkg update && pkg upgrade -y
pkg install -y python git ffmpeg termux-api
termux-setup-storage  # grant storage permission

Step 1: Test Microphone and Audio Playback on Android

What this step does: Verifies that your device microphone and speakers work correctly through Termux before connecting them to the voice assistant.

On-device assistants need reliable access to the microphone and speakers. On Android, Termux provides utilities to record audio and play media. This avoids complex audio dependencies and works on more devices.

These commands let you quickly test your microphone and audio playback without writing any code. This is useful to verify that your device permissions and audio paths are working before introducing Whisper or TTS.

• termux-microphone-record records from the device microphone to a .wav file

• termux-media-player plays audio files

• termux-tts-speak speaks text using the system TTS voice (fast fallback)

Run it now:

# Start a 4 second recording
termux-microphone-record -f in.wav -l 4 && termux-microphone-record -q

# Play back the captured audio
termux-media-player play in.wav

# Speak text via system TTS (fallback if you do not install a Python TTS)
termux-tts-speak "Hello, this is your on-device assistant running locally."

Step 2: Install and Run Whisper for ASR

What this step does: Converts recorded speech into text so the language model can understand what you said.

Whisper listens to your audio recording and converts it into text. Smaller versions like tiny or base run faster on most phones and are good enough for everyday commands.

Install Whisper:

pip install openai-whisper

If you run into installation issues, you can use Faster‑Whisper instead:

pip install faster-whisper

Below is a small Python script that takes the recorded audio file and turns it into text. It tries Whisper first, and if that isn’t available, it will automatically fall back to Faster‑Whisper.

# Convert recorded speech to text (asr_transcribe.py)
import sys

# Try Whisper, fallback to Faster-Whisper if needed
try:
    import whisper
    use_faster = False
except Exception:
    use_faster = True

if use_faster:
    from faster_whisper import WhisperModel
    model = WhisperModel("tiny.en")
    segments, info = model.transcribe(sys.argv[1])
    text = " ".join(s.text for s in segments)
    print(text.strip())
else:
    model = whisper.load_model("tiny.en")
    result = model.transcribe(sys.argv[1], fp16=False)
    print(result["text"].strip())

Run it now:

# Record 4 seconds and transcribe
termux-microphone-record -f in.wav -l 4 && termux-microphone-record -q
python asr_transcribe.py in.wav

Step 3: Install a Local LLM with MLC

What this step does: Installs and tests the on-device reasoning model that will generate responses to transcribed speech.

MLC compiles transformer models to mobile GPUs and Neural Processing Units, enabling on-device inference. You will run an instruction-tuned model with 4-bit or 8-bit weights for speed.

Install the command-line interface like this:

# Clone and install Python bindings (for scripting) and CLI
git clone https://github.com/mlc-ai/mlc-llm.git
cd mlc-llm
pip install -r requirements.txt
pip install -e python

We will use Llama 3 8B Instruct q4 because it offers strong reasoning while still running on many recent Android devices. If your phone has less memory or you want faster responses, you can swap in Phi-3.5 Mini (about 3.8B) without changing any code.

Download a mobile-optimized model:

mlc_llm download Llama-3-8B-Instruct-q4f16_1

We will use a short Python script to send text to the model and print the response. This lets us verify that the model is installed correctly before we connect it to audio.

# Local LLM text generation (local_llm.py)
from mlc_llm import MLCEngine
import sys

engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")
prompt = sys.argv[1] if len(sys.argv) > 1 else "Hello"
resp = engine.chat([{"role": "user", "content": prompt}])
# The engine may return different structures across versions
reply_text = resp.get("message", resp) if isinstance(resp, dict) else str(resp)
print(reply_text)

Run it now:

python local_llm.py "Summarize this in one sentence: building a local voice assistant on Android"

Step 4: Local Text-to-Speech (TTS)

What this step does: Turns the model’s text responses into spoken audio so the assistant can talk back.

This step converts the text returned by the model into spoken audio so the assistant can talk back. It uses the built-in Android Text-to-Speech voice and requires no additional Python packages.

termux-tts-speak "Hello, I am running entirely on your device."

This is the voice output method we will use throughout the tutorial.

Step 5: The Core Voice Loop

What this step does: Connects speech recognition, language model reasoning, and speech synthesis into a single interactive conversation loop.

This loop ties together recording, transcription, response generation, and playback.

# Core voice loop tying ASR + LLM + TTS (voice_loop.py)
import subprocess, os

def run(cmd): return subprocess.check_output(cmd).decode().strip()

print("Listening...")
subprocess.run(["termux-microphone-record", "-f", "in.wav", "-l", "4"]) ; subprocess.run(["termux-microphone-record", "-q"])
text = run(["python", "asr_transcribe.py", "in.wav"])
reply = run(["python", "local_llm.py", text])
try:
    subprocess.run(["python", "speak_xtts.py", reply]); subprocess.run(["termux-media-player", "play", "out.wav"])
except:
    subprocess.run(["termux-tts-speak", reply])

Run:

python voice_loop.py

Step 6: Tool Calling (Make It Act)

What this step does: Enables the assistant to perform actions – not just reply – by calling real functions on your device.

Tool calling lets the assistant perform actions, not just answer. When the model recognizes an action request, it outputs a small JSON instruction, and your code runs the corresponding function. You show the model which tools exist and how to call them. The program intercepts calls and runs the corresponding code.

Example use case:

You say: "Schedule a meeting tomorrow at 3 PM with John."

The assistant:

1. Transcribes what you said.

2. Detects that this is not a question, but an action request.

3. Calls the add_event() function with the correct parameters.

4. Confirms: "Okay, I scheduled that."

Here’s the structure of how tool calls will work:

• Define Python functions such as add_event, control_light

• Provide a schema for the model to output when it wants to call a tool

• Detect that schema in the LLM output and execute the function

# Tool calling functions (tools.py)
import json

def add_event(title: str, date: str) -> dict:
    # Replace with actual calendar integration
    return {"status": "ok", "title": title, "date": date}

TOOLS = {
    "add_event": add_event,
}

def run_tool(call_json: str) -> str:
    """call_json: '{"tool":"add_event","args":{"title":"Dentist","date":"2025-11-10 10:00"}}'"""
    data = json.loads(call_json)
    name = data["tool"]
    args = data.get("args", {})
    if name in TOOLS:
        result = TOOLS[name](**args)
        return json.dumps({"tool_result": result})
    return json.dumps({"error": "unknown tool"})

Prompt the model to use tools:

# LLM wrapper enabling tool use (llm_with_tools.py)
from mlc_llm import MLCEngine
import json, sys

SYSTEM = (
    "You can call tools by emitting a single JSON object with keys 'tool' and 'args'. "
    "Available tools: add_event(title:str, date:str). "
    "If no tool is needed, answer directly."
)

engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")
user = sys.argv[1]
resp = engine.chat([
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": user},
])
print(resp.get("message", resp) if isinstance(resp, dict) else str(resp))

And then glue it together:

# Run LLM with tool call detection (run_with_tools.py)
import subprocess, json
from tools import run_tool

user = "Add a dentist appointment next Thursday at 10"
raw = subprocess.check_output(["python", "llm_with_tools.py", user]).decode().strip()

# If the model returned a JSON tool call, run it
try:
    data = json.loads(raw)
    if isinstance(data, dict) and "tool" in data:
        print("Tool call:", data)
        print(run_tool(raw))
    else:
        print("Assistant:", raw)
except Exception:
    print("Assistant:", raw)

Run it now:

python run_with_tools.py

Step 7: Memory and Personalization

What this step does: Allows the assistant to remember personal information you share so conversations feel continuous and adaptive.

A helpful assistant should feel like it learns alongside you. Memory allows the system to keep track of small details you mention naturally in conversation.

Without memory, every conversation starts from scratch. With memory, your assistant can remember personal facts (for example, birthdays, favorite music), your routines, device settings, or notes you mention in conversation. This unlocks more natural interactions and enables personalization over time.

You can start with a simple key-value store and expand over time. Your program reads memory before inference and writes back new facts after.

# Simple key-value memory store (memory.py)
import json
from pathlib import Path

MEM_PATH = Path("memory.json")

def mem_load():
    return json.loads(MEM_PATH.read_text()) if MEM_PATH.exists() else {}

def mem_save(mem):
    MEM_PATH.write_text(json.dumps(mem, indent=2))

def remember(key: str, value: str):
    mem = mem_load()
    mem[key] = value
    mem_save(mem)

Use memory in the loop:

# Voice loop with memory loading and updating (voice_loop_with_memory.py)
import subprocess, json
from memory import mem_load, remember

# 1) Record and transcribe
subprocess.run(["termux-microphone-record", "-f", "in.wav", "-l", "4"]) 
subprocess.run(["termux-microphone-record", "-q"]) 
user_text = subprocess.check_output(["python", "asr_transcribe.py", "in.wav"]).decode().strip()

# 2) Load memory and add as system context
mem = mem_load()
SYSTEM = "Known facts: " + json.dumps(mem)

# 3) Ask the model
from mlc_llm import MLCEngine
engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")
resp = engine.chat([
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": user_text},
])
reply = resp.get("message", resp) if isinstance(resp, dict) else str(resp)
print("Assistant:", reply)

# 4) Very simple pattern: if the user said "remember X is Y", store it
if user_text.lower().startswith("remember ") and " is " in user_text:
    k, v = user_text[9:].split(" is ", 1)
    remember(k.strip(), v.strip())

Run it now:

python voice_loop_with_memory.py

Step 8: Retrieval-Augmented Generation (RAG)

What this step does: Lets the assistant search your offline notes or documents at answer time, improving accuracy for personal tasks.

To use RAG, we first install a lightweight vector database, then add documents to it, and later query it when answering questions.

A language model cannot magically know details about your life, your work, or your files unless you give it a way to look things up.

Retrieval-Augmented Generation (RAG) bridges that gap. RAG allows the assistant to search your own stored data at query time. This means the assistant can answer questions about your projects, home details, travel plans, studies, or any personal documents you store completely offline.

RAG allows the assistant to reference your actual notes when answering, instead of relying only on the model's internal training.

Install the vector store:

pip install chromadb

Add and search your notes:

# Local vector DB indexing and querying (rag.py)
from chromadb import Client

client = Client()
notes = client.create_collection("notes")

# Add your documents (repeat as needed)
notes.add(documents=["Contractor quote was 42000 United States Dollars for the extension."], ids=["q1"]) 

# Query the local vector database
results = notes.query(query_texts=["extension quote"], n_results=1)
context = results["documents"][0][0]
print(context)

Use retrieved context in responses:

# LLM answering using retrieved context (llm_with_rag.py)
from mlc_llm import MLCEngine
from chromadb import Client

engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")
client = Client()
notes = client.get_or_create_collection("notes")

question = "What was the quoted amount for the home extension?"
res = notes.query(query_texts=[question], n_results=2)
ctx = "\n".join([d[0] for d in res["documents"]])

SYSTEM = "Use the provided context to answer accurately. If missing, say you do not know.\nContext:\n" + ctx
ans = engine.chat([
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": question},
])
print(ans.get("message", ans) if isinstance(ans, dict) else str(ans))

Run it now:

python rag.py
python llm_with_rag.py

Step 9: Multi-Step Agentic Workflow

What this step does: Combines listening, reasoning, memory, and tool usage into a multi-step routine that runs automatically.

Now that the assistant can listen, respond, remember facts, and call tools, we can combine those abilities into a small routine that performs several steps automatically.

Practical example: "Morning Briefing" on your phone

Goal: when you say "Give me my morning briefing and text it to my partner", the assistant will:

1. Read today's agenda from a local file,

2. summarize it,

3. speak it aloud, and

4. send the summary via SMS using Termux.

Diagram: Multi-step morning briefing workflow with retrieval, summary, speech output, and SMS action.

Prepare your agenda file

This file stores your events for the day. You can edit it manually, generate it, or sync it later if you want.

Create agenda.json in the same folder:

{
  "2025-11-03": [
    {"time": "09:30", "title": "Standup meeting"},
    {"time": "13:00", "title": "Lunch with Priya"},
    {"time": "16:30", "title": "Gym"}
  ]
}

Phone-integrated tools for this workflow:

# Phone-integrated agent tools (tools_phone.py)
import json, subprocess, datetime
from pathlib import Path

AGENDA_PATH = Path("agenda.json")

def load_today_agenda():
    today = datetime.date.today().isoformat()
    if not AGENDA_PATH.exists():
        return []
    data = json.loads(AGENDA_PATH.read_text())
    return data.get(today, [])

def send_sms(number: str, text: str) -> dict:
    # Requires Termux:API and SMS permission
    subprocess.run(["termux-sms-send", "-n", number, text])
    return {"status": "sent", "to": number}

def notify(title: str, content: str) -> dict:
    subprocess.run(["termux-notification", "--title", title, "--content", content])
    return {"status": "notified"}

Create the agent routine:

# Multi-step morning briefing agent (agent_morning.py)
import json, subprocess, os
from mlc_llm import MLCEngine
from tools_phone import load_today_agenda, send_sms, notify

PARTNER_PHONE = os.environ.get("PARTNER_PHONE", "+15551234567")

TOOLS = {
    "send_sms": send_sms,
    "notify": notify,
}

SYSTEM = (
  "You assist on a phone. You may emit a single-line JSON when an action is needed "
  "with keys 'tool' and 'args'. Available tools: send_sms(number:str, text:str), "
  "notify(title:str, content:str). Keep messages concise. If no tool is needed, answer in plain text."
)

engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")

agenda = load_today_agenda()
agenda = load_today_agenda()
agenda_text = "
".join(f"{e['time']} - {e['title']}" for e in agenda) or "No events for today."

user_request = "Give me my morning briefing and text it to my partner." "Give me my morning briefing and text it to my partner."

# 1) Ask LLM for a 2-3 sentence summary to speak
summary = engine.chat([
  {"role": "system", "content": "Summarize this agenda in 2-3 sentences for a morning briefing:"},
  {"role": "user", "content": agenda_text},
])
summary_text = summary.get("message", summary) if isinstance(summary, dict) else str(summary)
print("Briefing:
", summary_text)

# 2) Speak locally (prefer XTTS, fallback to system TTS)
try:
    subprocess.run(["python", "speak_xtts.py", summary_text], check=True)
    subprocess.run(["termux-media-player", "play", "out.wav"]) 
except Exception:
    subprocess.run(["termux-tts-speak", summary_text])

# 3) Ask LLM whether to send SMS and with what text, using tool schema
resp = engine.chat([
  {"role": "system", "content": SYSTEM},
  {"role": "user", "content": f"User said: '{user_request}'. Partner phone is {PARTNER_PHONE}. Summary: {summary_text}"},
])
msg = resp.get("message", resp) if isinstance(resp, dict) else str(resp)

# 4) If the model requested a tool, execute it
try:
    data = json.loads(msg)
    if isinstance(data, dict) and data.get("tool") in TOOLS:
        # Auto-fill phone number if missing
        if data["tool"] == "send_sms" and "number" not in data.get("args", {}):
            data.setdefault("args", {})["number"] = PARTNER_PHONE
        result = TOOLS[data["tool"]](**data.get("args", {}))
        print("Tool result:", result)
    else:
        print("Assistant:", msg)
except Exception:
    print("Assistant:", msg)

Run it now:

export PARTNER_PHONE=+15551234567
python agent_morning.py

This example is realistic on Android because it uses Termux utilities you already installed: local TTS for speech output, termux-sms-send for messaging, and termux-notification for a quick on-device confirmation. You can extend it with a Home Assistant tool later if you have a local server (for example, to toggle lights or set thermostat scenes).

Conclusion and Next Steps

Building a fully local voice assistant is an incremental process. Each step you added – speech recognition, text generation, memory, retrieval, and tool execution – unlocked new capabilities and moved the system closer to behaving like a real assistant.

You built a fully local voice assistant on your phone with:

• On-device Automatic Speech Recognition with Whisper (with Faster-Whisper fallback)

• On-device reasoning with MLC Large Language Model

• Local Text-to-Speech using the built-in system TTS

• Tool calling for real actions

• Memory and personalization

• Retrieval-Augmented Generation for document-based knowledge

• A simple agent loop for multi-step work

From here you can add:

• Wake word detection (for example, Porcupine or open wake word models)

• Device-specific integrations (for example, Home Assistant, smart lighting)

• Better memory schemas and calendars or contacts adapters

Your data never leaves your device, and you control every part of the stack. This is a private, customizable assistant you can expand however you like.

In the past year, frontier AI labs such as OpenAI, xAI, Anthropic, Meta, and Google have all released real-time voice services. Yet voice apps also have the highest requirements for latency, privacy, and customization. It’s difficult to have a one-size-fits-all voice AI solution.

In this article, we’ll explore how to use open-source technologies to create voice AI agents that utilize your custom knowledge base, voice style, actions, fine-tuned AI models, and run on your own computer.

What We’ll Cover:

• Prerequisites

• What it Looks Like

• Two Voice AI Approaches

• The Voice AI Orchestrator

  • Configure an ASR

  • Run and configure a VAD

  • Configure an LLM

  • Configure a TTS

  • Configure MCP and actions

• Local AI With LlamaEdge

• Conclusion

Prerequisites

You’ll need to have and know a few things to most effectively follow along with this tutorial:

• Access to a Linux-like system. Mac or Windows WSL suffice too.

• Be comfortable with command line (CLI) tools.

• Be able to run server applications on the Linux system.

• Have/get free API keys from Groq and ElevenLabs.

• Optional: be able to compile and build Rust source code.

• Optional: have/get an EchoKit device or assemble your own.

What it Looks Like

The key software component we will cover is the echokit_server project. It is an open-source agent orchestrator for voice AI applications. That means it coordinates services such as LLMs, ASR, TTS, VAD, MCP, search, knowledge/vector databases, and others to generate intelligent voice responses from user prompts.

The EchoKit server provides a WebSocket interface that allows compatible clients to send and receive voice data to and from it. The echokit_box project provides an ESP32-based firmware that can act as a client to collect audio from the user and play TTS-generated voice from the EchoKit server. You can see a couple of demos here. You can assemble your own EchoKit device or purchase one.

Of course, you can also use a pure software client that conforms to the echokit_server WebSocket interface. The project publishes a JavaScript web page that you can run locally to connect to your own EchoKit server as a reference.

In the rest of the article, I will discuss how it’s implemented and how to deploy the system for your own voice AI applications.

Two Voice AI Approaches

When OpenAI released its “realtime voice” services in October 2024, the consensus was that voice AI required “end-to-end” AI models. Traditional LLMs take text as input and then respond in text. The voice end-to-end models take voice audio data as input and respond in voice audio data as well. The end-to-end models could reduce latency since the voice processing, understanding, and generation are done in a single step.

But an end-to-end model is very difficult to customize. For example, it’s impossible to add your own prompt and knowledge to the context for each LLM request, or to act on the LLM's thinking or tool-call responses, or to clone your own voice for the response.

The second approach is to use an “agent orchestration” service to tie together multiple AI models, using one model’s output as the input for the next model. This allows us to customize or select each model and manipulate or supplement the model input at every step.

• The VAD model is used to detect conversation turns in the user's speech. It determines when the user is finished speaking and is now expecting a response.

• The ASR/STT model turns user speech into text.

• The LLM model generates a text response, including MCP tool calls.

• The TTS model turns the response text into voice.

The issue with multi-model and multi-step orchestration is that it can be slow. A lot of optimizations are needed for this approach to work well. For example, a useful technique is to utilize streaming input and output wherever possible. This way, each model doesn’t have to wait for the complete response from the upstream model.

The EchoKit server is a stream-everything, highly efficient AI model orchestrator. It is entirely written in Rust for stability, safety, and speed.

The Voice AI Orchestrator

The EchoKit server project is an open-source AI service orchestrator focused on real-time voice use cases. It starts up a WebSocket server that listens for streaming audio input and returns streaming audio responses.

You can build the echokit_server project yourself using the Rust toolchain. Or, you can simply download the pre-built binary for your computer.

# for x86 / AMD64 CPUs
curl -LO https://github.com/second-state/echokit_server/releases/download/v0.1.0/echokit_server-v0.1.0-x86_64-unknown-linux-gnu.tar.gz
unzip echokit_server-v0.1.0-x86_64-unknown-linux-gnu.tar.gz

# for arm64 CPUs
curl -LO https://github.com/second-state/echokit_server/releases/download/v0.1.0/echokit_server-v0.1.0-aarch64-unknown-linux-gnu.tar.gz
unzip echokit_server-v0.1.0-aarch64-unknown-linux-gnu.tar.gz

Then, run it as follows:

nohup ./echokit_server &

It reads the config.toml file from the current directory. At the top of the file, you can configure the port on which the WebSocket server listens. You can also specify a WAV file that is downloaded to the connected EchoKit client device as a welcome message.

addr = "0.0.0.0:8000"
hello_wav = "hello.wav"

Configure an ASR

When the EchoKit server receives the user's voice data, it first sends the data to an ASR service to convert it into text.

There are many compelling ASR models available today. The EchoKit server can work with any OpenAI-compatible API providers, such as OpenAI itself, x.ai, OpenRouter, and Groq.

In our example, we use Groq’s Whisper ASR service. Whisper is a state-of-the-art ASR model released by OpenAI. Groq provides specialized hardware chips to run it very fast. You will first get a free API key from Groq. Then, configure the ASR service as follows. Notice the “prompt” for the Whisper model. It is a tried-and-true prompt to reduce hallucination of the Whisper model.

[asr]
url = "https://api.groq.com/openai/v1/audio/transcriptions"
api_key = "gsk_XYZ"
model = "whisper-large-v3"
lang = "en"
prompt = "Hello\n你好\n(noise)\n(bgm)\n(silence)\n"

Run and configure a VAD

In order to carry out a voice conversation, participants must detect each other's intentions and speak only when a turn arises. VAD (Voice Activity Detection) is a specialized AI model used to detect activities and, in particular, when the speaker has finished and expects an answer.

In EchoKit, we have VAD detection on both the device and the server.

• Device-side VAD: It detects human language. The device ignores background noise, music, keyboard sounds, and dog barking. It only sends human voice to the server.

• Server-side VAD: It processes the audio stream in 100ms (0.1s) chunks. Once it detects that the speaker has finished, it sends all transcribed text to the LLM and starts waiting for the LLM’s response stream.

The server-side VAD is optional, since the device-side VAD can also generate “conversation turn” signals. But due to the limited computing resources on the device, adding the server-side VAD can dramatically improve the overall VAD performance.

We’re porting the popular Silero VAD project from Python to Rust, and creating the silero_vad_server project. Build the project as instructed. You can start the VAD server on your EchoKit server’s port 9094 as follows:

VAD_LISTEN=0.0.0.0:9094 nohup target/release/silero_vad_server &

You might be wondering: why port to Rust? While many AI projects are written in Python for ease of development, Rust applications are often much lighter, faster, and safer at deployment. So, we’ll leverage AI tools like RustCoder to port as much Python code as possible to Rust. The EchoKit software stack is largely written in Rust.

The VAD server is a WebSocket service that listens on port 9094. As we discussed, the EchoKit server will stream audio to this WebSocket and stop the ASR when a conversation turn is detected. Therefore, we’ll add the VAD service to the EchoKit server’s ASR config section in config.toml.

[asr]
url = "https://api.groq.com/openai/v1/audio/transcriptions"
api_key = "gsk_XYZ"
model = "whisper-large-v3"
lang = "en"
prompt = "Hello\n你好\n(noise)\n(bgm)\n(silence)\n"
vad_realtime_url = "ws://localhost:9094/v1/audio/realtime_vad"

Configure an LLM

Once the ASR service transcribes the user's voice into text, the next step in the pipeline is the LLM (Large Language Model). It’s the AI service that actually “thinks” and generates an answer in text.

Again, the EchoKit server can work with any OpenAI-compatible API providers for LLMs, such as OpenAI itself, x.ai, OpenRouter, and Groq. Since the voice service is highly sensitive to speed, we’ll choose Groq again. Groq supports a number of open-source LLMs. We’ll choose the gpt-oss-20b model released by OpenAI.

[llm]
llm_chat_url = "https://api.groq.com/openai/v1/chat/completions"
api_key = "gsk_XYZ"
model = "openai/gpt-oss-20b"
history = 20

The “history” field indicates how many messages should be kept in the context. Another crucial feature of an LLM application is the “system prompt,” where you instruct the LLM how it should “behave.” You can specify the system prompt in the EchoKit server config as well.

[[llm.sys_prompts]]
role = "system"
content = """
You are a comedian. Engage in lighthearted and humorous conversation with the user. Tell jokes when appropriate.

"""

Since Groq is very fast, it can process very large system prompts in under one second. You can add a lot more context and instructions to the system prompt. For example, you can give the application “knowledge” about a specific field by putting entire books into the system prompt.

Configure a TTS

Finally, once the LLM generates a text response, the EchoKit server will call a TTS (text to speech) service to convert the text into voice and stream it back to the client device.

While Groq has a TTS service, it’s not particularly compelling. ElevenLabs is a leading TTS provider that offers hundreds of voice characters. It can express emotions and supports easy voice cloning. In the config below, you’ll put in your ElevenLabs API key and select a voice.

[tts]
platform = "Elevenlabs"
token = "sk_xyz"
voice = "VOICE-ID-ABCD"

The ElevenLabs TTS models and API services are all great, but they are not open-source. A very compelling open-source TTS, known as GPT-SoVITS, is also available.

You can port GPT-SoVITS from Python to Rust and create an open-source API server project called gsv_tts. It allows easy cloning of any voice. You can run a gsv_tts API server by following its instructions. Then, you can configure the EchoKit server to stream text to it and receive streaming audio from it.

[tts]
platform = "StreamGSV"
url = "http://gsv_tts.server:port/v1/audio/stream_speech"
speaker = "michael"

Configure MCP and actions

Of course, an “AI agent” is not just about chatting. It is about performing actions on specific tasks. For example, the “US civics test prep” use case, which I shared as an example video at the beginning of this article, requires the agent to get exam questions from a database, and then generate responses that guide the user toward the official answer. This is accomplished using LLM tools and actions.

• The LLM detects that the user is requesting a new question.

• Instead of responding in natural language, it responds with a JSON structure that instructs the agent to "get a new question and answer."

• The EchoKit server intercepts this JSON response and retrieves the question and answer from a database.

• The EchoKit server sends the question and answer back to the LLM.

• The LLM formulates a natural language response based on the question and answer.

• The EchoKit server generates a voice response using its TTS service.

As you can see, the EchoKit server needs to perform a few extra steps behind the scenes before it responds in voice. The EchoKit server leverages the MCP protocol for this. The function to look up questions and answers is provided by an open-source MCP server called ExamPrepAgent.

The MCP protocol standardizes the tools and functions for LLMs to call. There are many MCP servers available for all kinds of different tasks. ExamPrepAgent is just one of them.

We are running this MCP server on port 8003. With the MCP server up and running, you only need to add the following configuration to EchoKit server’s config.toml.

[[llm.mcp_server]]
server = "http://localhost:8003/mcp"
type = "http_streamable"

With MCP integration, the EchoKit AI agent can now perform actions. It can call APIs to send messages, make payments, or even turn electronic devices on or off.

Local AI With LlamaEdge

You’ve now seen the open-source EchoKit device working with the open-source EchoKit server to understand and respond to users in voice. But the AI models we use, while also open-source, run on commercial cloud providers. Can we run AI models using open-source technologies at home?

LlamaEdge is an open-source, cross-platform API server for AI models. It supports many mainstream LLM, ASR, and TTS models across Linux, Mac, Windows, and many CPU/GPU architectures. It’s perfect for running AI models on home or office computers. It also provides OpenAI-compatible API endpoints, which makes them very easy to integrate into the EchoKit server.

To install LlamaEdge and its dependencies, run the following shell command. It will detect your hardware and install the appropriate software that can fully take advantage of your GPUs (if any).

curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install_v2.sh | bash -s

Then, download an open-source LLM model. I am using Google's Gemma model as an example.

curl -LO https://huggingface.co/second-state/gemma-3-4b-it-GGUF/resolve/main/gemma-3-4b-it-Q5_K_M.gguf

Download the cross-platform LlamaEdge API server.

curl -LO https://github.com/second-state/LlamaEdge/releases/latest/download/llama-api-server.wasm

Start an LLamaEdge API server with the Google Gemma LLM model. by default, it listens on localhost port 8080.

wasmedge --dir .:. --nn-preload default:GGML:AUTO:gemma-3-4b-it-Q5_K_M.gguf llama-api-server.wasm -p gemma-3

Test the OpenAI compatible API on that server.

curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"system", "content": "You are a helpful assistant. Try to be as brief as possible."}, {"role":"user", "content": "Where is the capital of Texas?"}]}'

Now, you can add this local LLM service to your EchoKit server configuration.

[llm]
llm_chat_url = "http://localhost:8080/v1/chat/completions"
api_key = "NONE"
model = "default"
history = 20

The LlamaEdge project supports more than LLMs. It runs the Whisper ASR model and the Piper TTS model as OpenAI-compatible API servers as well.

Conclusion

The voice AI agent software stack is complex and deep. EchoKit is an open-source platform that ties together and coordinates all those components. It provides a good vantage point for us to learn about the entire stack.

I can’t wait to see what you build!