This guide walks you through a comprehensive workflow for evaluating and selecting the best LLM for your needs. It’s designed for developers who want more than API demos. You’ll see how to design, test, and compare models using real examples and meaningful metrics.

By the end, you’ll understand not only how to benchmark models but why each step matters.

Table of Contents

• Why Public Benchmarks Aren’t Enough

• Step 1: Define the Task and Metrics

• Step 2: Prepare the Data and Generate Outputs

• Step 3: Automate Evaluation with a Judge LLM

• Step 4: Analyze, Visualize, and Interpret

• Step 5: Iterate and Scale Up

• Preparing a Test Dataset

• Cloud Providers and APIs for LLM Access

• Common Pitfalls to Avoid

• How to Run This End to End

• Conclusion: Turning Evaluation into Insight

Why Public Benchmarks Aren’t Enough

Public leaderboards like MMLU, HumanEval, and HellaSwag show how well models perform on general tests, but they don’t reflect the nuances of your real-world application. A model that scores 90% on reasoning might still fail to produce factual or brand-aligned answers for your domain.

For example, if you’re building a customer review summarizer, your goal isn’t just correctness, it’s tone, style, and reliability. You might value concise responses with minimal hallucination over creative but inconsistent writing.

That’s why you need a custom benchmark that mirrors your actual inputs and quality expectations.

Step 1: Define the Task and Metrics

To start, you’ll want to decide up front what success looks like for your application by translating a product need (for example, a short, factual review summarizer) into measurable criteria such as accuracy, factuality, conciseness, latency, and cost. Clear, specific goals make the rest of the pipeline meaningful and comparable.

Example task: Summarize user reviews into concise, factual one-liners.

Key Metrics

• Accuracy: Does the summary reflect the correct information?

• Factuality: Does it avoid hallucinations?

• Conciseness: Is it short yet meaningful?

• Latency: How long does it take per query?

• Cost: How much do API tokens add up per 1,000 requests?

These metrics will help balance technical trade-offs and real-world constraints.

Step 2: Prepare the Data and Generate Outputs

Now, we’ll build a small but representative test set and generate candidate outputs from each model you plan to evaluate. The purpose is to create comparable inputs and collect the raw outputs you will later score and analyze.

Requirements: Python 3.9+ and pandas installed (pip install pandas).

import pandas as pd

reviews = [
    "The camera quality is great but the battery dies fast.",
    "Love the design and performance, but it's overpriced.",
    "Fast processor, poor sound quality, average screen."
]
references = [
    "Good camera, poor battery.",
    "Excellent design but expensive.",
    "Fast but weak audio and display."
]

# Build a tiny DataFrame for quick iteration
df = pd.DataFrame({"review": reviews, "reference": references})
print("Sample data:")
print(df.head())  # sanity check: confirm shape/columns

Now, generate responses using multiple LLMs through OpenRouter, which unifies different APIs into one.

Requirements: OpenRouter API key set as YOUR_KEY, openrouter Python client installed (pip install openrouter), and access to the models you plan to test.

import openrouter
import time

# Initialize API client
client = openrouter.Client(api_key="YOUR_KEY")

# Replace these placeholders with whichever providers/models you can access
models = ["model-A", "model-B", "model-C"]

results = {}
for model in models:
    print(f"Evaluating {model}...")
    start = time.time()
    outputs = []
    for review in reviews:
        # Keep the prompt identical across models to reduce bias
        res = client.completions.create(
            model=model,
            messages=[{"role": "user", "content": f"Summarize this review: {review}"}]
        )
        outputs.append(res.choices[0].message.content.strip())
    # Store both the raw outputs and a coarse latency figure
    results[model] = {"outputs": outputs, "latency": time.time() - start}

print("Model outputs generated.")

Tip: Even a handful of examples per model can reveal consistent behavior patterns.

Step 3: Automate Evaluation with a Judge LLM

In this step, our goal is to replace slow, inconsistent manual labeling with a repeatable, programmatic judging step. We’ll use a fixed judge model and a short rubric so you can get machine-readable scores that reflect qualitative criteria like tone, clarity, and factuality.

Before we go on, let’s clear something up: what is a model-as-a-judge? A model-as-a-judge (MAAJ) uses one LLM to grade outputs from another LLM against task-specific criteria. By prompting the judge with a clear, consistent rubric, you get structured scores that are repeatable and machine-readable. This is useful for aggregation, tracking, and visualization.

We use a fixed rubric because it minimizes drift between runs, and JSON because it makes the output easy to parse and analyze programmatically.

Here are some tips for reliable judging:

• Use a judge model that follows instructions well and keep it fixed across evaluation runs.

• Calibrate the rubric: start with 2–4 criteria and a simple numeric scale (for example, 1–5).

• Avoid self-judging: prefer a judge from a different provider or model family where possible to reduce shared biases.

• For tie-breakers or fine-grained comparisons, consider pairwise judgments (ask the judge to pick the better of two candidates) and convert preferences into scores.

Requirements: An API key for your judge provider and the official SDK (for example pip install openai).

from openai import OpenAI
import json

client = OpenAI(api_key="YOUR_API_KEY")  # or load from env var

# Clear rubric keeps the judge consistent across runs
PROMPT = """
You are grading summaries on a scale of 1-5 for:
1. Correctness (alignment with the reference)
2. Conciseness (brevity and clarity)
3. Helpfulness (coverage of key points)
Return a JSON object with the scores.
"""

def evaluate(candidate, reference):
    # Provide both reference and candidate to the judge
    msg = f"Reference: {reference}\nCandidate: {candidate}"
    response = client.chat.completions.create(
        model="judge-model",  # keep the judge fixed for fair comparisons
        messages=[
            {"role": "system", "content": PROMPT},
            {"role": "user", "content": msg}
        ]
    )
    # Judge returns JSON; parse into a Python dict
    return json.loads(response.choices[0].message.content)

In the code above,

• The rubric in PROMPT defines scoring dimensions (for example: correctness, conciseness, helpfulness). The judge is instructed to return a JSON object.

• For each candidate and its reference, the judge receives both strings and applies the rubric.

• The judge’s JSON output is parsed with json.loads(...) and aggregated per model to compute averages or distributions.

You can loop through models to automatically gather structured scores.

import statistics

for model, data in results.items():
    scores = [evaluate(cand, ref) for cand, ref in zip(data["outputs"], references)]
    results[model]["scores"] = scores

    avg = {k: statistics.mean([s[k] for s in scores]) for k in scores[0]}
    print(f"\n{model} Average Scores:")
    for k, v in avg.items():
        print(f"  {k}: {v:.2f}")

Step 4: Analyze, Visualize, and Interpret

Our goal here is to turn raw numbers and judge scores into actionable insight. Visualization exposes trade-offs (cost vs. quality vs. latency), highlights variance and outliers, and helps you pick the model that best fits your constraints.

What to visualize and why:

• Latency bars: compare average response time per model. Good for quick performance triage.

• Cost bars: cost per 1,000 requests. Makes budget trade-offs visible.

• Quality distributions: box plots or histograms of judge scores. Shows variance and outliers.

• Quality vs cost scatter: quickly surfaces Pareto-efficient choices.

• Confusion matrices: for classification tasks. Shows where models disagree with ground truth.

• Radar charts: helpful when comparing 3 to 6 metrics across models at once.

The code below builds a simple bar chart from a results dictionary: models_list provides x-axis labels and latencies maps to the bar heights in seconds. You can replicate the pattern for cost or judge-based scores by swapping the y-values.

Requirements: matplotlib installed (pip install matplotlib).

import matplotlib.pyplot as plt

latencies = [results[m]['latency'] for m in results]
models_list = list(results.keys())

plt.bar(models_list, latencies)  # simple bar chart; add styling if needed
plt.title('Model Latency Comparison')
plt.ylabel('Seconds')
plt.show()

The chart is embedded here for reference:

Figure: Model latency comparison (seconds per batch).

Reflection Question: Which metric matters most for your use case: accuracy, speed, or cost?

Step 5: Iterate and Scale Up

At this point, we’ll move from small experiments to a repeatable, automated evaluation pipeline that can run at scale, track regression, and integrate with monitoring and CI. This step is about operationalizing the evaluation so you can confidently detect when a model update helps or harms your product.

Evaluation flow (high level):

1. Dataset (JSONL): a versioned test set with metadata (category, difficulty).

2. Prompt templates: standardized prompts or templates applied uniformly across models.

3. Model runners: parallel execution across a pool of models (cloud APIs or local hosts).

4. Judge + Metrics: compute structured scores (judge JSON) and classical metrics (accuracy, F1).

5. Storage & dashboards: persist results, visualize trends, alert on regressions.

Having this explicit flow helps you choose tooling. Below are two representative frameworks and how they map to the flow so you can see which stages they help with.

Figure: Evaluation pipeline – Data → Models → Judge → Insights → Refinement → Data

Some examples that map to the flow are:

• AWS FMEval: focuses on large-scale evaluation and experiment tracking. It covers dataset adapters, parallel model runners, built-in metrics, and native integration with AWS experiment storage and dashboards. Use it when your data lives on cloud storage and you want tight Bedrock or AWS integration for production evaluation runs.

• LangChain Eval: focuses on tight integration with application pipelines. It covers prompt templates, judge and metric hooks, and easy programmatic evaluators that plug directly into LangChain-based model runners. Use it when your evaluation should be embedded in development pipelines or when you already use LangChain for orchestration.

from fmeval import DataConfig, ModelRunner, EvaluationSet

cfg = DataConfig(dataset_uri="s3://your-dataset/reviews.jsonl")  # JSONL test set
runner = ModelRunner(model_id="model-id")       # pick a model to evaluate
eval_set = EvaluationSet(config=cfg, runner=runner)
# Run evaluation with a simple metric; swap in your custom metric as needed
eval_set.evaluate(metric="accuracy")
# Persist results for dashboards or regression tracking
eval_set.save("./results.json")

You’ll want to run scheduling and drift tracking regularly – for example, nightly or weekly evals on a fixed test set. Send an alert when a model update drops a score or increases latency beyond a threshold.

Preparing a Test Dataset

A well-prepared test dataset is the foundation of reliable model evaluation. Here are a few best practices, followed by a concrete example:

• Reflect real use cases: use authentic data from your domain such as customer queries, logs, or user reviews.

• Diversify examples: include easy, typical, and edge-case scenarios to measure robustness.

• Expert annotation: have domain experts provide clear reference outputs or ground truth labels.

• Keep it separate: ensure the test dataset is not reused from training or fine-tuning.

• Update regularly: add new examples to reflect changing user behavior or data drift.

• Version everything: track dataset versions, annotation changes, and evaluation notes.

• Quality over quantity: start small but ensure examples are accurate and representative.

Small JSONL test set

Create a line-delimited JSON (JSONL) file where each line is a JSON object with two required fields: input (the prompt) and reference (the expected output). This simple, tooling-friendly format is accepted by most evaluation frameworks and is easy to version, diff, and slice.

Optionally add metadata fields such as category, difficulty, or source to enable filtered analysis and targeted slicing during evaluation.

{"input": "The camera quality is great but the battery dies fast.", "reference": "Good camera, poor battery."}
{"input": "Love the design and performance, but it's overpriced.", "reference": "Excellent design but expensive."}
{"input": "Fast processor, poor sound quality, average screen.", "reference": "Fast but weak audio and display."}

Helper script to produce JSONL:

import json
samples = [
    {"input": "The camera quality is great but the battery dies fast.", "reference": "Good camera, poor battery."},
    {"input": "Love the design and performance, but it's overpriced.", "reference": "Excellent design but expensive."},
    {"input": "Fast processor, poor sound quality, average screen.", "reference": "Fast but weak audio and display."}
]
with open("reviews_test.jsonl", "w") as f:
    for row in samples:
        f.write(json.dumps(row) + "\n")
print("Wrote reviews_test.jsonl")

You can add fields like category or difficulty to filter and slice results later.

Even a compact, well-designed test set can highlight major model differences and guide better deployment decisions.

Cloud Providers and APIs for LLM Access

Before you can benchmark different large language models, you need reliable ways to access them. Most LLMs are hosted behind APIs or cloud platforms that expose standard interfaces for sending prompts and receiving outputs. Choosing the right provider affects not only which models you can test, but also your results for latency, throughput, and cost.

Now, we’ll look at some of the main options for accessing LLMs. These range from commercial APIs like OpenAI and Anthropic, to open-source options like Hugging Face, and enterprise platforms like AWS Bedrock and Azure OpenAI.

Understanding these platforms will help you design realistic benchmarks that reflect the infrastructure you’ll actually deploy in production.

• OpenAI and Anthropic: Reliable APIs offering strong reasoning and creative models.

• Google Gemini and Cohere: Strong multimodal and enterprise-friendly options.

• OpenRouter: Simplifies access to multiple providers with a single API key.

• Hugging Face: Great for open-source experimentation and deployment flexibility.

• AWS Bedrock and Azure OpenAI: Enterprise-grade platforms with security, compliance, and scalability.

Use a unified testing approach for flexible experiments and a production cloud provider when you need compliance and scalability.

Once you’ve decided where to source your models, you can run consistent benchmarks across providers using a unified API interface. This helps make sure your comparisons reflect real deployment conditions.

Common Pitfalls to Avoid

Below are five common mistakes, why they matter, and what to do instead. Keep this checklist handy when designing experiments or reviewing results.

1. Using the same model as both generator and judge
Shared biases inflate scores and hide errors. Instead, you can use a separate judge (different provider, family, or size) and keep the judge fixed across runs.

2. Relying only on aggregate numbers
Averages hide tone, factuality issues, and edge-case failures. Instead, you should maintain a curated error-analysis set and do periodic manual spot checks.

3. Ignoring latency and cost
A high-scoring model may be too slow or expensive for production SLAs. Instead, you can track latency distributions and projected monthly cost alongside quality metrics.

4. Not versioning datasets or prompts
Silent changes break comparability and reproducibility. Make sure you store datasets and prompt templates in version control and log run metadata and data hashes for every evaluation.

5. Overfitting to the test set
Repeated tuning on a tiny set undermines generalization. Instead, keep a holdout set, rotate or refresh samples, and expand the dataset over time.

Conclusion: Turning Evaluation into Insight

Benchmarking helps you score models as well as understand them. Through this workflow, you’ve seen how to:

1. Define tasks and meaningful metrics.

2. Generate model outputs programmatically.

3. Evaluate using a judge model for consistency.

4. Visualize trade-offs to make data-driven choices.

As models evolve, your benchmarking pipeline becomes a living system. It helps you track progress, validate improvements, and justify decisions with evidence.

Choosing an LLM is no longer guesswork. It’s now a structured experiment grounded in real data. Each iteration builds intuition and confidence. Over time, you’ll know not just which model performs best, but why.

In this tutorial, I’ll walk you through the process step-by-step. You don’t need prior experience with machine learning models, as we’ll build up the system gradually and test each part as we go. By the end, you will have a fully local mobile voice assistant powered by:

• Whisper for Automatic Speech Recognition (ASR)

• Machine Learning Compiler (MLC) LLM for on-device reasoning

• System Text-to-Speech (TTS) using built-in Android TTS

Your assistant will be able to:

• Understand your voice commands offline

• Respond to you with synthesized speech

• Perform tool calling actions (such as controlling smart devices)

• Store personal memories and preferences

• Use Retrieval-Augmented Generation (RAG) to answer questions from your own notes

• Perform multi-step agentic workflows such as generating a morning briefing and optionally sending the summary to a contact

This tutorial focuses on Android using Termux (the terminal environment for Android) for a fully local workflow.

Table of Contents

• System Overview

• Requirements

• Step 1: Test Microphone and Audio Playback on Android

• Step 2: Install and Run Whisper for ASR

• Step 3: Install a Local LLM with MLC

• Step 4: Local Text-to-Speech (TTS)

• Step 5: The Core Voice Loop

• Step 6: Tool Calling (Make It Act)

• Step 7: Memory and Personalization

• Step 8: Retrieval-Augmented Generation (RAG)

• Step 9: Multi-Step Agentic Workflow

• Conclusion and Next Steps

System Overview

This diagram shows how your voice moves through the assistant: speech in → transcription → reasoning → action → spoken reply.

This pipeline describes the core flow:

• You speak into the microphone.

• Whisper converts audio into text.

• The local LLM interprets your request.

• The assistant may call tools (for example, send notifications or create events).

• The response is spoken aloud using the device’s Text-to-Speech system.

Key Concepts Used in This Tutorial

• Automatic Speech Recognition (ASR): Converts your speech into text. We use Whisper or Faster‑Whisper.

• Local Large Language Model (LLM): A reasoning model running on your phone using the MLC engine.

• Text‑to‑Speech (TTS): Converts text back to speech. We use Android’s built‑in system TTS.

• Tool Calling: Allows the assistant to perform actions (for example, sending a notification or creating an event).

• Memory: Stores personalized facts the assistant learns during conversation.

• Retrieval‑Augmented Generation (RAG): Lets the assistant reference your documents or notes.

• Agent Workflow: A multi‑step chain where the assistant uses multiple abilities together.

Requirements

What you should already be familiar with:

• Basic command line usage (running commands, navigating directories)

• Very basic Python (calling a function, editing a .py script)

You do not need to have:

• Machine learning experience

• A deep understanding of neural networks

• Prior experience with speech or audio models

Here are the tools and technologies you’ll need to follow along:

• An Android phone with Snapdragon 8+ Gen 1 or newer recommended (older devices will still work, but responses may be slower)

• Termux

• Python 3.9+ inside Termux

• Enough free storage (at least 4–6 GB) to store the model and audio files

Why these requirements matter:

Whisper and Llama models run on-device, so the phone must handle real‑time compute. MLC optimizes models for your device's GPU / NPU, so newer processors will run faster and cooler. And system TTS and Termux APIs let the assistant speak and interact with the phone locally.

If your phone is older or mid‑range, switch the model in Step 3 to Phi-3.5-Mini which is smaller and faster.

We’ll start by setting up your Android environment with Termux, Python, media access, and storage permissions so later steps can record audio, run models, and speak.

Run it now:

# In Termux
pkg update && pkg upgrade -y
pkg install -y python git ffmpeg termux-api
termux-setup-storage  # grant storage permission

Step 1: Test Microphone and Audio Playback on Android

What this step does: Verifies that your device microphone and speakers work correctly through Termux before connecting them to the voice assistant.

On-device assistants need reliable access to the microphone and speakers. On Android, Termux provides utilities to record audio and play media. This avoids complex audio dependencies and works on more devices.

These commands let you quickly test your microphone and audio playback without writing any code. This is useful to verify that your device permissions and audio paths are working before introducing Whisper or TTS.

• termux-microphone-record records from the device microphone to a .wav file

• termux-media-player plays audio files

• termux-tts-speak speaks text using the system TTS voice (fast fallback)

Run it now:

# Start a 4 second recording
termux-microphone-record -f in.wav -l 4 && termux-microphone-record -q

# Play back the captured audio
termux-media-player play in.wav

# Speak text via system TTS (fallback if you do not install a Python TTS)
termux-tts-speak "Hello, this is your on-device assistant running locally."

Step 2: Install and Run Whisper for ASR

What this step does: Converts recorded speech into text so the language model can understand what you said.

Whisper listens to your audio recording and converts it into text. Smaller versions like tiny or base run faster on most phones and are good enough for everyday commands.

Install Whisper:

pip install openai-whisper

If you run into installation issues, you can use Faster‑Whisper instead:

pip install faster-whisper

Below is a small Python script that takes the recorded audio file and turns it into text. It tries Whisper first, and if that isn’t available, it will automatically fall back to Faster‑Whisper.

# Convert recorded speech to text (asr_transcribe.py)
import sys

# Try Whisper, fallback to Faster-Whisper if needed
try:
    import whisper
    use_faster = False
except Exception:
    use_faster = True

if use_faster:
    from faster_whisper import WhisperModel
    model = WhisperModel("tiny.en")
    segments, info = model.transcribe(sys.argv[1])
    text = " ".join(s.text for s in segments)
    print(text.strip())
else:
    model = whisper.load_model("tiny.en")
    result = model.transcribe(sys.argv[1], fp16=False)
    print(result["text"].strip())

Run it now:

# Record 4 seconds and transcribe
termux-microphone-record -f in.wav -l 4 && termux-microphone-record -q
python asr_transcribe.py in.wav

Step 3: Install a Local LLM with MLC

What this step does: Installs and tests the on-device reasoning model that will generate responses to transcribed speech.

MLC compiles transformer models to mobile GPUs and Neural Processing Units, enabling on-device inference. You will run an instruction-tuned model with 4-bit or 8-bit weights for speed.

Install the command-line interface like this:

# Clone and install Python bindings (for scripting) and CLI
git clone https://github.com/mlc-ai/mlc-llm.git
cd mlc-llm
pip install -r requirements.txt
pip install -e python

We will use Llama 3 8B Instruct q4 because it offers strong reasoning while still running on many recent Android devices. If your phone has less memory or you want faster responses, you can swap in Phi-3.5 Mini (about 3.8B) without changing any code.

Download a mobile-optimized model:

mlc_llm download Llama-3-8B-Instruct-q4f16_1

We will use a short Python script to send text to the model and print the response. This lets us verify that the model is installed correctly before we connect it to audio.

# Local LLM text generation (local_llm.py)
from mlc_llm import MLCEngine
import sys

engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")
prompt = sys.argv[1] if len(sys.argv) > 1 else "Hello"
resp = engine.chat([{"role": "user", "content": prompt}])
# The engine may return different structures across versions
reply_text = resp.get("message", resp) if isinstance(resp, dict) else str(resp)
print(reply_text)

Run it now:

python local_llm.py "Summarize this in one sentence: building a local voice assistant on Android"

Step 4: Local Text-to-Speech (TTS)

What this step does: Turns the model’s text responses into spoken audio so the assistant can talk back.

This step converts the text returned by the model into spoken audio so the assistant can talk back. It uses the built-in Android Text-to-Speech voice and requires no additional Python packages.

termux-tts-speak "Hello, I am running entirely on your device."

This is the voice output method we will use throughout the tutorial.

Step 5: The Core Voice Loop

What this step does: Connects speech recognition, language model reasoning, and speech synthesis into a single interactive conversation loop.

This loop ties together recording, transcription, response generation, and playback.

# Core voice loop tying ASR + LLM + TTS (voice_loop.py)
import subprocess, os

def run(cmd): return subprocess.check_output(cmd).decode().strip()

print("Listening...")
subprocess.run(["termux-microphone-record", "-f", "in.wav", "-l", "4"]) ; subprocess.run(["termux-microphone-record", "-q"])
text = run(["python", "asr_transcribe.py", "in.wav"])
reply = run(["python", "local_llm.py", text])
try:
    subprocess.run(["python", "speak_xtts.py", reply]); subprocess.run(["termux-media-player", "play", "out.wav"])
except:
    subprocess.run(["termux-tts-speak", reply])

Run:

python voice_loop.py

Step 6: Tool Calling (Make It Act)

What this step does: Enables the assistant to perform actions – not just reply – by calling real functions on your device.

Tool calling lets the assistant perform actions, not just answer. When the model recognizes an action request, it outputs a small JSON instruction, and your code runs the corresponding function. You show the model which tools exist and how to call them. The program intercepts calls and runs the corresponding code.

Example use case:

You say: "Schedule a meeting tomorrow at 3 PM with John."

The assistant:

1. Transcribes what you said.

2. Detects that this is not a question, but an action request.

3. Calls the add_event() function with the correct parameters.

4. Confirms: "Okay, I scheduled that."

Here’s the structure of how tool calls will work:

• Define Python functions such as add_event, control_light

• Provide a schema for the model to output when it wants to call a tool

• Detect that schema in the LLM output and execute the function

# Tool calling functions (tools.py)
import json

def add_event(title: str, date: str) -> dict:
    # Replace with actual calendar integration
    return {"status": "ok", "title": title, "date": date}

TOOLS = {
    "add_event": add_event,
}

def run_tool(call_json: str) -> str:
    """call_json: '{"tool":"add_event","args":{"title":"Dentist","date":"2025-11-10 10:00"}}'"""
    data = json.loads(call_json)
    name = data["tool"]
    args = data.get("args", {})
    if name in TOOLS:
        result = TOOLS[name](**args)
        return json.dumps({"tool_result": result})
    return json.dumps({"error": "unknown tool"})

Prompt the model to use tools:

# LLM wrapper enabling tool use (llm_with_tools.py)
from mlc_llm import MLCEngine
import json, sys

SYSTEM = (
    "You can call tools by emitting a single JSON object with keys 'tool' and 'args'. "
    "Available tools: add_event(title:str, date:str). "
    "If no tool is needed, answer directly."
)

engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")
user = sys.argv[1]
resp = engine.chat([
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": user},
])
print(resp.get("message", resp) if isinstance(resp, dict) else str(resp))

And then glue it together:

# Run LLM with tool call detection (run_with_tools.py)
import subprocess, json
from tools import run_tool

user = "Add a dentist appointment next Thursday at 10"
raw = subprocess.check_output(["python", "llm_with_tools.py", user]).decode().strip()

# If the model returned a JSON tool call, run it
try:
    data = json.loads(raw)
    if isinstance(data, dict) and "tool" in data:
        print("Tool call:", data)
        print(run_tool(raw))
    else:
        print("Assistant:", raw)
except Exception:
    print("Assistant:", raw)

Run it now:

python run_with_tools.py

Step 7: Memory and Personalization

What this step does: Allows the assistant to remember personal information you share so conversations feel continuous and adaptive.

A helpful assistant should feel like it learns alongside you. Memory allows the system to keep track of small details you mention naturally in conversation.

Without memory, every conversation starts from scratch. With memory, your assistant can remember personal facts (for example, birthdays, favorite music), your routines, device settings, or notes you mention in conversation. This unlocks more natural interactions and enables personalization over time.

You can start with a simple key-value store and expand over time. Your program reads memory before inference and writes back new facts after.

# Simple key-value memory store (memory.py)
import json
from pathlib import Path

MEM_PATH = Path("memory.json")

def mem_load():
    return json.loads(MEM_PATH.read_text()) if MEM_PATH.exists() else {}

def mem_save(mem):
    MEM_PATH.write_text(json.dumps(mem, indent=2))

def remember(key: str, value: str):
    mem = mem_load()
    mem[key] = value
    mem_save(mem)

Use memory in the loop:

# Voice loop with memory loading and updating (voice_loop_with_memory.py)
import subprocess, json
from memory import mem_load, remember

# 1) Record and transcribe
subprocess.run(["termux-microphone-record", "-f", "in.wav", "-l", "4"]) 
subprocess.run(["termux-microphone-record", "-q"]) 
user_text = subprocess.check_output(["python", "asr_transcribe.py", "in.wav"]).decode().strip()

# 2) Load memory and add as system context
mem = mem_load()
SYSTEM = "Known facts: " + json.dumps(mem)

# 3) Ask the model
from mlc_llm import MLCEngine
engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")
resp = engine.chat([
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": user_text},
])
reply = resp.get("message", resp) if isinstance(resp, dict) else str(resp)
print("Assistant:", reply)

# 4) Very simple pattern: if the user said "remember X is Y", store it
if user_text.lower().startswith("remember ") and " is " in user_text:
    k, v = user_text[9:].split(" is ", 1)
    remember(k.strip(), v.strip())

Run it now:

python voice_loop_with_memory.py

Step 8: Retrieval-Augmented Generation (RAG)

What this step does: Lets the assistant search your offline notes or documents at answer time, improving accuracy for personal tasks.

To use RAG, we first install a lightweight vector database, then add documents to it, and later query it when answering questions.

A language model cannot magically know details about your life, your work, or your files unless you give it a way to look things up.

Retrieval-Augmented Generation (RAG) bridges that gap. RAG allows the assistant to search your own stored data at query time. This means the assistant can answer questions about your projects, home details, travel plans, studies, or any personal documents you store completely offline.

RAG allows the assistant to reference your actual notes when answering, instead of relying only on the model's internal training.

Install the vector store:

pip install chromadb

Add and search your notes:

# Local vector DB indexing and querying (rag.py)
from chromadb import Client

client = Client()
notes = client.create_collection("notes")

# Add your documents (repeat as needed)
notes.add(documents=["Contractor quote was 42000 United States Dollars for the extension."], ids=["q1"]) 

# Query the local vector database
results = notes.query(query_texts=["extension quote"], n_results=1)
context = results["documents"][0][0]
print(context)

Use retrieved context in responses:

# LLM answering using retrieved context (llm_with_rag.py)
from mlc_llm import MLCEngine
from chromadb import Client

engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")
client = Client()
notes = client.get_or_create_collection("notes")

question = "What was the quoted amount for the home extension?"
res = notes.query(query_texts=[question], n_results=2)
ctx = "\n".join([d[0] for d in res["documents"]])

SYSTEM = "Use the provided context to answer accurately. If missing, say you do not know.\nContext:\n" + ctx
ans = engine.chat([
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": question},
])
print(ans.get("message", ans) if isinstance(ans, dict) else str(ans))

Run it now:

python rag.py
python llm_with_rag.py

Step 9: Multi-Step Agentic Workflow

What this step does: Combines listening, reasoning, memory, and tool usage into a multi-step routine that runs automatically.

Now that the assistant can listen, respond, remember facts, and call tools, we can combine those abilities into a small routine that performs several steps automatically.

Practical example: "Morning Briefing" on your phone

Goal: when you say "Give me my morning briefing and text it to my partner", the assistant will:

1. Read today's agenda from a local file,

2. summarize it,

3. speak it aloud, and

4. send the summary via SMS using Termux.

Diagram: Multi-step morning briefing workflow with retrieval, summary, speech output, and SMS action.

Prepare your agenda file

This file stores your events for the day. You can edit it manually, generate it, or sync it later if you want.

Create agenda.json in the same folder:

{
  "2025-11-03": [
    {"time": "09:30", "title": "Standup meeting"},
    {"time": "13:00", "title": "Lunch with Priya"},
    {"time": "16:30", "title": "Gym"}
  ]
}

Phone-integrated tools for this workflow:

# Phone-integrated agent tools (tools_phone.py)
import json, subprocess, datetime
from pathlib import Path

AGENDA_PATH = Path("agenda.json")

def load_today_agenda():
    today = datetime.date.today().isoformat()
    if not AGENDA_PATH.exists():
        return []
    data = json.loads(AGENDA_PATH.read_text())
    return data.get(today, [])

def send_sms(number: str, text: str) -> dict:
    # Requires Termux:API and SMS permission
    subprocess.run(["termux-sms-send", "-n", number, text])
    return {"status": "sent", "to": number}

def notify(title: str, content: str) -> dict:
    subprocess.run(["termux-notification", "--title", title, "--content", content])
    return {"status": "notified"}

Create the agent routine:

# Multi-step morning briefing agent (agent_morning.py)
import json, subprocess, os
from mlc_llm import MLCEngine
from tools_phone import load_today_agenda, send_sms, notify

PARTNER_PHONE = os.environ.get("PARTNER_PHONE", "+15551234567")

TOOLS = {
    "send_sms": send_sms,
    "notify": notify,
}

SYSTEM = (
  "You assist on a phone. You may emit a single-line JSON when an action is needed "
  "with keys 'tool' and 'args'. Available tools: send_sms(number:str, text:str), "
  "notify(title:str, content:str). Keep messages concise. If no tool is needed, answer in plain text."
)

engine = MLCEngine(model="Llama-3-8B-Instruct-q4f16_1")

agenda = load_today_agenda()
agenda = load_today_agenda()
agenda_text = "
".join(f"{e['time']} - {e['title']}" for e in agenda) or "No events for today."

user_request = "Give me my morning briefing and text it to my partner." "Give me my morning briefing and text it to my partner."

# 1) Ask LLM for a 2-3 sentence summary to speak
summary = engine.chat([
  {"role": "system", "content": "Summarize this agenda in 2-3 sentences for a morning briefing:"},
  {"role": "user", "content": agenda_text},
])
summary_text = summary.get("message", summary) if isinstance(summary, dict) else str(summary)
print("Briefing:
", summary_text)

# 2) Speak locally (prefer XTTS, fallback to system TTS)
try:
    subprocess.run(["python", "speak_xtts.py", summary_text], check=True)
    subprocess.run(["termux-media-player", "play", "out.wav"]) 
except Exception:
    subprocess.run(["termux-tts-speak", summary_text])

# 3) Ask LLM whether to send SMS and with what text, using tool schema
resp = engine.chat([
  {"role": "system", "content": SYSTEM},
  {"role": "user", "content": f"User said: '{user_request}'. Partner phone is {PARTNER_PHONE}. Summary: {summary_text}"},
])
msg = resp.get("message", resp) if isinstance(resp, dict) else str(resp)

# 4) If the model requested a tool, execute it
try:
    data = json.loads(msg)
    if isinstance(data, dict) and data.get("tool") in TOOLS:
        # Auto-fill phone number if missing
        if data["tool"] == "send_sms" and "number" not in data.get("args", {}):
            data.setdefault("args", {})["number"] = PARTNER_PHONE
        result = TOOLS[data["tool"]](**data.get("args", {}))
        print("Tool result:", result)
    else:
        print("Assistant:", msg)
except Exception:
    print("Assistant:", msg)

Run it now:

export PARTNER_PHONE=+15551234567
python agent_morning.py

This example is realistic on Android because it uses Termux utilities you already installed: local TTS for speech output, termux-sms-send for messaging, and termux-notification for a quick on-device confirmation. You can extend it with a Home Assistant tool later if you have a local server (for example, to toggle lights or set thermostat scenes).

Conclusion and Next Steps

Building a fully local voice assistant is an incremental process. Each step you added – speech recognition, text generation, memory, retrieval, and tool execution – unlocked new capabilities and moved the system closer to behaving like a real assistant.

You built a fully local voice assistant on your phone with:

• On-device Automatic Speech Recognition with Whisper (with Faster-Whisper fallback)

• On-device reasoning with MLC Large Language Model

• Local Text-to-Speech using the built-in system TTS

• Tool calling for real actions

• Memory and personalization

• Retrieval-Augmented Generation for document-based knowledge

• A simple agent loop for multi-step work

From here you can add:

• Wake word detection (for example, Porcupine or open wake word models)

• Device-specific integrations (for example, Home Assistant, smart lighting)

• Better memory schemas and calendars or contacts adapters

Your data never leaves your device, and you control every part of the stack. This is a private, customizable assistant you can expand however you like.