This guide walks you through a comprehensive workflow for evaluating and selecting the best LLM for your needs. It’s designed for developers who want more than API demos. You’ll see how to design, test, and compare models using real examples and meaningful metrics.

By the end, you’ll understand not only how to benchmark models but why each step matters.

Table of Contents

• Why Public Benchmarks Aren’t Enough

• Step 1: Define the Task and Metrics

• Step 2: Prepare the Data and Generate Outputs

• Step 3: Automate Evaluation with a Judge LLM

• Step 4: Analyze, Visualize, and Interpret

• Step 5: Iterate and Scale Up

• Preparing a Test Dataset

• Cloud Providers and APIs for LLM Access

• Common Pitfalls to Avoid

• How to Run This End to End

• Conclusion: Turning Evaluation into Insight

Why Public Benchmarks Aren’t Enough

Public leaderboards like MMLU, HumanEval, and HellaSwag show how well models perform on general tests, but they don’t reflect the nuances of your real-world application. A model that scores 90% on reasoning might still fail to produce factual or brand-aligned answers for your domain.

For example, if you’re building a customer review summarizer, your goal isn’t just correctness, it’s tone, style, and reliability. You might value concise responses with minimal hallucination over creative but inconsistent writing.

That’s why you need a custom benchmark that mirrors your actual inputs and quality expectations.

Step 1: Define the Task and Metrics

To start, you’ll want to decide up front what success looks like for your application by translating a product need (for example, a short, factual review summarizer) into measurable criteria such as accuracy, factuality, conciseness, latency, and cost. Clear, specific goals make the rest of the pipeline meaningful and comparable.

Example task: Summarize user reviews into concise, factual one-liners.

Key Metrics

• Accuracy: Does the summary reflect the correct information?

• Factuality: Does it avoid hallucinations?

• Conciseness: Is it short yet meaningful?

• Latency: How long does it take per query?

• Cost: How much do API tokens add up per 1,000 requests?

These metrics will help balance technical trade-offs and real-world constraints.

Step 2: Prepare the Data and Generate Outputs

Now, we’ll build a small but representative test set and generate candidate outputs from each model you plan to evaluate. The purpose is to create comparable inputs and collect the raw outputs you will later score and analyze.

Requirements: Python 3.9+ and pandas installed (pip install pandas).

import pandas as pd

reviews = [
    "The camera quality is great but the battery dies fast.",
    "Love the design and performance, but it's overpriced.",
    "Fast processor, poor sound quality, average screen."
]
references = [
    "Good camera, poor battery.",
    "Excellent design but expensive.",
    "Fast but weak audio and display."
]

# Build a tiny DataFrame for quick iteration
df = pd.DataFrame({"review": reviews, "reference": references})
print("Sample data:")
print(df.head())  # sanity check: confirm shape/columns

Now, generate responses using multiple LLMs through OpenRouter, which unifies different APIs into one.

Requirements: OpenRouter API key set as YOUR_KEY, openrouter Python client installed (pip install openrouter), and access to the models you plan to test.

import openrouter
import time

# Initialize API client
client = openrouter.Client(api_key="YOUR_KEY")

# Replace these placeholders with whichever providers/models you can access
models = ["model-A", "model-B", "model-C"]

results = {}
for model in models:
    print(f"Evaluating {model}...")
    start = time.time()
    outputs = []
    for review in reviews:
        # Keep the prompt identical across models to reduce bias
        res = client.completions.create(
            model=model,
            messages=[{"role": "user", "content": f"Summarize this review: {review}"}]
        )
        outputs.append(res.choices[0].message.content.strip())
    # Store both the raw outputs and a coarse latency figure
    results[model] = {"outputs": outputs, "latency": time.time() - start}

print("Model outputs generated.")

Tip: Even a handful of examples per model can reveal consistent behavior patterns.

Step 3: Automate Evaluation with a Judge LLM

In this step, our goal is to replace slow, inconsistent manual labeling with a repeatable, programmatic judging step. We’ll use a fixed judge model and a short rubric so you can get machine-readable scores that reflect qualitative criteria like tone, clarity, and factuality.

Before we go on, let’s clear something up: what is a model-as-a-judge? A model-as-a-judge (MAAJ) uses one LLM to grade outputs from another LLM against task-specific criteria. By prompting the judge with a clear, consistent rubric, you get structured scores that are repeatable and machine-readable. This is useful for aggregation, tracking, and visualization.

We use a fixed rubric because it minimizes drift between runs, and JSON because it makes the output easy to parse and analyze programmatically.

Here are some tips for reliable judging:

• Use a judge model that follows instructions well and keep it fixed across evaluation runs.

• Calibrate the rubric: start with 2–4 criteria and a simple numeric scale (for example, 1–5).

• Avoid self-judging: prefer a judge from a different provider or model family where possible to reduce shared biases.

• For tie-breakers or fine-grained comparisons, consider pairwise judgments (ask the judge to pick the better of two candidates) and convert preferences into scores.

Requirements: An API key for your judge provider and the official SDK (for example pip install openai).

from openai import OpenAI
import json

client = OpenAI(api_key="YOUR_API_KEY")  # or load from env var

# Clear rubric keeps the judge consistent across runs
PROMPT = """
You are grading summaries on a scale of 1-5 for:
1. Correctness (alignment with the reference)
2. Conciseness (brevity and clarity)
3. Helpfulness (coverage of key points)
Return a JSON object with the scores.
"""

def evaluate(candidate, reference):
    # Provide both reference and candidate to the judge
    msg = f"Reference: {reference}\nCandidate: {candidate}"
    response = client.chat.completions.create(
        model="judge-model",  # keep the judge fixed for fair comparisons
        messages=[
            {"role": "system", "content": PROMPT},
            {"role": "user", "content": msg}
        ]
    )
    # Judge returns JSON; parse into a Python dict
    return json.loads(response.choices[0].message.content)

In the code above,

• The rubric in PROMPT defines scoring dimensions (for example: correctness, conciseness, helpfulness). The judge is instructed to return a JSON object.

• For each candidate and its reference, the judge receives both strings and applies the rubric.

• The judge’s JSON output is parsed with json.loads(...) and aggregated per model to compute averages or distributions.

You can loop through models to automatically gather structured scores.

import statistics

for model, data in results.items():
    scores = [evaluate(cand, ref) for cand, ref in zip(data["outputs"], references)]
    results[model]["scores"] = scores

    avg = {k: statistics.mean([s[k] for s in scores]) for k in scores[0]}
    print(f"\n{model} Average Scores:")
    for k, v in avg.items():
        print(f"  {k}: {v:.2f}")

Step 4: Analyze, Visualize, and Interpret

Our goal here is to turn raw numbers and judge scores into actionable insight. Visualization exposes trade-offs (cost vs. quality vs. latency), highlights variance and outliers, and helps you pick the model that best fits your constraints.

What to visualize and why:

• Latency bars: compare average response time per model. Good for quick performance triage.

• Cost bars: cost per 1,000 requests. Makes budget trade-offs visible.

• Quality distributions: box plots or histograms of judge scores. Shows variance and outliers.

• Quality vs cost scatter: quickly surfaces Pareto-efficient choices.

• Confusion matrices: for classification tasks. Shows where models disagree with ground truth.

• Radar charts: helpful when comparing 3 to 6 metrics across models at once.

The code below builds a simple bar chart from a results dictionary: models_list provides x-axis labels and latencies maps to the bar heights in seconds. You can replicate the pattern for cost or judge-based scores by swapping the y-values.

Requirements: matplotlib installed (pip install matplotlib).

import matplotlib.pyplot as plt

latencies = [results[m]['latency'] for m in results]
models_list = list(results.keys())

plt.bar(models_list, latencies)  # simple bar chart; add styling if needed
plt.title('Model Latency Comparison')
plt.ylabel('Seconds')
plt.show()

The chart is embedded here for reference:

Figure: Model latency comparison (seconds per batch).

Reflection Question: Which metric matters most for your use case: accuracy, speed, or cost?

Step 5: Iterate and Scale Up

At this point, we’ll move from small experiments to a repeatable, automated evaluation pipeline that can run at scale, track regression, and integrate with monitoring and CI. This step is about operationalizing the evaluation so you can confidently detect when a model update helps or harms your product.

Evaluation flow (high level):

1. Dataset (JSONL): a versioned test set with metadata (category, difficulty).

2. Prompt templates: standardized prompts or templates applied uniformly across models.

3. Model runners: parallel execution across a pool of models (cloud APIs or local hosts).

4. Judge + Metrics: compute structured scores (judge JSON) and classical metrics (accuracy, F1).

5. Storage & dashboards: persist results, visualize trends, alert on regressions.

Having this explicit flow helps you choose tooling. Below are two representative frameworks and how they map to the flow so you can see which stages they help with.

Figure: Evaluation pipeline – Data → Models → Judge → Insights → Refinement → Data

Some examples that map to the flow are:

• AWS FMEval: focuses on large-scale evaluation and experiment tracking. It covers dataset adapters, parallel model runners, built-in metrics, and native integration with AWS experiment storage and dashboards. Use it when your data lives on cloud storage and you want tight Bedrock or AWS integration for production evaluation runs.

• LangChain Eval: focuses on tight integration with application pipelines. It covers prompt templates, judge and metric hooks, and easy programmatic evaluators that plug directly into LangChain-based model runners. Use it when your evaluation should be embedded in development pipelines or when you already use LangChain for orchestration.

from fmeval import DataConfig, ModelRunner, EvaluationSet

cfg = DataConfig(dataset_uri="s3://your-dataset/reviews.jsonl")  # JSONL test set
runner = ModelRunner(model_id="model-id")       # pick a model to evaluate
eval_set = EvaluationSet(config=cfg, runner=runner)
# Run evaluation with a simple metric; swap in your custom metric as needed
eval_set.evaluate(metric="accuracy")
# Persist results for dashboards or regression tracking
eval_set.save("./results.json")

You’ll want to run scheduling and drift tracking regularly – for example, nightly or weekly evals on a fixed test set. Send an alert when a model update drops a score or increases latency beyond a threshold.

Preparing a Test Dataset

A well-prepared test dataset is the foundation of reliable model evaluation. Here are a few best practices, followed by a concrete example:

• Reflect real use cases: use authentic data from your domain such as customer queries, logs, or user reviews.

• Diversify examples: include easy, typical, and edge-case scenarios to measure robustness.

• Expert annotation: have domain experts provide clear reference outputs or ground truth labels.

• Keep it separate: ensure the test dataset is not reused from training or fine-tuning.

• Update regularly: add new examples to reflect changing user behavior or data drift.

• Version everything: track dataset versions, annotation changes, and evaluation notes.

• Quality over quantity: start small but ensure examples are accurate and representative.

Small JSONL test set

Create a line-delimited JSON (JSONL) file where each line is a JSON object with two required fields: input (the prompt) and reference (the expected output). This simple, tooling-friendly format is accepted by most evaluation frameworks and is easy to version, diff, and slice.

Optionally add metadata fields such as category, difficulty, or source to enable filtered analysis and targeted slicing during evaluation.

{"input": "The camera quality is great but the battery dies fast.", "reference": "Good camera, poor battery."}
{"input": "Love the design and performance, but it's overpriced.", "reference": "Excellent design but expensive."}
{"input": "Fast processor, poor sound quality, average screen.", "reference": "Fast but weak audio and display."}

Helper script to produce JSONL:

import json
samples = [
    {"input": "The camera quality is great but the battery dies fast.", "reference": "Good camera, poor battery."},
    {"input": "Love the design and performance, but it's overpriced.", "reference": "Excellent design but expensive."},
    {"input": "Fast processor, poor sound quality, average screen.", "reference": "Fast but weak audio and display."}
]
with open("reviews_test.jsonl", "w") as f:
    for row in samples:
        f.write(json.dumps(row) + "\n")
print("Wrote reviews_test.jsonl")

You can add fields like category or difficulty to filter and slice results later.

Even a compact, well-designed test set can highlight major model differences and guide better deployment decisions.

Cloud Providers and APIs for LLM Access

Before you can benchmark different large language models, you need reliable ways to access them. Most LLMs are hosted behind APIs or cloud platforms that expose standard interfaces for sending prompts and receiving outputs. Choosing the right provider affects not only which models you can test, but also your results for latency, throughput, and cost.

Now, we’ll look at some of the main options for accessing LLMs. These range from commercial APIs like OpenAI and Anthropic, to open-source options like Hugging Face, and enterprise platforms like AWS Bedrock and Azure OpenAI.

Understanding these platforms will help you design realistic benchmarks that reflect the infrastructure you’ll actually deploy in production.

• OpenAI and Anthropic: Reliable APIs offering strong reasoning and creative models.

• Google Gemini and Cohere: Strong multimodal and enterprise-friendly options.

• OpenRouter: Simplifies access to multiple providers with a single API key.

• Hugging Face: Great for open-source experimentation and deployment flexibility.

• AWS Bedrock and Azure OpenAI: Enterprise-grade platforms with security, compliance, and scalability.

Use a unified testing approach for flexible experiments and a production cloud provider when you need compliance and scalability.

Once you’ve decided where to source your models, you can run consistent benchmarks across providers using a unified API interface. This helps make sure your comparisons reflect real deployment conditions.

Common Pitfalls to Avoid

Below are five common mistakes, why they matter, and what to do instead. Keep this checklist handy when designing experiments or reviewing results.

1. Using the same model as both generator and judge
Shared biases inflate scores and hide errors. Instead, you can use a separate judge (different provider, family, or size) and keep the judge fixed across runs.

2. Relying only on aggregate numbers
Averages hide tone, factuality issues, and edge-case failures. Instead, you should maintain a curated error-analysis set and do periodic manual spot checks.

3. Ignoring latency and cost
A high-scoring model may be too slow or expensive for production SLAs. Instead, you can track latency distributions and projected monthly cost alongside quality metrics.

4. Not versioning datasets or prompts
Silent changes break comparability and reproducibility. Make sure you store datasets and prompt templates in version control and log run metadata and data hashes for every evaluation.

5. Overfitting to the test set
Repeated tuning on a tiny set undermines generalization. Instead, keep a holdout set, rotate or refresh samples, and expand the dataset over time.

Conclusion: Turning Evaluation into Insight

Benchmarking helps you score models as well as understand them. Through this workflow, you’ve seen how to:

1. Define tasks and meaningful metrics.

2. Generate model outputs programmatically.

3. Evaluate using a judge model for consistency.

4. Visualize trade-offs to make data-driven choices.

As models evolve, your benchmarking pipeline becomes a living system. It helps you track progress, validate improvements, and justify decisions with evidence.

Choosing an LLM is no longer guesswork. It’s now a structured experiment grounded in real data. Each iteration builds intuition and confidence. Over time, you’ll know not just which model performs best, but why.

Let me start by asking you a question: How would you test the performance of a piece of code or a function in Go? Well, you could use benchmark tests.

In this tutorial, I will show you how to use an awesome benchmarking tool that’s built into the Golang testing package.

Let’s go.

What Are Benchmark Tests?

In Go, benchmark tests are used to measure the performance (speed and memory usage) of functions or blocks of code. These tests are part of the Go testing framework and are written in the same files as unit tests, but they are specifically for performance analysis.

Example Use Case: Fibonacci Sequence

For this example, I'll be using the classic Fibonacci Sequence, which is determined by:

if (x < 2) 
   F(0) = 1
   F(2) = 2
else 
   F(x) = F(x-1) + F(x-2)

In practice, the sequence is:
1, 1, 2, 3, 5, 8, 13, etc.

This sequence is important because it appears in various parts of mathematics and nature as well, as shown below:

There are several ways to implement this code, and I'll be picking two of them for our benchmark testing: the recursive and iterative methods. The main objective of the functions is to provide a position and return the Fibonacci number at that position.

Recursive Method

// main.go
func fibRecursive(n uint) uint {
    if n <= 2 {
        return 1
    }
    return fibRecursive(n-1) + fibRecursive(n-2)
}

The function above is a recursive implementation of calculating the Fibonacci sequence. Now I’ll break it down step by step for you as a beginner in Go.

Here’s your function for calculating the Fibonacci numbers:

func fibRecursive(n uint) uint {
    if n <= 2 {
        return 1
    }
    return fibRecursive(n-1) + fibRecursive(n-2)
}

1. Function:

func fibRecursive(n uint) uint

• func: This keyword defines a function in Go.

• fibRecursive: This is the name of the function. It’s called fibRecursive because it calculates Fibonacci numbers using recursion.

• n uint: The function takes a single argument, n, which is of type uint (an unsigned integer). This represents the position of the Fibonacci sequence that we want to calculate.

• uint: The function returns a uint (unsigned integer) because Fibonacci numbers are non-negative integers.

2. Base Stage:

if n <= 2 {
    return 1
}

• The if statement checks if n is less than or equal to 2.

• In the Fibonacci sequence, the 1st and 2nd numbers are both 1. So, if n is 1 or 2, the function returns 1.

• This is called the base stage, and it stops the recursion from going infinitely deep.

3. Recursive Stage:

return fibRecursive(n-1) + fibRecursive(n-2)

• If n is greater than 2, the function calls itself twice:

  • fibRecursive(n-1): This will calculate the Fibonacci number for the position just before n.

  • fibRecursive(n-2): This will calculate the Fibonacci number for two positions before n.

• The function then adds these two results together, because every Fibonacci number is the sum of the two preceding numbers.

For more theory on recursion, check out these articles.

Iterative Method

// main.go

func fibIterative(position uint) uint {
    slc := make([]uint, position)
    slc[0] = 1
    slc[1] = 1

    if position <= 2 {
        return 1
    }

    var result, i uint
    for i = 2; i < position; i++ {
        result = slc[i-1] + slc[i-2]
        slc[i] = result
    }

    return result
}

This code implements an iterative approach to calculate the Fibonacci sequence in Go, which is different from the recursive approach. Here’s a breakdown of how it works:

1. Function:

func fibIterative(position uint) uint

• func: This keyword declares a function in Go.

• fibIterative: The name of the function suggests that it calculates Fibonacci numbers using iteration (a loop).

• position uint: The function takes one argument, position, which is an unsigned integer (uint). This represents the position of the Fibonacci sequence you want to calculate.

• uint: The function returns an unsigned integer (uint), which will be the Fibonacci number at the specified position.

2. Creating a Slice (Array-like structure):

slc := make([]uint, position)

• slc is a slice (a dynamic array in Go) that is created with the length of position. This slice will store Fibonacci numbers at each index.

3. Initial Values for Fibonacci Sequence:

slc[0] = 1
slc[1] = 1

• The first two Fibonacci numbers are both 1, so the first two positions in the slice (slc[0] and slc[1]) are set to 1.

4. Early Return for Small Positions:

if position <= 2 {
    return 1
}

• If the input position is 1 or 2, the function directly returns 1, because the first two Fibonacci numbers are always 1.

5. Iterative Loop:

var result, i uint
for i = 2; i < position; i++ {
    result = slc[i-1] + slc[i-2]
    slc[i] = result
}

• The loop starts from i = 2 and runs until it reaches the position.

• In each iteration, the Fibonacci number at index i is calculated as the sum of the two previous Fibonacci numbers (slc[i-1] and slc[i-2]).

• The result is stored both in result and in the slice slc[i] for future calculations.

6. Returning the Result:

return result

• Once the loop finishes, the variable result holds the Fibonacci number at the desired position, and the function returns it.

This is a more efficient approach to calculating Fibonacci numbers compared to recursion, especially when position is large, because it doesn’t repeat unnecessary calculations and we are proving by using benchmark tests. Let’s prove it.

How to Run the Benchmark Tests

Now, for the benchmark tests, let’s write some test. First, you will need to create a maintest.go file. In it, using Golang's documentation on benchmark tests, you can create the functions to be tested as follows:

// main_test.go

// Benchmark for Iterative Function
func BenchmarkFibIterative(b *testing.B) {
    for i := 0; i < b.N; i++ { 
        fibIterative(uint(10))
    }
}
// Benchmark for Recursive Function
func BenchmarkFibRecursive(b *testing.B) {
    for i := 0; i < b.N; i++ {
        fibRecursive(uint(10))
    }
}

Let's run the test for position 10 and then increase appropriately. To run the benchmark tests, you simply run the command go test -bench=NameoftheFunction.

If you want to know more about this command, check here. Let’s check the function for position 10:

func BenchmarkFibIterative(b *testing.B) {
    for i := 0; i < b.N; i++ { 
        fibIterative(uint(10))
    }
}

go test -bench=BenchmarkFibIterative
Results:
cpu: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
BenchmarkFibIterative-8         27715262                42.86 ns/op
PASS
ok      playground      2.617s

Let’s analyze with the help of this image:

According to the image, we have 8 cores for the tests, and no time limit (it will run until completion). It took 27_715_262 iterations and 1.651 seconds to complete the task.

func BenchmarkFibRecursive(b *testing.B) {
    for i := 0; i < b.N; i++ {
        fibRecursive(uint(10))
    }
}

go test -bench=BenchmarkFibRecursive
Results:
cpu: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
BenchmarkFibRecursive-8          6644950               174.3 ns/op
PASS
ok      playground      1.819s

Using the same image to analyze the result, in this case it took 6_644_950 iterations and 1.819 seconds to complete the task we have:

The benchmark results show that the iterative approach is significantly more efficient than the recursive approach for calculating the Fibonacci sequence.

For position 10, the iterative function ran approximately 27.7 million iterations in 1.651 seconds, while the recursive function managed only 6.6 million iterations in 1.819 seconds. The iterative method outperformed the recursive method both in terms of iterations and time, highlighting its efficiency.

To proven even further this, let’s try with the position 40 (4 times the previous value):

// Results for the Iterative Function
cpu: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
BenchmarkFibIterative-8          9904401               114.5 ns/op
PASS
ok      playground      1.741s

// Results for the Recursive Function
cpu: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
BenchmarkFibRecursive-8                4         324133575 ns/op
PASS
ok      playground      3.782s

The benchmark results clearly highlight the efficiency difference between the iterative and recursive approaches for calculating Fibonacci again.

The iterative function completed approximately 9.9 million iterations with an average execution time of 114.5 nanoseconds per operation, finishing the benchmark in 1.741 seconds. In stark contrast, the recursive function only completed 4 iterations with an average execution time of 324,133,575 nanoseconds per operation (over 324 milliseconds per call), taking 3.782 seconds to finish.

These results demonstrate that the recursive approach is far less efficient due to repeated function calls and recalculations, making the iterative method vastly superior in both speed and resource usage, especially as input size increases.

Just out of curiosity, I tried position 60 and it literally crashed the test:

// Results for the Iterative Function
cpu: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
BenchmarkFibIterative-8          7100899               160.9 ns/op

// Results for the Recursive Function
SIGQUIT: quit
PC=0x7ff81935f08e m=0 sigcode=0

goroutine 0 gp=0x3bf1800 m=0 mp=0x3bf26a0 [idle]:
runtime.pthread_cond_wait(0x3bf2be0, 0x3bf2ba0)
...

Conclusion

If your production code is running slowly or is unpredictably slower, you can use this technique, combined with pprof or other tools from the built-in testing package, to identify and test where your code is performing poorly and work on how to optimize it.

Remember: Code that is beautiful to the eyes is not necessarily more performant.

Reference

• Recursive & Iterative functions to Fibonacci’s sequence here.

• Benchmark testing here.

Homework

This article explains why for some small numbers, the recursive strategy is better. Can you find a better way to improve the recursive function? (Tip: use Dynamic Programming).