How to Evaluate and Select the Right LLM for Your GenAI Application

Every day, we learn something new about generative AI applications – how they behave, where they shine, and where they fall short. As Large Language Models (LLMs) rapidly evolve, one thing becomes increasingly clear: selecting the right model for your use case is critical.

Different LLMs can behave very differently for the same prompt. Some excel at coding, others at reasoning, summarization, or conversational tasks. For example, I use ChatGPT for general inquiries, formatting text, or light research, while preferring Claude for deeper coding assistance.

This highlights a key idea that there is no single “best” model.

Here’s an example where Claude explains which Claude model should be used for specific use cases.

In this article, I’ll walk you through a practical and repeatable methodology to evaluate and select an LLM for a real-world GenAI application, based on techniques used in enterprises.

What We’ll Cover:

1. What we’ll cover:

2. Prerequisites

3. What’s the Goal Here?

4. Why Do LLMs Perform Differently?

   • 1. Training Data and Domain

   • 2. Fine-Tuning and RAG

   • 3. Architecture Differences

5. When Do You Need to Evaluate an LLM?

   • 1. Before You Start Building

   • 2. When Upgrading an Existing Application to a New Model

6. Key Factors to Evaluate

   • 1. Accuracy and Consistency

   • 2. Latency

   • 3. Cost

   • 4. Ethical and Responsible AI Considerations

   • 5. Context Window

7. How to Evaluate LLMs in Practice

   • Step 1: Curate a Dataset

   • Step 2: Standardize Your Evaluation Setup

   • Step 3: Perform Statistical Analysis

   • Step 4: Perform the Evaluation

   • Step 5: Log Everything

   • Step 6: Review and Reporting

8. Mini Case Study

   • Requirements

   • Dataset Design

   • Evaluation

   • Human Review

   • Decision

9. Don’t Forget the Business Use Case

10. Conclusion

Prerequisites

To fully understand and grasp the concepts discussed in this tutorial, it’ll be helpful to have the following background knowledge:

1. Experience building or working with LLM-based applications: You should be familiar with how LLMs are used in real-world applications, such as chatbots or RAG systems.

2. Familiarity with prompt engineering concepts: A basic understanding of how prompts influence model responses will help when evaluating correctness and behavior.

3. Basic programming knowledge: Some examples involve structured evaluation outputs and metrics, so familiarity with reading code or data formats like tables or JSON is beneficial.

What’s the Goal Here?

This article does not simply list frameworks. Instead, it provides clear, experience-driven guidelines from someone who has applied these techniques in enterprise applications and successfully shared findings.

While there is a lot of theoretical or example-based content available on LLM evaluation, what is often missing is practical guidance. Real-world use cases vary significantly and are rarely straightforward.

In this article, I will share implementable and practical insights that you can apply directly to your own projects.

Why Do LLMs Perform Differently?

Before diving into how to select or evaluate models, an important question arises: why do LLMs perform differently in the first place?

Below are some common reasons.

1. Training Data and Domain

The quality, diversity, and domain of training data play a major role in model performance.

For example, models trained heavily on GitHub or GitLab repositories tend to perform better at programming tasks, while those trained on academic papers or general web data may excel at reasoning or summarization.

2. Fine-Tuning and RAG

Most real-world applications are domain-specific, not generic.

For example, when implementing an employee facilitation system, each company has its own rules and policies. To handle such domain-specific requirements, two common approaches are used:

• Fine-tuning

• Retrieval-Augmented Generation (RAG)

RAG doesn’t change the behavior of the model. Instead, it provides additional domain context using retrieved data. Fine-tuning, on the other hand, is more sophisticated and involves training the model itself on domain-specific data.

If you want to learn more about the difference between Fine-tuning & RAG, here’s a helpful article by IBM.

3. Architecture Differences

Although most LLMs are built on transformer architectures, their performance can still vary significantly.

For example, OpenAI’s ChatGPT and Google Gemini are both transformer-based models, yet they differ in performance due to factors such as:

• The number of parameters

• Differences in training datasets

(Reference)

Now that we understand why LLMs differ, let’s move on to when and why evaluation becomes necessary.

When Do You Need to Evaluate an LLM?

Model evaluation becomes essential in the following scenarios.

1. Before You Start Building

If you’re building a production-grade GenAI application, early model selection is critical.

At this stage, you should clearly define the problem: the application’s scope, your expected number of users, any latency expectations, and privacy requirements.

You should also identify non-negotiable requirements (SLOs). For example, perhaps you need accuracy to be above 90% and latency below 2 seconds.

You’ll need to consider cost implications as well, such as funding constraints at early stages, expected user growth, and request volume and scaling.

Common evaluation factors include:

• Speed and latency

• Accuracy and reliability

• Data privacy and compliance

2. When Upgrading an Existing Application to a New Model

Another common use case is upgrading a model when the application is already in production.

In this scenario:

• Core metrics usually remain the same

• The features will be already implemented and also benchmarked on existing model.

• There is already a baseline performance threshold that must be preserved

Upgrading a model is not always straightforward. System prompts that worked well previously may behave very differently with a new model.

From personal experience, after upgrading an LLM, responses that were previously well formatted suddenly became inconsistent and poorly structured.

When an application is live, evaluation focuses on regression testing and measurable improvement:

• Existing features and prompts must be revalidated

• Metrics should be evaluated feature by feature

• Improvements should be data-driven, not anecdotal

Key Factors to Evaluate

These are the most important factors to evaluate when you’re choosing a model for your task:

1. Accuracy and Consistency

Accuracy and consistency are in most cases the most important factors when building LLM-based applications.

Accuracy refers to whether the responses generated by the model are correct or not, while consistency measures the model’s tendency to produce the same response when given the same input multiple times. Ideally, a model should demonstrate both accurate and consistent behavior.

For example, consider a RAG application where a user asks a question. If the model generates the correct answer on the first attempt, an incorrect answer on the second attempt, and then the correct answer again on the third attempt, this indicates that the responses are not consistent even if accuracy is occasionally achieved.

When selecting an LLM, ask yourself the following questions:

• Does the model hallucinate on simple or complex queries?

• Are responses consistent across multiple runs?

• Does accuracy degrade for edge cases?

2. Latency

Alongside accuracy, it is important to consider the performance of your application. From a user’s perspective, a system with high latency or slow performance can lead to negative feedback or decreased usage, even if the responses are accurate.

For example, consider a streaming-response RAG application that delivers answers chunk by chunk. If the first chunk arrives after 15 seconds and the complete response after 60 seconds, this indicates poor performance from a user experience standpoint.

When evaluating LLMs, ask yourself the following questions:

• How quickly does the model respond?

• Is latency predictable under load?

3. Cost

LLMs are not free, and each token comes with a price. So it’s important to consider cost when selecting a model. You should perform proper calculations and assessments to estimate the expected load. Consider how many requests you’ll make per minute and the size of each request, as this will directly impact your overall expenses.

When evaluating LLMs, ask yourself the following questions:

• What is the cost per request or per token?

• Is the model viable for your expected traffic, especially in early-stage or proof-of-concept phases?

Here’s a reference for pricing from OpenAI as an example.

4. Ethical and Responsible AI Considerations

With generative AI, it has become even more critical to enforce ethical constraints and implement responsible AI. Without these guidelines and restrictions, models can produce content that is harmful to society, which should never be tolerated.

For example, your application should not provide assistance for harmful requests, such as “How to make a bomb.”

When evaluating LLMs, ask yourself the following questions:

• Does the model adhere to safety and community guidelines?

• Are harmful, biased, or disallowed requests properly rejected?

Responsible AI is not optional. It’s a shared responsibility across developers, product owners, and managers. Ignoring ethical considerations can harm both the product and society.

5. Context Window

If your application processes large documents or relies on long conversations, the context window becomes a critical factor.

The context window includes both input and output tokens, not just the response.

Examples:

• GPT-3: 4K tokens

• GPT-3.5 Turbo: 8.1K tokens

You can read more about context window here.

How to Evaluate LLMs in Practice

Step 1: Curate a Dataset

Dataset curation is the most important step when evaluating LLMs.

For each feature of your application, curate a representative dataset that includes:

• Real user queries (if the application is already in production)

• Carefully designed synthetic queries (if it’s not)

At early stages, real user data may not be available or may not cover all scenarios. Synthetic datasets created manually or through automation help fill those gaps.

I have discussed this process in more detail in a previous article. You can read it if you’d like to learn more.

The following table illustrates the different categories of queries you might include in your dataset. It shows the type of queries, their purpose, and example questions for each category. This helps ensure that your dataset provides broad coverage of the application’s behavior, from simple requests to complex reasoning and out-of-scope handling.

Step 2: Standardize Your Evaluation Setup

To ensure a fair evaluation, it’s important to keep all elements of the setup constant. The only thing that should change is the model being tested.

Keep the dataset constant

Don’t change your test data for each execution. Using the same dataset ensures that both models are evaluated on exactly the same queries, providing a fair comparison of results.

Keep prompts and evaluation scripts constant

System prompts and evaluation scripts should remain unchanged. LLMs can behave differently even on the same prompt, so keeping these constant ensures a fair assessment.

Keep evaluation rules and thresholds constant

If your evaluation includes thresholds – such as an accuracy requirement or a similarity threshold (for example, cosine similarity ≥ 80%) don’t change these between models. This ensures that each model is measured by the same standards.

Change only one variable: the model under test

The model being evaluated should be the only variable in your experiment.

These principles apply whether your evaluation is manual or automated, and they help ensure that results are objective, reproducible, and unbiased.

Manual evaluation involves a human reviewing the response to each query and marking it as passing or failing. This approach is helpful for assessing qualitative aspects, such as user experience, tone, and readability. But manual evaluation isn’t scalable: time constraints and reviewer fatigue make it impractical for large datasets.

For large-scale testing, automated evaluation is more practical. Scripts or tools can run queries, compare responses against expected results, and calculate metrics. This can be done using LLM-as-a-judge approaches or rule-based techniques like cosine similarity.

Even with automation, human oversight is still necessary. LLMs can hallucinate or misinterpret prompts, so humans shift from direct testers to reviewers or managers, validating results and ensuring the evaluation process remains accurate.

Step 3: Perform Statistical Analysis

Once tests are executed and you have all results, its time to do some statistical analysis. Avoid making intuition-based decision making. The decision should be mapped and tracked with numbers or statistics

Your evaluation results should be in the following forms so you can more easily perform statistical analysis:

• Pass/fail thresholds

• Numeric scores

• Percentage-based success rates

Even for subjective aspects such as tone, define expectations upfront:

• What qualifies as a “professional” tone?

• What wording is unacceptable?

Clear definitions reduce bias and improve reproducibility.

Your results after statistical analysis should be looking like following table. In it, each feature or metric has a score / percentage. This table shows an example of aggregated performance across all evaluation metrics for two models, including average latency. It helps visualize trade-offs and supports data-driven model selection.

Step 4: Perform the Evaluation

For applications with multiple features, automation becomes essential.

While manual evaluation is possible, it’s time-consuming and error-prone. A common approach includes:

• Generating a response from the application

• Comparing it with a ground truth or reference answer

• Using a separate evaluation model or rule-based approach to score the response

This enables large-scale, repeatable evaluations.

Available Frameworks and Tools for Evaluation

When implementing LLM evaluation, you can either build custom scripts or use existing frameworks and tools. Each approach has its advantages depending on your project and team requirements.

1. Custom Scripts

Custom scripts give you full control over the evaluation process. You aren’t dependent on any framework and can design the evaluation to match your application’s exact needs.

For example, in one project, I built an LLM evaluation script using LangChain with custom prompt templates. I also compared it against the evaluators provided by LangChain. Surprisingly, the custom script produced better results because I had more control over the prompts and evaluation logic.

A simplified example of a custom script I used for one of projects is below, in which i used LangChain and Azure Open AI using TypeScript to implement a RAG Evaluator:

import * as dotenv from "dotenv";
import { AzureChatOpenAI } from "@langchain/openai";
import { PromptTemplate } from "@langchain/core/prompts";

dotenv.config();

const evaluationModel = new AzureChatOpenAI();

/**
 * LLM-as-a-Judge evaluation function
 * Compares an AI-generated response against a reference answer.
 */
export async function evaluateResponse({
  question,
  actualResponse,
  referenceResponse,
}: {
  question: string;
  actualResponse: string;
  referenceResponse: string;
}) {
  // Placeholder prompt – replace with your actual evaluation instructions
  const promptTemplate = `
<<INSERT YOUR EVALUATION PROMPT HERE>>

Question: {question}
AI Response: {actualResponse}
Reference: {referenceResponse}
`;

  const prompt = PromptTemplate.fromTemplate(promptTemplate);

  const formattedPrompt = await prompt.format({
    question,
    actualResponse,
    referenceResponse,
  });

  // Invoke the evaluation model
  let result;
  try {
    result = await evaluationModel.invoke(formattedPrompt);
  } catch {
    // Retry once after 20 seconds if invocation fails
    await new Promise((resolve) => setTimeout(resolve, 20000));
    result = await evaluationModel.invoke(formattedPrompt);
  }
  return result;
}

2. Existing Frameworks

Frameworks provide pre-built functionality for evaluation, logging, and comparison, which can save time and improve reproducibility. Some popular options include:

• MLflow – Popular for end-to-end AI workflows, including experiment tracking, evaluation, and comparison.

• Comet – Provides robust experiment tracing and evaluation dashboards.

• RAGAS – Specifically designed for evaluating RAG (retrieval-augmented generation) applications, offering structured evaluation and logging.

Frameworks are particularly useful if:

• Your team is already using one (for example, MLflow for AI experiments)

• There’s a company or client requirement to adopt a specific framework

• You want scalable, repeatable evaluation with logging and dashboards without the need of doing extra work on logging and scaling

In my experience, sticking to custom scripts may be preferable for maximum flexibility, domain-specific control, or one-off experiments.

Step 5: Log Everything

As your evaluations run, make sure you log everything that matters:

• Query

• Model used

• Response

• Expected behavior

• Scores per metric

These logs are critical for traceability, decision-making, and revisiting experiments later. CSV is a practical format that is easy to query and analyze.

Step 6: Review and Reporting

Once your results are compiled, review them carefully.

For example:

• Model A: Accuracy = 85%, Completeness = 75%, Latency = 8 seconds

• Model B: Accuracy = 87%, Completeness = 78%, Latency = 16 seconds

If latency is a non-negotiable requirement, Model A will be preferable despite a slight drop in accuracy.

Create a summary report that includes key metrics, comparative analysis, and any final recommendations. This report becomes a decision artifact that can be shared with stakeholders.

Mini Case Study

Let’s consider a mini case study of selecting an LLM for a RAG application that answers questions related to company policies and employee benefits.

Requirements

1. Responses must be under 5 seconds

2. Responses must be complete

3. Responses must be accurate at least 85% of the time

4. Responsible AI considerations must be enforced

Dataset Design

In this mini case study, the dataset is designed to cover all critical evaluation scenarios for the RAG application. This ensures that we can make informed decisions based on objective metrics.

Case 1: Accuracy – Testing the model for correctness and completeness to determine if it produces the responses we require.

• Question: The user query

• Expected answer: The ground truth response for comparison

• Accuracy: Whether the model returned the correct answer (pass/fail)

• Completeness: Whether all parts of the answer were addressed

• Latency: Time taken to generate the response

Case 2: Guardrails – Testing whether the model upholds responsible AI guidelines.

• Question: A potentially unsafe query that violates community guidelines

• Rejection to Answer: True/False depending on whether the model correctly refused to answer

Case 3: Out-of-scope – Testing whether the application stays within its instructed system prompts and scope.

• Question: A query unrelated to the system’s scope

• Rejection to Answer: True/False depending on correct refusal

Curating the dataset in this way ensures that it covers all aspects of model behavior. The results generated from this dataset provide a solid foundation for making reliable, informed evaluation decisions.

The table below shows a small snippet of a dataset covering the above discussed cases:

Evaluation

To validate the dataset, you can use the LLM-as-a-judge evaluation technique. In this approach, you use an LLM to evaluate another LLM’s output based on rules defined in a prompt.

This technique is useful because direct string matching isn’t reliable, as LLM responses often vary even for the same question. By using another LLM as a judge, you can objectively assess correctness while accounting for natural variance in responses.

Here’s how it works:

1. You define an evaluation prompt that includes:

   • The question

   • The expected response (reference answer)

   • The actual response from the model under test

   • Evaluation rules to determine correctness, completeness, or adherence to guidelines

The judge LLM compares the actual response to the reference and outputs a structured result, typically in JSON. This result indicates whether the response is correct, incomplete, incorrect, or contains additional information.

This allows you to automate evaluation at scale while keeping results interpretable and consistent.

Example: LLM-as-a-Judge Evaluator

Below is a simplified implementation using LangChain, Azure OpenAI, and a custom prompt:

import * as dotenv from "dotenv";
import { AzureChatOpenAI } from "@langchain/openai";
import { PromptTemplate } from "@langchain/core/prompts";

dotenv.config();

const evaluationModel = new AzureChatOpenAI();

/**
 * LLM-as-a-Judge evaluation function
 * Compares an AI-generated response against a reference answer.
 */
export async function evaluateResponse({
  question,
  actualResponse,
  referenceResponse,
}: {
  question: string;
  actualResponse: string;
  referenceResponse: string;
}) {
  const prompt = PromptTemplate.fromTemplate(`
You are an impartial AI evaluator.

Your task is to evaluate whether the AI-generated response correctly answers the given question,
based on the provided reference answer.

Question:
{question}

AI Generated Response:
{actualResponse}

Reference Answer:
{referenceResponse}

Evaluation Rules (Mandatory):
1. The AI-generated response must correctly answer the question using the reference.
2. Minor wording differences are acceptable if meaning is preserved.
3. If additional information is present but does not contradict the reference, mention it in reasoning but do NOT mark incorrect.
4. If the response is empty, null, or contains errors, mark the evaluation as "Failed".

Return the evaluation strictly as a JSON object with the following keys:
- "reasoning": Explanation comparing the response to the reference
- "value": One of "Yes", "No", or "Failed"
- "cause":
    - "N/A" if value is "Yes"
    - "incomplete" if reference information is missing
    - "incorrect" if response contradicts the reference
    - "additional info" if extra unrelated information is present
  `);

  const formattedPrompt = await prompt.format({
    question,
    actualResponse,
    referenceResponse,
  });

  let result;
  try {
    result = await evaluationModel.invoke(formattedPrompt);
  } catch {
    // Simple retry mechanism for transient failures
    await new Promise((resolve) => setTimeout(resolve, 20000));
    result = await evaluationModel.invoke(formattedPrompt);
  }

  const cleanedResponse = String(result.content)
    .replace(/^```json\s*/, "")
    .replace(/\s*```$/, "")
    .trim();

  return JSON.parse(cleanedResponse);
}

Human Review

After automated evaluation, you’ll need to perform your own review. You should do the following:

• Check edge cases or nuanced responses that the judge LLM might misinterpret

• Filter out false positives or negatives

• Add comments or explanations where necessary

Even with an LLM-as-a-judge, human oversight is essential because LLMs can hallucinate. In this workflow, the human acts as a reviewer or manager, rather than manually scoring every response.

Decision

Once all results are compiled and the summary is generated, you can get a clear picture of which model is preferable. Take the table below as an example:

As you can see, in most metrics, Model B performs slightly better than Model A, with around a 2% improvement. But since our initial requirements specified a latency under 5 seconds and a minimum accuracy of 85%, Model A is favored due to its significantly lower response time, despite the marginal difference in accuracy.

Don’t Forget the Business Use Case

A common mistake when evaluating LLMs is overlooking the business use case when choosing a model. It’s easy to rely only on human judgment without setting clear evaluation rules, rush decisions without properly designing tests, and not dedicate enough effort to creating well-thought-out datasets and evaluation plans.

So just make sure you take these factors into consideration and you should be able to choose the right model for your use case.

Conclusion

As GenAI systems mature and become deeply embedded in production workflows, LLM evaluation becomes a core engineering discipline.

By treating model selection as an engineering problem rather than a subjective choice, you can build applications that are faster, safer, more reliable, and easier to evolve over time.

You can reuse the same methodology whenever models change, ensuring your GenAI application continues to meet its goals as the ecosystem evolves.

Hope you’ve all found this helpful and interesting. Keep learning!