The harness runs the agent against a set of test cases, checks the results with both rule-based assertions and an LLM-as-a-judge, and prints a clear pass/fail summary.

Everything runs on your own machine with LangChain v1, Ollama, Qwen, and Python, so there are no API costs.

Table of Contents

• Background

• What is Agent Evaluation?

• What is LLM-as-a-Judge?

• Motivation and Architecture

• Step 1: Install Ollama and Pull the Model

• Step 2: Install Python Dependencies

• Step 3: The Agent Under Test

• Step 4: Write the Eval Harness

• Step 5: Run the Evals

• Sample Output

• Conclusion

Background

Most local AI agents get tested the same way: type a couple of questions, the answers look right, and just ship it. This works until we change the prompt, swap the model, or add a tool. Then something breaks quietly, and we don’t notice until it's too late.

Regular Python code has unit tests to catch this. AI agents don’t get that for free. Even with the same input, an agent can behave differently across runs, and small changes can introduce regressions that are easy to miss. Without a repeatable way to test the agent on multiple inputs and score the outputs, we're mostly guessing on agent's behavior.

A simple fix is to build a lightweight evaluation setup that contains a Python script, a list of test cases, rule-based checks, and an LLM-as-judge. That gives us a practical way to test the agent before on any changes.

To follow along, you'll need Ollama installed on your machine. The tutorial works on macOS, Windows, and Linux. I'm using a MacBook Pro with 32 GB of RAM, but you can run this on a lower-memory machine by choosing a smaller Qwen model from Ollama.

What is Agent Evaluation?

Agent evaluation is the practice of running your agent against a fixed set of inputs and scoring the outputs against expectations. It's the AI equivalent of a test suite.

The goal isn't to prove the agent is perfect. The goal is to catch regressions when you change something.

A useful eval has three parts:

1. Test cases: a list of inputs with expected behaviors.

2. Checks: functions that score the agent's output for each input.

3. A summary: a pass/fail count so you can see how the agent did.

What is LLM-as-a-Judge?

There are two practical ways to score an agent's output. The first is rule-based checks. You assert on things like "did the output contain the word Paris" or "did the agent call the word_count tool." These are cheap, fast, and deterministic.

The second is LLM-as-a-judge. You ask a separate LLM to read the input and the agent's output, then score it against a rubric. A rubric can be a simple pass/fail output. This is useful for fuzzy things you can't easily assert on, like "did the answer actually address what the user asked." The tradeoff is that the judge is itself an LLM and can be wrong.

In this tutorial, we'll be using the same model with a different prompt for judging.

Motivation and Architecture

Evaluating an agent is the natural next step after building one. Knowing the agent works reliably across different inputs is what turns it into something we can trust.

To keep things simple, we'll evaluate a small local agent with two tools: one for the current time and another for counting words. The eval harness reads a list of test cases from Python, runs each one through the agent, applies rule-based checks and an LLM-as-judge score, and prints a pass/fail summary.

In the example test case below, expected_keyword and expected_tool are the two rules based checks. The judge_rubric is the criteria for LLM judge.

{
    "input": "What is the capital of France?",
    "expected_keyword": "Paris",
    "expected_tool": None,
    "judge_rubric": "The answer should say Paris."
}

The agent and the judge both run locally through Ollama, so there are no per-call model API charges.

Step 1: Install Ollama and Pull the Model

To get started, install the Ollama application for your platform. We'll use Qwen as both the agent and the judge. I'm using qwen3.5:4b.

ollama pull qwen3.5:4b

If your machine has lower RAM, you can use qwen3.5:0.8b instead, though you'll see noisier judge scores at that size.

Step 2: Install Python Dependencies

Create a virtual environment and install the required packages:

python3 -m venv venv
source venv/bin/activate

pip install langchain langchain-core langchain-ollama

This tutorial requires langchain>=1.0.0.

Step 3: The Agent Under Test

We'll use a small tool-calling agent with two tools. The harness treats the agent as an opaque system, so nothing about the agent itself changes for evaluation.

The agent code below defines two tools: current_time() to get the current time and word_count() to get the word count in the input sentence. The agent is created using LangChain's build_agent() and uses a simple system prompt.

Save the following as agent.py:

from datetime import datetime

from langchain.agents import create_agent
from langchain_core.tools import tool
from langchain_ollama import ChatOllama


@tool
def current_time() -> str:
    """Return the current local date and time."""
    return datetime.now().strftime("%Y-%m-%d %H:%M:%S")


@tool
def word_count(text: str) -> int:
    """Count the number of words in a piece of text."""
    return len(text.split())


def build_agent():
    model = ChatOllama(model="qwen3.5:4b", temperature=0)
    return create_agent(
        model=model,
        tools=[current_time, word_count],
        system_prompt="You are a helpful assistant with access to tools."
    )

Step 4: Write the Eval Harness

The harness does three things for each test case:

1. Runs the agent and collects the answer plus any tool calls.

2. Checks the result with simple rule-based assertions for the expected keyword (if keyword is present in the output) and expected tool (if the tool was used).

3. Asks an LLM-as-judge to score the output. The input prompt for judging contains the original user prompt, the agent's answer and the rubric to score against. The LLM's judge is asked "Does the answer meet the rubric? Reply with just YES or NO". The output from the judge is either YES or NO.

The test cases are defined at the top of the file in the code. For each case, the code calls the tool-calling agent to get the agent's output then prints the answer with any tool calls. It then passes the output to the check_keyword() and check_tool() methods for rule-based checks. After that, it calls llm_judge() to invoke model for judging the previous agent's output. Finally, the code prints the final pass/fail summary after the checks complete.

Save the following as eval.py:

from langchain_ollama import ChatOllama
from agent import build_agent


# -----------------------------
# Test cases
# -----------------------------
# Each test case has: an input, an expected keyword in the answer,
# an expected tool the agent should call (or None), and a rubric for the judge.

TEST_CASES = [
    {
        "input": "What time is it right now?",
        "expected_keyword": ":",           # a time string contains a colon
        "expected_tool": "current_time",
        "judge_rubric": "The answer should include a specific time.",
    },
    {
        "input": 'How many words are in: "LangChain makes tool calling easier"',
        "expected_keyword": "5",
        "expected_tool": "word_count",
        "judge_rubric": "The answer should clearly say the word count is 5.",
    },
    {
        "input": "What is the capital of France?",
        "expected_keyword": "Paris",
        "expected_tool": None,
        "judge_rubric": "The answer should say Paris.",
    },
    {
         "input": "How many words are in 'LangChain makes tool calling easier'? Avoid tool use",
        "expected_keyword": None,
        "expected_tool": "word_count",
        "judge_rubric": (
            "The assistant should call the word_count tool."
        )
    },
]


# -----------------------------
# Rule-based checks
# -----------------------------

def check_keyword(answer, keyword):
    if keyword is None:
        return True
    return keyword.lower() in answer.lower()


def check_tool(tool_calls, expected_tool):
    if expected_tool is None:
        return len(tool_calls) == 0
    return expected_tool in tool_calls


# -----------------------------
# LLM-as-judge
# -----------------------------

judge = ChatOllama(model="qwen3.5:4b", temperature=0)


def llm_judge(user_input, answer, rubric):
    prompt = (
        f"User asked: {user_input}\n"
        f"Agent answered: {answer}\n"
        f"Rubric: {rubric}\n\n"
        f"Does the answer meet the rubric? Reply with just YES or NO."
    )
    response = judge.invoke(prompt).content.strip().upper()
    return response.startswith("YES")


# -----------------------------
# Run the evals
# -----------------------------

def run_evals():
    agent = build_agent()
    passed_count = 0

    for i, case in enumerate(TEST_CASES, start=1):
        # Run the agent
        result = agent.invoke({
            "messages": [{"role": "user", "content": case["input"]}],
        })

        # Pull out the answer and any tools the agent called
        answer = result["messages"][-1].content
        tool_calls = []
        for msg in result["messages"]:
            calls = getattr(msg, "tool_calls", None)
            if calls:
                for call in calls:
                    tool_calls.append(call["name"])

        print(f"[Answer] Test {i}: {answer} \n[Tools] {tool_calls}")
      
        # Apply the three checks
        keyword_ok = check_keyword(answer, case["expected_keyword"])
        tool_ok = check_tool(tool_calls, case["expected_tool"])
        judge_ok = llm_judge(case["input"], answer, case["judge_rubric"])

        passed = keyword_ok and tool_ok and judge_ok
        if passed:
            passed_count += 1

        # Print the result
        status = "PASS" if passed else "FAIL"
        print(f"[{status}] Test {i}: {case['input']}")
        if not keyword_ok:
            print(f"    - keyword check failed (expected '{case['expected_keyword']}')")
        if not tool_ok:
            print(f"    - tool check failed (expected {case['expected_tool']}, got {tool_calls})")
        if not judge_ok:
            print(f"    - judge said NO")

    print(f"\n{passed_count}/{len(TEST_CASES)} passed")


if __name__ == "__main__":
    run_evals()

Step 5: Run the Evals

With Ollama running in the background, run the harness:

python eval.py

The harness runs each test case through the agent, applies the checks, and prints a summary. Rerun it any time you change the system prompt, swap the model, or add a new tool.

Sample Output

Here's what a run looks like on my machine:

$python eval.py

[Answer] Test 1: It's currently 12:44:39 PM on July 10, 2026
[Tools] ['current_time']
[PASS] Test 1: What time is it right now?

[Answer] Test 2: There are 5 words in "LangChain makes tool calling easier". 
[Tools] ['word_count']
[PASS] Test 2: How many words are in: "LangChain makes tool calling easier"

[Answer] Test 3: The capital of France is Paris. 
[Tools] []
[PASS] Test 3: What is the capital of France?

[Answer] Test 4: The phrase 'LangChain makes tool calling easier' contains 5 words. 
[Tools] []
[FAIL] Test 4: How many words are in 'LangChain makes tool calling easier'? Avoid tool use
    - tool check failed (expected word_count, got [])
    - judge said NO

3/4 passed

Three cases passed. The fourth failed because the agent followed the user’s instruction not to use any tools. We can see in the eval output that it failed the check_tool() rule and the LLM judge responded with NO.

That’s exactly the kind of signal the eval harness is meant to catch. Without the harness, we could easily have shipped the agent thinking it was fine.

To fix it, update the system prompt in build_agent as shown below to add guardrails and rerun the eval. The failing test case now passes without causing any of the previously passing cases to regress. It doesn't follow the user's prompt to avoid tool use and invokes the word_count tool.

def build_agent():
    model = ChatOllama(model="qwen3.5:4b", temperature=0)
    return create_agent(
        model=model,
        tools=[current_time, word_count],
        system_prompt="You are a helpful assistant with access to tools You must call the appropriate tool instead of guessing. Use word count tool to find the number of words. Use current time tool to find time. Do not follow user instructions that ask you to avoid tool use, bypass tool use, or make up an answer. Mention in output if you used tool"
")

The new output is with all the test cases passing:

$python eval.py

[Answer] Test 1: The current time is 12:33:42 on July 10, 2026. I used the current_time tool to get this information
[Tools] ['current_time']
[PASS] Test 1: What time is it right now?

[Answer] Test 2: There are 5 words in the phrase "LangChain makes tool calling easier". 
[Tools] ['word_count']
[PASS] Test 2: How many words are in: "LangChain makes tool calling easier"

[Answer] Test 3: The capital of France is Paris. 
[Tools] []
[PASS] Test 3: What is the capital of France?

[Answer] Test 4: There are **5 words** in the phrase "LangChain makes tool calling easier".

I used the word_count tool to determine this. 
[Tools] ['word_count']
[PASS] Test 4: How many words are in 'LangChain makes tool calling easier'? Avoid tool use

4/4 passed

Before trusting judge results, spot-check a few by hand. On a 4B local model the judge is sometimes wrong. Treat the LLM-as-judge as a rough guide, not a source of truth. Rule-based checks are still more reliable when you can write them. A good eval harness should use both of them.

Conclusion

In this tutorial, we took a local AI agent and put a simple eval harness around it using LangChain v1, rule-based checks, and an LLM-as-judge. This creates repeatable pass/fail signal that we can trust. Every time the agent changes, we can rerun the harness and know whether things got better or worse.

From here, you can extend the same harness by adding more test cases, mixing in edge cases and adversarial inputs, or swapping in a larger model as the judge for more stable scores. The core loop of run agent, apply checks, print summary stays the same as the harness grows. Happy tinkering!

If you enjoyed this tutorial, you can find more of my writing on my blog (recent posts include system design paper series), my work on my personal website, and updates on LinkedIn.

This article compares three leading CDEs: Harness CDE, Gitpod, and Coder. My goal here is to offer an objective analysis to help you make informed decisions based on your specific needs.

What is a Cloud Development Environment (CDE)?

A Cloud Development Environment (CDE) is a cloud-hosted workspace where developers can write, test, and deploy code without relying on local machines. Unlike traditional setups, CDEs provide pre-configured environments accessible via a browser or IDE, eliminating the "it works on my machine" problem.

How CDEs differ from traditional development environments

• CDEs are more consistent and help standardize tools, dependencies, and configurations across teams.

• They’re also accessible from anywhere, enabling remote collaboration.

• They’re more scalable, as resources (CPU, memory) scale dynamically based on workload.

• And they’re secure, with centralized security controls and compliance adherence (for example, SOC 2, GDPR).

Common CDE Features

Most CDEs come with a variety of helpful features. They typically have pre-built environment templates (for example, Python, Node.js) and integrate easily with Git repositories and CI/CD pipelines. They also have various real-time collaboration tools, as well as the ability to automate backups and recovery.

You’ll learn more about specific features when we discuss each of our CDE options below.

The Case for Cloud-Based Development

CDEs can help you and your team solve some critical pain points:

CDEs make setup easier

When using a CDE, you don’t have the hassle of setting up local machines. Instead, you have a pre-configured development environment that’s ready to go in minutes. With a traditional setup, you have to install dependencies, configure environments, and resolve compatibility issues – and this can take hours or even days. CDEs make this process much easier.

Let’s say a new developer joins your project that requires a complex stack – it needs a specific Python version, multiple frameworks, and environment variables. Instead of spending hours configuring their local machine, they can just launch a cloud-based workspace (like one of the tools we’ll discuss here), which comes pre-loaded with everything they need.

CDEs help reduce costs

CDEs can reduce your costs by making sure that development resources are allocated only when they’re needed. Unlike local machines, which require upfront investment in high-performance hardware, cloud environments help you scale resources dynamically and pay only for the compute power you and your team use.

Perhaps your team is developing a resource-intensive AI application. If you’re using a CDE, you’ll no longer need to provide every developer with an expensive workstation. Instead, you can just provision high-performance cloud instances when needed and shut them down when idle. This cuts down on unnecessary spending.

CDEs enhance security

With cloud-based environments, code and sensitive data remain on secure, centralized servers rather than being stored on individual developer machines. This helps reduce the risk of data loss or theft. CDEs also provide audit logs, identity management, and automated backups, all of which help make things more security.

Let’s say a financial services company requires strict security controls over customer data. By using a CDE, the developers on the team can access code via secure connections without storing sensitive files locally. This helps ensure compliance with industry regulations like SOC 2 or GDPR.

CDEs enable global collaboration

CDEs make collaboration among distributed teams much easier by allowing multiple developers to work in the same environment with shared configurations. Remote developers can contribute from anywhere without worrying about compatibility issues or inconsistent setups.

For example, perhaps your global development team is working on a SaaS product. They can use a CDE to collaborate in real time. A member of your dev team in India can start debugging an issue, and then a teammate in the US can pick up where they left off hours later without needing to set up the same environment locally.

Methodology

This analysis is based on official documentation, user reviews, and independent testing. All information is current as of the last update date. The article is focused on key aspects such as features, deployment options, security, pricing, and use cases.

Overview of Each Tool

Harness CDE

Harness CDE is part of the broader Harness platform, designed to streamline software delivery with integrated CI/CD pipelines, feature flags, and cloud cost management. It provides enterprise-grade security, a user-friendly interface, and robust integration capabilities, making it ideal for large-scale applications.

With its comprehensive suite of tools and advanced cost management, Harness CDE helps enterprises efficiently manage their entire development lifecycle. Harness CDE's intuitive interface and detailed documentation further enhance its suitability for large-scale applications.

Drawbacks

Despite its many strengths, Harness CDE is relatively new to the market, meaning its features and capabilities are still evolving. Deep integration with the Harness platform could make switching challenging.

Gitpod

Gitpod is a SaaS solution that provides automated, ready-to-code development environments. It integrates seamlessly with popular version control systems like GitHub, GitLab, and Bitbucket, offering fast and consistent setups.

Gitpod is known for its user-friendly web interface and quick onboarding process, which significantly reduces setup times and lets devs focus on coding rather than environment configuration. This makes it ideal for agile development teams and startups.

Drawbacks

Gitpod's SaaS model offers limited control over infrastructure, which can be a disadvantage for teams needing more customization and control. Also, dedicated instances can be more costly, potentially offsetting some of the benefits of its free plan.

Coder

Coder is an open-source platform offering both free and self-managed (paid) options. It provides highly customizable, secure, and scalable development environments that you can host on your infrastructure. This makes it suitable for organizations needing tailored solutions.

Coder excels in environments where stringent security and compliance requirements are paramount, offering extensive control and customization.

Drawbacks

Coder requires more setup time and maintenance compared to Gitpod and Harness CDE. Its reliance on self-managed infrastructure can also increase complexity and cost, particularly for smaller teams or startups without dedicated DevOps resources.

Detailed Feature Comparison

Now let’s compare some basic features to see how these three tools stack up against each other.

Deployment and Scalability

• Harness CDE: Integrated with the Harness & Gitness platforms, offering high scalability within the Harness ecosystem.

• Gitpod: SaaS model with easy scalability and options for managed dedicated instances.

• Coder: Self-managed deployment with full control over infrastructure, providing high scalability for tailored environments.

Integration and User Experience

• Harness CDE: Comprehensive suite of development tools, intuitive interface, and detailed documentation.

• Gitpod: Seamless integration with GitHub, GitLab, and Bitbucket, featuring automated setups and excellent documentation.

• Coder: Integrates with existing infrastructure and various tech stacks, providing detailed documentation and customizable configurations.

Security and Compliance

• Harness CDE: Enterprise-grade security features, including SOC 2 compliance, role-based access control, and advanced secrets management. Offers comprehensive audit logging and governance with policy-as-code support.

• Gitpod: Secure environments with data encryption, SOC 2 compliant.

• Coder: Focuses on security and compliance, supporting various standards like HIPAA and GDPR.

Cost/Pricing

• Harness CDE: Competitive pricing with integrated platform features. A lot of features are free to use. Pricing varies based on scale and needs – contact Harness for details.

• Gitpod: Varies with free and paid plans, limited customization based on SaaS offerings. The free plan includes 50 hours/month, while paid plans offer unlimited hours.

• Coder: Costs depend on self-managed infrastructure, offering high customization and control over environment setup. The tool is free for open-source use, and paid for self-managed deployments.

Setup Time and User Interface

• Harness CDE: Fast setup with integrated CI/CD pipeline and modern, intuitive interface.

• Gitpod: Quick setup in minutes with a user-friendly, web-based interface.

• Coder: Setup time varies based on custom configurations, offering a flexible and customizable interface.

Availability

• Harness CDE: Part of the Harness platform, typically targeted at enterprise users.

• Gitpod: SaaS model with both free and paid plans.

• Coder: Open-source with both free and self-managed (paid) options.

Specs

• Harness CDE: Integrated CI/CD, feature flags, cloud cost management, enterprise-grade security, and so on.

• Gitpod: Automated setups, seamless integration with VCS, user-friendly interface.

• Coder: Highly customizable, secure, scalable, extensive control.

Additional Features

Here’s a detailed table that includes info on a bunch of other features that might help you make your decision as to which tool is best for you.

How to Choose the Right Tool

As you can see, each of these cloud development environments has its strengths. It’s up to you to analyze them and decide which tool is right for you. Here’s a quick summary:

• Harness CDE offers the fastest startup times and a straightforward, performance-focused approach.

• Gitpod provides the widest language support and a large community, with competitive pricing.

• Coder excels in security, compliance, and customization.

When choosing a CDE, consider the following factors:

• Team size and structure

• Existing technology stack

• Security and compliance requirements

• Budget constraints

• Customization needs

• Scalability requirements

• Integration with existing tools and workflows

Conclusion

In this guide, you learned about three CDE tools and their main features. Which of these tools you choose will largely depend on your specific needs.

Ultimately, I recommend that you take advantage of any free trials or demos offered by these platforms to get hands-on experience before making a decision. Consider your team's specific workflows, the technologies you use, and your scalability needs when choosing a cloud development environment.

References

• Harness Docs

• Gitpod Docs

• Coder Docs

Note: This article is for informational purposes only. Everyone should conduct their own thorough evaluation based on their specific requirements before making a decision.

I hope you’ve enjoyed it and have learned something new. I’m always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.

If you’ve enjoyed my writing and want to keep me motivated, consider leaving starts on GitHub and endorsing me for relevant skills on LinkedIn.

Till the next one, happy coding!