<![CDATA[ Artificial Intelligence - freeCodeCamp.org ]]> https://www.freecodecamp.org/news/ https://cdn.freecodecamp.org/universal/favicons/favicon.png <![CDATA[ Artificial Intelligence - freeCodeCamp.org ]]> https://www.freecodecamp.org/news/ Eleventy Wed, 07 Oct 2026 06:32:30 +0000 60 <![CDATA[ How AI Is Changing Email Deliverability: A Technical Guide to Sender Reputation and Inbox Placement ]]> https://www.freecodecamp.org/news/ai-email-deliverability-explained/ 6ac38287d6fd64daae93bf3c Mon, 05 Oct 2026 10:57:11 +0000 Sending an email doesn't always mean it will reach the recipient's inbox. Sometimes, an email is sent successfully by an application but ends up in the spam folder instead.

This can be a real problem for developers, especially when they're sending important messages such as password-reset links, account verification codes, or payment confirmations.

This is where email deliverability comes into the picture. Email providers don't just check whether an email has been sent. They also examine who sent it, how it was sent, and whether it looks trustworthy.

To do this, providers use techniques such as sender reputation, email authentication, and AI-powered spam filters. As AI becomes more involved in this process, understanding how these systems work is becoming increasingly important for developers.

What We'll Cover Here:

How Email Providers Traditionally Evaluated Sender Reputation

Before exploring how modern filtering works, we should look at the established systems that still form the foundation of email sorting.

What Is Sender Reputation?

At the core of these systems lies sender reputation, which is an ongoing assessment of a sender's trustworthiness. This trust score is calculated based on historical sending behaviour, cryptographic authentication, and how previous recipients have responded to your messages.

The Signals Behind Traditional Email Filtering

To build this reputation, traditional spam filtering relies on several specific, measurable signals.

First, IP reputation tracks the historical behaviour specifically associated with your server's IP address. Alongside this, domain reputation evaluates the historical trust tied to the domain name used in your sender address. These identifiers are then cross-referenced with bounce rates, as repeatedly sending messages to invalid addresses strongly indicates poor list hygiene.

Spam complaints can also hurt sender reputation when recipients repeatedly mark messages as spam.

Email authentication is another important part of the process. Three protocols are commonly used here:

  1. SPF (Sender Policy Framework) tells receiving servers which servers are allowed to send email for a domain.

  2. DKIM (DomainKeys Identified Mail) adds a digital signature to outgoing messages, allowing the receiving server to verify that the message was authorised and wasn't changed in transit.

  3. DMARC (Domain-based Message Authentication, Reporting and Conformance) builds on SPF and DKIM by allowing domain owners to specify how receiving servers should handle messages that fail authentication and by providing reports about those failures.

Because these signals are so reliable, traditional filtering has effectively utilized rules and statistical techniques for years. AI isn't replacing every existing mechanism here. These foundational signals absolutely still matter, but modern email providers can now evaluate much more than just a sender's technical configuration.

How AI Is Changing the Way Email Providers Detect Spam

Building upon those traditional signals, artificial intelligence introduces an entirely new layer of contextual analysis.

From Fixed Rules to Machine Learning

Historically, rule-based systems flagged messages using predefined conditions, such as known malicious signatures or universally suspicious links.

Machine learning models, on the other hand, dynamically learn evolving patterns from massive collections of labeled messages. This shift allows providers to adapt to new threats instantly without waiting for manual rule updates.

How AI Recognises Suspicious Email Behaviour

By leveraging this dynamic learning, machine learning systems evaluate multiple signals simultaneously rather than checking them sequentially. These comprehensive models analyze message content, looking closely at suspicious wording alongside structural anomalies. Simultaneously, they scrutinize the characteristics of all embedded links, attachments, and the domains hosting them.

This deep inspection is paired with an analysis of sending frequency, where any sudden spikes in volume immediately trigger closer inspection. The AI cross-references this activity with your historical sender behaviour and incorporates real-time recipient interactions to create a holistic profile of the email's intent.

Why Context Matters More Than Individual Keywords

Because these systems evaluate data holistically, context matters far more than individual keywords. Consider two separate emails containing the word "free." One could be a legitimate account notification from a developer community, while the other might combine deceptive links with erratic sending patterns.

Modern filtering evaluates these characteristics collectively, meaning spam detection is no longer about simply identifying a single suspicious word. Instead, it focuses on recognizing suspicious patterns across text, senders and historical behaviour.

In fact, recent upgrades to Google's spam filters include RETVec (Resilient & Efficient Text Vectorizer), an AI model that vectorizes text to capture the underlying meaning of words. This technology allows Gmail to effectively detect manipulative text patterns, like spaced-out characters or homoglyphs, while significantly reducing false positives.

Sender Reputation in the Age of AI: What Has Actually Changed?

With this advanced contextual analysis in play, the concept of sender reputation has fundamentally evolved.

Reputation is no longer simply a permanent, static score assigned to an email address. Instead, it's a fluid evaluation where your sending patterns, authentication failures, and recipient responses continuously influence how your traffic is filtered.

Because AI systems monitor these trends in real-time, sudden increases in sending volume will almost always trigger additional, aggressive scrutiny. Machine learning excels at identifying this type of unusual behaviour, which would be incredibly difficult to reliably detect using simple, static rules.

Why Good Authentication Doesn't Guarantee Inbox Placement

While establishing a solid technical foundation is necessary, it's no longer sufficient on its own. SPF, DKIM and DMARC establish important cryptographic proof of your email's authenticity, but authentication alone doesn't prove that a message is actually wanted or trustworthy.

Why Reputation Can Change Over Time

This dynamic nature explains why reputation can fluctuate dramatically over time. If a previously reliable domain suddenly starts dispatching massive volumes of unsolicited messages, its stellar historical reputation won't protect the new, anomalous traffic from immediate AI intervention.

Each provider maintains its own independent filtering infrastructure, meaning there's no single, universal AI-generated reputation score governing the entire internet.

Inbox Placement: Why the Same Email Can Have Different Outcomes

Because these filtering architectures are decentralized, the exact same email can experience vastly different outcomes depending on where it lands.

Providers like Gmail, Outlook, and Yahoo all operate entirely independent filtering infrastructures with unique internal policies. Consequently, an authenticated message might easily reach the primary inbox of one recipient while being silently routed to the spam folder of another.

This discrepancy happens because each provider places a different weighted value on your domain reputation, sending history, and specific user engagement signals.

For example, if I send an identical newsletter to both Gmail and Outlook users, the message will pass the same DNS authentication checks everywhere. But their respective AI systems evaluate the content, sender history, and internal user metrics differently, leading to distinct inbox placement results. Therefore, inbox placement can never be absolutely guaranteed by any single authentication setting.

What Developers Can Do to Improve Email Deliverability

Knowing that these systems are complex and fragmented, developers must take proactive steps to align their infrastructure with AI expectations.

Configure SPF, DKIM and DMARC Correctly

The first step is to configure your email authentication records correctly. For example, an SPF record is published as a DNS TXT record and identifies which servers are authorised to send email for your domain. A simplified example might look like this:

v=spf1 include:_spf.example.com ~all

The exact value depends on the email service you use, so you should use the SPF record provided by your email provider rather than copying this example directly.

DKIM works differently. Your email provider generates a cryptographic key pair. The public key is published in your domain's DNS records, while the private key is used to sign outgoing messages. Receiving servers can then use the public key to verify the signature.

DMARC connects these mechanisms. A basic monitoring record might look like:

v=DMARC1; p=none; rua=mailto:dmarc@example.com

Here, p=none tells receiving servers to monitor authentication failures without asking them to reject or quarantine those messages, while rua specifies an address for aggregate reports.

These records are only examples. The correct values depend on your email infrastructure, so always follow the documentation provided by your email service.

Monitor Bounces and Spam Complaints

Beyond authentication, you should monitor how recipients and receiving providers respond to your messages. Hard bounces, spam complaints, and sudden changes in delivery rates can reveal problems with an email list or sending setup.

For Gmail recipients, Google Postmaster Tools provides eligible senders with information about metrics such as spam rates, authentication and domain or IP reputation. Microsoft provides SNDS for monitoring IP addresses that send mail to Microsoft's consumer email services. Yahoo also provides sender guidance and resources through its Sender Hub.

These tools don't guarantee inbox placement, but they can help you identify delivery problems instead of relying only on whether your application reports that an email was successfully sent.

Maintain Consistent Sending Patterns

To avoid sudden changes in sending behaviour, you should keep your email volume relatively consistent and scale it gradually as your application grows. A domain that normally sends a few hundred emails a day, for example, may attract additional scrutiny if it suddenly starts sending thousands without an established sending history.

For a new domain or email account, some senders use a warm-up platform to gradually increase sending activity and build a history of email traffic. But warm-up is only one part of the process. It doesn't replace proper SPF, DKIM, or DMARC configuration, good list hygiene, or responsible sending practices and it can't guarantee inbox placement.

Test Inbox Placement Across Providers

Finally, test important emails across more than one provider. A successful SMTP response only tells you that the receiving server accepted the message. It doesn't guarantee that the message reached the primary inbox.

For example, you could send a test password-reset email to Gmail, Outlook, and Yahoo accounts and check whether the message arrives in the inbox, spam folder, or another filtered location. This can help reveal provider-specific delivery problems.

For ongoing monitoring, tools such as Google Postmaster Tools, Microsoft SNDS, and Yahoo Sender Hub can provide additional information about sender reputation and delivery-related signals.

The Limitations of AI-Powered Email Filtering

Despite these powerful monitoring tools and advanced algorithms, it's important to acknowledge what AI can't do flawlessly.

Machine learning drastically improves pattern recognition, but it doesn't make spam classification infallible. Legitimate emails frequently suffer from false positives, where critical messages are incorrectly classified as junk due to an algorithmic misjudgment.

Spammers also constantly modify their tactics, forcing these models to perpetually adapt to changing behaviour. This constant evolution is compounded by limited transparency, as email providers deliberately don't disclose the exact mathematical weights of their filtering models to prevent abuse.

Also, some filtering capabilities are inherently limited by privacy considerations, as providers must balance message analysis with strict data protection regulations. Consequently, an entirely legitimate password-reset email might still be flagged simply because an underlying model detected a temporary, unexpected variance in your sending volume.

Wrap Up

While occasional false positives are inevitable, AI has undeniably made email filtering vastly more capable of analyzing complex, nuanced contexts. Still, traditional sender reputation remains crucially important, working hand-in-hand with strict authentication and responsible sending practices.

Developers should internalize the reality that a successful network delivery is entirely different from successful inbox placement. Ultimately, while AI helps email providers decide which messages deserve the user's attention, developers still carry the responsibility of giving those intelligent systems consistently good reasons to trust their infrastructure.

]]> <![CDATA[ How to Separate Operational Intent from the Executor ]]> https://www.freecodecamp.org/news/separate-operational-intent-from-executor/ 6ac242565423a0b19b9f5c6d Sun, 04 Oct 2026 12:11:02 +0000 In my previous article, I argued that software automation needs a layer between operational intent and execution.

The reason is simple: the specification should describe what success means, and the executor should decide how to achieve it.

That separation creates a useful possibility.

If the operational specification is independent from the execution mechanism, then more than one executor should be able to satisfy the same specification.

That sounds straightforward. But in practice, it raises several difficult questions:

Can two different executors achieve the same operational outcome?

How do we compare them?

What must remain stable when the executor changes?

Which differences are acceptable?

What evidence should each executor produce?

How do we know whether executor independence is real or just theoretical?

These questions matter because modern software systems rarely keep one execution mechanism forever.

Teams change:

CI/CD platforms
cloud providers
deployment systems
infrastructure tools
orchestration engines
incident automation
AI agents

If changing the executor also changes the meaning of the operation, then the system isn't really specification-driven. The executor still owns too much of the intent.

In this article, I’ll show you how to separate an operational specification from its executors, run the same specification through two different implementations, collect evidence, compare outcomes, and identify where executor independence breaks down.

The goal isn't to prove that two executors behave identically internally.

The goal is to determine whether they can satisfy the same operational contract.

Prerequisites

You should be comfortable with:

  • software architecture

  • interfaces and dependency inversion

  • TypeScript or a similar language

  • CI/CD and deployment concepts

  • observability

  • basic testing

  • operational specifications

You don't need Kubernetes, Terraform, or any specific cloud platform. The examples are intentionally small and in-memory so the architecture stays visible.

Table of Contents

What Executor Independence Actually Means

Executor independence means that the operational specification remains valid even if the mechanism that performs the work changes.

For example, suppose the specification says:

Deploy Orders version v42.

Constraints:
- at least 3 replicas available
- error rate <= 1%
- p95 latency <= 400 ms
- rollback must remain possible

One executor may implement that using:

Kubernetes rolling deployment

Another may use:

blue/green deployment

Another may use:

a managed cloud deployment service

And later, an AI agent may generate its own execution plan.

If every executor can be evaluated against the same specification, then the specification is doing its job.

Conceptually:

                    ┌── Executor A
Operational Spec ───┼── Executor B
                    ├── Executor C
                    └── AI Agent

The specification stays stable while the execution mechanism changes.

That's executor independence.

Why Executor Independence Matters

Software teams replace tools constantly. A deployment workflow might move from:

Jenkins
↓
GitHub Actions
↓
Argo CD
↓
Kubernetes operator

If every migration requires rediscovering:

what success means
what constraints matter
what evidence is required
when rollback is allowed

then the operational meaning was never truly independent from the old tool.

This creates several risks.

  1. Tool lock-in: The system may be technically portable while the operational rules are not.

  2. Hidden behavior changes: A new executor may preserve the same deployment steps but lose an important constraint.

  3. Reimplementation drift: Teams may recreate the old behavior approximately rather than exactly.

  4. Audit gaps: It becomes difficult to prove that the new executor preserves the same operational contract.

Executor independence gives you a stronger migration target: preserve the specification, replace the mechanism.

Start with One Stable Operational Specification

To test executor independence, first define something stable.

For example:

type DeploymentSpec = {
  service: string;
  version: string;
  minReplicas: number;
  maxErrorRate: number;
  maxP95LatencyMs: number;
};

Then:

const spec: DeploymentSpec = {
  service: "orders",
  version: "v42",
  minReplicas: 3,
  maxErrorRate: 0.01,
  maxP95LatencyMs: 400,
};

This specification shouldn't contain:

kubectl
helm
terraform
AWS
Azure
Argo
GitHub Actions

Those belong to executors.

The specification describes the operational contract.

Define an Executor Contract

Now define the minimum interface an executor must satisfy.

For example:

type ExecutionEvidence = {
  deployedVersion: string;
  availableReplicas: number;
  errorRate: number;
  p95LatencyMs: number;
};

interface DeploymentExecutor {
  execute(
    spec: DeploymentSpec
  ): Promise<ExecutionEvidence>;
}

This interface doesn't say how deployment happens.

It only says:

given a specification,
perform the operation,
return evidence

In this article, evidence means the observable facts produced or collected after execution that let us evaluate what actually happened. For a deployment, that might include the version that is running, the number of available replicas, the measured error rate, and p95 latency.

Evidence isn't the executor's opinion that the operation succeeded. It's the data we can compare against the specification.

That's important. If the executor interface includes tool-specific concepts, portability starts leaking.

For example, this would be more coupled:

interface DeploymentExecutor {
  executeKubectlCommand(
    namespace: string,
    manifestPath: string
  ): Promise<void>;
}

Now the interface already assumes Kubernetes.

That's not executor-independent.

Build a First Executor

Let’s create a simple in-memory executor.

class RollingDeploymentExecutor
  implements DeploymentExecutor {
  async execute(
    spec: DeploymentSpec
  ): Promise<ExecutionEvidence> {
    return {
      deployedVersion: spec.version,
      availableReplicas:
        spec.minReplicas,
      errorRate: 0.004,
      p95LatencyMs: 280,
    };
  }
}

This executor simulates a rolling deployment.

Internally, you can imagine that it:

starts new replicas
waits for health
gradually replaces old replicas

But none of that appears in the specification. The executor owns the mechanism.

Build a Second Executor

Now create a different strategy.

class BlueGreenExecutor
  implements DeploymentExecutor {
  async execute(
    spec: DeploymentSpec
  ): Promise<ExecutionEvidence> {
    return {
      deployedVersion: spec.version,
      availableReplicas:
        spec.minReplicas + 2,
      errorRate: 0.003,
      p95LatencyMs: 260,
    };
  }
}

This executor may conceptually:

create a parallel environment
verify it
switch traffic
keep the old environment available

Its internal process is different, but its evidence shape is the same. That means both can be evaluated against the same operational specification.

Run the Same Specification Through Both Executors

Now execute both.

const rolling =
  new RollingDeploymentExecutor();

const blueGreen =
  new BlueGreenExecutor();

const rollingEvidence =
  await rolling.execute(spec);

const blueGreenEvidence =
  await blueGreen.execute(spec);

At this point, we have:

same specification
different executors
different internal behavior
different evidence values

The important question isn't whether they performed the same steps. They didn't. The question is if they both satisfied the same operational contract.

Compare Evidence, Not Internal Steps

Executor independence depends on comparing outcomes rather than implementation details.

Suppose:

Rolling deployment:
replicas = 3
error rate = 0.4%
p95 = 280 ms

Blue/green:
replicas = 5
error rate = 0.3%
p95 = 260 ms

Those outputs aren't identical, but both may be conformant. That matters.

Executor independence doesn't require:

same commands
same number of steps
same topology
same timing
same infrastructure

It requires:

same operational intent
satisfied constraints
required evidence
acceptable outcome

This is similar to interface-based programming. Two implementations can behave differently internally while satisfying the same contract.

Normalize Executor-Specific Evidence

Real executors often return different evidence formats.

Suppose executor A returns:

{
  "readyReplicas": 3,
  "image": "orders:v42",
  "latencyP95": 280
}

Executor B returns:

{
  "instancesHealthy": 5,
  "releaseVersion": "v42",
  "p95Ms": 260
}

These can't be compared directly. You need adapters.

For example:

type CanonicalEvidence = {
  version: string;
  availableReplicas: number;
  p95LatencyMs: number;
};

Adapter A:

function normalizeRolling(
  raw: {
    readyReplicas: number;
    image: string;
    latencyP95: number;
  }
): CanonicalEvidence {
  return {
    version:
      raw.image.split(":")[1],
    availableReplicas:
      raw.readyReplicas,
    p95LatencyMs:
      raw.latencyP95,
  };
}

This adapter translates the rolling executor's native output into the canonical evidence model. It extracts the version from the image tag, maps readyReplicas to availableReplicas, and renames latencyP95 to p95LatencyMs.

The important point is that the adapter doesn't change the operational meaning. It only converts executor-specific field names and formats into the shared representation expected by the specification layer.

Adapter B:

function normalizeBlueGreen(
  raw: {
    instancesHealthy: number;
    releaseVersion: string;
    p95Ms: number;
  }
): CanonicalEvidence {
  return {
    version:
      raw.releaseVersion,
    availableReplicas:
      raw.instancesHealthy,
    p95LatencyMs:
      raw.p95Ms,
  };
}

This adapter does the same job for the blue/green executor. Its raw output uses different names, but those values represent the same operational concepts: version, available capacity, and p95 latency.

With both adapters in place, the rest of the system no longer needs to understand the executor-specific shapes. It can evaluate both results using the same canonical model.

Now both executors produce evidence that can be evaluated using the same model.

This is an important architectural boundary. Executor-specific evidence stays near the executor, while canonical evidence belongs to the specification layer.

Separate Execution Evidence from Conformance

The executor should produce evidence. It shouldn't decide whether the operation was successful from the perspective of the specification.

That distinction prevents another form of coupling.

For example, avoid:

return {
  success: true,
};

A generic success flag tells you very little.

Instead, return evidence:

return {
  deployedVersion: "v42",
  availableReplicas: 3,
  errorRate: 0.004,
  p95LatencyMs: 280,
};

Then evaluate it separately.

type ConformanceResult = {
  conformant: boolean;
  failures: string[];
};

function evaluateConformance(
  spec: DeploymentSpec,
  evidence: ExecutionEvidence
): ConformanceResult {
  const failures: string[] = [];

  if (
    evidence.deployedVersion !==
    spec.version
  ) {
    failures.push(
      "wrong-version"
    );
  }

  if (
    evidence.availableReplicas <
    spec.minReplicas
  ) {
    failures.push(
      "insufficient-replicas"
    );
  }

  if (
    evidence.errorRate >
    spec.maxErrorRate
  ) {
    failures.push(
      "error-rate-too-high"
    );
  }

  if (
    evidence.p95LatencyMs >
    spec.maxP95LatencyMs
  ) {
    failures.push(
      "latency-too-high"
    );
  }

  return {
    conformant:
      failures.length === 0,
    failures,
  };
}

The evaluator receives two things: the specification that defines what should be true, and the evidence that describes what was actually observed.

It checks each constraint independently. A wrong version adds wrong-version, too few replicas adds insufficient-replicas, and the error-rate and latency checks work the same way.

At the end, conformant is true only when no failures were recorded. Returning the individual failure names is useful because it explains why an execution didn't conform instead of collapsing everything into a generic false.

This is why the executor should return facts rather than a final verdict. The same evidence can be reevaluated later if the specification changes, if an audit needs to reconstruct the decision, or if you want to compare multiple executors using exactly the same rules.

Now:

executor → evidence

specification + evidence → conformance

This separation will become important later.

What Counts as Equivalent Execution?

Two executors don't need to produce identical evidence. They need to produce evidence that satisfies the same specification.

Suppose:

Executor A
replicas: 3
error rate: 0.4%
latency: 280 ms

Executor B
replicas: 5
error rate: 0.3%
latency: 260 ms

Both may pass.

Now suppose executor B produces:

replicas: 2
error rate: 0.3%
latency: 260 ms

It fails one constraint.

That means:

Executor A:
conformant

Executor B:
non-conformant

The executors are still independent implementations, but only one satisfied the specification for this execution.

That distinction matters.

Executor independence doesn't guarantee executor correctness. It only gives you a stable contract against which correctness can be evaluated.

Where Executor Independence Usually Breaks

There are several common failure modes.

Tool-Specific Fields in the Specification

For example:

kubernetesNamespace: production
helmChart: orders

If those are truly implementation details, they shouldn't live in the operational specification.

Executor-Owned Success Rules

If each executor defines its own thresholds, the specification is no longer authoritative.

Missing Evidence Normalization

Different executors may expose different concepts that are never mapped into a common model.

Hidden Preconditions

Executor A may require an approval. Executor B may not.

If approval is part of operational intent, that rule shouldn't live only inside one executor.

Hidden Recovery Behavior

One executor may rollback automatically, while another may leave the failed state running.

If recovery behavior matters, the specification should express that expectation.

Semantic Mismatch

Two tools may use the same words differently.

For example:

healthy
ready
available
running

Those concepts need explicit definitions. Otherwise executor portability is superficial.

How to Test Executor Portability

You can test executor independence explicitly.

Start with a suite of specifications.

For example:

const cases: DeploymentSpec[] = [
  {
    service: "orders",
    version: "v42",
    minReplicas: 3,
    maxErrorRate: 0.01,
    maxP95LatencyMs: 400,
  },
  {
    service: "payments",
    version: "v18",
    minReplicas: 5,
    maxErrorRate: 0.005,
    maxP95LatencyMs: 250,
  },
];

Then run each specification against every executor.

const executors:
  DeploymentExecutor[] = [
    new RollingDeploymentExecutor(),
    new BlueGreenExecutor(),
  ];

for (const spec of cases) {
  for (const executor of executors) {
    const evidence =
      await executor.execute(spec);

    const result =
      evaluateConformance(
        spec,
        evidence
      );

    console.log({
      spec: spec.service,
      executor:
        executor.constructor.name,
      result,
    });
  }
}

This produces a useful matrix:

               Rolling    BlueGreen
Orders v42     PASS       PASS
Payments v18   PASS       FAIL

Now you have evidence about executor portability.

That is much stronger than assuming that both tools support deployments, so they're equivalent.

Why This Matters for AI Agents

AI agents make executor independence even more interesting.

An AI agent may generate a new plan every time.

For example:

Execution 1:
scale
deploy
verify
route

Execution 2:
create parallel environment
verify
switch traffic

Execution 3:
deploy canary
observe
expand rollout

The plans differ, and the executor behavior is dynamic. That makes step-by-step equivalence unrealistic.

But the specification can still remain stable.

For example:

Deploy Orders v42.

Constraints:
replicas >= 3
error rate <= 1%
latency <= 400 ms

Evidence:
running version
replicas
error rate
latency

The AI agent can choose any acceptable plan, and its result is still evaluated using the same contract.

This creates a powerful boundary:

agent autonomy
inside
operational constraints

The agent can optimize execution. It can't silently redefine success.

A Small End-to-End Example

Let’s put the pieces together.

Start with the specification:

const spec: DeploymentSpec = {
  service: "orders",
  version: "v42",
  minReplicas: 3,
  maxErrorRate: 0.01,
  maxP95LatencyMs: 400,
};

Create two executors:

const executors:
  DeploymentExecutor[] = [
    new RollingDeploymentExecutor(),
    new BlueGreenExecutor(),
  ];

Run them:

for (const executor of executors) {
  const evidence =
    await executor.execute(spec);

  const conformance =
    evaluateConformance(
      spec,
      evidence
    );

  console.log(
    executor.constructor.name,
    evidence,
    conformance
  );
}

Possible result:

RollingDeploymentExecutor

evidence:
version = v42
replicas = 3
error rate = 0.004
p95 = 280

conformance:
PASS

and:

BlueGreenExecutor

evidence:
version = v42
replicas = 5
error rate = 0.003
p95 = 260

conformance:
PASS

The executors didn't behave identically, and they didn't need to. They satisfied the same operational specification.

Now imagine the second executor reports:

replicas = 2

The result becomes:

BlueGreenExecutor

conformance:
FAIL

reason:
insufficient-replicas

That's exactly what we want: the specification remains stable, and the executor changes.

Evidence reveals whether the execution satisfies the contract.

A Practical Workflow

If you want to test executor independence in a real system, I would use this sequence.

1. Pick One Operation

For example:

deploy service
restore backup
rotate certificate
scale worker pool

2. Extract the Operational Specification

Define:

objective
constraints
evidence requirements
recovery expectations

3. Remove Tool-Specific Language

Look for:

kubectl
Terraform
GitHub Actions
AWS CLI
specific resource IDs

Keep only what is truly part of the operational intent.

4. Define an Executor Interface

Make the executor responsible for:

performing the operation
producing evidence

5. Build the First Adapter

Wrap the current implementation, don't rewrite it.

6. Build a Second Executor

Use:

another tool
another deployment strategy
a simulator
a test double
an AI agent

7. Normalize Evidence

Map executor-specific observations into a canonical evidence model.

8. Evaluate Conformance Separately

Don't let the executor decide whether it passed.

9. Compare Portability

Run the same specifications through multiple executors.

10. Investigate Differences

Ask:

Is the executor wrong?

Is the specification incomplete?

Is evidence missing?

Are the concepts actually equivalent?

Those differences are useful. They expose hidden coupling.

What Executor Independence Does Not Mean

Executor independence doesn't mean that all tools are interchangeable.

Different executors may have different:

capabilities
costs
latency
failure modes
security models
operational complexity

A specification may also require a capability that one executor simply can't provide.

For example:

zero-downtime deployment

may be feasible in one platform and impossible in another.

That's not a failure of the specification. It's useful information. The executor can't satisfy the contract.

Executor independence also doesn't mean ignoring implementation details. Implementation details still matter for:

performance
security
cost
reliability
maintainability

The point is narrower: the operational meaning shouldn't depend on one specific execution mechanism.

From Replaceable Tools to Stable Operational Intent

Software infrastructure changes constantly.

Tools come and go, execution strategies evolve, cloud platforms change, and agents become more capable.

If operational intent is embedded inside each executor, every change risks becoming a semantic migration.

But if intent is represented independently:

Operational specification
          ↓
     stable contract
          ↓
   replaceable executor

then the system becomes easier to evolve.

This also gives us something else:

evidence from multiple executors
evaluated against the same specification

At that point, we can stop asking:

Did the tool finish?

and start asking:

To what degree did this execution conform to the specification?

That's the next problem.

Conclusion

Executor independence isn't about pretending every tool is the same. It's about protecting operational intent from implementation churn.

A specification describes:

what should happen
what constraints must hold
what evidence is required

An executor decides:

how to make it happen

Then execution produces evidence. And evidence can be evaluated independently.

That gives us:

Specification
      ↓
Executor A ──→ Evidence A
Executor B ──→ Evidence B
      ↓
Evaluation

Two executors can use completely different strategies and still satisfy the same operational contract.

That's a useful property for ordinary automation.

It becomes even more important when executors are autonomous agents whose internal plans may change from one run to the next.

But once multiple executors can operate against the same specification, a new question becomes unavoidable:

How should we measure conformance between the specification and what actually happened?

That's where I want to go next.

]]> <![CDATA[ How to Diagnose and Fix AI Inference Latency on Kubernetes ]]> https://www.freecodecamp.org/news/how-to-diagnose-and-fix-ai-inference-latency-on-kubernetes/ 6abca2f9f944d026311c7f7b Wed, 30 Sep 2026 05:49:45 +0000 It's Thursday, around quarter past two. Your team shipped an internal assistant two weeks ago. The demo went well enough that someone in finance asked whether it could read contracts. Word spread. Today, for the first time, everyone is using it at once.

The support channel has the same complaint arriving in six different tones. Is it down? Mine is just spinning. It worked this morning. Somebody has posted a screenshot of a loading indicator with no caption, which you suspect they're enjoying.

So you open the dashboard.

Nodes ready. Pods running. CPU at 20%. No restarts, nothing in CrashLoopBackOff, no alert fired. By every signal Kubernetes gives you, the system is healthy.

It is not healthy. Somebody is eleven seconds into waiting for the first word of an answer, and they're about to go back to doing it the old way, and they're not coming back.

That is the failure this piece is about. Not the outage. The quiet one, where everything is technically up and the product is still unusable.

Table of Contents

Your First Instinct is Wrong

The instinct at that moment is to suspect the model. Wrong version, bad quantization, context too long, or that someone changed the system prompt.

It is almost never the model.

Walk one request end to end and you find four stages. It gets routed to a replica. It waits in a queue. The model reads the prompt. The model streams an answer. Only the last two involve the model doing anything you are paying for. The first two are queueing and routing, which is to say infrastructure.

When people say their model is slow, the model is usually fine. The wait is somewhere else.

You Are Not the Only One

It's worth knowing before you go hunting that this is the most common complaint in the field, not an unlucky configuration on your part.

A practitioner survey of 200 people running AI in production found that nearly half, 49.5%, named latency at peak load as their single hardest scaling problem. Not accuracy. Not hallucination. Not cost per token. Latency, at the exact moment people are trying to use the thing.

Two more figures from the same research are worth holding alongside it. 59.5% said running inference closer to users or decision points is critical or very important. 45.5% still serve from a single cloud region. People know proximity matters and have not managed to act on it, which tells you something about what multi-region GPU capacity costs to stand up.

The same research mentions organizations mandating sub-250ms response times alongside 99.9% availability. Read that carefully, because taken literally it is impossible. No meaningful LLM response completes in a quarter of a second. It has to mean time to first token (how long before the first word of the answer appears), and that is the right thing to hold yourself to anyway. Once tokens are flowing at a readable pace, people stop counting. The silence before the first one is what loses them.

The Thing Nobody Told You When You Deployed It

Here's the part that explains everything else.

A normal web request is short, small, and costs roughly what the last one cost. Kubernetes scheduling assumes that. Service load balancing assumes it. The Horizontal Pod Autoscaler assumes it. Your ingress controller's default timeout assumes it. Nobody wrote the assumption down, because for fifteen years it was simply true.

An LLM request breaks it in three places.

It runs in two phases with completely different profiles. Prefill: the model reads the entire prompt in one pass. Compute-bound, bursty, and this is what decides how long your user stares at nothing. Then decode: one token at a time, each needing the accumulated state of every token before it. That state is the KV cache. It lives in GPU memory next to the weights, and it grows as the answer grows.

So requests are not short, they can run for a minute. They do not cost the same, since a prompt with a document pasted into it can cost fifty times what a one-liner costs. And the genuinely scarce resource is GPU memory, which does not appear on a single dashboard you inherited.

That is why your monitoring said everything was fine. It was measuring the wrong machine.

Where the Eleven Seconds Went

Follow the request through the four stages and the failure modes fall out in order. These six are the ones practitioners name most often, and they map neatly onto the path.

Stage One, Routing: Your Load Balancer is Guessing

Round-robin stops being fair the moment request costs diverge. Three heavy prompts land on one pod while its neighbour handles one-liners. The loaded pod's queue grows, its tail latency climbs, and the fleet average still looks completely reasonable, which is why nobody noticed before today.

Long-lived connections make it worse. With HTTP/2 or keepalive, a client opens one connection and sends everything down it, and a Kubernetes Service balances per connection rather than per request. All that traffic pins to a single backend and stays there.

Then there is the cost most teams have never considered. If a replica already holds the prefix of this prompt in its KV cache, routing there skips real prefill work. Since most production prompts share a long system preamble, that is not an edge case, it is most of your traffic. Round-robin throws the benefit away at random, every time.

Stage Two, The Queue: Nothing is Coming to Help

Your autoscaler could add capacity. It will be late, and the reason is unglamorous: a new replica has to pull a multi-gigabyte serving image, fetch 20GB to 70GB of weights over the network, and load them into GPU memory. Minutes, not seconds, and that assumes a node with a free card already exists. If it does not, add provisioning time, and add whether your provider has stock in that zone today.

There is a second problem stacked on the first. The signal is usually wrong. CPU tells you nothing here, because the CPU is idle while the GPU works. GPU utilization is barely better: it reports that a kernel was executing during the sample window, and a server handling one request and a server handling sixty both read 100%. It cannot tell you whether anyone is waiting.

Queue depth can. vLLM already exposes running and waiting request counts as Prometheus metrics. Scale on those:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-server
spec:
  scaleTargetRef:
    name: llm-server
  minReplicaCount: 2
  maxReplicaCount: 8
  cooldownPeriod: 600
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(vllm:num_requests_waiting{app="llm-server"})
        threshold: "5"

Check the metric names against your vLLM version. Several have been renamed across releases, and the failure is silent.

Two values there are deliberate. The minimum of 2, because a cold start means you cannot sit at zero and expect to serve. The ten minute cooldown, because scaling down eagerly means paying that cold start again shortly afterwards.

Which leads to the uncomfortable conclusion about autoscaling inference. Reactive scaling is permanently late by however long your cold start is, and no trigger fixes that. The fix is to keep more spare capacity than feels comfortable. Run extra replicas at all times, scale up earlier than you think you need to, and scale down more slowly. This also means scale to zero isn't a good fit for user-facing inference. Save it for batch jobs, where nobody is sitting there waiting for a response.

Stage Three, Prefill: The GPUs Are There, and Useless

Kubernetes hands out CPU in millicores and GPUs whole. A pod asks for one card and gets the entire card, whether it needs 10% of it or all of it.

The waste is the obvious problem. Fragmentation is the one that ruins a Thursday. A 70B model in 16-bit needs roughly 140GB of weights, so a replica needs several GPUs on one node. You can have six free GPUs spread two-two-one-one across four nodes, and the replica sits in Pending indefinitely. Capacity on paper, none of it usable, and no alert, because nothing is broken.

Sharing a card gives three options, each with a catch you should know before choosing. Time-slicing is easy to enable and gives no memory isolation, so one greedy pod can take down its neighbour. Fine for development, not for anything customer-facing. MIG gives real hardware isolation but fixed slice geometry, which means predicting your workload mix before you have one. Dynamic Resource Allocation is the structurally correct answer and brings accelerator requests properly into the core Kubernetes API.

None of these are on by default. Each needs a deliberate decision from somebody who understands the workload, who is usually not in the room when the cluster gets built.

Stage Four, Decode: Your Ingress is Cutting People Off

Many ingress controllers ship with a 60 second read timeout and response buffering enabled.

The timeout terminates long generations mid-sentence. Users report that as a crash, and you cannot reproduce it, because your test prompt is short. The buffering holds tokens and releases them in clumps, which destroys the feel of streaming while the model is behaving perfectly.

Two config lines, written for a workload that no longer exists.

And Underneath All four: Bursty Traffic and Nobody's Name on the Problem

Internal AI tools don't get steady traffic. They get sudden spikes: when everyone starts work at 9am, in the hour after a company all-hands meeting, or when someone shares the link in a busy Slack channel and says it's actually quite good. A fixed replica count handles exactly one of those. Most teams set it once during a quiet week and never revisit it, so they get both waste and collapse on the same day.

Then the sixth failure mode, which is not technical at all, and which is the reason the other five survive for quarters.

Platform engineering owns the cluster. The cluster is green. From where they sit, the job is done and done well. Application developers can see latency is bad and cannot see why, because every cause sits in scheduling, scaling and routing that they neither control nor observe. MLOps owns the model artifact and almost nothing that determines how it performs at quarter past two on a Thursday.

Three teams, all telling the truth, and the problem living in the gap between their on-call rotas, where no alert is configured and no dashboard points.

Meanwhile, the Platform Did Not Stand Still

The encouraging part of this story is that the gaps above are known, and the fixes have been shipping.

Dynamic Resource Allocation brings accelerator hardware requests into the core Kubernetes API, so the scheduler can reason about the device rather than counting opaque units. Kueue adds job queueing and multi-tenant GPU quota alongside CPU and memory, which is how you stop a research job starving the endpoint your users depend on. Gateway API Inference Extension and llm-d add model-aware routing: request criticality, dynamic balancing from live model metrics, and awareness of cache state. That last one is the direct answer to the guessing load balancer in stage one.

Published benchmarks give a sense of scale: Kueue reducing multi-stage workload makespan by up to 15%, dynamic accelerator slicing cutting mean job completion time by 36%, and Gateway API plus llm-d improving tail time to first token by up to 90% under heavy load.

Treat those as headroom rather than forecast. Up to 90% is measured on a configuration chosen to demonstrate the improvement, and your gain depends entirely on how poor your baseline is. If you already run least-outstanding-requests balancing with a warm cache, expect far less. If you are on stock round-robin behind a default ingress, possibly something dramatic, because that baseline is genuinely bad.

The direction is what matters. The biggest published gain is in tail time to first token, which is precisely what half the surveyed practitioners named as their worst problem. Somebody built the fix for the thing people were complaining about.

How This Got Here in the First Place

Worth a short detour, because it explains why the tooling arrived late.

When generative AI landed in enterprise planning documents, the confident position was that Kubernetes could not hold it. Container orchestration was built for small fungible units of CPU and memory, cheap restarts, interchangeable replicas. AI needed scheduled specialized silicon carrying enormous state that takes minutes to place. A purpose-built platform was coming.

It never arrived. AI went into the microservices stack and Kubernetes absorbed it.

The survey data explains why. More than half of enterprises do not train models at all, and only 7% deploy a model on any given day. Meanwhile 82% of container users run Kubernetes in production, and 66% of organizations hosting generative AI already serve it there. The industry spent years designing for the workload almost nobody has, while everybody else quietly downloaded weights and put them behind an endpoint.

What actually converged was not a place to run AI. It was a standard way to run it, with the location left negotiable. Which is exactly why your problems today are placement, routing and capacity rather than anything model-shaped.

What To Do Once the Fire is Out

The path from working to reliable comes down to five things: capacity, performance, placement, resilience and ownership.

Capacity

Requests per second is meaningless when one request is 50 tokens and the next is 8,000. Plan in tokens per second and track prompt and completion separately, because they stress different parts of the system. Prompt tokens are a compute burst during prefill. Completion tokens are a sustained drip during decode that holds GPU memory for the life of the response.

Load test with your worst realistic prompt at your worst realistic concurrency, and plan from that number. A benchmark built on short prompts gives a reassuring figure that evaporates in week one.

On supply, treat GPUs as something you reserve ahead of time rather than request when needed. Keep a committed baseline and burst on top. Find out your provider's quota and regional stock position before the evening you need it, and use Kueue so one team cannot quietly consume the pool.

Performance

Two numbers matter most. Time to first token (TTFT) is how long a user waits between sending a prompt and seeing the first word of the answer. It decides whether people trust the product. Inter-token latency is the gap between each word after that, and it decides whether the answer feels smooth to read.

Track both at p95 and p99, not the average. These are percentiles. p95 is the time that 95% of requests come in under, so it shows what your slowest 1 in 20 users experience. p99 does the same for the slowest 1 in 100. An average can look healthy while those users wait far too long, and they're the ones who give up.

Next, tune the serving layer. This is the software that loads the model and handles requests, such as vLLM. It's where you'll get the biggest improvements for the least effort and cost. Continuous batching keeps the GPU busy without making early requests wait for a batch to fill. Quantization roughly halves memory footprint where the quality tradeoff is acceptable, and freed memory becomes concurrent requests. Set maximum context length to what your product actually needs rather than what the model supports. If the model handles 128K and your longest genuine prompt is 6K, you are reserving KV cache for a scenario that never occurs. One config line, and it can substantially raise effective concurrency.

Placement

Taint GPU nodes so general workloads cannot land on them and only inference pods with matching tolerations schedule there. A logging sidecar should never be the reason a model replica cannot place.

Keep weights close. Node-local NVMe or a zone-local cache turns a 40GB network pull into a fast local read, which feeds directly into cold start, autoscaling lag, and the peak-hour latency you spent Thursday afternoon on. This is the highest-leverage unglamorous fix on the list.

Respect topology. Cards connected by NVLink behave very differently from cards that merely share a PCIe bus, and for a sharded model that interconnect sits on the critical path of every token generated.

Then be honest about geography. If your users are in London and your GPUs are in Virginia, every token crosses an ocean, and a well-tuned model behind a long network path is still a slow product.

Resilience

Readiness should pass only when the model is loaded and genuinely able to serve. Use a startup probe with a generous failure threshold, otherwise liveness kills the pod partway through loading weights and you get a restart loop that presents as a mystery and costs an hour.

Set a PodDisruptionBudget so a routine node upgrade cannot remove half your replicas at once.

Raise the termination grace period past the 30 second default and add a preStop hook. A generation can easily run longer than that, so otherwise every rollout severs in-flight streams and your users experience deployments as random failure.

Shed load deliberately. Past a queue depth threshold, return a fast busy-try-again rather than accepting a request that will hang for two minutes. This feels wrong to engineers and is right for users. People forgive a quick honest retry. Nobody forgives a frozen screen.

Have somewhere to fall back to: a second region, a smaller model covering the common cases, or an external API at the edge of disaster. Given that 45.5% of practitioners run from a single region, this is the most commonly skipped item on the list, which makes it the most likely candidate for your next incident.

Ownership

Agree on the numbers and write them down: time to first token, inter-token latency, error rate, queue time. Put them on one dashboard showing cluster and model metrics side by side, and make sure both teams open the same link rather than maintaining separate versions of reality.

Split responsibility explicitly. Platform owns the ability to hit the target: capacity, scheduling, routing, scaling, placement. Application owns how the model is configured and called: context length, batching parameters, prompt size, retry behaviour. When the number slips, both show up, and the conversation starts from the same graph instead of two that disagree.

Four Things Your Monitoring Still Isn't Telling You

Latency, traffic, errors and saturation are still necessary and no longer sufficient. Four more belong on the board.

Token consumption rate, split prompt and completion, because that is your real unit of capacity. Cost per request by tenant and query type, because we run eight GPUs is not an answer to what a feature costs per user. Output quality, tracking hallucination rate and guardrail drift, because latency work that degrades quality is not a win. Model attribution, recording which checkpoint and which prompt version produced a response, because when quality moves you need to know what changed.

That last one connects to a shift worth noticing. Model registries have become peers to container registries, and prompts and agent configurations are increasingly versioned as GitOps artifacts in the same delivery pipelines as everything else.

The prompt is code now. It changes production behaviour, it can break things, and it should go through review, versioning and rollback like anything else that does. Plenty of teams are still editing prompts in a web console and wondering why quality shifted on a Tuesday.

The Thing That is Still Unsettled

The substrate has converged, and the speed of it is the evidence. A Kubernetes AI conformance programme launched in late 2025 with 18 platforms. Within roughly four months it had grown to 31 and extended to agentic workloads, covering the major hyperscalers, the enterprise and private cloud distributions, GPU neoclouds and edge networks. Companies that agree on almost nothing agreed on this.

The layer above has not converged at all. AI gateways, evaluation platforms, agent control planes, agent observability. All advancing faster in commercial products than in any standard body, and all sitting exactly where the interesting business logic is heading.

The risk is worth naming plainly. You can be perfectly portable at the Kubernetes layer and completely locked in at the layer where you actually build. Your pods will migrate between clouds beautifully while your agent definitions, eval suites and gateway policies stay exactly where they are, because there is nowhere standard to move them to.

That is not a reason to avoid those tools. They solve real problems today. It is a reason to know which of your components has an exit and which does not, and to make that a decision you wrote down rather than one you discover during a renewal negotiation.

Monday Morning

Start with one measurement. Time a cold start end to end, from pod created to first token served, and write the number down. That figure is your autoscaling floor, and your minimum replica count falls directly out of it.

Put time to first token and queue depth on your main dashboard at p99. Replace CPU-based autoscaling with a queue-depth trigger and minimums that respect the cold start. Check your ingress read timeout and buffering against a real long streaming response rather than a test prompt. Fix the termination grace period so deployments stop cutting people off.

Then, over the next few weeks: taint your GPU nodes, move weights to node-local or zone-local storage, set maximum context length to what you actually use, and find out whether you are still on round-robin. You probably are.

And before the quarter ends, decide who owns the end-to-end latency number, and say it out loud in a room with the other team present.

Back to Thursday

The dashboard was never lying. It was answering a different question.

It was telling you whether the containers were alive, which they were. It could not tell you that requests were queueing behind a badly routed batch, that no new replica was coming for four minutes, that six GPUs were free and unusable, or that the ingress was buffering tokens it should have been streaming. Nothing in the standard toolkit is pointed at any of that.

Getting a model to answer on Kubernetes takes an afternoon. Getting it to answer well when everyone logs in at once is a different discipline, and it looks far more like ordinary infrastructure engineering than the current conversation suggests. Scheduling, routing, capacity, ownership. We have been solving those since long before any of this carried the AI label. The skills are already in the building. They need pointing at the right metric.

A green cluster is where the work starts. What counts is what the person typing the prompt sees.

I hope you’ve enjoyed this and learned something new. I’m always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.

If you’ve enjoyed my writing and want to keep me motivated, consider leaving stars on GitHub and endorsing me for relevant skills on LinkedIn.

Till the next one, happy exploring!

]]> <![CDATA[ Why Software Automation Needs a Layer Between Intent and Execution ]]> https://www.freecodecamp.org/news/software-automation-intent-execution-layer/ 6abb57d97eb0e171af50644b Tue, 29 Sep 2026 06:16:57 +0000 Software automation is good at turning instructions into actions.

A pipeline can deploy a release, Terraform can provision infrastructure, a runbook can restart a service, or an agent can inspect telemetry and choose a remediation step.

But all of those systems share the same weakness: they're usually better at executing instructions than at representing the intent behind those instructions.

That gap is easy to ignore while automation is simple. If a script only runs one command, the script itself may be enough to explain what should happen.

But as systems become more distributed, more policy-driven, and more autonomous, execution logic starts carrying responsibilities it was never designed to own.

A deployment workflow may now encode:

what should be deployed
where it may run
which constraints must hold
which evidence must be checked
who may approve the change
when rollback is required
which failures are acceptable

At that point, the automation is no longer just executing. It's also becoming the place where operational intent is defined.

And that's a problem.

Because when intent and execution are fused together, changing the executor can also change the meaning of the operation.

In my previous article, I introduced the idea of an executable operational specification: a machine-readable description of what an operation is supposed to achieve and how its outcome can be evaluated.

In this article, I want to go one level deeper.

The central question is:

What should exist between human intent and the system that executes it?

I will explore:

  • why automation tools often become accidental specifications

  • why intent and execution evolve at different speeds

  • what gets lost when intent is embedded in pipelines and scripts

  • why desired state is useful but still incomplete

  • how to introduce an explicit operational intent layer

  • how preconditions, constraints, evidence, and recovery rules fit into that layer

  • how execution plans differ from operational specifications

  • how this structure helps with AI agents

  • and what this separation still can't solve

The goal isn't to add another abstraction for its own sake. The goal is to make operational decisions easier to understand, evaluate, and change without rewriting the meaning of the operation every time the execution mechanism changes.

Prerequisites

You should be comfortable with:

  • key software architecture concepts

  • CI/CD

  • infrastructure automation

  • observability

  • distributed systems

  • basic policy concepts

  • TypeScript or a similar language

You don't need a specific orchestration tool or cloud platform. The examples are deliberately small and tool-independent.

Table of Contents

Automation Tools Often Become Accidental Specifications

Consider a deployment workflow. It may begin as something simple:

steps:
  - build
  - test
  - deploy

Then production requirements appear.

You add:

approval
region validation
health checks
canary percentage
rollback logic
error-rate thresholds
latency thresholds
artifact verification

Eventually, the pipeline becomes the only place where the operation is fully described.

That creates an accidental architecture:

Operational intent
      ↓
pipeline configuration
      ↓
execution

The pipeline is now doing two jobs: it describes what the operation means and how the operation is executed.

Those responsibilities are related, but they're not identical.

Suppose this workflow says:

deploy:
  image: orders:v42

verify:
  errorRateBelow: 0.01

rollback:
  whenVerificationFails: true

That may work perfectly. But now imagine moving the deployment to another platform.

If the operational rules only exist inside this pipeline, you must rediscover and reimplement them while moving the executor.

The real migration becomes:

move execution mechanism
+
reconstruct operational intent

That's much more dangerous than just changing tooling.

Intent and Execution Change at Different Speeds

Operational intent is often relatively stable.

For example:

Deploy the approved version.

Maintain availability.

Do not exceed the error-rate threshold.

Keep rollback possible.

Do not deploy outside the approved region.

Those expectations may remain valid for years. But execution mechanisms change much more frequently.

A team may move through:

shell scripts
Jenkins
GitHub Actions
Argo CD
Kubernetes operators
cloud-native deployment services
AI agents

If intent is tightly coupled to those mechanisms, every tooling change risks becoming an operational redesign.

That creates unnecessary churn.

You want:

stable intent
     ↓
replaceable execution

not:

new executor
     ↓
rediscover the meaning of the operation

This separation becomes more important as automation becomes more sophisticated.

The more intelligence the executor contains, the easier it becomes for intent to disappear inside implementation details.

What Gets Lost When Intent Lives Inside the Executor

When operational intent is embedded directly in scripts and pipelines, several things become harder.

1. Review

A reviewer may see:

if: error_rate < 0.01

but not know whether that threshold is:

a business requirement
an SRE policy
a temporary rollout rule
a historical workaround

The value is executable. Its meaning isn't explicit.

2. Auditability

Six months later, somebody asks:

Why was this deployment allowed?

The answer may require reconstructing:

pipeline version
feature flags
runtime variables
dashboards
approvals
operator decisions

3. Portability

Changing tools means reimplementing rules.

4. Testing

You can test whether the pipeline works. But it's harder to test whether the operational intent itself is complete.

5. Governance

Authorization rules may become buried in workflow code.

For example:

production changes require approval

may be implemented as a platform-specific branch protection rule instead of being represented as part of the operation.

6. AI Execution

An AI agent may generate a valid sequence of commands while violating an unstated operational constraint.

The problem isn't that the executor is weak. It's that the intent isn't independently represented.

Desired State Helps, But It's Not the Whole Intent

Many modern systems already use desired-state models.

For example:

replicas: 3
image: orders:v42

That's useful.

The system can compare:

desired state
vs.
observed state

and reconcile the difference.

But operational intent often includes more than final configuration.

Suppose the desired state is:

orders:v42
replicas: 3

The operation may also require:

do not reduce availability below 99.9%
do not deploy if payment dependency is degraded
keep error rate below 1%
require approval for production
rollback if latency exceeds 400 ms

Those aren't simply target-state properties.

Some are:

preconditions
runtime constraints
evidence requirements
governance rules
recovery rules

Desired state is part of operational intent, but it's not always the entire operational intent.

The Missing Layer

A more explicit architecture looks like this:

Human / organizational intent
             ↓
Operational specification
             ↓
Execution planning / binding
             ↓
Executor
             ↓
Actions
             ↓
Evidence
             ↓
Evaluation

The operational specification is the missing layer.

Its job is not to execute commands. Its job is to preserve the meaning of the operation independently from the mechanism performing it.

That layer can contain:

objective
scope
preconditions
constraints
required evidence
authorization rules
recovery conditions

The executor then answers a different question:

How can I satisfy this specification using the tools available to me?

That's a cleaner separation.

A Small Deployment Example

Suppose we want to deploy version v42 of the Orders service.

The operational objective is:

Deploy Orders v42 to production.

But that's not enough.

We also know:

the payment dependency must be healthy
at least 3 replicas must remain available
error rate must remain below 1%
p95 latency must remain below 400 ms
production deployment requires approval
rollback must remain possible

We can represent those concerns separately.

type DeploymentIntent = {
  service: string;
  version: string;
  environment: string;
  preconditions: {
    paymentDependencyHealthy: boolean;
    approved: boolean;
  };
  constraints: {
    minAvailableReplicas: number;
    maxErrorRate: number;
    maxP95LatencyMs: number;
  };
  recovery: {
    rollbackOnViolation: boolean;
  };
};

Then:

const intent: DeploymentIntent = {
  service: "orders",
  version: "v42",
  environment: "production",
  preconditions: {
    paymentDependencyHealthy: true,
    approved: true,
  },
  constraints: {
    minAvailableReplicas: 3,
    maxErrorRate: 0.01,
    maxP95LatencyMs: 400,
  },
  recovery: {
    rollbackOnViolation: true,
  },
};

Notice what's missing. There's no:

kubectl
Terraform
GitHub Actions
Argo
cloud API

The intent doesn't yet care how the deployment happens.

That's deliberate.

Separate Preconditions from Execution Steps

A precondition answers:

Is this operation allowed to start?

That's different from:

Which step should run first?

For example:

payment dependency must be healthy
production approval must exist
artifact must be signed
maintenance window must be open

These aren't execution instructions. They're conditions that should be true before execution begins.

A simple evaluator might look like this:

type Preconditions = {
  paymentDependencyHealthy: boolean;
  approved: boolean;
};

function canStart(
  preconditions: Preconditions
): boolean {
  return (
    preconditions.paymentDependencyHealthy &&
    preconditions.approved
  );
}

If:

canStart({
  paymentDependencyHealthy: false,
  approved: true,
});

returns:

false

the operation shouldn't begin.

That decision shouldn't depend on whether the executor is Kubernetes, a script, or an AI agent. The precondition belongs to the operational intent.

Separate Constraints from the Mechanism

Constraints answer what must remain true while or after this operation runs.

Examples:

replicas >= 3
error rate <= 1%
latency <= 400 ms
cost increase <= 20%
region must remain us-east-1

An executor may use many different strategies to satisfy them.

For example, to maintain three replicas:

Kubernetes may use a rolling deployment
another platform may use blue/green deployment
an agent may temporarily scale capacity before switching traffic

The mechanism differs, but the constraint does not.

That's exactly why the constraint should live outside the executor.

Make Evidence Requirements Explicit

A constraint is only useful if you can evaluate it.

Suppose you define:

error rate <= 1%

Then you need a source of evidence.

For example:

Prometheus
Datadog
CloudWatch
OpenTelemetry
custom application metrics

The specification shouldn't necessarily hard-code one vendor. But it should say that evidence is required.

For example:

type EvidenceRequirement = {
  name: string;
  metric: string;
};

const evidenceRequirements:
  EvidenceRequirement[] = [
    {
      name: "availability",
      metric: "availableReplicas",
    },
    {
      name: "errors",
      metric: "errorRate",
    },
    {
      name: "latency",
      metric: "p95LatencyMs",
    },
  ];

The executor or evidence collector can decide where those values come from. The specification defines what must be known. That distinction is important.

Define Recovery Before Execution Starts

Many operational workflows define rollback only after something goes wrong. But that's too late. Recovery is part of operational intent.

For example:

If error rate exceeds 1%, stop rollout.

If p95 latency exceeds 400 ms, restore previous version.

If evidence is unavailable, require human review.

These aren't just implementation details. They express acceptable operational behavior.

A simple representation might be:

type RecoveryPolicy = {
  rollbackOnConstraintViolation: boolean;
  requireHumanReviewWhenEvidenceMissing:
    boolean;
};

The executor can implement rollback differently. The policy still belongs to the operation.

Operational Specification vs Execution Plan

This distinction is important.

An operational specification answers:

What should happen?

What must be true?

What evidence is required?

What happens if constraints fail?

An execution plan answers:

Which actions will we perform?

In what order?

Using which tools?

For example:

Specification

Deploy Orders v42.

Keep at least 3 replicas available.

Error rate <= 1%.

Rollback if the constraint is violated.

Execution plan

1. Scale Orders to 6 replicas.
2. Deploy v42 to 1 replica.
3. Route 10% of traffic.
4. Wait 5 minutes.
5. Check metrics.
6. Increase traffic.
7. Remove old replicas.

The plan is one possible way to satisfy the specification.

Another executor might produce a different plan. That's useful. It means the same operational intent can survive changes in tooling and execution strategy.

Why This Matters for AI Agents

This separation becomes especially important when the executor is an AI agent.

A traditional script usually has a predictable path while an agent may decide dynamically.

For the same incident, it might choose:

restart
scale
reroute
disable feature
change configuration

If correctness is defined by following one exact sequence, dynamic execution becomes difficult to verify.

But if correctness is defined by an operational specification, the agent has flexibility.

For example:

Objective:
restore checkout availability

Preconditions:
payment database reachable

Constraints:
authentication remains enabled
no payment duplication
error rate <= 1%
cost increase <= 20%

Evidence:
checkout success rate
duplicate payment count
error rate
cost delta

Recovery:
escalate if constraints cannot be satisfied

The agent can choose different actions, but its behavior is still bounded.

That gives us:

Intent
↓
Specification
↓
Agent-generated plan
↓
Execution
↓
Evidence
↓
Evaluation

The agent is free to decide how. It's not free to redefine what success means.

That distinction will become increasingly important as autonomous systems gain more operational authority.

A Minimal TypeScript Model

We can combine these ideas into a small generic model.

type OperationalSpec<TContext, TEvidence> = {
  name: string;

  canStart(
    context: TContext
  ): boolean;

  evaluate(
    evidence: TEvidence
  ): EvaluationResult;

  recovery: RecoveryPolicy;
};

type EvaluationResult = {
  conformant: boolean;
  failures: string[];
};

type RecoveryPolicy = {
  rollbackOnConstraintViolation: boolean;
};

For a deployment:

type DeploymentContext = {
  paymentDependencyHealthy: boolean;
  approved: boolean;
};

type DeploymentEvidence = {
  version: string;
  replicas: number;
  errorRate: number;
  p95LatencyMs: number;
};

Then:

const deploymentSpec:
  OperationalSpec<
    DeploymentContext,
    DeploymentEvidence
  > = {
    name: "deploy-orders-v42",

    canStart: (context) =>
      context.paymentDependencyHealthy &&
      context.approved,

    evaluate: (evidence) => {
      const failures: string[] = [];

      if (evidence.version !== "v42") {
        failures.push(
          "wrong-version"
        );
      }

      if (evidence.replicas < 3) {
        failures.push(
          "insufficient-replicas"
        );
      }

      if (evidence.errorRate > 0.01) {
        failures.push(
          "error-rate-too-high"
        );
      }

      if (
        evidence.p95LatencyMs > 400
      ) {
        failures.push(
          "latency-too-high"
        );
      }

      return {
        conformant:
          failures.length === 0,
        failures,
      };
    },

    recovery: {
      rollbackOnConstraintViolation:
        true,
    },
  };

Now execution can be handled elsewhere.

For example:

interface DeploymentExecutor {
  execute(
    spec: typeof deploymentSpec
  ): Promise<DeploymentEvidence>;
}

That executor could be backed by:

Kubernetes
a cloud platform
a test harness
a local simulator
an AI agent

The specification remains the reference point.

A Practical Workflow

If you want to introduce this separation into an existing automation system, start with one operation.

1. Pick a Repeated Operation

For example:

deploy service
rotate certificate
scale worker pool
restore backup

2. Extract the Objective

Write down what the operation is trying to achieve. Avoid commands.

3. Extract Preconditions

Ask:

What must already be true before this operation can begin?

4. Extract Constraints

Ask:

What must remain true during or after execution?

5. Define Evidence

For every constraint, identify the observation required to evaluate it.

6. Define Recovery

Decide what should happen when evidence shows that a constraint was violated.

7. Leave Execution Separate

Let the current automation tool remain the executor. Don't rewrite everything at once.

8. Evaluate the Existing Executor

Ask whether the current script, pipeline, or agent satisfies the extracted specification.

This is useful because the separation can be introduced incrementally. You don't need a new operations platform to begin.

What This Separation Doesn't Solve

Separating intent from execution doesn't automatically make operations correct.

You can still have:

wrong objectives
bad thresholds
missing evidence
misleading metrics
conflicting constraints
unsafe executors
security bugs
incorrect recovery policies

The specification itself can be wrong. That matters.

A formal representation doesn't make an assumption true. It only makes the assumption easier to inspect and evaluate.

There's also a cost. More explicit operational intent means more artifacts to maintain.

If specifications aren't versioned and reviewed, they can become stale. If evidence definitions are weak, conformance results can become misleading. And some operational decisions remain difficult to formalize.

Human judgment doesn't disappear. The value comes from making more of that judgment explicit.

From Automation Logic to Operational Contracts

There's another way to think about this separation.

A mature operational specification starts to behave like a contract.

It says:

This operation may begin under these conditions.

It must preserve these constraints.

It must produce this evidence.

Its outcome will be evaluated using these rules.

If it fails, these recovery conditions apply.

That's much stronger than:

run this workflow

The workflow is an implementation. The operational contract describes what that implementation is responsible for achieving.

This gives us a structure like:

Intent
↓
Operational contract
↓
Execution plan
↓
Executor
↓
Evidence
↓
Evaluation

That separation creates room for something important:

multiple execution strategies
for the same operational intent

And once that's possible, another question appears:

Can two different executors satisfy the same operational specification?

That's the question I want to explore next.

Conclusion

Automation has made software operations faster and more repeatable.

But execution is only one part of an operation. We still need to know:

why the operation is allowed
what outcome is intended
what constraints must hold
what evidence is required
what happens when the constraints fail

When all of that logic lives inside a pipeline, script, or agent, the executor becomes the accidental owner of operational meaning.

That coupling makes systems harder to review, migrate, audit, and verify.

A better separation is:

Intent
↓
Operational specification
↓
Execution plan
↓
Executor
↓
Evidence
↓
Evaluation

The specification describes what success means. The executor decides how to achieve it.

That distinction becomes increasingly valuable as execution mechanisms become more dynamic. Especially when those executors are AI agents.

Because the more freedom we give a system to decide how work gets done, the more explicit we need to be about what the work is supposed to achieve.

]]> <![CDATA[ How Executable Operational Specifications Can Make Software Automation Verifiable ]]> https://www.freecodecamp.org/news/executable-operational-specifications-software-automation/ 6ab54b145ca9fc6a257ae996 Thu, 24 Sep 2026 16:08:52 +0000 Modern software systems are increasingly automated. We automate deployments, infrastructure changes, scaling, incident response, and data pipelines.

And now, with AI agents, we are starting to automate operational decisions too. That sounds like progress, but it creates a problem that is easy to miss:

We are getting better at executing operations without necessarily getting better at specifying what those operations are supposed to achieve.

A deployment pipeline can run successfully and still violate an important business constraint.

An infrastructure script can finish without error and still leave the system in the wrong state.

An AI agent can complete a sequence of actions and still produce an outcome that nobody can verify afterward.

In many systems, operational intent still lives across:

runbooks
tickets
Slack messages
CI/CD configuration
Terraform files
dashboards
monitoring rules
human memory

Those artifacts are useful. But they are not the same thing as an executable operational specification.

An executable operational specification describes what should happen in a way that can later be evaluated against what actually happened.

That distinction becomes increasingly important as execution gets faster, more distributed, and more autonomous.

In this article, I'll explore:

  • why automation alone is not enough,

  • the difference between execution logic and operational intent,

  • what an executable operational specification is,

  • why observability does not solve this problem by itself,

  • how specifications can make operational behavior verifiable,

  • how this applies to CI/CD, infrastructure, incident response, and AI agents,

  • what a minimal specification might look like,

  • and what problems this approach still does not solve.

The central idea is simple:

If a system can execute an operation automatically, we should also be able to describe what successful execution means independently of the tool performing it.

Prerequisites

You should be comfortable with:

  • software architecture

  • CI/CD

  • infrastructure automation

  • observability

  • distributed systems

  • basic testing concepts

  • automation workflows

You do not need to use any particular cloud provider, deployment platform, or orchestration tool.

The ideas in this article are deliberately tool-independent.

Table of Contents

Automation Solves Execution, Not Intent

Consider a deployment pipeline. It might:

build the application
run tests
build an image
push the image
deploy it
wait for readiness
mark the pipeline as successful

If every step completes, the pipeline turns green. But what did the pipeline actually prove?

Usually, something like:

the configured steps completed successfully

That is useful. But it is not necessarily the same as:

the intended operational outcome was achieved

Suppose the deployment succeeds technically, but:

the wrong image version was deployed
only two replicas are running instead of three
the error rate increased
a required feature flag is disabled
the service is healthy but cannot reach a dependency
the rollout violated a regional constraint

The executor did its job. The operation still failed in a broader sense. This is the gap between execution success and operational conformance.

Automation tells us:

The steps ran.

What we often need to know is:

Did the resulting system satisfy the intended conditions?

Those are different questions.

Operational Knowledge Is Usually Fragmented

Most production systems already contain operational knowledge. The problem is that it is scattered.

For example, a deployment rule might exist partly in:

GitHub Actions
Terraform
Kubernetes manifests
Grafana dashboards
PagerDuty alerts
a runbook
a ticket
a senior engineer's memory

One artifact might define how many replicas should exist. Another might define what error rate is acceptable. A third might explain when rollback is required. A fourth might describe which regions are allowed.

No single representation says:

This is the operation we intend to perform.

These are its constraints.

This is the evidence we require.

This is how we determine whether it succeeded.

That makes operations harder to reason about. It also makes automation brittle.

When the rules are distributed across tools, the executor often becomes the de facto specification.

And once that happens, it becomes difficult to ask whether the executor behaved correctly.

The logic that performs the action and the logic that defines success are effectively the same thing.

Execution Logic and Operational Intent Are Different Things

Imagine a deployment script:

kubectl set image \
  deployment/orders \
  orders=registry.example.com/orders:2026.09.19

This tells Kubernetes how to perform an action. It does not fully describe why the action is acceptable.

The operational intent might be closer to:

Deploy version 2026.09.19 of the Orders service.

Constraints:

- production must keep at least 3 available replicas
- the error rate must remain below 1%
- p95 latency must stay below 400 ms
- deployment is allowed only in us-east-1
- rollback must remain possible for 30 minutes

The command and the intent are related. But they are not the same artifact.

This distinction matters because several different executors might be able to satisfy the same intent.

For example:

Kubernetes
Nomad
a cloud deployment service
a custom deployment controller
an AI-operated platform

If the operational goal is expressed independently, the executor becomes replaceable.

If the goal is embedded inside executor-specific code, replacing the executor may also mean rediscovering the intent.

What Is an Executable Operational Specification?

An executable operational specification is a machine-readable description of an operational objective that can be evaluated against observed evidence.

At a minimum, it should answer questions like:

What are we trying to achieve?

What constraints must hold?

What evidence should be collected?

How do we determine whether the operation conforms?

For example:

operation: deploy-orders-service

target:
  service: orders
  version: 2026.09.19
  environment: production

constraints:
  minAvailableReplicas: 3
  maxErrorRate: 0.01
  maxP95LatencyMs: 400
  region: us-east-1

evidence:
  - deployedVersion
  - availableReplicas
  - errorRate
  - p95Latency
  - region

This is intentionally simple. The important part is not the YAML syntax.

The important part is that the specification defines success independently from the mechanism used to perform the deployment.

That gives us a structure like:

Operational intent
        ↓
Specification
        ↓
Executor
        ↓
Execution
        ↓
Evidence
        ↓
Evaluation

The specification becomes a stable point of reference.

A Simple Deployment Example

Suppose we want to deploy version 2026.09.19 of an Orders service.

The operation succeeds only if:

the correct version is running
at least three replicas are available
error rate stays below 1%
p95 latency stays below 400 ms

We can express that as:

type DeploymentSpec = {
  service: string;
  version: string;
  minAvailableReplicas: number;
  maxErrorRate: number;
  maxP95LatencyMs: number;
};

For example:

const spec: DeploymentSpec = {
  service: "orders",
  version: "2026.09.19",
  minAvailableReplicas: 3,
  maxErrorRate: 0.01,
  maxP95LatencyMs: 400,
};

Now suppose execution produces evidence:

type DeploymentEvidence = {
  deployedVersion: string;
  availableReplicas: number;
  errorRate: number;
  p95LatencyMs: number;
};

For example:

const evidence: DeploymentEvidence = {
  deployedVersion: "2026.09.19",
  availableReplicas: 3,
  errorRate: 0.004,
  p95LatencyMs: 280,
};

We can evaluate the evidence against the specification:

function conforms(
  spec: DeploymentSpec,
  evidence: DeploymentEvidence
): boolean {
  return (
    evidence.deployedVersion ===
      spec.version &&
    evidence.availableReplicas >=
      spec.minAvailableReplicas &&
    evidence.errorRate <=
      spec.maxErrorRate &&
    evidence.p95LatencyMs <=
      spec.maxP95LatencyMs
  );
}

Then:

console.log(
  conforms(spec, evidence)
);

returns:

true

Now imagine the pipeline technically succeeds but only two replicas remain available:

const evidence: DeploymentEvidence = {
  deployedVersion: "2026.09.19",
  availableReplicas: 2,
  errorRate: 0.004,
  p95LatencyMs: 280,
};

The executor may still report success.

The specification does not:

conforms → false

That is the distinction we want.

Turn Success Criteria into Evidence Requirements

A specification becomes useful when its claims can be evaluated.

Suppose the specification says:

error rate must remain below 1%

Then the system needs evidence for:

error rate

If it says:

at least 3 replicas must remain available

then it needs evidence for:

available replica count

This sounds obvious, but it introduces an important discipline:

Every operational requirement should imply some form of observable evidence.

For example:

Requirement Evidence
correct version deployed running image/version
minimum replicas available replica count
error rate below threshold request/error metrics
latency below threshold latency metrics
correct region runtime placement
no schema regression schema validation result

This relationship matters because vague operational goals are difficult to automate safely.

Consider:

Deploy safely.

What evidence proves that?

The statement is too ambiguous.

A better specification decomposes "safely" into conditions that can be checked.

Why Observability Alone Is Not Enough

At this point you might ask:

Isn't this just observability?

Not exactly. Observability helps answer:

What is happening?

A specification helps answer:

What should be happening?

Those are complementary questions.

A dashboard might tell you:

error rate = 1.4%

That is an observation.

But whether 1.4% is acceptable depends on an expected condition.

The specification might say:

maxErrorRate = 1%

Now you can evaluate:

observed: 1.4%
expected: <= 1%

result: non-conformant

Without the expected condition, the metric is just a number. Without the metric, the specification cannot be verified. You need both.

Conceptually:

Specification
     +
Observed evidence
     ↓
Evaluation

Specifications Make Automation Verifiable

Automation without an independent specification is difficult to verify.

Consider a script that performs:

scale service
restart pods
change routing
wait
finish

If the script is also the only place where expected outcomes are encoded, then asking whether it behaved correctly becomes circular.

You are effectively asking:

Did the automation do what the automation says it should do?

A separate specification gives you another reference point.

Now you can ask:

What was intended?

What did the executor do?

What evidence did execution produce?

Did the evidence satisfy the specification?

This makes operations easier to audit and test. It also makes failures more informative.

Instead of:

deployment failed

you can potentially say:

deployment execution completed

but specification failed because:

availableReplicas:
expected >= 3
observed = 2

That is much more useful.

Keep the Specification Independent from the Executor

One of the strongest properties of this model is executor independence.

Suppose the specification says:

Deploy Orders version 2026.09.19

Keep:
- >= 3 replicas
- error rate <= 1%
- p95 latency <= 400 ms

One executor might use Kubernetes. Another might use a managed cloud platform. Another might use a custom orchestrator.

The specification should not need to change simply because the executor changed.

Conceptually:

                 ┌── Kubernetes Executor
Specification ───┼── Cloud Executor
                 ├── Custom Executor
                 └── AI Agent

Each executor produces evidence. Each execution is evaluated against the same intent.

That gives you a useful separation:

what should happen

from:

how it happens

This separation is common in other areas of software engineering. Interfaces separate callers from implementations. SQL separates queries from storage mechanics. Desired-state systems separate target state from reconciliation logic.

Operational specifications apply a similar idea to operational workflows.

A Minimal TypeScript Model

A simple generic model might look like this:

type Constraint<T> = {
  name: string;
  evaluate(
    evidence: T
  ): boolean;
};

type OperationalSpec<T> = {
  name: string;
  constraints: Constraint<T>[];
};

For deployment evidence:

type Evidence = {
  version: string;
  replicas: number;
  errorRate: number;
};

You can define:

const deploymentSpec:
  OperationalSpec<Evidence> = {
    name: "deploy-orders",
    constraints: [
      {
        name: "correct-version",
        evaluate: (evidence) =>
          evidence.version ===
          "2026.09.19",
      },
      {
        name: "minimum-replicas",
        evaluate: (evidence) =>
          evidence.replicas >= 3,
      },
      {
        name: "error-rate",
        evaluate: (evidence) =>
          evidence.errorRate <= 0.01,
      },
    ],
  };

Then evaluate every constraint:

function evaluate<T>(
  spec: OperationalSpec<T>,
  evidence: T
) {
  return spec.constraints.map(
    (constraint) => ({
      constraint: constraint.name,
      passed:
        constraint.evaluate(
          evidence
        ),
    })
  );
}

For:

const evidence: Evidence = {
  version: "2026.09.19",
  replicas: 2,
  errorRate: 0.003,
};

you might get:

correct-version    PASS
minimum-replicas   FAIL
error-rate         PASS

That is more useful than a single generic success or failure flag. It tells you exactly which part of the intended operation did not conform.

How This Applies to CI/CD

CI/CD systems already contain some declarative elements.

For example:

steps:
  - test
  - build
  - deploy

But these steps mostly describe execution order. A specification can add operational expectations around them.

For example:

Deployment objective:
release version X

Constraints:
tests passed
artifact digest matches approved build
minimum replicas remain available
error rate stays below threshold
rollback remains possible

The pipeline still performs the work. The specification defines the conditions the pipeline must satisfy. This also makes pipeline replacement easier.

If you move from:

GitHub Actions

to:

GitLab CI

or:

Argo

the executor changes.

The operational objective does not necessarily need to.

How This Applies to Infrastructure

Infrastructure-as-code already gives us desired state. That is closely related to this idea.

For example:

resource "aws_instance" "app" {
  instance_type = "t3.medium"
}

But operational intent often extends beyond configuration state.

You may also care about:

service availability
cost limits
regional restrictions
security controls
capacity
latency
backup freshness

Those constraints may live outside the IaC definition. An operational specification can bring them together.

For example:

Provision application environment

Required:
3 instances
region = us-east-1
monthly projected cost < $500
encryption enabled
backup age < 24h

The executor may use Terraform.

The evidence may come from:

cloud APIs
cost systems
security scanners
backup metadata

The specification gives those sources a common purpose.

How This Applies to Incident Response

Incident response is another place where execution and intent are often mixed together.

A runbook might say:

restart service
clear cache
scale replicas

But the actual operational objective might be:

restore checkout availability

while:
avoiding duplicate payments
preserving order state
keeping error rate below threshold

That difference matters.

If restarting the service does not restore checkout availability, the runbook technically executed but the operation failed.

An executable specification could express recovery conditions:

checkout success rate > 99%
payment duplication = 0
queue backlog < threshold
error rate < 1%

Then incident automation can be evaluated by the result it achieves, not just the actions it performs.

How This Applies to AI Agents

This becomes even more important with AI agents. Traditional automation usually follows predefined execution logic.

An AI agent may choose the execution path dynamically. For example, an operations agent might decide to:

inspect metrics
restart a service
change capacity
modify a feature flag
reroute traffic

The exact sequence may vary from one incident to another. That makes executor-level validation harder.

You cannot always verify the agent by checking whether it followed one exact script. But you can still verify operational intent.

For example:

Objective:
restore API availability

Constraints:
do not disable authentication
do not lose queued requests
error rate < 1%
p95 latency < 500 ms
cost increase < 20%

The agent may choose different actions and the specification remains stable.

That creates a useful control structure:

Human / organizational intent
        ↓
Operational specification
        ↓
Agent
        ↓
Actions
        ↓
Evidence
        ↓
Evaluation

The more autonomous execution becomes, the more valuable this separation becomes.

What Should Go Into an Operational Specification?

A useful specification often includes several categories.

Objective

What should be achieved?

For example:

Deploy Orders service version 2026.09.19

Scope

Where does the operation apply?

For example:

environment: production
region: us-east-1
service: orders

Constraints

What must remain true?

For example:

available replicas >= 3
error rate <= 1%
p95 latency <= 400 ms

Required Evidence

What must be observed?

For example:

running version
replica count
error rate
latency

Evaluation Rules

How do we decide whether execution conforms?

For example:

version must match exactly
replicas must be >= 3
error rate must be <= 0.01

Recovery Conditions

What should happen if conformance fails?

For example:

stop rollout
restore previous route
require human approval

Not every specification needs all of these.

But separating them makes operational intent much clearer.

What Should Stay Out of the Specification?

A specification should not become another implementation script. That means avoiding unnecessary executor-specific mechanics.

For example, this is probably too implementation-specific:

run kubectl command X
wait 10 seconds
call endpoint Y
run shell command Z

Those belong in an executor. The specification should focus on the desired operational outcome.

For example:

service version = 2026.09.19
available replicas >= 3
health checks passing

A useful rule is:

If changing the execution tool forces you to rewrite the specification, the specification may contain too much implementation detail.

Some executor-specific constraints are unavoidable. But the default should be to keep intent and mechanism separate.

A Practical Workflow

If I were introducing executable operational specifications into an existing system, I would start small.

1. Pick One Important Operation

For example:

deploy service
rotate certificate
restore backup
scale worker pool

2. Write Down the Objective

Ask:

What does success actually mean?

Not:

Which commands do we run?

3. Identify Constraints

For example:

minimum availability
maximum error rate
security requirements
regional restrictions
cost limits

4. Identify Evidence

For each constraint, ask:

What observation would prove or disprove this condition?

5. Separate the Executor

Keep the mechanism that performs the work independent from the specification.

6. Evaluate After Execution

Collect evidence and compare it against the specification.

7. Report Conformance

Prefer:

3 constraints passed
1 constraint failed

over:

operation failed

8. Improve the Specification

Missing evidence and ambiguous constraints will become visible quickly.

That is useful.

The specification becomes better as operational knowledge becomes explicit.

What Executable Specifications Do Not Solve

Executable specifications are not a complete operations architecture.

They do not automatically solve:

bad requirements
incorrect metrics
missing observability
distributed transactions
security failures
poor executor implementations
organizational ownership
conflicting business goals

They also introduce their own risks:

  • A bad specification can encode the wrong objective.

  • An incomplete specification can create false confidence.

  • A stale specification can become another source of drift.

  • And not every operational decision can be reduced to a simple threshold.

Human judgment still matters. The goal is not to eliminate judgment. The goal is to make operational intent more explicit and more testable.

From Automated Operations to Verifiable Operations

Software operations have spent years becoming more automated. That trend will continue. But increasing automation creates a new question:

How do we know the automation achieved the right outcome?

Execution logs, pipeline success, and agent confidence are not enough. We need something to compare execution against. That is where an executable operational specification becomes useful.

It gives us:

intent
↓
constraints
↓
evidence requirements
↓
execution
↓
observed evidence
↓
evaluation

That structure turns an operation from:

something happened

into:

something happened,
we know what was expected,
we collected evidence,
and we can evaluate the result.

That is a much stronger foundation for automation.

Conclusion

The more software operations we automate, the more important it becomes to separate what we want from how a tool executes it.

Pipelines are executors.

Infrastructure tools, scripts, and AI agents are executors. They can all perform actions.

But the operational objective should exist independently from the mechanism carrying it out.

An executable operational specification gives us a way to describe that objective in terms of:

desired outcome
constraints
required evidence
evaluation rules

Then execution becomes something we can verify instead of merely observe.

This matters today for deployments, infrastructure, and incident response.

It will matter even more as operational systems become increasingly autonomous.

Because automation can tell us that an action was executed.

What we really need to know is whether the system ended up where it was supposed to be.

]]> <![CDATA[ How to Turn a RECIST Line into a 3D Tumor Segmentation Mask ]]> https://www.freecodecamp.org/news/how-to-turn-a-recist-line-into-a-3d-tumor-segmentation-mask/ 6aadae08b3c208add9878388 Fri, 18 Sep 2026 21:32:56 +0000 A radiologist can mark a tumor on a CT scan by drawing a straight line across it. Creating a complete 3D segmentation requires outlining the tumor across the slices where it appears, which is more time-consuming.

This tutorial explains how Lumina works. It's a system that uses a CT scan and one RECIST line to produce a 3D segmentation mask of the marked tumor.

Lumina was developed for the FLARE 2026 pan-cancer segmentation challenge. The system was designed to run on a CPU with an 8 GB memory limit and a 60-second inference limit.

This tutorial covers the main design decisions, implementation details, and experiments that shaped the final system.

What We'll Cover:

What Lumina Does

A radiologist can measure a tumor by drawing a line across its longest visible diameter. This is a RECIST measurement (Response Evaluation Criteria in Solid Tumors), which is widely used to measure tumor response during cancer treatment.

A RECIST line measures a 2D diameter marker. It doesn't describe the tumor's full 3D shape.

Axial CT image of the abdomen with a green diameter marker placed across a liver lesion.

Lumina uses this measurement to prompt a 3D segmentation model.

The task can be described as:

Input: A 3D CT scan and a 2D RECIST line marking one tumor.

Output: A 3D mask of that tumor, with one label for each voxel.

A voxel is the 3D equivalent of a pixel. Each voxel represents a small volume of tissue in the CT scan.

The RECIST line tells the model which tumor to segment. The model then predicts the tumor's three-dimensional extent.

This makes the segmentation problem more specific because the model doesn't need to identify every possible tumor in the scan.

Lumina Pipeline Overview

The complete process has several stages. We start with a 3D CT scan and a RECIST line marking one tumor. We then convert the line into additional input channels, crop and resample the image, and pass the three channels through a 3D segmentation network.

The network produces a probability map, which we convert into the final 3D tumor mask through resampling, thresholding, and connected-component selection.

Diagram showing the Lumina pipeline from a CT scan and RECIST line to a 3D tumor segmentation mask.

The main stages are:

  1. Prepare the CT scan and RECIST marker.

  2. Encode the RECIST line as additional network input.

  3. Crop and resample the image to a common grid.

  4. Predict the tumor probability map with a 3D segmentation network.

  5. Refine the probability map and convert it into a binary mask.

  6. Control CPU inference so the full case stays within runtime and memory limits.

Prerequisites

You'll get more from this tutorial if you have some experience with:

  • Python

  • NumPy

  • PyTorch

  • Basic convolutional neural networks.

No detailed medical background is required. The tutorial explains medical imaging concepts as it introduces them.

The implementation uses NumPy, SciPy, PyTorch, and MONAI, a medical imaging framework built on PyTorch.

Step 1: Review Your Data Before Coding

Our data was provided as .npz files, which is NumPy's compressed array format.

Each file contains fields similar to these:

imgs       # the CT scan, a 3D array
recist     # the marker lines, same shape, one integer per tumour
spacing    # how many millimetres apart the voxels are
origin     # where the scan sits in the scanner's coordinates
direction  # how the scan is rotated
gts        # the ground-truth segmentation, training files only

Before building the model, we inspected the image values, array shapes, and spatial metadata.

Two details were especially significant.

The Scans Were Already Brightness-Adjusted

CT scanners normally store images in Hounsfield units. Water is about 0 HU, while bone can exceed 1000 HU.

In this dataset, the scans had already been converted to a fixed 0–255 range. The original Hounsfield-unit values weren't available.

This meant that we couldn't apply the usual CT windowing process to the original values. Instead, we measured the intensity distribution of the data provided.

For example, 53.4% of the voxels had a value of exactly 0, corresponding to air outside the body.

Coordinate Order Matters

The spacing array is stored as (X, Y, Z), while NumPy arrays are indexed as (Z, Y, X).

If you mix these two conventions, spatial measurements can be incorrect.

We converted the coordinates once at the input boundary and used the (Z, Y, X) convention internally:

def _to_zyx(vec3, order):
    vec3 = np.asarray(vec3, dtype=float).ravel()
    if order == "xyz":
        return vec3[::-1].copy()      # (x, y, z) -> (z, y, x)
    if order == "zyx":
        return vec3.copy()
    raise ValueError(f"unknown geometry_order {order!r}")

The important lesson is to inspect several files before designing the preprocessing pipeline.

Check:

  • Array shapes

  • Intensity ranges

  • Voxel spacing

  • Coordinate conventions

  • Metadata

  • Available labels

Avoid assuming the data follows conventions from another CT dataset or tutorial.

Step 2: Turn the RECIST Line into Network Input

A neural network receives a stack of input channels. The CT scan provides the first channel. We then represent the RECIST line in a form the network can use.

We use three channels:

CT image and the two RECIST prompt channels used as input to Lumina: the RECIST line and endpoint heatmap.

  • CT scan

  • RECIST line, drawn 3 voxels thick

  • Endpoint heatmap, containing a Gaussian around each endpoint

The two endpoints define the measured diameter and provide the network with the location and length of the RECIST measurement.

The endpoint information is represented using Gaussian functions. Each Gaussian has a high value near the endpoint and gradually decreases with distance.

def _endpoint_heatmap(endpoints_zyx, shape, sigma):
    d, h, w = (int(s) for s in shape)
    heat = np.zeros((d, h, w), dtype=np.float32)

    rad = max(1, int(np.ceil(3 * sigma)))
    two_s2 = 2.0 * sigma * sigma

    for z0, y0, x0 in np.asarray(endpoints_zyx, float):
        zc, yc, xc = int(round(z0)), int(round(y0)), int(round(x0)))

        zl, zr = max(0, zc - rad), min(d, zc + rad + 1)
        yl, yr = max(0, yc - rad), min(h, yc + rad + 1)
        xl, xr = max(0, xc - rad), min(w, xc + rad + 1)

        zz, yy, xx = np.mgrid[
            zl:zr, yl:yr, xl:xr
        ].astype(np.float32)

        g = np.exp(
            -((zz-z0)**2 + (yy-y0)**2 + (xx-x0)**2) / two_s2
        )

        np.maximum(
            heat[zl:zr, yl:yr, xl:xr],
            g,
            out=heat[zl:zr, yl:yr, xl:xr]
        )

    return heat

The Gaussian is calculated only in a small region around each endpoint.

With sigma = 1.5, values become very small several voxels away from the endpoint. Calculating the Gaussian over the complete volume would therefore perform unnecessary work.

Draw the Line at the Required Resolution

We also redraw the RECIST line from its two endpoints at the resolution the network requires.

We don't create the line at one resolution and then resize it with the image. Resizing a thin line can make it thinner or cause it to disconnect. Recreating the line from its endpoints keeps the prompt consistent with the image resolution.

Step 3: Align All Tumors on a Common Grid

CT scans can have different voxel spacings.

For example, one scan may have thin slices while another may have much thicker slices. The physical size represented by the same number of voxels can therefore vary between scans.

For each marked tumor, we'll create a fixed-size crop and resample it to a fixed voxel spacing.

Our configuration is:

target_spacing: [2.5, 1.0, 1.0]     # mm per voxel: z, y, x
crop_size:      [64, 160, 160]      # voxels: z, y, x

The in-plane dimensions correspond to a physical field of view of:

160 × 160 mm

The network therefore receives a consistent 64 × 160 × 160 input.

We use a target spacing derived from the training data. The 2.5 mm slice spacing was close to the median slice spacing in the training set.

We use a 160 mm crop based on the distribution of lesion sizes in the training data. It covers the 99th percentile of lesion extent.

Intensity Normalization

We also normalize the CT intensity values using statistics calculated from the training crops:

intensity_mean: 96.88
intensity_std: 79.00

The normalization is:

image = (image - 96.88) / 79.00

We calculate these statistics using only the training data.

We use the same values for validation and test data. We don't recalculate these statistics from the validation or test sets, because that would allow information from those sets to influence preprocessing.

For Lumina, the training-data statistics are a mean of 96.88 and a standard deviation of 79.00. We store these values in the configuration and use the same preprocessing during training, validation, and inference.

Step 4: Build the 3D Segmentation Network

Here, we'll use DynUNet from MONAI.

DynUNet is a 3D U-Net architecture based on ideas from nnU-Net and supports configurable network depth, kernels, strides, and residual connections.

Simplified DynUNet architecture showing the encoder, decoder, bottleneck, and skip connections used to preserve spatial information.

A U-Net has two main parts.

  • The encoder gradually reduces the spatial resolution while learning increasingly high-level features.

  • The decoder then restores the spatial resolution to produce a segmentation map.

Skip connections pass fine spatial details from the encoder to the decoder. This helps the decoder recover the tumor's boundary when it creates the final segmentation.

For Lumina, the network receives three channels:

  • CT

  • RECIST line

  • Endpoint heatmap.

The model configuration is:

from monai.networks.nets import DynUNet

# 5 levels; the first downsample skips z (see below).
kernels = [[3, 3, 3]] * 5
strides = [[1, 1, 1], [1, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]]

model = DynUNet(
    spatial_dims=3,
    in_channels=3,          # CT + line + endpoint blobs
    out_channels=1,         # one probability per voxel
    kernel_size=kernels,
    strides=strides,
    upsample_kernel_size=strides[1:],
    filters=features,
    res_block=True,
)

The strides list is an important part of the configuration. Our voxels have different sizes: they're 2.5 mm apart between slices but only 1.0 mm apart within each slice. If we reduce all three dimensions at every level, we lose inter-slice detail too quickly. We'll therefore keep the z dimension unchanged during the first downsampling step. This helps preserve useful information in all three directions.

Lumina's main design work focuses on input representation, preprocessing, training strategy, post-processing, and CPU inference behavior.

Step 5: Use a Loss Function That Includes Boundary Information

During training, the network needs a way to measure how different its prediction is from the ground-truth segmentation. This measurement is called a loss function. The network adjusts its weights to reduce this loss.

For Lumina, we combine two common losses: Dice loss and binary cross-entropy (BCE).

Dice Loss

Dice loss focuses on the overlap between the predicted tumor region and the ground-truth region.

It's useful when we want the overall size and shape of the predicted region to match the reference segmentation.

Binary Cross-entropy

Binary cross-entropy works at the voxel level. For every voxel, the network predicts a probability between 0 and 1.

The ground-truth value is:

  • 1 if the voxel belongs to the tumor

  • 0 if the voxel belongs to the background.

BCE penalizes the network when its predicted probability differs from the ground-truth value.

For example, if a tumor voxel has a predicted probability close to1, the penalty is small. If the network confidently predicts a tumor voxel as background, the penalty is larger.

We combine the two losses by adding them:

loss = dice_loss + bce_loss

The two terms provide different training signals:

  • Dice loss: encourages good overall region overlap.

  • BCE: encourages correct individual voxel predictions.

Adding Boundary Information

Dice and BCE don't give the lesion boundary any special treatment.

This matters because a segmentation can have good overall overlap while still having an inaccurate boundary.

Lumina is evaluated using both:

  • Dice, which measures region overlap

  • NSD (Normalized Surface Dice), which measures surface agreement within a specified distance

The official challenge tolerance is 1 mm.

To give the network additional information about the boundary, we add a boundary-band loss term.

We create a thin band around the ground-truth surface by expanding and shrinking the target mask:

shell = dilate(target) & ~erode(target)

loss = (
    dice_loss
    + bce_loss
    + 0.5 * bce_loss_on(shell)
)

The three terms therefore have different roles:

  • Dice loss: overall region overlap

  • BCE: voxel-level prediction accuracy

  • Boundary-band loss: extra attention to voxels near the lesion surface.

The additional boundary term is weighted by 0.5.

Comparing Boundary Losses

MONAI also provides HausdorffDTLoss, which uses a distance-transform-based formulation.

We can compare the computational cost of the different approaches during training:

Loss Time per step
Dice + cross-entropy 7 ms
Ours (boundary band) 203 ms
MONAI HausdorffDTLoss 957 ms

The fast path for the Hausdorff distance loss required the cupy library, which couldn't be built in our training environment. Its CPU implementation was considerably more expensive.

Our boundary-band approach uses morphological operations based on max-pooling and has a lower computational cost.

The boundary term improved Dice on large tumors by 0.0136. On small tumors, the results were mixed: 15 cases improved and 13 worsened.

Step 6: Transform the Probability Map into a Segmentation Mask

The segmentation network produces a probability map. Each voxel in this map contains a value between 0 and 1, representing how likely that voxel is to belong to the tumor. We can now convert this probability map into the final 3D segmentation mask. There are three steps:

  1. Resample the probability map back to the original CT grid.

  2. Apply a threshold to create a binary mask.

  3. Keep the connected component associated with the RECIST line.

Resample the Probability Map

During preprocessing, we crop and resample the CT image to a fixed size of 64 × 160 × 160 voxels. The network therefore produces its prediction on this processed grid.

Before creating the final mask, we resample the probability map back to the original CT image grid.

prob_original = resample_to_original_grid(
    probability_map,
    original_image
)

We do this before thresholding so that the probability values can be interpolated on the original grid. This allows the final boundary to be represented more accurately.

Apply the Threshold

The network produces a probability between 0 and 1 for every voxel. We use a threshold of 0.35 to convert this probability map into a binary segmentation mask.

mask = prob_original >= 0.35

Voxels with a probability of at least 0.35 become part of the predicted tumor, while the remaining voxels are treated as background.

The threshold of 0.35 was selected during Lumina's validation experiments and is part of the final inference pipeline.

Keep the Tumor Connected to the RECIST Line

The thresholded mask can contain small disconnected regions. Some of these regions may not belong to the tumor.

Because we know where the tumor was marked, we use the RECIST line to select the relevant connected component.

components = connected_components(mask)

tumor_mask = select_component(
    components,
    recist_line
)

We retain the component that intersects the RECIST line as the final tumor segmentation.

If the RECIST line doesn't intersect any component after thresholding, we use the component closest to the line midpoint as a fallback.

The order of these operations is important:

Probability map
      ↓
Resample to original CT grid
      ↓
Apply threshold (0.35)
      ↓
Connected-component selection
      ↓
Final 3D tumor mask

Why Do We Resample Before Thresholding?

We resample the probability map before thresholding so that the probability values can be interpolated on the original CT grid. During development, we also tested thresholding before resampling. That approach produced small changes at the lesion boundary because interpolation was applied to an already binary mask.

Using the probability map, preserves more information during interpolation and gives the final segmentation a more precise boundary.

The result is a 3D binary mask aligned with the original CT scan, ready for evaluation or visualization.

Step 7: Accelerate and Ensure Consistency in CPU Inference

The CPU and memory limits influence the inference design.

We use several techniques to keep inference within the challenge constraints.

Pin the Thread Count

CPU operations can produce small numerical differences when calculations run with different thread counts.

Near a segmentation threshold such as 0.35, very small changes in probability can affect whether a voxel is included in the final mask.

We therefore explicitly set the PyTorch thread count:

torch.set_num_threads(8)

Using the same thread configuration across runs helps keep the inference process reproducible.

Budget the Inference Passes

We use an ensemble because combining predictions from several model passes improves the segmentation results.

The ensemble uses predictions from different inputs, including flipped versions of the image and a separately trained SegResNet model.

The number of lesions can vary from one scan to another. If we use four passes for every lesion, a scan with five lesions requires 20 model passes. This can exceed the challenge's runtime limit.

We therefore set a maximum number of passes for each scan and adjust the number of passes based on the number of marked lesions.

A simplified version is:

want = max(
    1,
    min(len(members) + 1, cap // max(len(ids), 1))
)
Number of tumors Passes per tumor
1 4
2 3
3 2
More than 3 1
5c02d6d1-2f89-45e3-a5d6-06524acdb5ea

The plan is determined by the number of markers before inference starts. This keeps the amount of computation predictable and avoids making the result depend on how much time happens to remain during execution.

The measured results were:

Setting Score
No ensemble 0.7242
Cap 4 0.7324
Cap 6 0.7361
No cap (4 passes always) 0.7410

The unrestricted ensemble produced the highest score but didn't satisfy the runtime constraint. A cap of 6 retained much of the ensemble improvement while keeping inference within the required limit.

Handle Individual Failures

When processing multiple cases, one failed case shouldn't stop the entire pipeline.

If a case can't be processed, we generate an empty mask and log the error. The pipeline then continues with the remaining cases.

This allows the complete batch to finish even when an individual case has a problem.

The Results

On the 217 hidden test scans, Lumina achieved:

  • Dice: 0.7619

  • NSD: 0.6094

The NSD value uses the official 1 mm tolerance.

The median inference time per case was 20.3 seconds. The slowest observed runtime was 31.8 seconds.

Peak memory usage was 2.22 GB inside the 8 GB container.

The Docker image was approximately 559 MB.

Qualitative results

Large lesion with a clear boundary

The first example is a large lesion with a volume of 272.9 cm³. It achieved:

  • DSC: 0.961

  • NSD: 0.850

CT images of a well-circumscribed large lesion with a clear boundary; predicted segmentation closely follows the reference, with DSC 0.961 and NSD 0.850.

The lesion is well circumscribed, with a clear interface with the surrounding fat.

The predicted contour follows the reference boundary closely across the visible extent of the lesion.

Medium lesion with an uncertain boundary

The second example is a medium lesion with a volume of 28.4 cm³. It achieved:

  • DSC: 0.563

  • NSD: 0.114

CT images of a medium lesion where the prediction covers a larger region than the reference annotation; the boundary is unclear, with DSC 0.563 and NSD 0.114.

The prediction covers a larger area than the reference annotation.

Radiologist review indicated that this type of case can have uncertainty in the reference boundary itself. Therefore, a low numerical score doesn't by itself establish that the predicted contour is clinically unacceptable.

What the Radiologist Review Showed

We also reviewed 30 lesions with a radiologist to better understand the types of errors Lumina made.

The review showed that lesion size wasn't the only factor affecting segmentation.

Lesions with clear boundaries were generally easier to segment. More difficult cases had poorly defined margins or a similar appearance to the surrounding tissue.

The main errors were:

  • Under-segmentation: part of the lesion was missed.

  • Over-segmentation: surrounding tissue was included.

  • Boundary errors: the predicted contour did not follow an unclear or infiltrative margin.

The review also showed that numerical metrics should be interpreted together with the visible lesion boundary.

In some cases, the reference boundary itself was difficult to define. A low score therefore didn't necessarily mean that the predicted contour was clinically unacceptable.

This is consistent with the lower surface accuracy observed in difficult lesions, where small boundary differences can strongly affect NSD.

Three Lessons That Apply to Other Projects

1. Make Your Validation Data Representative

A validation set should reflect the distribution that the final system will be evaluated on.

Our internal validation split came from the training data. Its median tumor volume was 814 mm³.

The competition scoring set had a median tumor volume of 16,805 mm³, which was much larger.

This difference affected how some improvements appeared during development.

For example, the boundary loss showed little improvement on the internal validation set for several epochs. On the public validation distribution, which contained larger lesions, its effect was more useful.

If an improvement targets a particular part of the data distribution, that distribution should be represented in validation.

2. Measure the Limits of Your Preprocessing

The 160 mm crop was an important design choice in Lumina.

Some large lesions extended beyond this field of view.

Before changing the crop size, we measured the error introduced by cropping and resampling.

We passed the ground-truth masks through the same crop-and-resample pipeline and measured the resulting surface score.

For large lesions, the geometry ceiling was 0.9699 NSD, while the model achieved 0.5353 NSD.

This showed that the preprocessing pipeline preserved the lesion surface reasonably well. Preprocessing alone did not explain the remaining gap.

We also tested several approaches for increasing the field of view, including:

  • Overlapping tiles

  • Adaptive zoom

  • A larger crop

These changes didn't improve the final score.

Measuring the preprocessing ceiling helped us focus further development on the model and inference pipeline.

3. Record Where Your Numbers Come From

Model development involves many configuration values:

  • Thresholds

  • Crop sizes

  • Voxel spacing

  • Loss weights

  • Sampling weights

  • Ensemble settings

  • Runtime limits

Each value should have a clear source.

For example, record:

  • What was measured

  • Which dataset was used

  • When the measurement was made

  • Which alternatives were tested

  • Why the final value was selected.

This makes it easier to reproduce experiments and understand design decisions later.

Conclusion

Lumina shows that a single RECIST line can reconstruct a tumor's 3D extent from a CT scan.

The system combines prompt-based input channels, a fixed physical crop, a 3D DynUNet, boundary-aware training, probability-based post-processing, and controlled CPU inference.

On the 217 held-out cases, Lumina achieved a Dice score of 0.7619 and an NSD of 0.6094 at the official 1 mm tolerance.

The median inference time was 20.3 seconds, with a peak memory usage of 2.22 GB, keeping the system within the challenge constraints.

The experiments also showed that boundary accuracy remains an important area for improvement, particularly for large and less clearly defined lesions.

The geometry-ceiling analysis showed that the crop and resampling pipeline preserved the lesion surface well, indicating that further improvements should focus on segmentation accuracy and robustness.

Overall, Lumina demonstrates a practical approach for converting a 2D RECIST measurement into a 3D lesion segmentation while meeting strict computational constraints.

]]> <![CDATA[ How AI Coding Assistants Can Help You Debug Without Writing the Code for You ]]> https://www.freecodecamp.org/news/how-ai-coding-assistants-can-help-you-debug-without-writing-the-code-for-you/ 6aadadc0f205881df958e884 Fri, 18 Sep 2026 21:31:44 +0000 AI coding assistants have become really good at fixing code.

Paste an error into an AI tool and, within seconds, you'll get a corrected implementation. That's useful when you simply want to get something working.

But when you're learning to program, there's another question worth asking: did the AI help you understand the problem, or did it just remove the problem for you?

That difference matters.

Debugging isn't only about arriving at working code. It's also about understanding why something failed, identifying the incorrect assumption, making a change, and verifying that the change actually fixed the problem.

I explored this while using Coddy.tech, an interactive coding-learning platform that combines coding exercises, test feedback, debugging tools, hints, and an AI tutor called Bugsy.

Rather than looking only at whether the AI could solve a programming problem, I tried to examine something different: how much assistance should an AI coding tutor provide before it simply gives away the answer?

In this article, I'll explore that question, propose a simple framework for AI-assisted debugging, and use some of my hands-on experiments with Coddy to see how these ideas work in practice.

Table of Contents

Debugging Is More Than Producing Correct Code

Let's start with a simple Python function:

def calculate_average(numbers):
    total = 0

    for number in numbers:
        total += number

    return total / (len(numbers) - 1)


scores = [80, 90, 70, 100]

print(calculate_average(scores))

The above program runs without any syntax errors or exceptions, but the result is wrong.

The four scores total is 340, so the expected average is:

340 / 4 = 85

Instead, the function calculates:

340 / 3

because of this line:

return total / (len(numbers) - 1)

An AI assistant could immediately respond with:

return total / len(numbers)

The problem is solved. But for someone learning programming, the AI has performed most of the important reasoning for them.

A different response could be:

Your logic calculates the total correctly. But take a closer look at the divisor. How many number of values are actually present in numbers?

Now the developer still has to investigate the logic.The small difference represents two very different approaches to AI assistance.

How Developers Actually Debug

When we debug manually, we usually perform some version of this process:

Debug → Isolate → Reason → Fix → Verify

Suppose this test fails:

assert calculate_average([80, 90, 70, 100]) == 85

We usually inspect the actual result. Then check if total contains the expected value. If the total is correct, we verify the division.

Eventually, we notice that four values are being divided as though only three existed.

This whole process creates understanding.

If an AI assistant immediately rewrites the function, the code becomes valid, but much of that reasoning disappears. This suggests that coding assistants designed for learning nees more than code-generation ability. They need a strategy for deciding how much help to provide.

A Framework for AI-Assisted Debugging

One way I think about this is through five stages:

Context → Diagnosis → Hint → Verification → Explanation

Each stage serves a different purpose.

1. Context

Before suggesting a solution, an assistant needs to understand what you're trying to accomplish.

That context might include:

  • the requirement or problem statement

  • the current code

  • expected output

  • actual output

  • compiler or runtime errors

  • failed tests

  • previous attempts

Without this information, technically valid advice can still be wrong for the actual requirement.

Consider:

def is_adult(age):
    return age > 18

Is this implementation correct? We don't know.

If the requirement says that a person must be older than 18, then it's correct.

But if the requirement says that a person is considered an adult at age 18 or older, then we have a boundary-condition bug.

The code itself doesn't contain enough information to make that determination. The requirement supplies the missing context.

2. Diagnosis

Once enough context is available, the assistant can identify the likely source of the problem.

Diagnosis should answer what appears to be wrong. It doesn't necessarily need to answer what exact code should replace it.

For our average example, the AI assistant could say:

The total is being calculated correctly, but the number of elements used in the division does not match the number of values in the list.

That alone narrows the problem without completely solving it.

3. Hint

If diagnosis isn't enough, the assistant can provide a more specific hint, like:

Check what len(numbers) returns for the sample input and compare it with the divisor in your return statement.

Now you have a concrete debugging step but still have to make the correction.

This creates something like a hint ladder:

Observation → Direction → Stronger Hint → Explanation → Solution

AI assistance doesn't need to be binary. There are useful levels between providing no help and revealing the complete logic/implementation.

4. Verification

Fixing the failure isn't enough.

After correcting your implementation, you might test:

assert calculate_average([80, 90, 70, 100]) == 85

assert calculate_average([10, 20]) == 15

assert calculate_average([5]) == 5

Everything appears fine.

But then try:

calculate_average([])

Now you have another problem: division by zero.

The original bug is fixed, but verification exposes another condition you hadn't considered.

Any Useful AI assistant shouldn't only help you make one failing example pass. Instead it should also encourage you to think about what else could fail.

5. Explanation

After you reach the solution, AI can reinforce the concept:

An average is calculated by dividing the sum by the number of elements/values. Because the length of the list is four elements, subtracting one from its length caused the total to be divided by three instead of four.

At this point, the explanation reinforces the reasoning rather than replacing it.

Progressive Assistance Matters

Imagine someone is implementing this requirement: A person is considered an adult at age 18 or older.

They write:

def is_adult(age):

    return age > 18

Instead of immediately replacing > with >=, an AI tutor could increase the assistance level. The first hint might bee:

Check your boundary condition.

If the learner still struggles:

What should happen when age is exactly 18?

And then:

Your comparison currently excludes the boundary value itself.

Only if necessary does the assistant finally show: return age >= 18.

Instead of Problem → AI → Answer, we get Problem → Observation → Hint → Reasoning → Attempt → Verification → Explanation.

That's a very different learning experience.

I Tried This Learning Loop in Coddy

I wanted to see how the ideas translate into an actual coding-learning environment, so I experimented with Coddy.tech.

I started with a beginner Python challenge about line comments.

There's a straightforward requirement: comment out a print("Goodbye!") line without deleting it so that below line is printed:

Hello, Python!

The exercise wasn't particularly interesting from a programming perspective. What caught my attention was everything surrounding the code.

In the same workspace I had access to the challenge requirements, browser-based Python editor, Run Code, test results, expected output, multiple hints, solution access, an option to explain the challenge, and Coddy's AI tutor, Bugsy.

That creates several ways to respond to a failure instead of immediately asking AI for the solution.

Test Feedback Before AI

I intentionally entered an incorrect solution and executed the code.

Coddy's test area connected the failure back to the requirement, telling me that I needed to add the comment symbol at the beginning of the Goodbye line without deleting it.

The expected output was also displayed:

Hello, Python!

From a testing perspective, it's useful even though it seems simple. The learner isn't only asking: Does the code execute? They're also asking: Does the implementation produce the behavior required by the exercise?

Those two aren't the same questions. A program can execute successfully and still be functionally incorrect.

Showing the expected behavior introduces that distinction early.

Progressive Hints

The same exercise also provided multiple hint levels.

The first hint directed me toward adding # at the beginning of the appropriate line, while additional hints remains available.

This creates another path: Attempt → Test feedback → Hint 1 → Hint 2 → Hint 3 → Solution.

The learner doesn't necessarily need to jump directly from failure to the complete answer. That supports the progressive-assistance model we discussed earlier.

Coddy provides multiple layers of feedback, including test results, expected output, progressive hints, AI assistance, and solution access

Testing Bugsy With My Incorrect Code

Next, I opened Bugsy while the incorrect code was still in the editor.

This gave more interesting result.

Bugsy understood the objective of the exercise and directed me towards commenting out the Goodbye line.

But it also noticed another problem in my current implementation: the Hello statement had an incorrectly formed closing quote/parenthesis.

That second problem matters most because it wasn't simply the concept being taught by the exercise.

It comes from my current code. Bugsy appeared to be responding to both the challenge context and what I had actually written in the editor.

This illustrates why context was the first element of the framework: Context → Diagnosis → Hint → Verification → Explanation

Consider this code outside the exercise:

print("Goodbye!")

print("Hello, Python!")

There's nothing inherently wrong with it.

You need a complex requirement to know that Goodbye! shouldn't appear and specifically, that you're supposed to comment out the line rather than delete it.

That's where integrating AI becomes interesting in the learning environment.

Separating Help From the Solution

Another detail that i found interesting: Bugsy provided guidance while keeping "Reveal Solution" locked as a separate action.

That creates a useful difference between help me move forward and reveal the answer.

The distinction may not be perfect (we'll come back to that) but I like the underlying design idea.

An AI tutor doesn't necessarily need to treat every request for help as a request to reveal complete implementation.

Moving to a Harder Challenge

A beginner comments exercise can only tell us basic things. So I tried a medium-level Python challenge involving more reasoning.

The task was to implement:

find_book_descriptions(catalog, query)

The function needed to search a two-dimensional library catalog.

Each book contains an ID and description.

The implementation needed to:

  • iterate through the books

  • perform case-insensitive matching

  • search both the ID and description

  • collect matching descriptions

  • join multiple results with newline characters

  • return "No books found." when there were no matches

This gave me a much better environment for testing the assistance.

I intentionally created a broken implementation containing a mixture of Python and pseudocode.

When I ran it, multiple test cases failed.

Coddy showing multiple test cases failing,assuming the input will be in different each time

Those Test Cases Shows the Behavior too, Not Just Failure

The test panel showed multiple test cases along with arguments, program output, and expected output. Which is important.

Instead of seeing only failure, you can investigate the relationship between Input → Actual behavior → Expected behavior.

That's basically a testing workflow.

A single successful example doesn't necessarily mean that an implementation satisfies the complete requirement. Different inputs may expose different defects.

Debugging Without Immediately Asking AI

The same challenge also had a separate Debug option that I used on the broken implementation.

Instead of correcting the entire program or explaining the whole implementation, the Debug panel surfaced the immediate Python failure:

SyntaxError: invalid syntax (main.py, line 5)

I liked the separation. Not every programming problem needs generative AI.

If Python already knows where parsing failed, exposing that information gives you an opportunity to investigate independently.

At this point, I had three different feedback mechanisms:

Mechanism Question it helps answer
Test Cases Does my implementation behave as expected?
Debug Where is execution currently failing?
Bugsy What may be wrong with my approach, and how can I move forward?
The same broken implementation produces different levels of assistance: Debug identifies the immediate syntax failure, while Bugsy analyzes the broader structure and logic of the solution.

Then I Asked Bugsy

I gave the same broken implementation to Bugsy.

This time the response went beyond identifying the syntax error. Bugsy recognized that the implementation was mixing Python with pseudocode.

It navigated me toward several changes, including creating a list for matching descriptions, iterating through each book, separating the book ID and description, using lowercase comparisons for case-insensitive searching, and appending the description rather than the query.

It also identified a more interesting control-flow problem.

The "No books found" decision shouldn't happen while individual books are still being searched. Why?

Imagine the first book doesn't match but the second one does.

If the program concludes "No books found" while still inside the search loop, it may make that decision before looping through the rest of the catalog.

That's not just syntax correction. It also requires understanding the relationship between the requirement and the control flow.

This is where contextual AI assistance becomes more interesting than a generic error explanation.

But How Much Help Is Too Much?

The medium challenge also exposed a limitation, or at least an important tradeoff.

Bugsy didn't stop after identifying the problematic areas. It provided a fairly detailed structure showing how the function could be implemented.

From a productivity perspective, that's very useful. If I'm an experienced developer trying to finish something quickly, I highly appreciate it.

But if I'm trying to learn the concept, I'm less convinced that more information is always better.

Consider below two responses.

Approach A

Here is the corrected implementation...

Approach B

Your "No books found" condition is being evaluated while you're still searching the catalog.

What could happen if the first book doesn't match, but the second book does?Both can eventually lead to correct code.

But Approach B requires you to reason about control flow.

This exposes a difficult problem for AI tutors.

They potentially have two goals:

Help the learner succeed

and

Preserve enough difficulty to get the learner to think

Those goals can conflict.

An AI assistant capable of generating the complete solution still has to decide whether generating it is actually the most useful thing to do.

Different Learners May Need Different Amounts of Help

The appropriate amount of assistance also depends on who's asking.

A beginner learning loops for the first time may benefit from progressive hints. An experienced developer debugging unfamiliar library behavior may simply want the answer.

So perhaps the ideal interaction shouldn't always be:

Here's how to fix it.

It could begin by understanding intent:

Do you want a hint, an explanation, or the corrected implementation?

That's a relatively small UX decision, but it changes the role of the AI.

AI Isn't the Entire Learning System

After spending more time exploring Coddy, another thing became clearer: Bugsy isn't the only learning experience.

The platform also separates activities into areas such as Journey, Practice, Projects, and Missions.

In the Python Journey I explored, lessons were organized through a syllabus and progression path.

The interface also included XP, levels, streaks, daily missions, and a leaderboard.

Coddy’s Python Journey combines a structured syllabus with practice, projects, missions, XP-based progress, and daily learning goals

Those may sound like gamification features rather than AI features. But that's exactly why they're worth discussing.

Learning programming requires repetition and encouragement to help make learning interesting and fun

AI can explain why a loop fails. But understanding that explanation once doesn't mean you'll correctly implement a different loop tomorrow.

You still need to practice. This gives us two complementary systems.

  1. Learning progression: Journey → Practice → Projects → Repetition

  2. Assistance when something goes wrong: Run Code → Test Feedback → Debug/Hints → Bugsy → Solution

I think this distinction matters when evaluating AI-learning products.

The question shouldn't only be how capable is the AI?

We should also ask what is the learner doing before and after asking the AI?Are they building stronger fundamentals by using it, or becoming more dependent on AI?

Coding Assistants Should Be Tested Differently

Most evaluations of coding assistants naturally focus on whether they produce correct code which is important.

But for an AI system intended to support learning, I think we need additional test cases.

For example:

Scenario What I would evaluate
Syntax error Does it correctly locate the problem?
Runtime error Does it explain why execution failed?
Logic error Can it diagnose the problem without unnecessarily rewriting everything?
Boundary condition Does it understand values such as 0, empty input, or equality boundaries?
Wrong algorithm Can it guide the learner toward the right concept?
Repeated wrong attempts Does the assistance adapt?
Correct implementation Does it recognize that nothing needs fixing?
Alternative valid implementation Does it accept a solution different from the reference answer?

The final two are particularly interesting.

Correct Code Is Also a Test Case

Consider:

def square(number):
    return number * number

Suppose this completely satisfies the requirement.

What happens if I still ask the AI for help?

A poor assistant might suggest unnecessary changes because it feels obligated to produce something.

A better assistant should be able to say:

Your implementation already satisfies the stated requirement.

This is closely related to something we encounter when testing generative AI systems: false positives.

Being helpful doesn't always mean finding something wrong. Sometimes being helpful means recognizing that nothing needs fixing.

Alternative Solutions Matter

Programming problems also rarely have only one valid implementation.

Consider:

def is\_even(number):

return number % 2 == 0

Someone else might write:

def is\_even(number):

if number % 2 == 0:

return True

return False

The first is more concise, but both satisfy the requirement.

An AI learning assistant shouldn't confuse different from the reference solution with incorrect.

That's an important test case for any coding-learning system.

Repeated Failure Is Another Test

Suppose the learner receives a hint and submits another incorrect solution.

What should happen? Repeating the exact same hint may not help. Immediately revealing the entire solution may be too aggressive.

Instead, assistance could become progressively more specific.

For example:

Attempt 1

Look closely at the operation you're using to determine whether the number is even.

Attempt 2

Division gives you the quotient. Think about which operation tells you the remainder.

Attempt 3

In Python, % returns the remainder after division. Try using it with 2.

This is an interesting evaluation dimension for AI tutors because the evaluation isn't only about correctness, but also about adaptation.

Learner asking AI tutor Bugsy mutiple times to explain the challenege and Bugsy explaining differently everytime without revealing entire codeRepeated requests for help are another useful test for an AI tutor. Here, I asked Bugsy about the same beginner challenge in different ways to observe whether its explanation changed or became more specific

In this example, the second request produced another explanation of the same underlying problem, while also pointing out the issue in my current Hello statement.

This raises another useful evaluation question: should repeated requests simply produce another explanation, or should the level of assistance adapt based on the learner's previous interaction?

AI Coding Assistants Have Boundary Conditions Too

Traditional software testing spends a lot of time around boundaries.

  • What happens at zero?

  • What happens at the maximum value?

  • What happens when input is empty?

  • What happens exactly at the threshold?

AI coding assistants have boundaries too, but many of them are behavioral.

  • How little context can we provide before the assistant starts guessing?

  • How much assistance can it provide before it effectively gives away the exercise?

  • When should a hint become an explanation?

  • When should an explanation become code?

  • What happens after repeated failures?

  • What happens when the learner produces a different but valid implementation?

And when should the AI simply say: I don't have enough information yet.

These aren't only educational questions. They're quality-engineering questions.

A Practical Framework for Evaluating AI Coding Assistance

After these experiments, I come back to the five stages introduced earlier:

  1. Context: Does the assistant understand what the developer is actually trying to accomplish?

  2. Diagnosis: Can it identify why the current implementation fails?

  3. Hint: Can it provide enough direction without unnecessarily revealing the complete solution?

  4. Verification: Does the environment help the developer validate the correction against additional scenarios?

  5. Explanation: Does the interaction leave the developer understanding why the final implementation works?

Together:

Context → Diagnosis → Hint → Verification → Explanation

A coding assistant that performs well across those dimensions is doing more than generating code. It's participating in the debugging process.

Where Coddy Fits

This is why I found Coddy interesting to explore. The most interesting part isn't simply that it has an AI tutor.

AI can be attached to almost any coding interface today. In fact not only just to coding interfaces, but to almost anything in general.

The more interesting combination is: Structured learning + coding exercises + executable code + test feedback + debugging + contextual AI assistance.

Each component serves a different purpose.

Structured learning provides direction. Exercises require application. Execution provides immediate feedback. Test cases compare implementation against expected behavior. Debugging exposes technical failures. And hints provide incremental assistance.

Bugsy can provide additional contextual guidance. And the complete solution remains another level of assistance.

In the exercises I tried, that produced a workflow closer to:

Learn → Code → Run → Fail → Inspect → Debug → Ask for Help → Retry

rather than:

Problem → Ask AI → Copy Answer

That specific difference is important.

At the same time, my medium-level experiment showed that contextual AI can still provide a substantial amount of implementation guidance very quickly.

How much the AI reveals (and when it reveals it) remains an important design decision.

Less AI isn't always the goal. None of this means developers should avoid AI-generated code. There are plenty of situations where generating the implementation immediately is exactly what we want.

Experienced engineers may use AI to:

  • generate boilerplate

  • create unit tests

  • refactor repetitive code

  • understand unfamiliar libraries

  • prototype implementations

  • explain legacy code

  • create documentation

In those situations, speed may be the primary objective

But compare these two requests:

Help me finish this implementation.

and:

Help me understand why my implementation fails.

They may involve exactly the same code. But they represent completely different goals. A useful AI coding assistant should ideally recognize that difference.

Wrapping up

The most impressive AI coding assistant may not always be the one that produces the most code. Sometimes it may be the one that knows when not to produce code.

Good debugging assistance should help developers move from:

“My code doesn't work.”

to:

“I understand why my code didn't work.”

That requires more than code generation.

It requires context, diagnosis, progressive assistance, verification, and explanation.

My experiment with Coddy showed why integrating the AI with the coding environment can be useful: Bugsy could respond to both the exercise and the code I was working with, while test cases, debugging, hints, and solution access provided different levels of assistance.

It also exposed the harder question: when an AI knows how to solve the problem, how much of that solution should it reveal?

As AI becomes more deeply integrated into programming education, I think evaluating whether an assistant generates correct code will remain important.

But we should also measure something harder: did the developer leave the interaction understanding the problem better than when they entered it?

For an AI tutor, that may ultimately be the more meaningful test.

If you would like to experiment with the features discussed in this article, you can explore them on Coddy.tech.

]]> <![CDATA[ How to Migrate a Legacy Monolith Incrementally Without a Big-Bang Rewrite ]]> https://www.freecodecamp.org/news/migrate-legacy-monolith-incrementally/ 6aac7747d406d7c207351312 Thu, 17 Sep 2026 23:27:03 +0000 Large legacy migrations often fail long before the final cutover.

The failure usually starts when the migration is framed as a single event. Move the application. Move the database. Move all the users. Switch the traffic. Turn the old system off.

That creates a dangerous assumption: the legacy system and the new system need to exchange places all at once.

They usually don't.

If you already understand the legacy behavior, protect it with characterization tests, create migration-friendly boundaries, and compare old and new implementations, you have another option.

You can migrate one capability at a time. That changes the problem completely.

Instead of:

legacy monolith
      ↓
complete rewrite
      ↓
big-bang cutover

you can move toward:

legacy monolith
      ↓
one capability extracted
      ↓
small percentage of traffic
      ↓
observe
      ↓
expand
      ↓
repeat

The goal isn't to make the migration slower. The goal is to make each change smaller, observable, and reversible.

In this tutorial, I'll show you how to migrate a legacy monolith incrementally by:

  • choosing a safe first migration slice

  • defining a boundary between legacy and new code

  • routing requests between implementations

  • using the Strangler Fig pattern

  • migrating by business capability instead of technical layer

  • keeping old and new implementations running together

  • introducing progressive traffic

  • detecting failures before full cutover

  • designing rollback paths

  • handling data ownership carefully

  • removing migrated legacy behavior

  • using AI without turning an incremental migration into an automated rewrite

The examples use TypeScript, but the approach applies to most languages, runtimes, and architectures.

The objective is simple: make migration a sequence of controlled changes instead of one irreversible event.

Prerequisites

To follow along, you should be comfortable with:

  • TypeScript or a similar language

  • API and service boundaries

  • integration testing

  • dependency injection

  • routing and reverse proxies

  • database transactions

  • observability

  • incremental refactoring

  • legacy modernization

You should also already understand the behavior of the capability you want to migrate.

Ideally, you know:

  • its inputs

  • its outputs

  • its important business rules

  • its side effects

  • its dependencies

  • its external contracts

  • how you'll detect behavioral differences

If you haven't reached that point yet, migration may be premature.

Table of Contents

Why Big-Bang Migrations Are So Risky

Imagine a legacy commerce application.

It contains:

customers
orders
payments
inventory
shipping
invoicing
notifications
reporting

The modernization plan says:

replace the monolith

That sounds like one project.

Operationally, it may mean changing:

runtime
framework
database
deployment model
API contracts
authentication
networking
observability
data model
business logic
external integrations

at the same time.

If the final cutover fails, the number of possible causes is enormous.

For example:

Did pricing change?

Did the database migration lose data?

Is the payment provider failing?

Did authentication behave differently?

Did the new runtime change date handling?

Did a timeout become shorter?

Did an event stop being published?

Did the new deployment configuration fail?

This is one of the central problems with big-bang migration: too many variables change together.

Incremental migration tries to reduce the number of changing variables at each step.

Think in Migration Slices, Not Applications

Instead of asking:

How do we migrate this monolith?

ask:

What's the smallest meaningful business capability we can move independently?

For example:

Calculate Order Total
Generate Invoice
Send Order Confirmation
Create Shipment
Renew Subscription
Approve Customer

A migration slice should ideally have:

clear input
clear output
known side effects
understood dependencies
observable behavior
a rollback path

That gives you something concrete to move.

For example:

Generate Invoice

might become:

input:
orderId

behavior:
load order
calculate taxes
generate invoice number
create invoice

side effects:
store invoice
publish invoice.created

output:
invoice

That's much easier to migrate than:

billing module

or:

src/services/

Business capabilities make better migration units than folders.

Choose the First Capability Carefully

The first slice matters.

I would usually avoid starting with the most critical capability in the system.

You want something meaningful enough to validate the migration approach, but not so dangerous that a mistake creates catastrophic consequences.

A useful first slice often has:

moderate traffic
limited external dependencies
clear behavior
good test coverage
few transactional boundaries
low blast radius

For example:

Generate Customer Statement

may be a better first migration candidate than:

Authorize Payment

The first migration is partly technical work, but it's also a learning exercise.

You're validating:

routing
deployment
observability
rollback
data access
testing
team workflow

before applying the pattern to more critical capabilities.

Create a Boundary Between Legacy and New

Suppose the legacy application has:

async function generateInvoice(
  orderId: string
) {
  // legacy implementation
}

Before migration, introduce a boundary:

interface InvoiceGenerator {
  generate(
    orderId: string
  ): Promise<Invoice>;
}

The legacy implementation becomes:

class LegacyInvoiceGenerator
  implements InvoiceGenerator {
  async generate(
    orderId: string
  ): Promise<Invoice> {
    // existing behavior
  }
}

The new implementation becomes:

class NewInvoiceGenerator
  implements InvoiceGenerator {
  async generate(
    orderId: string
  ): Promise<Invoice> {
    // migrated behavior
  }
}

Now the caller doesn't need to know which implementation is active.

That creates an important capability:

replace implementation
without replacing caller

which is one of the foundations of incremental migration.

Use the Strangler Fig Pattern

A common way to describe incremental replacement is the Strangler Fig pattern.

Instead of replacing the entire application at once, new behavior gradually grows around the old system.

Conceptually:

            incoming request
                   │
                   ↓
                router
              /        \
             /          \
      legacy path     new path

At first:

legacy: 100%
new:      0%

Later:

legacy: 95%
new:      5%

Then:

legacy: 50%
new:     50%

Eventually:

legacy:  0%
new:    100%

At that point, the old implementation for that capability can be removed.

The key is that the replacement happens gradually. The legacy application continues serving parts of the system while the new implementation takes over others.

Migrate Capabilities, Not Technical Layers

One tempting migration strategy is:

move database
then move services
then move APIs
then move UI

That can create long periods where every capability spans both old and new architecture.

For example:

new API
↓
legacy service
↓
new database
↓
legacy event publisher

This is sometimes unavoidable.

But whenever possible, I prefer vertical slices.

A vertical slice might be:

Generate Invoice

request
↓
application logic
↓
persistence
↓
events
↓
response

That capability can move as one coherent unit.

Then:

Create Shipment

can move separately.

Then:

Renew Subscription

and so on.

This gives you working migrated capabilities earlier. It also reduces the number of temporary cross-system dependencies.

Keep Legacy and New Implementations Running Together

During an incremental migration, coexistence is normal.

For some period of time, you may have:

LegacyInvoiceGenerator
NewInvoiceGenerator

both deployed.

That's not duplication by accident. It's part of the migration strategy.

The important question is how requests choose between them.

You may use:

feature flag
tenant
user group
request header
region
percentage rollout
specific account IDs

For example:

class InvoiceRouter {
  constructor(
    private readonly legacy:
      InvoiceGenerator,
    private readonly migrated:
      InvoiceGenerator
  ) {}

  async generate(
    orderId: string,
    useMigrated: boolean
  ) {
    if (useMigrated) {
      return this.migrated.generate(
        orderId
      );
    }

    return this.legacy.generate(
      orderId
    );
  }
}

This is deliberately simple. The important part is that routing is explicit. You know which implementation handled each request.

Route Traffic Explicitly

Avoid migration logic that's difficult to observe.

For example:

try {
  return await newService.call();
} catch {
  return legacyService.call();
}

This may look resilient, but it can hide failures.

Suppose the new implementation fails 40% of the time. If every failure silently falls back to legacy, users may see no problem. But the migration isn't healthy.

The problem is that the first version mixes two decisions together: which implementation should receive the request and what should happen when that implementation fails. Because the fallback happens inside the catch, the migrated path can fail repeatedly without producing an explicit routing signal that you can measure.

A better approach is to make the routing decision first, record it, and then call the selected implementation. That separates migration policy from error handling and gives you a clear record of how much traffic actually reached each path.

For example:

const route =
  migrationPolicy.route(request);

metrics.increment(
  `invoice.route.${route}`
);

if (route === "migrated") {
  return migrated.generate(
    request.orderId
  );
}

return legacy.generate(
  request.orderId
);

Now you can measure:

requests routed to legacy
requests routed to migrated
migration failures
fallback count
latency
business outcomes

Migration should be observable as a first-class system behavior.

Start with Internal or Low-Risk Traffic

Before routing a large percentage of customers to the migrated path, start with safer traffic.

For example:

development
test environments
internal users
staff accounts
test tenants
specific low-risk customers

This lets you validate:

deployment
routing
observability
data access
external integrations
failure handling

with lower risk.

You can then expand.

For example:

internal users
↓
1% production
↓
5%
↓
10%
↓
25%
↓
50%
↓
100%

The exact percentages aren't important, but the principle is.

Each increase should happen because the previous stage produced enough evidence.

Progressively Increase Production Traffic

Suppose you have:

10,000 invoice requests/day

Instead of switching all requests:

legacy → new

at once, route:

1%

first.

That gives roughly:

100 real requests/day

through the migrated path.

Now monitor:

error rate
latency
output differences
side effects
customer-visible failures
business metrics

If the system behaves correctly, increase traffic. If it doesn't, reduce or disable migrated routing.

The migration becomes a controlled experiment. That's very different from a cutover event.

Use Differential Testing Before and During Rollout

The previous article in this series focused on differential testing. That technique becomes especially useful here.

Differential testing means running the legacy and migrated implementations with the same input and comparing their observable behavior. Depending on the capability, that may include return values, errors, state changes, and side effects.

The goal isn't to prove that the implementations are internally identical. It's to detect meaningful behavioral differences before those differences reach all of your production traffic.

Before live routing, you can compare:

same input
↓
legacy result

same input
↓
new result

During rollout, you can also sample real traffic and compare behavior where it is safe to do so.

For example:

real request
      │
      ├────→ active implementation
      │
      └────→ shadow implementation

Then compare:

output
errors
side effects
business state

This gives you evidence before increasing traffic.

A rollout decision can then be based on:

divergence
error rate
latency
business outcomes

instead of:

It seems fine.

A Small End-to-End Invoice Migration Example

The individual pieces are easier to understand when you see them working together.

Here's a deliberately small, in-memory example based on the invoice capability we've been using throughout the article. It doesn't include a real database, reverse proxy, queue, or deployment platform. The point is to show the migration control flow in one place.

Start with a shared contract:

type InvoiceInput = {
  orderId: string;
  subtotal: number;
};

type Invoice = {
  orderId: string;
  total: number;
};

interface InvoiceGenerator {
  generate(
    input: InvoiceInput
  ): Promise<Invoice>;
}

The legacy implementation calculates the invoice total like this:

class LegacyInvoiceGenerator
  implements InvoiceGenerator {
  async generate(
    input: InvoiceInput
  ): Promise<Invoice> {
    return {
      orderId: input.orderId,
      total: input.subtotal * 1.21,
    };
  }
}

Now imagine we've migrated that capability into a new implementation:

class MigratedInvoiceGenerator
  implements InvoiceGenerator {
  async generate(
    input: InvoiceInput
  ): Promise<Invoice> {
    const tax =
      input.subtotal * 0.21;

    return {
      orderId: input.orderId,
      total: input.subtotal + tax,
    };
  }
}

The code is different, but the intended behavior is the same.

Next, define a deterministic rollout function. This example assigns each orderId to a bucket from 0 to 99 so the same order always follows the same route:

function bucketFor(
  value: string
): number {
  const sum = [...value].reduce(
    (total, char) =>
      total + char.charCodeAt(0),
    0
  );

  return sum % 100;
}

function shouldUseMigrated(
  orderId: string,
  percentage: number
): boolean {
  return (
    bucketFor(orderId) < percentage
  );
}

If percentage is 10, roughly 10% of IDs will be assigned to the migrated path.

Now add some tiny in-memory metrics:

const metrics = {
  legacyRequests: 0,
  migratedRequests: 0,
  mismatches: 0,
};

Then put the legacy and migrated implementations behind one migration-aware entry point:

class IncrementalInvoiceService {
  migratedEnabled = true;
  rolloutPercentage = 10;

  constructor(
    private readonly legacy:
      InvoiceGenerator,
    private readonly migrated:
      InvoiceGenerator
  ) {}

  async generate(
    input: InvoiceInput
  ): Promise<Invoice> {
    const legacyResult =
      await this.legacy.generate(
        structuredClone(input)
      );

    const migratedResult =
      await this.migrated.generate(
        structuredClone(input)
      );

    if (
      migratedResult.orderId !==
        legacyResult.orderId ||
      migratedResult.total !==
        legacyResult.total
    ) {
      metrics.mismatches += 1;
    }

    const useMigrated =
      this.migratedEnabled &&
      shouldUseMigrated(
        input.orderId,
        this.rolloutPercentage
      );

    if (useMigrated) {
      metrics.migratedRequests += 1;
      return migratedResult;
    }

    metrics.legacyRequests += 1;
    return legacyResult;
  }
}

This small service combines several ideas from the article.

First, it runs both implementations with the same input and compares their results. Because this example is entirely in memory and has no external side effects, doing that is safe.

Second, it routes only a percentage of requests to the migrated result.

Third, it records how many requests used each path and how many behavioral mismatches occurred.

You can exercise it with a few requests:

const service =
  new IncrementalInvoiceService(
    new LegacyInvoiceGenerator(),
    new MigratedInvoiceGenerator()
  );

for (let i = 1; i <= 100; i++) {
  await service.generate({
    orderId: `order-${i}`,
    subtotal: 1000,
  });
}

console.log(metrics);

You might see something like:

legacyRequests:   89
migratedRequests: 11
mismatches:        0

The exact split may not be exactly 90/10 with only 100 inputs because the bucket function is intentionally simple. The important point is that routing is deterministic, measurable, and controlled by rolloutPercentage.

If the migrated implementation starts producing differences, the mismatch counter gives you an observable signal.

And if you decide the rollout should stop, rollback is explicit:

service.migratedEnabled = false;

From that point forward, all returned responses come from the legacy implementation again.

This is intentionally a simplified example. A production system would need stronger routing, real metrics, error handling, persistent state, and careful treatment of side effects.

In particular, you shouldn't blindly execute both implementations if generating an invoice sends email, writes to two production databases, charges a customer, or publishes externally visible events. In those cases, the shadow path needs recording adapters, isolated infrastructure, or another mechanism that lets you compare behavior without duplicating real effects.

But the control loop is the same:

same input
↓
compare legacy and migrated behavior
↓
route a small percentage
↓
observe
↓
expand or roll back

That is incremental migration in its smallest useful form.

Design Rollback Before You Need It

Rollback shouldn't be invented during an incident. Before moving traffic, ask what happens if the migrated path fails.

For routing-level migrations, rollback may be simple:

migration flag = false

and traffic returns to:

legacy implementation

For example:

if (
  featureFlags.useNewInvoices
) {
  return migrated.generate(
    orderId
  );
}

return legacy.generate(orderId);

If the migrated path behaves incorrectly:

useNewInvoices = false

Rollback is almost immediate.

But rollback becomes more complicated when:

data format changes
new data is written
events differ
external systems are updated
legacy code cannot read new records

In those cases, rollback may require more than flipping a feature flag. You might need backward-compatible schemas so both versions can read the same records, compensating actions for external side effects, replayable events, reconciliation jobs, or a short period where the legacy system remains able to consume data written by the new path.

For higher-risk migrations, it can also help to define a rollback boundary in advance. For example: traffic can return to legacy until a new schema version is written, or after a particular external event is emitted, recovery requires compensation instead of a simple rollback. The important part is knowing when rollback is still reversible and when you've crossed into a different recovery strategy.

That's why rollback design needs to happen before deployment.

Treat Data Migration as a Separate Problem

Application migration and data migration are related, but they aren't the same problem.

Suppose the legacy system stores:

{
  "customer_type": "P",
  "status": 2
}

while the new system stores:

{
  "customerType": "PREMIUM",
  "status": "APPROVED"
}

You now need to answer:

Which database is authoritative?

Can both systems read the same data?

Do we transform on read?

Do we migrate records in batches?

Do we replicate changes?

When does ownership change?

These decisions should be explicit. Otherwise the application migration may appear successful while the data boundary remains ambiguous.

Be Careful with Dual Writes

One common transition strategy is:

write to legacy database
+
write to new database

This is called dual writing, and it looks simple.

For example:

await legacyOrders.save(order);
await newOrders.save(order);

But what happens if:

legacy write succeeds
new write fails

Now the two systems disagree.

Or:

legacy write fails
new write succeeds

Same problem.

Dual writes create a distributed consistency problem.

If you use them, you need to think about:

retries
idempotency
reconciliation
ordering
partial failure
monitoring

Sometimes a safer approach is:

single authoritative write
↓
change event
↓
replication

or a transactional outbox.

There's no universal solution. The important point is not to treat dual writing as a trivial migration technique.

Decide Who Owns the Data

During coexistence, data ownership can become confusing.

Imagine:

legacy system writes customers

new system writes invoices

both systems read orders

That may be perfectly reasonable, but it should be documented.

For each migrated capability, define:

system of record
write owner
readers
replication direction
consistency expectations

For example:

Invoices

Write owner:
new system

Source of truth:
new database

Legacy access:
read-only adapter

Replication:
new → legacy reporting store

Now the architecture has an explicit direction.

Without ownership rules, migrations often create permanent synchronization problems.

Observe Business Behavior, Not Just Infrastructure

During rollout, teams often monitor:

CPU
memory
latency
HTTP 500s
database connections

Those are important. But they're not enough.

Suppose:

HTTP 200 rate = 99.99%

while:

invoice totals are wrong

Infrastructure monitoring says:

healthy

But the business system is not healthy.

Migration observability should include domain signals.

For example:

orders processed
payments authorized
invoices generated
discount distribution
failed renewals
average invoice total
events published

If you know normal business behavior, unusual changes can expose migration defects that technical metrics miss.

Know When a Migration Slice Is Complete

A capability isn't fully migrated just because traffic reached 100%.

Before declaring it complete, I would verify:

100% traffic on new path
acceptable error rate
acceptable latency
behavioral differences resolved
side effects verified
data ownership established
rollback window completed
legacy callers removed
legacy writes stopped
observability in place

Then ask:

Is the legacy implementation still serving any purpose?

If not, remove it.

Leaving both implementations permanently active creates:

maintenance cost
confusion
duplicate bugs
unclear ownership
future migration debt

Incremental migration should eventually simplify the system, not permanently duplicate it.

Remove the Legacy Path

This step is often delayed.

Teams migrate traffic but leave the old path in place:

just in case

Months later:

nobody knows whether it is still used

Before deleting it, verify:

routing metrics show zero traffic
no callers depend on it
data dependencies are removed
rollback period is complete
operational documentation is updated

Then remove:

legacy implementation
legacy feature flags
legacy database access
unused integration code
temporary compatibility layers

Deletion is part of migration.

A migration that only adds new architecture without removing old architecture can increase complexity rather than reduce it.

How to Use AI During an Incremental Migration

AI can help with many parts of this process.

For example, it can inspect the legacy codebase and help answer:

Which modules implement this capability?

Which callers depend on it?

Which database tables does it touch?

Which external services does it call?

Which side effects occur?

Which feature flags already exist?

Which paths need adapters?

A useful prompt might be:

Analyze the Generate Invoice capability.

Identify:

1. entry points,
2. business rules,
3. persistence dependencies,
4. external integrations,
5. side effects,
6. callers,
7. data ownership,
8. possible migration seams.

Do not redesign the system.

Return evidence for each finding using file paths
and relevant code references.

AI can also help compare migration changes.

For example:

Compare the legacy and migrated implementations.

Identify possible behavioral differences in:

- return values,
- errors,
- side effects,
- persistence,
- event ordering,
- retries,
- idempotency,
- transaction boundaries.

Do not assume the new implementation is correct.

This is useful because migration involves a lot of repetitive analysis, and AI can accelerate that analysis.

Don't Let AI Turn the Migration into a Rewrite

There's a common failure mode.

You ask:

Help me migrate this legacy capability.

The model responds with:

new architecture
new domain model
new API
new event model
new database schema
new validation layer
new framework

At that point, you're no longer migrating one capability, you're redesigning it.

Sometimes redesign is necessary, but it should be intentional.

During incremental migration, I prefer prompts with explicit constraints.

For example:

Migrate this capability without intentionally changing
observable behavior.

Preserve:

- inputs,
- outputs,
- errors,
- side effects,
- ordering where relevant,
- transactional behavior.

Only introduce the minimum structural changes required
to run it in the target environment.

List any behavior you cannot preserve with confidence.

That keeps the transformation narrow.

AI should help reduce mechanical effort. It shouldn't silently expand project scope.

A Practical Incremental Migration Workflow

Here's the workflow I would use.

1. Understand the Capability

Identify:

inputs
outputs
rules
side effects
dependencies
unknowns

2. Characterize Existing Behavior

Protect important behavior with:

characterization tests
integration tests
contract tests

3. Refactor for Migration

Create:

seams
adapters
explicit dependencies
clear orchestration

without intentionally changing behavior.

4. Build the New Implementation

Implement the capability in the target environment. Keep its observable contract clear.

5. Differentially Test Old and New

Compare:

outputs
errors
side effects
business state

using representative cases.

6. Introduce Explicit Routing

Allow requests to choose:

legacy
or
migrated

through an observable migration policy.

7. Start with Safe Traffic

Use:

internal users
test tenants
selected customers

8. Increase Traffic Gradually

For example:

1%
5%
10%
25%
50%
100%

only when evidence supports the next stage.

9. Monitor Technical and Business Metrics

Observe both:

system health
business behavior

10. Keep Rollback Available

Make returning to the legacy path fast and understood.

11. Transfer Data Ownership

Explicitly define which system owns:

writes
reads
replication

12. Remove the Legacy Path

After the migration has stabilized:

delete old implementation
remove temporary routing
remove obsolete dependencies

Then choose the next capability.

What Incremental Migration Doesn't Solve

Incremental migration reduces risk, but it doesn't eliminate complexity.

You may still need to deal with:

distributed transactions
shared databases
old schemas
tight coupling
unsupported runtimes
poor test coverage
organizational ownership
regulatory constraints

There are also systems where partial migration is extremely difficult.

For example:

highly stateful systems
strongly coupled desktop applications
large transactional batch systems
systems with shared global state

Sometimes the migration boundary needs to be larger.

The principle remains the same:

Make the smallest reversible change that produces useful migration progress.

Incremental doesn't always mean tiny. It means controlled.

The Complete Legacy Modernization Workflow

This article closes the workflow we've been building throughout this series.

We started with a basic problem:

How do you modernize a legacy application without accidentally turning the project into a rewrite?

The first step was understanding.

Legacy system
↓
investigate
↓
map behavior and dependencies

Then characterization.

observed behavior
↓
tests
↓
behavioral safety net

Then refactoring.

entangled capability
↓
seams and boundaries
↓
migration-friendly structure

Then differential testing.

legacy implementation
        +
new implementation
        ↓
behavior comparison

And finally incremental migration.

Understand
↓
Characterize
↓
Refactor
↓
Migrate
↓
Compare
↓
Route
↓
Observe
↓
Expand
↓
Remove legacy

The sequence matters.

If you skip understanding, you may migrate the wrong behavior.

If you skip characterization, you may not notice behavioral changes.

If you skip refactoring, the migration boundary may remain too large.

If you skip comparison, differences remain hidden.

If you skip incremental rollout, you discover problems at full blast radius.

Each step reduces a different kind of uncertainty.

Conclusion

Modernizing a legacy application doesn't require replacing everything at once.

In many cases, the safer strategy is to create a path where old and new implementations can coexist temporarily.

Move one capability, then compare it.

Route a small amount of traffic and observe what happens.

Increase traffic when the evidence supports it, and roll back when it doesn't.

Transfer ownership explicitly, then remove the legacy path.

And repeat.

The full workflow becomes:

Understand
↓
Characterize
↓
Refactor
↓
Migrate incrementally
↓
Compare behavior
↓
Progressively route traffic
↓
Observe
↓
Remove legacy

AI can make every stage faster.

It can help map code, identify dependencies, generate adapters, compare implementations, analyze failures, and inspect migration diffs.

But speed isn't the same as confidence.

The important decisions still require engineering judgment:

What behavior matters?

What can change?

What should remain compatible?

What is the migration boundary?

What evidence is enough?

When is rollback necessary?

When can the legacy path be removed?

Those aren't code-generation questions. They're migration decisions.

And that's the larger lesson behind this entire series.

AI makes it increasingly cheap to produce new code. But that doesn't make legacy modernization trivial. It makes the quality of the decisions around the code more important.

Because the safest migration is rarely the one that changes the most software. It's the one that lets you change the system while continuously knowing what changed, why it changed, and whether it's safe to keep going.

]]> <![CDATA[ How to Use Differential Testing During a Legacy Migration ]]> https://www.freecodecamp.org/news/differential-testing-legacy-migration/ 6aa8200e59dfce663a0e73bd Mon, 14 Sep 2026 16:25:50 +0000 The most dangerous moment in a legacy migration isn't necessarily when you start writing the new implementation. It's when the new implementation looks finished.

The code compiles, the tests pass, the architecture is cleaner, and the new service responds faster.

Then everybody starts asking the same question:

Can we switch traffic now?

That's where confidence becomes difficult.

A new implementation can pass its own test suite and still behave differently from the system it is replacing.

Maybe rounding changed, or null values are handled differently, or an error became a successful response.

Maybe records are sorted differently, or a side effect happens in a different order, or a business rule you never documented was lost during the migration.

This is why, during a legacy migration, I like having another source of evidence: run the old and new implementations with the same inputs and compare what they do.

That's the basic idea behind differential testing. Instead of asking only if the new system passes its tests, you also ask: given the same input, where does the new system behave differently from the old one?

Those differences become evidence.

Some are bugs, some are intentional improvements, some are harmless representation differences, and some reveal behavior nobody knew existed.

In this tutorial, I'll show you how to use differential testing during a legacy migration to:

  • Compare old and new implementations

  • Define what should be considered equivalent

  • Normalize outputs before comparing them

  • Handle timestamps and other nondeterministic values

  • Compare errors and side effects

  • Run differential tests automatically

  • Introduce tolerances where exact equality doesn't make sense

  • Analyze mismatches

  • Use shadow traffic in production safely

  • Use AI to classify divergences without letting it decide correctness

  • Determine when the new implementation is ready for cutover

The examples use TypeScript and Vitest, but the approach applies to most languages and migration strategies.

The goal isn't to prove that two implementations are internally identical. It's to obtain evidence that they are behaviorally equivalent where equivalence matters.

Prerequisites

To follow along here, you should be comfortable with:

  • TypeScript or a similar language

  • unit and integration testing

  • asynchronous code

  • API and service boundaries

  • legacy modernization

  • basic observability concepts

You should also already have some understanding of the capability being migrated.

Ideally, you know its inputs, outputs, important business rules, external contracts, side effects, and known areas of uncertainty.

Differential testing works best after you've already created a boundary around the capability you want to migrate.

Table of Contents

What Differential Testing Actually Tells You

Imagine that your legacy application calculates the final price of an order.

The legacy implementation looks like this:

type Order = {
  subtotal: number;
  customerType: "STANDARD" | "PREMIUM";
  country: string;
};

function legacyCalculateTotal(order: Order): number {
  let total = order.subtotal;

  if (order.customerType === "PREMIUM") {
    total *= 0.9;
  }

  if (order.country === "AR") {
    total -= 500;
  }

  return Math.max(total, 0);
}

During the migration, you create a new implementation:

function newCalculateTotal(order: Order): number {
  const premiumDiscount =
    order.customerType === "PREMIUM"
      ? order.subtotal * 0.1
      : 0;

  const countryAdjustment =
    order.country === "AR"
      ? 500
      : 0;

  return Math.max(
    order.subtotal -
      premiumDiscount -
      countryAdjustment,
    0
  );
}

The implementations look different. And that's fine. What matters is whether they produce equivalent behavior.

A simple differential test can run both:

import { describe, expect, it } from "vitest";

describe("order total migration", () => {
  it("matches the legacy implementation", () => {
    const order: Order = {
      subtotal: 10000,
      customerType: "PREMIUM",
      country: "AR",
    };

    const legacy =
      legacyCalculateTotal(order);

    const migrated =
      newCalculateTotal(order);

    expect(migrated).toBe(legacy);
  });
});

For this input:

legacy → 8500
new    → 8500

Good. But one matching example proves very little.

The value comes from systematically asking:

same input
↓
legacy implementation ──→ result A

same input
↓
new implementation ─────→ result B

compare A and B

Every mismatch gives you something to investigate.

Start with One Observable Boundary

Don't begin by comparing entire applications. To start, choose one capability.

For example:

Calculate Order Total
Generate Invoice
Approve Customer
Renew Subscription
Calculate Commission
Create Shipment

Suppose the migration boundary is:

interface OrderProcessor {
  process(order: Order): Promise<ProcessedOrder>;
}

Now you have two implementations:

LegacyOrderProcessor

NewOrderProcessor

That is a useful differential boundary, because both receive the same conceptual input, and both produce the same conceptual output.

You can compare them without requiring their internal architecture to match.

That matters because migrations often change structure intentionally.

The legacy implementation might be:

controller
→ service
→ SQL
→ provider SDK

while the new implementation might be:

use case
→ repository
→ gateway
→ events

Differential testing shouldn't care. It should care about observable behavior.

Run the Legacy and New Implementations with the Same Input

Suppose both implementations expose:

interface OrderProcessor {
  process(order: Order): Promise<ProcessedOrder>;
}

You can create:

const legacyProcessor =
  new LegacyOrderProcessor();

const newProcessor =
  new NewOrderProcessor();

Then:

it("produces the same processed order", async () => {
  const input: Order = {
    id: "order-1",
    subtotal: 10000,
    customerType: "PREMIUM",
    country: "US",
  };

  const legacy =
    await legacyProcessor.process(
      structuredClone(input)
    );

  const migrated =
    await newProcessor.process(
      structuredClone(input)
    );

  expect(migrated).toEqual(legacy);
});

Notice the use of:

structuredClone(input)

That matters if either implementation mutates its input.

Without separate copies, the first execution could influence the second.

You want:

same initial state

not:

new implementation receives state modified by legacy implementation

That kind of contamination can create misleading results.

Don't Compare Raw Output Blindly

The first version of a differential test is often:

expect(newResult).toEqual(legacyResult);

Sometimes that's exactly right. But other times it's wrong.

Imagine the legacy system returns:

{
  "id": "order-1",
  "total": 9000,
  "status": "PROCESSED",
  "generatedAt": "2026-09-09T10:00:01.231Z",
  "requestId": "legacy-f93a"
}

The new system returns:

{
  "requestId": "new-b517",
  "status": "PROCESSED",
  "generatedAt": "2026-09-09T10:00:01.416Z",
  "total": 9000,
  "id": "order-1"
}

A raw object comparison may fail because:

requestId differs
timestamp differs

But the business behavior might be equivalent.

You need to decide which fields are part of the meaningful contract.

Maybe:

id
total
status

matter.

While:

generatedAt
requestId

don't need exact equivalence.

That leads to normalization.

Normalize Values Before Comparing Them

Normalization means transforming outputs into a common representation before comparing them.

The goal isn't to change the business meaning of the data. It's to remove differences that are expected and irrelevant to the comparison, such as generated request IDs or timestamps, so the test can focus on the fields that actually define the behavior you care about.

In practice, that often means creating a canonical representation: a smaller, stable shape that contains only the meaningful fields you want to compare.

For example:

type ProcessedOrder = {
  id: string;
  total: number;
  status: string;
  generatedAt: string;
  requestId: string;
};

function normalizeOrder(
  order: ProcessedOrder
) {
  return {
    id: order.id,
    total: order.total,
    status: order.status,
  };
}

Here, ProcessedOrder contains both business-relevant fields and values that may legitimately differ between executions.

The normalizeOrder() function keeps id, total, and status, while leaving out generatedAt and requestId. That means two results can still be considered equivalent even if they were generated at slightly different times or used different request identifiers.

Now compare:

expect(
  normalizeOrder(migrated)
).toEqual(
  normalizeOrder(legacy)
);

This makes your equivalence rule explicit.

You're saying:

These fields define relevant behavior for this comparison.

Normalization can also handle:

  • ordering

  • casing

  • optional fields

  • timestamps

  • generated identifiers

  • numeric formatting

  • provider-specific metadata

But normalization must be deliberate. If you remove too much, you can hide real migration bugs.

Handle Timestamps and Other Nondeterministic Values

Legacy systems contain many nondeterministic values.

For example:

timestamps
UUIDs
random tokens
request IDs
trace IDs
database-generated IDs
unordered collections
provider-generated references

If you compare those values exactly, your differential suite may fail constantly.

One option is dependency control.

Dependency control means moving a nondeterministic source, such as the current time or an ID generator, behind an interface that you can replace during tests.

Instead of letting each implementation read the real clock independently, you inject the same controlled clock into both. That gives them the same value and removes time itself as a source of meaningless divergence.

Suppose the code uses:

new Date()

You can replace that dependency with a clock:

interface Clock {
  now(): Date;
}

Then both implementations receive:

const clock = {
  now: () =>
    new Date(
      "2026-09-09T10:00:00.000Z"
    ),
};

Now time becomes deterministic.

The same technique can work for ID generation:

interface IdGenerator {
  next(): string;
}

Then tests can provide:

const ids = {
  next: () => "fixed-id",
};

If controlling nondeterminism is impractical, normalize it out only when it's not part of the behavior you need to protect.

Compare Business Meaning, Not Just JSON

Two systems can return different representations while expressing the same business state.

Imagine you have this in your legacy system:

{
  "status": 2
}

And this in your new one:

{
  "status": "APPROVED"
}

Raw comparison says:

different

Business comparison may say:

equivalent

You can create a semantic normalizer:

function normalizeStatus(
  status: number | string
) {
  if (status === 2) {
    return "APPROVED";
  }

  return status;
}

Here, the normalizer translates the legacy numeric value 2 into the business meaning used by the new implementation: "APPROVED".

It doesn't claim that every number and string are interchangeable. It encodes one explicit equivalence rule that you've already decided is valid for this migration.

Then:

expect(
  normalizeStatus(newResult.status)
).toBe(
  normalizeStatus(legacyResult.status)
);

This is especially useful when migration intentionally changes:

database schema
API representation
enumerations
provider-specific formats
internal identifiers

The important question becomes:

Does the observable business meaning remain equivalent?

Not:

Are the bytes identical?

Compare Errors as Part of the Contract

Success responses aren't the whole behavior. Errors matter too.

Suppose the legacy implementation rejects a missing customer:

throw new Error("Customer not found");

The new implementation accidentally returns:

return null;

These two implementations behave very differently for the same invalid input.

The legacy version fails explicitly, while the new version silently returns a value that a caller may interpret as a successful result.

If your differential tests only exercise cases where a valid customer exists, both implementations may appear equivalent and this contract change will remain invisible.

That's why failure behavior has to be compared too.

Create cases that capture errors:

async function captureResult<T>(
  operation: () => Promise<T>
) {
  try {
    return {
      type: "success" as const,
      value: await operation(),
    };
  } catch (error) {
    return {
      type: "error" as const,
      error:
        error instanceof Error
          ? error.message
          : String(error),
    };
  }
}

The helper wraps an asynchronous operation and converts both possible outcomes into data.

If the operation succeeds, it returns an object with type: "success" and the returned value. If the operation throws, the catch block converts that exception into an object with type: "error" and a readable error message.

This gives both implementations the same comparison shape, so the test can compare success versus failure explicitly instead of letting an exception stop the test before the two behaviors can be evaluated.

Now:

const legacy =
  await captureResult(() =>
    legacyProcessor.process(input)
  );

const migrated =
  await captureResult(() =>
    newProcessor.process(input)
  );

expect(migrated.type).toBe(legacy.type);

If errors are contractually important, compare:

error category
HTTP status
error code
retryability
validation details

Don't necessarily compare exact wording unless clients depend on it.

Compare Side Effects, Too

One of the easiest migration mistakes is preserving the return value while losing a side effect.

Suppose both implementations return:

{
  "status": "PROCESSED"
}

But the legacy version also:

persists the order
publishes an event
creates a payment
writes an audit entry

and the new version forgets the audit entry.

Response-level differential testing won't catch that. So you'll want to capture side effects.

For example:

type Effect =
  | {
      type: "payment";
      orderId: string;
      amount: number;
    }
  | {
      type: "event";
      name: string;
      orderId: string;
    };

A test adapter can record them:

class RecordingPaymentGateway {
  effects: Effect[] = [];

  async charge(
    orderId: string,
    amount: number
  ) {
    this.effects.push({
      type: "payment",
      orderId,
      amount,
    });
  }
}

Instead of sending a real payment request, this adapter records what the application attempted to do in the effects array.

You can apply the same idea to event publication:

class RecordingEvents {
  effects: Effect[] = [];

  async publish(
    name: string,
    orderId: string
  ) {
    this.effects.push({
      type: "event",
      name,
      orderId,
    });
  }
}

The application still calls its payment and event dependencies as usual. The test doubles simply capture those calls as structured data instead of performing the real external actions.

After running the legacy and migrated implementations with their own recording adapters, you can compare the two recorded effect lists and verify that both systems attempted the same observable side effects.

Now the differential test can compare:

expect(newEffects).toEqual(legacyEffects);

Again, exact ordering should only be required if ordering matters.

Use Tolerances When Exact Equality Is Wrong

Some domains shouldn't use exact equality.

Imagine a migrated calculation produces:

legacy → 34.333333333
new    → 34.333333334

Is that a migration bug? Maybe not.

Floating-point calculations may justify a tolerance.

For example:

expect(newResult).toBeCloseTo(
  legacyResult,
  6
);

Or define an explicit comparator:

function withinTolerance(
  a: number,
  b: number,
  tolerance: number
) {
  return Math.abs(a - b) <= tolerance;
}

Then:

expect(
  withinTolerance(
    migrated.total,
    legacy.total,
    0.01
  )
).toBe(true);

But tolerances should come from domain requirements. Don't use them just to make failing tests disappear.

For financial systems, one cent can matter. For scientific calculations, a much smaller numerical difference may matter.

Equivalence is a business and engineering decision.

Build a Reusable Differential Test Harness

Once you compare more than a few cases, you can create a reusable harness.

For example:

type DifferentialResult<T> = {
  input: T;
  equivalent: boolean;
  legacy: unknown;
  migrated: unknown;
};

async function compareImplementations<
  TInput,
  TOutput
>(
  input: TInput,
  legacy: (
    input: TInput
  ) => Promise<TOutput>,
  migrated: (
    input: TInput
  ) => Promise<TOutput>,
  normalize: (
    output: TOutput
  ) => unknown
): Promise<
  DifferentialResult<TInput>
> {
  const legacyResult =
    await legacy(
      structuredClone(input)
    );

  const migratedResult =
    await migrated(
      structuredClone(input)
    );

  const normalizedLegacy =
    normalize(legacyResult);

  const normalizedMigrated =
    normalize(migratedResult);

  return {
    input,
    equivalent:
      JSON.stringify(
        normalizedLegacy
      ) ===
      JSON.stringify(
        normalizedMigrated
      ),
    legacy: normalizedLegacy,
    migrated: normalizedMigrated,
  };
}

The harness does four things.

First, it runs the legacy and migrated implementations with separate clones of the same input, so one execution can't mutate the data seen by the other.

Second, it passes both outputs through the same normalize() function. That applies the equivalence rules in one place instead of repeating them in every test.

Third, it compares the normalized results and records whether they're equivalent.

Finally, it returns the input and both normalized outputs together. That makes a failed comparison easier to inspect because the test report can show exactly which case diverged and what each implementation produced.

Then:

const result =
  await compareImplementations(
    input,
    legacyProcessor.process.bind(
      legacyProcessor
    ),
    newProcessor.process.bind(
      newProcessor
    ),
    normalizeOrder
  );

expect(result.equivalent).toBe(true);

For real systems, I would usually avoid relying on JSON.stringify() as the final equality mechanism.

The example keeps the harness readable.

In production-quality tooling, use a proper structural or domain-specific comparator.

The important part is that comparison logic becomes centralized.

Generate Test Cases from Real Behavior

Hand-written examples are useful. But migrations often fail on cases nobody thought to write manually.

Useful sources of inputs include:

existing test fixtures
historical incidents
production-safe request samples
database records
boundary values
previous bug reports
known customer scenarios

Suppose production shows these order shapes:

const cases: Order[] = [
  {
    subtotal: 0,
    customerType: "STANDARD",
    country: "US",
  },
  {
    subtotal: 500,
    customerType: "PREMIUM",
    country: "AR",
  },
  {
    subtotal: 10000,
    customerType: "STANDARD",
    country: "AR",
  },
];

The first block is the test data. It captures a small set of representative input shapes that you've observed in real usage or reconstructed safely from production behavior.

The next block is the test itself. it.each(cases) tells Vitest to run the same differential comparison once for every input in that array.

That separates two concerns: defining realistic cases and defining how every case should be evaluated.

Now:

it.each(cases)(
  "matches legacy behavior",
  async (input) => {
    const legacy =
      await legacyProcessor.process(
        structuredClone(input)
      );

    const migrated =
      await newProcessor.process(
        structuredClone(input)
      );

    expect(
      normalizeOrder(migrated)
    ).toEqual(
      normalizeOrder(legacy)
    );
  }
);

Real examples help expose assumptions that synthetic test data often misses. But production data must be handled carefully.

Remove or anonymize:

personal data
credentials
tokens
financial identifiers
confidential business data

The objective is to preserve useful behavioral shapes, not copy sensitive production information into test fixtures.

Classify Every Difference

A differential failure doesn't automatically mean that the new implementation is wrong.

Suppose you find 200 mismatches. Classify them.

I like categories such as:

migration defect
legacy defect intentionally preserved
intentional behavior change
representation difference
nondeterministic difference
test/comparator defect
unknown

For example:

Input:
subtotal = 5000

Legacy:
discount = 0

New:
discount = 500

Classification:
unknown

Investigation reveals that the new implementation changed:

amount > 5000

to:

amount >= 5000

Now you need a decision.

Was that:

accidental migration change

or:

intentional bug fix

Differential testing exposes the decision. It doesn't make the decision for you.

That's one of its greatest benefits.

How to Use AI to Investigate Differential Failures

Large migrations can produce hundreds or thousands of differences. And AI can help triage them.

Suppose you have:

{
  "input": {
    "subtotal": 5000,
    "country": "AR"
  },
  "legacy": {
    "total": 4500
  },
  "new": {
    "total": 4000
  }
}

You can give the model:

  • the input

  • both outputs

  • relevant legacy code

  • relevant migrated code

  • the comparator rules

Then ask:

Analyze this differential test failure.

Identify the smallest behavioral difference that could
explain the mismatch.

Compare the legacy and migrated implementations.

Return:

1. observed difference,
2. relevant legacy branch,
3. relevant migrated branch,
4. likely cause,
5. evidence supporting the cause,
6. additional test cases that could confirm it.

Do not decide which behavior is correct.
Do not modify the code yet.

That last instruction matters. AI can be very useful for locating why two implementations diverge. It shouldn't silently turn that diagnosis into a business decision.

Don't Let AI Decide Which Behavior Is Correct

Imagine the legacy system does this:

Customer age 65 → no discount
Customer age 66 → discount

The new system does:

Customer age 65 → discount
Customer age 66 → discount

AI may look at the code and say:

The new implementation appears more logical because senior discounts typically begin at age 65.

That's irrelevant.

The business rule might be:

age > 65

for a reason. Or the legacy behavior might contain a bug.

You need evidence.

Use:

requirements
existing tests
production behavior
business owners
historical tickets
commit history
contracts

AI can help gather and summarize that evidence. It shouldn't invent the rule.

Differential testing is valuable because it tells you that there's a difference before you accidentally turn that difference into production behavior.

How to Use Shadow Traffic Safely

Once offline differential tests look good, you can sometimes compare behavior with real traffic. This is often called shadowing or traffic mirroring.

The pattern looks like:

real request
    │
    ├────────────→ legacy system
    │                  │
    │                  ↓
    │             real response
    │
    └────────────→ new system
                       │
                       ↓
                  shadow result

The user still receives:

legacy response

while the new system processes a copy of the request.

Then you compare:

legacy output
vs.
shadow output

This can reveal cases that your test suite never captured.

For example:

unexpected null combinations
rare customer states
unusual international data
old records
large values
unusual sequence patterns

But shadow execution requires careful design, especially when the operation has side effects.

How to Prevent Shadow Execution from Duplicating Side Effects

Imagine shadowing:

POST /payments

If both systems really execute the payment, you have a serious problem.

The same applies to:

send email
create shipment
charge card
modify inventory
publish event
write external record

The shadow implementation shouldn't perform destructive or externally visible effects unless they're safely isolated.

One approach is to replace real gateways with recording adapters:

class ShadowPaymentGateway
  implements PaymentGateway {
  calls: PaymentRequest[] = [];

  async charge(
    request: PaymentRequest
  ) {
    this.calls.push(request);

    return {
      paymentId: "shadow",
    };
  }
}

The new implementation still tries to execute:

payment

but instead of charging a real card, the shadow adapter records:

what would have been sent

You can then compare that intent with the legacy side effect.

This distinction is important:

compare behavior

does not mean:

duplicate production effects

Measure Divergence Instead of Waiting for Perfection

When running thousands of comparisons, a binary:

pass / fail

may not tell the whole story.

You can measure divergence.

For example:

Requests compared:     100,000
Equivalent:             99,620
Different:                 380

Divergence rate:          0.38%

Then classify those 380:

250 timestamp differences
80 known intentional changes
30 comparator problems
15 migration defects fixed
5 still unexplained

After normalization:

meaningful unresolved divergence:
5 / 100,000
= 0.005%

Now the conversation becomes much more concrete.

Instead of:

I think the migration is ready.

you can say:

We compared 100,000 representative executions and have five unresolved behavioral differences.

Whether that's acceptable depends on what those five cases are.

One incorrect financial transaction can matter more than 100 harmless formatting differences.

So don't evaluate only the percentage. Evaluate the severity.

How to Know When You're Ready for Cutover

Differential testing doesn't give you a universal threshold. But it can give you evidence.

Before cutover, I would want to answer questions such as:

Have Important Input Classes Been Compared?

Not only happy paths.

Include:

boundaries
errors
historical bugs
large values
missing values
rare states

Are Meaningful Differences Classified?

Avoid:

we have 47 unexplained mismatches

Are Critical Differences Resolved?

Especially:

money
authorization
state transitions
data integrity
external contracts
idempotency

Are Intentional Differences Documented?

If the new behavior intentionally differs, that should be explicit.

Are Side Effects Equivalent?

Not only responses.

Have Production-like Cases Been Tested?

Synthetic fixtures alone may not be enough.

Can the Migration Be Rolled Back?

Differential confidence reduces risk. It doesn't eliminate the need for rollback.

If you can answer these questions, you're much closer to a controlled cutover.

A Practical Differential Testing Workflow

Here's the workflow I would use.

1. Pick One Capability

For example:

Process Order
Calculate Invoice
Approve Customer

Don't compare the whole platform at once.

2. Define the Observable Contract

List what matters:

return value
status
error
database state
events
external calls

3. Create Legacy and New Adapters

Expose both implementations through the same conceptual interface.

4. Define Normalization Rules

Decide how to handle:

timestamps
generated IDs
ordering
representation changes
optional values

Do this before looking at lots of failures. Otherwise you may weaken the comparator simply to make results pass.

5. Compare Known Cases

Begin with:

existing tests
characterization cases
edge cases
historical bugs

6. Capture Side Effects

Use recording or fake adapters where necessary.

7. Automate the Harness

Produce structured output for every mismatch.

For example:

{
  "caseId": "case-493",
  "equivalent": false,
  "legacy": {},
  "migrated": {},
  "difference": {}
}

8. Classify Differences

Use categories:

defect
intentional change
normalization issue
nondeterminism
unknown

9. Add Representative Real-World Cases

Use anonymized or safely reconstructed production patterns.

10. Shadow Real Traffic When Appropriate

Only after controlling side effects and privacy risk.

11. Measure Divergence

Track both:

frequency
severity

12. Resolve Unknowns Before Cutover

The most dangerous category is often not:

different

It is:

different and nobody knows why

What Differential Testing Can't Prove

Differential testing has an important limitation: it compares the new system against the old one.

That means the legacy system becomes a behavioral reference. But the legacy system may already be wrong.

Suppose:

legacy output = wrong
new output    = same wrong result

The differential test passes, but that doesn't make the behavior correct.

This is why differential testing should complement:

specification tests
characterization tests
business requirements
security testing
performance testing
contract testing
domain review

It answers:

Did behavior change?

It doesn't automatically answer:

Is this the right behavior?

That distinction matters. The legacy application is evidence, it's not absolute truth.

Differential Testing Turns Migration Risk into Evidence

There's another reason I like this technique. Without differential testing, migration discussions can become subjective.

One person says:

The new implementation looks ready.

Another says:

I do not trust it yet.

Both may have reasonable instincts, but neither statement is very measurable.

Differential testing changes the conversation.

Now you can say:

12,000 cases compared
47 differences found
31 representation differences
9 intentional behavior changes
6 migration defects fixed
1 unresolved

That is a much better engineering discussion. You're converting uncertainty into observable differences. Then you can decide what to do with them.

Conclusion

A legacy migration is not complete because the new implementation passes its own tests.

The harder question is whether it preserves the behavior that matters from the system it is replacing.

Differential testing gives you another way to answer that question.

Run both implementations with the same inputs.

Compare outputs.

Compare errors.

Compare side effects.

Normalize only the differences that truly do not matter.

Investigate everything else.

And when possible, use representative production behavior to discover cases your test suite did not anticipate.

The migration sequence now becomes:

Understand
↓
Characterize
↓
Refactor
↓
Migrate
↓
Compare
↓
Cut over

AI can accelerate this process too.

It can help build comparators, analyze failures, group similar divergences, inspect code paths, and suggest additional test cases.

But it should not decide which implementation is correct.

That still requires evidence, domain knowledge, and engineering judgment.

The purpose of differential testing is not to eliminate uncertainty completely.

It is to make uncertainty visible before you switch production traffic.

Because during a migration, discovering that the new system behaves differently is useful.

Discovering it after the old system has been turned off is much more expensive.

]]> <![CDATA[ How to Build a Self-Evaluating AI System: Automated Testing and Evaluation Pipelines for LLM Applications ]]> https://www.freecodecamp.org/news/build-a-self-evaluating-ai-system-automated-testing-and-evaluation-pipelines-for-llm-apps/ 6aa41d147411afb20c713931 Fri, 11 Sep 2026 15:24:04 +0000 So you shipped your AI feature and it works in demos. Your team is impressed. Then a user asks a question slightly outside your test cases and the model confidently returns something completely wrong.

The truth about building with Large Language Models is that traditional software testing falls apart. You can't write a simple assert output ==expected when your system generates different text every time it runs.

Most tutorials out there will teach you how to build a chatbot or wire up a RAG pipeline and then they just...stop. "Deploy to production" they say, as if the hard part is over. But the hard part is actually knowing whether your AI is any good and catching it when it stops being good.

In this article, I'll walk you through building a complete evaluation pipeline. We'll also cover three different evaluation strategies that work at different levels of cost and depth.

What We'll Cover:

What You'll Need

To follow along, you should have Python 3.10+ and some basic experience calling an LLM API. It doesn't matter if you're using OpenAI, Anthropic, or a local model because the evaluation patterns work the same way.

You'll also need an OpenAI API key for the LLM-as-judge examples (we're using gpt-4o-mini since it's cheap and good enough for scoring).

If you already have an LLM-powered app you want to evaluate, even a tiny one, that's perfect. If not, the examples are self-contained so you can still follow everything.

Grab the dependencies here:

pip install openai numpy pandas scikit-learn python-dotenv

Why Traditional Testing Breaks Down for LLM Applications

If you've written tests for regular software, you know the drill. Function goes in, value comes out, you assert they match. Clean, simple, and done.

But LLMs break that entire model. And not in one way, but in several that compound on each other.

First, the outputs aren't deterministic. You can send the exact same prompt twice and get back different wording. Even setting temperature=0 doesn't fully save you because model providers update their models behind the scenes. The same API call in January and March might behave differently.

Second, there's no single right answer. If your app summarizes a document, what does a correct summary even look like? Two humans would write different summaries and both could be perfectly good. You can't assertEqual your way through that.

And third, nothing breaks visibly and there's no error, crash, or red line in your logs. The model just quietly returns a polished, confident wrong answer. Your uptime dashboard says 100% while your users are getting nonsense. This is the one that really gets you when an LLM fails.

So you can't just test LLM apps the way you test a REST API. You need scoring instead of pass/fail. You need to evaluate batches of outputs not individual ones. And you need something that runs continuously because the quality can drift over time without you changing a single line of code.

The Three Layers of LLM Evaluation

The approach I've landed on after a lot of trial and error uses three layers stacked from cheap-and-fast to expensive-and-thorough.

  1. Layer 1 is deterministic checks. Think of these as bouncers at the door. Is the output valid JSON when it should be? Is it suspiciously short or absurdly long? Does it contain a hallucinated URL? These checks are instant, free and catch more problems than you'd expect.

  2. Layer 2 is LLM-as-judge. This is where you use a separate LLM call to grade your main LLM's output. "Was this answer relevant? Was it accurate? Did it actually help?" A model like gpt-4o-mini is surprisingly good at scoring other models' work as long as you give it a clear rubric.

  3. Layer 3 is human evaluation. Real people reviewing real outputs. You don't do this on every response, as that would be impossibly slow. But you do it periodically, to make sure your automated layers haven't drifted away from what good actually means.

The trick is knowing when to use which layer, and we'll build each one of them.

The Three Layers of LLM Evaluation: Deterministic checks, LLM-as-Judge, and Human Evaluation

How to Build Layer 1: Deterministic Checks

When I first started building eval pipelines, I skipped straight to the fancy stuff: LLM judges, embedding similarity scores, the works. Meanwhile, my app was occasionally returning completely empty strings and I didn't notice for two weeks. Two weeks!

That's why I now start every project with deterministic checks. They're dead simple: no ML and no API calls, just plain Python asking basic sanity questions about the output. Does it exist? Is it the right format? Is it suspiciously short? Did the model hallucinate a URL?

You might be thinking these are too basic to matter. I thought so too. Then I ran them on a month of production logs and found that roughly a third of the bad outputs I'd missed would've been caught by checks you could write in five minutes.

Here's the DeterministicEvaluator class I now drop into every project on day one:

import json
import re
from dataclasses import dataclass


@dataclass
class EvalResult:
    """Holds the result of a single evaluation check."""
    check_name: str
    passed: bool
    score: float  # 0.0 to 1.0
    details: str


class DeterministicEvaluator:
    """Layer 1: Fast, rule-based checks for LLM outputs."""

    def check_json_validity(self, output: str) -> EvalResult:
        """Verify the output is valid JSON when JSON is expected."""
        try:
            json.loads(output)
            return EvalResult("json_validity", True, 1.0, "Valid JSON")
        except json.JSONDecodeError as e:
            return EvalResult("json_validity", False, 0.0, f"Invalid JSON: {e}")

    def check_length_bounds(
        self, output: str, min_chars: int = 10, max_chars: int = 5000
    ) -> EvalResult:
        """Check that output length falls within acceptable bounds."""
        length = len(output)
        if length < min_chars:
            return EvalResult(
                "length_bounds", False, 0.0,
                f"Too short: {length} chars (minimum: {min_chars})"
            )
        if length > max_chars:
            return EvalResult(
                "length_bounds", False, 0.0,
                f"Too long: {length} chars (maximum: {max_chars})"
            )
        return EvalResult("length_bounds", True, 1.0, f"Length OK: {length} chars")

    def check_no_hallucinated_links(self, output: str) -> EvalResult:
        """Detect URLs in output that the model may have fabricated."""
        url_pattern = r'https?://[^\s\)\]\}\"\'<>]+'
        urls = re.findall(url_pattern, output)
        if urls:
            return EvalResult(
                "no_hallucinated_links", False, 0.0,
                f"Found {len(urls)} URLs that may be hallucinated: {urls[:3]}"
            )
        return EvalResult("no_hallucinated_links", True, 1.0, "No URLs found")

    def check_required_sections(
        self, output: str, required: list[str]
    ) -> EvalResult:
        """Verify that required sections or keywords appear in the output."""
        missing = [s for s in required if s.lower() not in output.lower()]
        if missing:
            score = 1.0 - (len(missing) / len(required))
            return EvalResult(
                "required_sections", False, score,
                f"Missing sections: {missing}"
            )
        return EvalResult("required_sections", True, 1.0, "All sections present")

    def check_no_refusal(self, output: str) -> EvalResult:
        """Detect if the model refused to answer when it should not have."""
        refusal_phrases = [
            "i cannot", "i can't", "i'm unable to", "as an ai",
            "i don't have access", "i'm not able to"
        ]
        output_lower = output.lower()
        for phrase in refusal_phrases:
            if phrase in output_lower:
                return EvalResult(
                    "no_refusal", False, 0.0,
                    f"Possible refusal detected: '{phrase}'"
                )
        return EvalResult("no_refusal", True, 1.0, "No refusal detected")

    def run_all(self, output: str, config: dict = None) -> list[EvalResult]:
        """Run all deterministic checks and return results."""
        config = config or {}
        results = [
            self.check_length_bounds(
                output,
                config.get("min_chars", 10),
                config.get("max_chars", 5000)
            ),
            self.check_no_hallucinated_links(output),
            self.check_no_refusal(output),
        ]
        if config.get("expect_json"):
            results.append(self.check_json_validity(output))
        if config.get("required_sections"):
            results.append(
                self.check_required_sections(output, config["required_sections"])
            )
        return results


if __name__ == "__main__":
    evaluator = DeterministicEvaluator()

    # Test with a normal output
    good_output = "Python is a high-level programming language known for its readability."
    results = evaluator.run_all(good_output)
    for r in results:
        print(f"  {r.check_name}: {'PASS' if r.passed else 'FAIL'} ({r.details})")

    # Test with a suspicious output
    bad_output = "Visit https://fake-docs.example.com/api for more details."
    results = evaluator.run_all(bad_output)
    for r in results:
        print(f"  {r.check_name}: {'PASS' if r.passed else 'FAIL'} ({r.details})")

Every one of these checks runs in under a millisecond and they cost nothing. But don't let the simplicity fool you because the hallucinated links check alone has saved me from shipping fabricated documentation URLs to users more times than I'd like to admit.

Also one thing worth stressing is that these are starting points. The generic checks above work for any LLM app. But the biggest wins come from domain-specific ones. If your app generates SQL, add a syntax parser. If it drafts emails, verify that there's a subject line and a greeting. If it outputs code, try running it through a linter.

Every check you add here is one fewer bad output that reaches the expensive layers downstream or worse, your users.

How to Build Layer 2: LLM-as-Judge Evaluation

Alright, so your output passes the sanity checks: it's valid JSON, reasonable length, no fabricated links. But here's a question Layer 1 can't answer: is the response actually helpful?

An output can be perfectly structured, pass every deterministic check, and still be completely useless to the person reading it. "The capital of France is Berlin" is valid text, correct length, no hallucinated URLs...but it's also wrong.

This is where things get a little meta. The idea behind LLM-as-judge is that you make a separate LLM call whose only job is to read your main model's output and score it. Yes, you're using AI to grade AI. It sounds like asking one student to grade another student's homework. But it actually works surprisingly well, and research from labs like Anthropic and Google have shown that LLM judges correlate strongly with human evaluators when you give them clear scoring criteria.

The key phrase there is "clear scoring criteria." Without that, this whole approach falls apart.

How to Design Scoring Rubrics

If you tell an LLM "rate this from 1 to 10," you'll get back scores that are all over the place. A 7 on one run becomes a 5 on the next. The scores are essentially meaningless because the model has no shared definition of what each number means.

The fix is a rubric with concrete anchor descriptions. Here's one for helpfulness.

Score 1 - The response is completely irrelevant, incorrect, or harmful.
Score 2 - The response addresses the topic but contains major errors or omissions.
Score 3 - The response is partially correct but misses key information.
Score 4 - The response is correct and helpful with minor issues.
Score 5 - The response is comprehensive, accurate, and directly addresses the question.

Now notice how each level describes something you could point to in the output, not a vibe. "Completely off-topic" is observable. "Kind of bad" is not. That specificity is what makes the judge consistent across runs.

How to Implement the Judge

Here's the full LLMJudge class. I'll walk through the important design decisions after.

import json
import os
from openai import OpenAI
from dataclasses import dataclass

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))


@dataclass
class JudgeResult:
    """Holds the result of an LLM judge evaluation."""
    criterion: str
    score: int
    max_score: int
    reasoning: str


RUBRICS = {
    "relevance": {
        "description": "Does the response directly address the user's question?",
        "levels": {
            1: "Completely off-topic or addresses a different question entirely.",
            2: "Tangentially related but misses the core question.",
            3: "Addresses the question but includes significant irrelevant content.",
            4: "Directly addresses the question with minor tangents.",
            5: "Precisely and completely addresses the question asked.",
        },
    },
    "accuracy": {
        "description": "Is the factual content of the response correct?",
        "levels": {
            1: "Contains critical factual errors that would mislead the reader.",
            2: "Multiple factual errors on important points.",
            3: "Mostly accurate but contains one notable error.",
            4: "Accurate with only trivial imprecisions.",
            5: "Completely accurate with no factual errors.",
        },
    },
    "completeness": {
        "description": "Does the response cover all important aspects of the question?",
        "levels": {
            1: "Addresses less than 20 percent of what the question requires.",
            2: "Covers some aspects but misses major required components.",
            3: "Covers the basics but lacks depth on important points.",
            4: "Comprehensive coverage with minor gaps.",
            5: "Thoroughly covers all aspects the question requires.",
        },
    },
}


class LLMJudge:
    """Layer 2: Uses a separate LLM to evaluate response quality."""

    def __init__(self, model: str = "gpt-4o-mini"):
        self.model = model

    def evaluate(
        self, question: str, response: str, criterion: str
    ) -> JudgeResult:
        """Evaluate a single response on a single criterion."""
        rubric = RUBRICS[criterion]
        levels_text = "\n".join(
            f"Score {score}: {desc}"
            for score, desc in rubric["levels"].items()
        )

        judge_prompt = f"""You are an expert evaluator. Your job is to score an AI assistant's response.

CRITERION: {rubric['description']}

SCORING RUBRIC:
{levels_text}

USER QUESTION:
{question}

AI RESPONSE:
{response}

Evaluate the response on the criterion above. You must respond with valid JSON only:
{{"score": <integer 1-5>, "reasoning": "<2-3 sentence explanation>"}}"""

        judge_response = client.chat.completions.create(
            model=self.model,
            messages=[{"role": "user", "content": judge_prompt}],
            temperature=0.0,
            response_format={"type": "json_object"},
        )

        result = json.loads(judge_response.choices[0].message.content)
        return JudgeResult(
            criterion=criterion,
            score=result["score"],
            max_score=5,
            reasoning=result["reasoning"],
        )

    def evaluate_all(
        self, question: str, response: str, criteria: list[str] = None
    ) -> list[JudgeResult]:
        """Evaluate a response across all specified criteria."""
        criteria = criteria or list(RUBRICS.keys())
        return [self.evaluate(question, response, c) for c in criteria]


if __name__ == "__main__":
    judge = LLMJudge()

    question = "What is a Python decorator and when should you use one?"
    good_response = (
        "A Python decorator is a function that takes another function as input "
        "and extends its behavior without modifying it. You define a decorator "
        "with the @decorator_name syntax above a function definition. Use "
        "decorators when you need to add cross-cutting concerns like logging, "
        "authentication checks, or caching to multiple functions without "
        "duplicating code in each one."
    )

    results = judge.evaluate_all(question, good_response)
    for r in results:
        print(f"  {r.criterion}: {r.score}/{r.max_score} - {r.reasoning}")

There are a few things worth calling out in this code.

  1. Temperature is zero: You're not asking the judge to be creative. You want the same input to produce the same score every time, or as close to it as possible.

  2. The output is structured JSON: I learned this one the hard way. If you let the judge respond in free text, you end up writing fragile parsing code to extract the score. Force JSON output and your life gets much easier.

  3. The rubric is baked into every prompt: The judge never uses its own idea of what good means. It always scores against your rubric and that's what makes it reproducible.

How to Handle Judge Reliability

Even with all of that, a single judge call can be noisy. I've seen the same response score a 4 on one call and a 3 on the next. If you're making decisions based on those scores, that variance matters. Two things can help you with that.

The first is multi-judge consensus. This means you run the same evaluation three times and take the median. Yes, it costs 3x as much. But the scores become much more stable, and for CI/CD gating decisions, stability matters more than saving a few cents.

The second is calibration sets. You keep a small set of responses (maybe 20-30) where you already have reliable human scores. Run your judge on these periodically. If the judge starts disagreeing with the humans, something changed and you need to investigate.

We can look at this consensus implementation that shows how to handle that:

import numpy as np


def evaluate_with_consensus(
    judge: LLMJudge,
    question: str,
    response: str,
    criterion: str,
    num_judges: int = 3,
) -> JudgeResult:
    """Run multiple judge evaluations and return the median."""
    results = [
        judge.evaluate(question, response, criterion)
        for _ in range(num_judges)
    ]
    scores = [r.score for r in results]
    median_score = int(np.median(scores))
    median_result = min(results, key=lambda r: abs(r.score - median_score))
    return JudgeResult(
        criterion=criterion,
        score=median_score,
        max_score=5,
        reasoning=f"Consensus ({scores}): {median_result.reasoning}",
    )

How to Build Layer 3: Human Evaluation Loops

I once had an LLM judge giving a response 5/5 on accuracy, 5/5 on relevance, 4/5 on completeness. The scores looked perfect until a colleague actually read the response and said, "This is technically correct but it would confuse the hell out of anyone who isn't already an expert." And he was right.

The answer used jargon the user wouldn't know, buried the key point three paragraphs deep, and read like a textbook instead of a helpful reply.

That's the ceiling of automated evaluation. LLM judges are great at detecting factual errors and structural problems, but they have blind spots around tone, clarity for a specific audience, and the subtle difference between "correct" and "actually helpful." Those blind spots are where human evaluation comes in.

Now, to be clear, this doesn't mean hiring a team to review every single response. That doesn't scale and you don't need it. The goal is narrower: get a small batch of human scores on a regular schedule and use those scores as a reality check on your automated layers.

How to Build a Lightweight Annotation Interface

You really don't need Label Studio or some fancy annotation platform for this. You only need a Python script that shows a response and asks for a score.

Here's how this works at a high level: the script takes a question-response pair, displays it in the terminal, asks the reviewer to score it on a 1-5 scale, and saves the result to a file. Each annotation gets stored as a single line of JSON called JSONL format which makes it easy to load back later, run analysis on, or feed into a dashboard.

import json
import random
from pathlib import Path
from dataclasses import dataclass, asdict


@dataclass
class Annotation:
    """A single human annotation for an LLM response."""
    question: str
    response: str
    annotator: str
    score: int
    notes: str


class AnnotationCollector:
    """Collects and stores human evaluations."""

    def __init__(self, output_file: str = "annotations.jsonl"):
        self.output_path = Path(output_file)

    def collect_annotation(
        self, question: str, response: str, annotator: str
    ) -> Annotation:
        """Present a question-response pair and collect a human score."""
        print("\n" + "=" * 60)
        print(f"QUESTION: {question}")
        print("-" * 60)
        print(f"RESPONSE: {response}")
        print("-" * 60)
        print("Score this response (1-5):")
        print("  1 = Terrible  2 = Poor  3 = Acceptable  4 = Good  5 = Excellent")

        while True:
            try:
                score = int(input("Score: "))
                if 1 <= score <= 5:
                    break
                print("Please enter a number between 1 and 5.")
            except ValueError:
                print("Please enter a valid number.")

        notes = input("Notes (optional, press Enter to skip): ").strip()

        annotation = Annotation(
            question=question,
            response=response,
            annotator=annotator,
            score=score,
            notes=notes,
        )
        self.save(annotation)
        return annotation

    def save(self, annotation: Annotation) -> None:
        """Append annotation to JSONL file."""
        with open(self.output_path, "a") as f:
            f.write(json.dumps(asdict(annotation)) + "\n")

    def load_all(self) -> list[Annotation]:
        """Load all saved annotations."""
        annotations = []
        if self.output_path.exists():
            with open(self.output_path) as f:
                for line in f:
                    data = json.loads(line)
                    annotations.append(Annotation(**data))
        return annotations

Let me walk through what's happening in this script.

The Annotation dataclass is just a container that holds everything about a single review, the original question, the model's response, who reviewed it, the score they gave, and any notes they added. Nothing fancy, but having a structured format means you can easily compare scores across reviewers later.

The collect_annotation method is where the actual review happens. It prints the question and response to the terminal with some visual separators so the reviewer can read them clearly then prompts for a score.

The while true loop with input validation is important here. It keeps asking until the reviewer gives a valid number between 1 and 5 so you don't end up with garbage data in your annotations file.

The save method appends each annotation as a single JSON line to an annotations.jsonl file. I'm using JSONL (one JSON object per line) instead of a regular JSON array because it's append-friendly. You can add new annotations without reading and rewriting the entire file, which matters when you're collecting hundreds of reviews over time.

And load_all reads everything back, parsing each line into an Annotation object. This is what you'd call when you want to analyze your annotations, compare them to your LLM judge scores, or calculate agreement between reviewers.

In practice, you'd use this by feeding it a batch of question-response pairs from your production logs or golden dataset. You might run it during a weekly review session where a team member spends 30 minutes scoring 20-30 responses. That small investment gives you a reliable ground truth to calibrate your automated layers against.

How to Calculate Inter-Annotator Agreement

Now here's a problem you'll hit quickly: you ask two people to score the same response and they give it different scores. Is the response ambiguous or is your rubric ambiguous?

You need a way to measure this, and Cohen's Kappa is the standard tool for that. It basically tells you how much two annotators agree, adjusted for the amount of agreement you'd expect just by chance.

from sklearn.metrics import cohen_kappa_score


def measure_agreement(
    scores_annotator_1: list[int], scores_annotator_2: list[int]
) -> dict:
    """Calculate inter-annotator agreement using Cohen's Kappa."""
    kappa = cohen_kappa_score(scores_annotator_1, scores_annotator_2)

    interpretation = "poor"
    if kappa > 0.8:
        interpretation = "almost perfect"
    elif kappa > 0.6:
        interpretation = "substantial"
    elif kappa > 0.4:
        interpretation = "moderate"
    elif kappa > 0.2:
        interpretation = "fair"

    exact_agreement = sum(
        a == b for a, b in zip(scores_annotator_1, scores_annotator_2)
    ) / len(scores_annotator_1)

    return {
        "cohens_kappa": round(kappa, 3),
        "interpretation": interpretation,
        "exact_agreement": round(exact_agreement, 3),
    }


if __name__ == "__main__":
    # two annotators scored the same 10 responses
    annotator_a = [5, 4, 3, 4, 5, 2, 3, 4, 5, 4]
    annotator_b = [5, 4, 4, 4, 5, 3, 3, 4, 5, 3]

    agreement = measure_agreement(annotator_a, annotator_b)
    print(f"Cohen's Kappa: {agreement['cohens_kappa']}")
    print(f"Interpretation: {agreement['interpretation']}")
    print(f"Exact Agreement: {agreement['exact_agreement']:.0%}")

You would want a Kappa above 0.6. Anything below that and your rubric is the problem, not your annotators. Go back and add more concrete examples to each score level. Keep refining until people consistently agree. It usually takes two or three rounds of iteration.

How to Build the Regression Testing Pipeline

We can look at a scenario that's probably happened to you or other people you know: you tweak a prompt to fix one bad output you noticed. It works and that specific output is better now. You later ship it and week later, you find out the change broke three other responses you never thought to check.

This is incredibly common. The only way out is regression testing. If you've done traditional software development, you might already know what regression testing means. It's the practice of re-running a fixed set of tests every time you make a change, specifically to make sure you didn't break something that was already working.

The word regression literally means going backwards: your system was handling a question correctly and now after your change, it isn't.

In regular software, regression tests are usually unit tests or integration tests. For LLM applications, it works a bit differently. Instead of checking for exact outputs, you're scoring a batch of responses and comparing those scores against a previous run. If the scores drop, something regressed. The idea is the same but the mechanism is built around scoring rather than pass/fail assertions.

How to Create Golden Datasets

A golden dataset is just a curated list of questions that represent what your app actually needs to handle. You run your system against this list every time something changes (new prompt, new model, or updated retrieval logic) and compare the scores to your last run.

import json
from pathlib import Path
from dataclasses import dataclass, asdict


@dataclass
class GoldenExample:
    """A single test case in the golden dataset."""
    id: str
    question: str
    reference_answer: str
    category: str
    difficulty: str  # "easy", "medium", "hard"
    criteria: list[str]  # which criteria to evaluate


class GoldenDataset:
    """Manages a curated evaluation dataset."""

    def __init__(self, filepath: str = "golden_dataset.json"):
        self.filepath = Path(filepath)
        self.examples: list[GoldenExample] = []
        if self.filepath.exists():
            self.load()

    def add(self, example: GoldenExample) -> None:
        """Add a new example to the dataset."""
        self.examples.append(example)
        self.save()

    def get_by_category(self, category: str) -> list[GoldenExample]:
        """Filter examples by category."""
        return [e for e in self.examples if e.category == category]

    def save(self) -> None:
        """Persist dataset to disk."""
        data = [asdict(e) for e in self.examples]
        with open(self.filepath, "w") as f:
            json.dump(data, f, indent=2)

    def load(self) -> None:
        """Load dataset from disk."""
        with open(self.filepath) as f:
            data = json.load(f)
            self.examples = [GoldenExample(**item) for item in data]

    def summary(self) -> dict:
        """Return dataset statistics."""
        categories = {}
        for e in self.examples:
            categories[e.category] = categories.get(e.category, 0) + 1
        return {
            "total_examples": len(self.examples),
            "categories": categories,
        }

Some things I've learned about building these is that you should start with 50 to 100 examples. That's enough to catch meaningful regressions without making each eval run take forever.

Also, make sure you include edge cases – those weird questions that tripped up your model before. If your dataset is 90% easy questions, you won't notice when hard questions start failing.

Finally, treat this as a living document. Every time something breaks in production, turn it into a golden dataset example. Over a few months, your dataset evolves from generic test questions into a detailed map of exactly where your app is fragile.

How to Run Evaluations in CI/CD

Now let's wire everything together. This RegressionPipeline class runs your system against the golden dataset, scores every response, and compares the results to a previous run.

import json
from datetime import datetime, timezone
from dataclasses import dataclass, asdict


@dataclass
class EvalRun:
    """Records the results of one full evaluation run."""
    run_id: str
    timestamp: str
    model: str
    prompt_version: str
    total_examples: int
    avg_scores: dict  # criterion -> average score
    pass_rate: float  # percentage of examples above threshold
    failures: list[dict]  # examples that scored below threshold


class RegressionPipeline:
    """Runs evaluation against golden dataset and detects regressions."""

    def __init__(
        self,
        deterministic_eval: "DeterministicEvaluator",
        llm_judge: "LLMJudge",
        threshold: float = 3.5,
    ):
        self.det_eval = deterministic_eval
        self.judge = llm_judge
        self.threshold = threshold

    def run(
        self,
        golden_dataset: "GoldenDataset",
        generate_fn: callable,
        model_name: str,
        prompt_version: str,
    ) -> EvalRun:
        """Run full evaluation pipeline against golden dataset.

        Args:
            golden_dataset: The dataset to evaluate against.
            generate_fn: A function that takes a question string and
                         returns the model's response string.
            model_name: Identifier for the model being tested.
            prompt_version: Identifier for the prompt version.
        """
        all_scores = {}
        failures = []

        for example in golden_dataset.examples:
            # Generate response
            response = generate_fn(example.question)

            # Layer 1: Deterministic checks
            det_results = self.det_eval.run_all(response)
            det_failures = [r for r in det_results if not r.passed]

            if det_failures:
                failures.append({
                    "id": example.id,
                    "question": example.question,
                    "layer": "deterministic",
                    "details": [r.details for r in det_failures],
                })
                continue

            # Layer 2: LLM judge
            judge_results = self.judge.evaluate_all(
                example.question, response, example.criteria
            )

            for result in judge_results:
                if result.criterion not in all_scores:
                    all_scores[result.criterion] = []
                all_scores[result.criterion].append(result.score)

                if result.score < self.threshold:
                    failures.append({
                        "id": example.id,
                        "question": example.question,
                        "layer": "llm_judge",
                        "criterion": result.criterion,
                        "score": result.score,
                        "reasoning": result.reasoning,
                    })

        avg_scores = {
            criterion: sum(scores) / len(scores)
            for criterion, scores in all_scores.items()
        }

        total_evaluated = len(golden_dataset.examples)
        pass_count = total_evaluated - len(failures)

        return EvalRun(
            run_id=f"eval_{datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')}",
            timestamp=datetime.now(timezone.utc).isoformat(),
            model=model_name,
            prompt_version=prompt_version,
            total_examples=total_evaluated,
            avg_scores=avg_scores,
            pass_rate=pass_count / total_evaluated if total_evaluated else 0,
            failures=failures,
        )

    def compare_runs(self, baseline: EvalRun, current: EvalRun) -> dict:
        """Compare two evaluation runs to detect regressions."""
        regressions = {}
        improvements = {}

        for criterion in current.avg_scores:
            if criterion in baseline.avg_scores:
                diff = current.avg_scores[criterion] - baseline.avg_scores[criterion]
                if diff < -0.2:  # Score dropped by more than 0.2
                    regressions[criterion] = {
                        "baseline": baseline.avg_scores[criterion],
                        "current": current.avg_scores[criterion],
                        "change": round(diff, 3),
                    }
                elif diff > 0.2:
                    improvements[criterion] = {
                        "baseline": baseline.avg_scores[criterion],
                        "current": current.avg_scores[criterion],
                        "change": round(diff, 3),
                    }

        return {
            "verdict": "REGRESSION" if regressions else "PASS",
            "regressions": regressions,
            "improvements": improvements,
            "pass_rate_change": current.pass_rate - baseline.pass_rate,
        }

Now you can hook this into your CI/CD pipeline so it runs whenever someone changes a prompt or model config. If compare_runs returns REGRESSION, the build fails. No one deploys until they figure out what went wrong.

How to Know If Your AI Actually Got Better: Statistical Significance

So you tweaked your prompt and the average score went from 3.8 to 4.0. Time to celebrate, right? Maybe. Or maybe that 0.2 improvement is just random noise.

With a golden dataset of 50-100 examples, variance alone can easily produce score differences that big. You need an actual statistical test to know if the change is real.

A quick primer if you haven't done statistics in a while. A paired t-test is a way to compare two sets of measurements that are linked together. In our case, each pair is the same question scored under two different versions of your system: the old prompt and the new prompt.

The test looks at every pair, calculates how much the score changed for each question, and then asks: "Are these changes consistently in one direction or are they scattered randomly?"

If the changes are consistent (most questions scored higher with the new prompt), the test gives you a low p-value which means the improvement is likely real. If the changes are all over the place (some questions got better, some got worse, no clear pattern), the p-value will be high which means you can't be confident that the new version is actually better.

The reason we use a paired t-test instead of a regular one is that it accounts for question difficulty. Some questions are inherently harder than others, and pairing ensures we're measuring the change per question rather than just comparing two unrelated batches of scores.

Here's how to implement this:

from scipy import stats
import numpy as np


def is_improvement_significant(
    scores_before: list[float],
    scores_after: list[float],
    alpha: float = 0.05,
) -> dict:
    """Test whether a score improvement is statistically significant.

    Uses a paired t-test since the same questions are evaluated in both runs.
    """
    t_stat, p_value = stats.ttest_rel(scores_after, scores_before)
    mean_diff = np.mean(scores_after) - np.mean(scores_before)

    return {
        "mean_before": round(np.mean(scores_before), 3),
        "mean_after": round(np.mean(scores_after), 3),
        "mean_difference": round(mean_diff, 3),
        "p_value": round(p_value, 4),
        "is_significant": p_value < alpha,
        "direction": "improvement" if mean_diff > 0 else "regression",
        "recommendation": (
            "Safe to deploy"
            if p_value < alpha and mean_diff > 0
            else "Do not deploy - change is not a significant improvement"
        ),
    }


if __name__ == "__main__":
    # scores on 20 golden examples, before and after a prompt change
    before = [3, 4, 3, 5, 4, 3, 4, 4, 3, 5, 4, 3, 4, 3, 4, 5, 3, 4, 4, 3]
    after =  [4, 4, 4, 5, 5, 3, 4, 5, 4, 5, 4, 4, 4, 4, 5, 5, 4, 4, 5, 4]

    result = is_improvement_significant(before, after)
    print(f"Mean: {result['mean_before']} -> {result['mean_after']}")
    print(f"p-value: {result['p_value']}")
    print(f"Significant: {result['is_significant']}")
    print(f"Recommendation: {result['recommendation']}")

If the p-value comes back below 0.05, there's less than a 5% chance the improvement is just luck. That's when you ship. Anything above that and your improvement might just be noise, so don't deploy it no matter how good the averages look.

How to Put It All Together: The Complete Evaluation Architecture

Let's connect all three layers into a single orchestrator. This is the class that ties everything together. It runs deterministic checks first, escalates to LLM judging if those pass, and optionally brings in human evaluation for calibration.

class EvaluationOrchestrator:
    """Coordinates all three evaluation layers into a single pipeline."""

    def __init__(self):
        self.det_eval = DeterministicEvaluator()
        self.llm_judge = LLMJudge()
        self.annotation_collector = AnnotationCollector()

    def evaluate_response(
        self,
        question: str,
        response: str,
        run_human_eval: bool = False,
    ) -> dict:
        """Run the complete evaluation pipeline on a single response."""

        # Layer 1: Deterministic (always runs, every request)
        det_results = self.det_eval.run_all(response)
        det_passed = all(r.passed for r in det_results)

        if not det_passed:
            return {
                "status": "FAIL",
                "layer": "deterministic",
                "details": [r for r in det_results if not r.passed],
                "recommendation": "Fix structural issues before deeper eval",
            }

        # Layer 2: LLM Judge (runs on sample or in CI)
        judge_results = self.llm_judge.evaluate_all(question, response)
        avg_score = sum(r.score for r in judge_results) / len(judge_results)

        if avg_score < 3.5:
            return {
                "status": "FAIL",
                "layer": "llm_judge",
                "avg_score": avg_score,
                "details": judge_results,
                "recommendation": "Response quality below threshold",
            }

        # Layer 3: Human eval (periodic calibration)
        if run_human_eval:
            annotation = self.annotation_collector.collect_annotation(
                question, response, annotator="reviewer"
            )
            return {
                "status": "PASS" if annotation.score >= 4 else "REVIEW",
                "layer": "human",
                "automated_score": avg_score,
                "human_score": annotation.score,
            }

        return {
            "status": "PASS",
            "layer": "llm_judge",
            "avg_score": avg_score,
            "details": judge_results,
        }

What I Wish I Knew Earlier

I want to close with some things I wish someone had told me before I started building eval systems.

First, don't build all three layers at once. Start with just the deterministic checks, and then ship them. You'll be surprised how many issues they catch on their own, and the process of writing them forces you to actually define what correct output means for your app. Add the LLM judge when you need it and then add human eval later.

Second, check your judge against humans once a month. Run your LLM judge on 20-30 responses that already have human scores. If the judge has drifted more than 0.5 points on average, something changed: maybe the judge model was updated, or maybe your rubric doesn't cover a new failure mode. Either way, you need to recalibrate.

Third, every production failure becomes a test case. This is maybe the most useful habit. Something breaks? Great, that's a new golden dataset example. Over a few months, your dataset stops being a generic test suite and becomes a detailed map of every way your app has ever failed.

And finally, don't chase perfect eval scores. I've seen teams tweak prompts endlessly to push their eval scores from 4.2 to 4.5 only to discover that their rubric had a blind spot and users were still unhappy. The scores are a tool, not a goal, so human evaluation exists to catch what the numbers miss.

Wrapping Up

We covered a lot of ground in this article, so let me bring it all together. The core problem is that LLM applications fail differently from traditional software. There's no crash, no error log, and no stack trace. Just a confident, well-formatted, wrong answer.

And because the outputs aren't deterministic, you can't test them with simple assertions. You need a different approach entirely.

That approach is a layered evaluation pipeline:

  • Layer 1 (Deterministic Checks) handles the basics: is the output valid, the right length, and free of hallucinated URLs? These are fast, free, and catch more problems than you'd expect.

  • Layer 2 (LLM-as-Judge) brings in semantic evaluation: is the response actually relevant, accurate, and complete? By giving a judge model a clear rubric with concrete scoring criteria, you get surprisingly reliable and automated quality scores.

  • Layer 3 (Human Evaluation) keeps the whole system calibrated. A small batch of human reviews on a regular schedule catches the subtle issues that automated scoring misses, like tone, clarity, and the difference between "correct" and "genuinely helpful."

On top of those three layers, you learned how to build a regression testing pipeline with golden datasets so you can catch quality drops before they reach production. You also learned how to use statistical significance testing to make sure your improvements are real and not just noise.

If there's one thing I'd want you to take away, it's this: start small. Don't try to build all of this in a weekend. Drop the DeterministicEvaluator class into your project today: that takes five minutes and it'll immediately start catching things you're currently missing. Then add the LLM judge when you're ready for deeper evaluation. Then layer in human review and regression testing as your app matures.

The teams that ship reliable AI products aren't the ones with the fanciest models. They're the ones who built the scaffolding to know when those models are failing and who catch it before their users do.

]]> <![CDATA[ What Is an Agent Harness? The Architecture Behind Claude Code, DeepSeek Harness, and Hermes Agent ]]> https://www.freecodecamp.org/news/what-is-an-agent-harness/ 6aa41926c7a41a4b7462a57d Fri, 11 Sep 2026 15:07:18 +0000 On August 13, 2026, DeepSeek published a GitHub repository called deepseek-harness. Within two days, it had passed 95,386 stars and 8,826 forks (a vanity metric on its own, but a spike this fast signals something more than luck). This is among the fastest growth curves a developer tool has posted on GitHub in 2026 (Flowtivity, deepseek-ai/deepseek-harness).

Nine months earlier, a solo Austrian engineer named Mario Zechner shipped something close to the opposite: a coding agent called Pi with four built-in tools and almost nothing else. Pi took roughly a year of organic growth to cross 91,600 stars, without the launch spike. Just a slow, compounding climb from engineers who tried it and stayed (earendil-works/pi).

So here we have two wildly different growth curves, with two wildly different design philosophies. And underneath both of them, we have the same word: harness.

If you build with AI agents in any capacity, that word is now unavoidable, and most explanations of it are either marketing copy or a diagram with too many arrows.

This article defines what an agent harness is, then compares ten of the most popular agent harnesses to date, from Claude Code to DeepSeek Harness to Pi, against the same five-part architecture.

By the end, you'll understand why the term replaced "framework" in developer conversation this year and how the loudest 2026 harnesses differ underneath their branding. You'll also have a 60-line Python harness to run yourself along with a breakdown of the stack layers around it (MCP, orchestration, observability), plus a decision guide for picking one for your team.

Table of contents

What is an Agent Harness?

A harness is the runtime shell wrapped around an LLM model. The model itself only does one thing: given a stream of text and a list of available tools, it predicts what to say or which tool to call next. The harness handles everything else.

This unglamorous, boring plumbing includes the loop that calls the model, the code that executes tools, the memory that manages context over 40 turns, and the sandbox that protects your filesystem. It's the infrastructure that decides if an agent recovers from a failed tool call or just hangs.

Diagram of an agent harness: a model core at the center surrounded by a tool router, memory layer, planning layer, and sandbox boundary, connected in a feedback loop that feeds the model's output back in as the next turn's input.

Figure 1: The five parts every agent harness has to implement, drawn as a loop around a model core. The model sits at the center and predicts only the next message or tool call. Around it: a tool router that dispatches calls to the filesystem, shell, or external APIs, a memory layer that decides what context survives into the next turn, a planning layer that breaks a large task into steps before execution starts, and a sandbox boundary that constrains what the tools are allowed to touch.

The loop arrow shows the model's output feeding back in as the next turn's input, which is what turns a single prediction into an agent that keeps working until the task is done.

One practitioner definition captures the same shape from a different angle. A harness supplies everything a model doesn't do on its own: the loop that carries a goal from plan into action, access to tools like the terminal or file system, a memory layer that survives across turns, coordination for any subagents it spins up, and the permission rules that bound what it's allowed to touch (CellCog).

Concretely, when you type a request into Claude Code, Cursor, or Aider, here's what happens, in order:

  1. The harness assembles a prompt: your request, the system instructions, and a list of tool schemas the model can call.

  2. The model responds, usually with a mix of reasoning text and one or more tool calls (read_file, run_bash, edit, or whatever the harness exposes).

  3. The harness executes each tool call, ideally inside a sandbox, and captures the output.

  4. The harness appends the tool output back into the conversation and calls the model again.

  5. The loop repeats, sometimes for dozens of turns, until the model produces a final answer, or until the harness hits a turn limit, a cost limit, or a human interrupts it.

That five-step loop, sometimes called the agent loop or the ReAct loop (after the 2022 paper that first described reasoning and acting as one interleaved process: Yao et al.), is the part every harness on the market shares.

What varies, and what determines whether a given harness is good at its job, is everything wrapped around step 3 and step 4: how good the planning is before execution starts, how the memory decides what to keep and what to drop as the context fills up, how isolated the sandbox is, and whether the harness can spin up a second, smaller version of itself to handle a sub-task without polluting the main conversation.

When any one of those four goes wrong, the symptoms look identical from the outside: the agent stalls, forgets what it was doing, or burns through your context window on a task that should take five turns.

From Agent Frameworks to Agent Harnesses: What Changed

The word "framework" dominated agent conversation from 2023 through 2025: tools like LangChain, AutoGen, and CrewAI. Frameworks in that era were libraries. You imported components, chose your own model calls, and wrote the orchestration logic yourself. They gave you building blocks.

A harness is a different kind of product. It ships the loop already built, and that loop is opinionated about memory, planning, and safety. You then interact with it by running a command.

Anthropic's Claude Code made this shift undeniable through 2025: a terminal-native agent that plans, edits files, runs tests, and commits code without you writing any orchestration logic.

By 2026, the ship-the-loop pattern showed up across the ten harnesses profiled in the table below, from Claude Code to DeepSeek Harness to Cline, and "harness" became the word everyone started using to describe that shape, distinct from a framework you assemble yourself.

You can see it in the naming: DeepSeek's own repository is called deepseek-harness, echoing the same framework-to-harness shift that Claude Code introduced.

LangChain's Deep Agents shows that the industry now treats "harness" as its own architectural layer, released as an attempt to reverse-engineer what made Claude Code's harness effective and rebuild it as an open, model-agnostic library.

LangChain's own account of the project traces it back to one question, in Harrison Chase's words: "What about Claude Code made it general purpose, and could we abstract out and generalize those characteristics?"

LangChain has an obvious incentive here too: it's pitching an alternative to the tool it's studying, and the four mechanisms it names still hold up regardless of who names them.

Deep Agents packages four specific mechanisms that Claude Code's harness relies on:

  • A planning tool that forces the model to write out its steps before touching any files. This cuts down on the model quietly drifting off task over a long session.

  • A virtual filesystem and sandbox that gives the agent structured, isolated read and write access to a repository.

  • Subagent delegation, where the main agent spins up a smaller agent with its own clean context window to handle an isolated piece of work, then reports back a summary.

  • Context and memory management, including middleware that compresses conversation history and offloads large tool outputs so a long session doesn't blow through the model's context window (LangChain).

That list is worth memorizing because those four mechanisms (planning, sandboxing, delegation, and context management) are the engineering problems every serious harness has to solve, whether or not Deep Agents remains the harness people point to. Everything else is branding.

The Agent Harness Solutions at a Glance

The table below covers the harnesses pulling the most developer attention as of August 2026 and what each one bets its architecture on.

Harness Built by Optimized for Notable fact
Claude Code Anthropic End-to-end coding sessions: plan, edit, test, commit Popularized the planning-tool-plus-subagent pattern that competitors now copy
DeepSeek Harness (dsh) DeepSeek AI Total runtime modularity Passed 95,000 GitHub stars in 2 days. Every component, models, tools, sandboxes, UI, is a swappable plugin (GitHub).
Deep Agents LangChain Model-agnostic reproduction of Claude Code's harness patterns Ships as an open-source library plus a CLI, and works with any tool-calling model (LangChain)
Hermes Agent Nous Research A persistent, self-improving assistant that lives across channels Reaches platforms including Telegram, Slack, Discord, WhatsApp, and email from one process, with a growing public hub of shareable skills (Nous Research, GitHub)
Pi Mario Zechner / Earendil Inc. Radical minimalism: four built-in tools, everything else is an opt-in TypeScript extension Over 91,600 GitHub stars from organic, non-launch growth (GitHub)
Oh-My-Pi (omp) Can Bölük A maximalist fork of Pi that bakes in an IDE: LSP diagnostics, a debugger via DAP, persistent execution kernels Rewrote Pi's engine in Rust. Ships 60-plus model providers and 31 built-in tools (GitHub).
CellCog CellCog A general-purpose super-agent harness pointed at knowledge work broadly Ranked #1 on DeepResearch Bench as of August 2026 (score 55.78), with native video, image, and document output built into the same engine (CellCog)
OpenHands All Hands AI An open, dockerized autonomous software engineer with bash, browser, and test execution built in Formerly named OpenDevin. Docker is the default sandbox, isolating each session's shell commands and file writes from the host (OpenHands Docs).
Aider Paul Gauthier and contributors Git-native pair programming, where every agent step is a clean, reviewable commit Long-running favorite for engineers who want a tight diff-review loop
Cline Cline Bot Inc. and contributors A model-agnostic, approval-gated VS Code extension Every file edit and command pauses for your sign-off before it runs, by default

A few of these are coding-specific, and a few (such as CellCog and Hermes Agent especially) are trying to generalize the harness pattern past code and into broader knowledge work.

A harness built for coding can assume a repository, a test suite, and a diff as its unit of work. A harness built for general knowledge work has to invent an equivalent structure for research, writing, and multi-step business tasks, which is a harder, less standardized problem.

If you're evaluating a harness for anything beyond code, ask first: what's its unit of work, and did anyone build the equivalent of a diff for it, or just assume one exists?

Three Competing Philosophies for How a Harness Should Work

Strip away the marketing, and three different engineering bets sit underneath the 2026 agent harness boom.

Three-column diagram comparing agent harness design philosophies: DeepSeek Harness's plugin kernel with swappable modules, Claude Code and Deep Agents' four fixed mechanisms (planning, virtual filesystem, subagents, context compression), and Hermes Agent's compounding skill library.

Figure 2: Three bets on how to build a harness, shown as three parallel columns. Column one, DeepSeek Harness, centers on a plugin kernel where models, sandboxes, memory, and the UI are all interchangeable modules.

Column two, Claude Code and Deep Agents, centers on four fixed mechanisms: planning, virtual filesystem, subagents, and context compression.

Column three, Hermes Agent, centers on a compounding skill library that grows every time the agent solves something new. The three columns share only the base loop from Figure 1. Everything above that loop is a different bet on what makes an agent reliable over long sessions.

Bet One: Everything is a Plugin.

DeepSeek Harness is built on a meta-framework called Cordis, whose design is described in DeepSeek's own paper "A Programming Paradigm for Spatiotemporal Composability," which boils down to one idea: everything can be swapped at runtime (deepseek-ai/deepseek-harness).

In practice, that means the model, sandbox, session storage, scheduling loop, and even the UI theme are all swappable modules. The harness also ships a "creator mode" for inspecting the running system, testing Cordis plugins in memory, and combining them into new configurations (DeepSeek).

The bet here: no single architecture wins forever, so the winning move is to make architecture itself a configuration file.

Bet Two: a Small, Fixed Set of Mechanisms, Executed Well.

Claude Code and, following it, LangChain's Deep Agents bet the opposite way: pick four mechanisms (planning, sandboxed filesystem access, subagent delegation, and context compression) and invest in making each one reliable.

Every mechanism on this list is familiar enough that rivals borrow it wholesale: the table above credits Claude Code with popularizing the planning-plus-subagent pattern other harnesses now copy. The bet works because all four run together on every task. Skip one, and the others cover for it, for a while, until a long session finds the gap.

Bet Three: Memory That Compounds.

Hermes Agent bets that the biggest unsolved problem is what happens between sessions. Most harnesses reset to a blank context on every new conversation. Hermes instead offers to save the approach as a reusable skill when it solves something non-trivial. It then checks that skill library before reasoning from scratch on a similar future request so it can get faster at recurring tasks the longer you use it (Nous Research).

That's an advantage, as well as a risk: a skill library that grows unchecked can turn into debt that outlives the reason it was written. Paired with native scheduling and channel integrations across platforms like Telegram, Slack, and Discord, the design goal is closer to a standing assistant that lives on a server than a tool you open for one session and close.

A fourth bet sits underneath all three: Pi and Oh-My-Pi argue that most of what the other harnesses build in is unnecessary weight, and that four tools plus an extension system beat a feature-complete platform for engineers who know what they want.

Pi's climb past 91,600 GitHub stars, driven by organic word of mouth rather than a launch campaign, suggests that bet has staying power.

All four bets are defensible. They optimize against different failure modes: DeepSeek Harness optimizes against architectural lock-in, Claude Code and Deep Agents optimize against unreliable long-session behavior, Hermes optimizes against repeated work across sessions, and Pi optimizes against bloat.

So before you pick one, ask which failure mode costs you time today. The answer will help you choose the correct agent harness.

Build a Minimal Harness in Under 60 Lines of Python

The example below builds the five-step loop from Figure 1 with Anthropic's Messages API: a model, three tools, and a loop that keeps calling the model until it stops asking for tool calls. You'll see every failure mode this section talks about waiting inside these 60 lines.

import subprocess
from anthropic import Anthropic

client = Anthropic()

TOOLS = [
    {
        "name": "read_file",
        "description": "Read a UTF-8 text file from the working directory.",
        "input_schema": {
            "type": "object",
            "properties": {"path": {"type": "string"}},
            "required": ["path"],
        },
    },
    {
        "name": "write_file",
        "description": "Write content to a file, overwriting it if it exists.",
        "input_schema": {
            "type": "object",
            "properties": {
                "path": {"type": "string"},
                "content": {"type": "string"},
            },
            "required": ["path", "content"],
        },
    },
    {
        "name": "run_bash",
        "description": "Run a shell command inside the sandbox directory and return its output.",
        "input_schema": {
            "type": "object",
            "properties": {"command": {"type": "string"}},
            "required": ["command"],
        },
    },
]

def execute_tool(name, tool_input):
    if name == "read_file":
        return open(tool_input["path"]).read()
    if name == "write_file":
        with open(tool_input["path"], "w") as f:
            f.write(tool_input["content"])
        return f"wrote {len(tool_input['content'])} bytes to {tool_input['path']}"
    if name == "run_bash":
        result = subprocess.run(
            tool_input["command"],
            shell=True,
            cwd="./sandbox",
            capture_output=True,
            text=True,
            timeout=30,
        )
        return result.stdout + result.stderr
    raise ValueError(f"unknown tool: {name}")

def run_harness(task, max_turns=15):
    messages = [{"role": "user", "content": task}]

    for _ in range(max_turns):
        response = client.messages.create(
            model="claude-sonnet-5",
            max_tokens=4096,
            tools=TOOLS,
            messages=messages,
        )
        messages.append({"role": "assistant", "content": response.content})

        if response.stop_reason != "tool_use":
            return response.content[0].text

        tool_results = []
        for block in response.content:
            if block.type == "tool_use":
                output = execute_tool(block.name, block.input)
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": output,
                })
        messages.append({"role": "user", "content": tool_results})

    return "stopped: hit max_turns without a final answer"

Run run_harness("Write a Python script in sandbox/hello.py that prints the first 10 Fibonacci numbers, then run it and show me the output.") and watch the turns unfold: the model writes the file, calls run_bash to execute it, reads the output, and only then produces a final text answer. Every production harness in the tables above is a more engineered version of this same shape.

Claude Code adds a planning step before turn one and a permission gate before every run_bash equivalent. Deep Agents adds a virtual filesystem, plus a middleware layer that compresses messages before it grows past the model's context window. DeepSeek Harness makes the TOOLS list and the model client themselves swappable at runtime.

The gap between this toy loop and a serious one sits entirely in reliability engineering: what happens when a tool call fails, what happens at turn 50, and what stops the sandbox from touching anything outside ./sandbox.

Nothing in execute_tool catches a malformed response or a tool that errors out, so a single bad tool call can loop the model back onto the same broken result turn after turn. Add a retry path yourself, or the harness keeps doing this by default.

Two things in this example deserve a closer look. First, cwd="./sandbox" is a load-bearing safety boundary: without it, run_bash can execute anything the host user can, which is why every serious harness runs tool execution inside a container or a restricted directory. It's an easy line to delete by accident during a refactor, and a dangerous one to lose.

Second, max_turns=15 exists because nothing here tells the model to stop on its own. If you skip it, a harness with no turn limit and no cost limit will keep looping and keep spending tokens for as long as the model keeps asking for tools. If you forget that line during a refactor, the failure looks identical from the outside: a job that never returns, and a token bill that keeps climbing until someone kills the process by hand.

The Agent Harness Solution Stack

A harness doesn't run alone. Three adjacent layers show up in almost every production agent deployment, and knowing where each one starts and stops keeps you from asking a harness to solve a problem that belongs one layer over. Skip that mapping, and you'll spend a week debugging the harness for a bug that lives in the sandbox instead.

Four-layer diagram of the agent stack: the Model Context Protocol at the bottom, the agent harness loop above it, orchestration frameworks (LangGraph, CrewAI, AG2, Mastra, DSPy) on top, with observability tools (Langfuse, LangSmith) and sandboxing tools (E2B, Modal) shown as side panels.

Figure 3: Four horizontal layers, stacked bottom to top. The bottom layer, the protocol layer, is the Model Context Protocol (MCP). This is the shared standard that lets any harness talk to any external tool or data source the same way. The second layer up is the harness itself, the loop from Figure 1.

The third layer, orchestration frameworks, sits above single-agent harnesses and coordinates multiple agents or long-running stateful workflows: LangGraph, CrewAI, AG2, Mastra, and DSPy live here.

The top layer, drawn as two side panels rather than a fourth horizontal band, is observability and sandboxing: tools like Langfuse and LangSmith watch all the layers below them, and E2B and Modal provide the isolated execution environment the harness's sandbox runs inside.

The Protocol Layer: MCP

The Model Context Protocol is an open standard, originally introduced by Anthropic in November 2024, for connecting a model to external tools, files, and data sources in one consistent way (Anthropic). By late 2025, it had moved to the Agentic AI Foundation under the Linux Foundation, backed by Anthropic, OpenAI, and Block (Wikipedia).

A harness typically loads its tool list from an MCP config, not from code you write by hand:

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/Users/you/project"]
    },
    "postgres": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-postgres", "postgresql://localhost/mydb"]
    }
  }
}

Every MCP server you add here becomes available as tools inside TOOLS, without you writing a single new execute_tool branch. Write the integration once, and any MCP-compatible harness like Claude Code, Deep Agents, DeepSeek Harness, or one you build yourself can use it. That's the entire argument for the protocol layer in one sentence.

The Orchestration Layer

A harness runs one agent through a single loop. The moment you need multiple agents cooperating on a stateful, long-running workflow with dedicated roles like a planner, researcher, and reviewer, you enter orchestration framework territory. Choosing the wrong framework here will cost you months instead of a few lines of code.

  1. LangGraph, which models a multi-agent workflow as a graph with checkpointing and time-travel debugging, and is widely used for stateful production workflows at regulated companies (GitHub)

  2. CrewAI, built around defining agents by role and letting them collaborate on a shared task

  3. AG2, the community-maintained successor to Microsoft's original AutoGen project, which moved in 2026 to an async, event-driven runtime built around composable middleware (GitHub, pickaxe.co)

  4. Mastra, a TypeScript-first agent framework that crossed 22,000 GitHub stars and 300,000 weekly npm downloads after reaching version 1.0 in January 2026 (pickaxe.co, GitHub)

  5. DSPy from Stanford NLP, which treats prompt engineering as something closer to compilation than hand-authorship, optimizing prompts against a metric (GitHub)

Observability and Sandboxing

Once an agent makes tool calls on its own, you need to see what it did and where it did it, to avoid debugging blindly. Langfuse and LangSmith trace every model call, tool call, and token cost across a session, which is how you debug a harness that failed on turn 34 (GitHub, LangChain).

Braintrust and Arize Phoenix add rigorous evaluation on top of that tracing, so you can regression-test a harness's behavior the same way you'd test a codebase (Braintrust, Arize-ai/phoenix). And for the sandbox itself, the isolated environment where run_bash-style tool calls execute, E2B and Modal provide disposable micro-VMs that let a harness run untrusted code without touching the host machine (GitHub, Modal).

Why the Hype Curve and the Adoption Curve Diverge

Line chart comparing two GitHub star growth curves over about 400 days: DeepSeek Harness spiking to over 95,000 stars within two days of its August 2026 launch, versus Pi's steady, unbroken climb to over 91,600 stars across a full year with no launch spike.

Figure 4: Two GitHub star growth curves plotted on the same axes over roughly 400 days. The DeepSeek Harness curve is nearly vertical: flat at zero, then a near-instant spike to 95,000-plus stars within the first two days after its August 13, 2026 launch, then flattening out.

The Pi curve is the opposite shape: a shallow, steady, almost straight-line climb from its August 2025 release to over 91,600 stars a year later, with no single spike anywhere on the line. Both curves end up in roughly the same place.

The point of putting these two curves on one chart is that the shape getting there is different for each: one curve reflects a coordinated launch and a well-timed announcement. The other reflects a year of engineers individually deciding, one at a time, that the tool was worth keeping installed.

A launch spike tells you a project generated attention. Sustained use tells you whether the tool is still open in a terminal six months later, and those are different questions with different causes.

DeepSeek Harness's 95,000 stars in two days is a verifiable number (Flowtivity), but it's also driven largely by timing, distribution, and a well-known model lab's existing audience.

Pi's climb to a similar star count carries a different kind of signal: nobody coordinated a launch for it a year in. It accumulated through word of mouth among engineers who tried a four-tool coding agent, kept using it, and told other engineers.

A tool picked off a launch-week spike can look just as capable on day one and still leave a team stranded three months later if the maintainers move on to the next announcement. Neither number outweighs the other, but if you're choosing a harness to bet a team's workflow on, research the curve's shape, not just its current height.

A steep spike with a flattening tail tells you a project has an active community forming, worth watching before you commit production workflows to it. A long, shallow, unbroken climb tells you engineers kept it installed after the excitement wore off, which is a stronger, if slower, signal.

How to Choose a Harness for Your Team

Match the harness to the failure mode in front of you, not whatever's trending this week. Picking based on stars instead of your bottleneck is the mistake that costs a team weeks of migration work later.

  • You need one agent finishing one coding task reliably, end to end: Start with Claude Code, Deep Agents, or Aider if you want the tightest, most reviewable diff-per-commit loop you can get. All three implement the planning-plus-sandbox pattern from Figure 1 well.

  • You're worried about vendor or architecture lock-in and expect to swap models frequently: DeepSeek Harness's plugin-everything design and Deep Agents' model-agnosticism both directly target this concern. A harness that hardcodes one provider's SDK into its core is the wrong choice here, regardless of how capable that provider's model is today.

  • The same categories of tasks keep recurring across weeks or months, and you want the agent to get faster at them over time by building on what it knows: Hermes Agent's compounding skill library is built specifically for this pattern, especially if you also want it reachable from the chat platforms your team lives in.

  • You want the smallest possible audit surface area, and you are comfortable writing your own extensions for anything missing: Pi's four-tool core, or Oh-My-Pi if you specifically want IDE-grade tooling, LSP diagnostics, and a debugger, layered on top of that same minimal foundation.

  • You need several agents coordinating on a long-running, stateful process: That question sits a layer above the harness. Move up to LangGraph, CrewAI, AG2, or Mastra.

Whichever you pick, treat the observability layer as non-optional from day one. A harness that fails on turn 30 of an unattended run is a debugging nightmare without a trace. The same failure with Langfuse or LangSmith attached turns into a five-minute fix. Skip this step to save an afternoon of setup, and you'll pay for it the first time an agent fails mid-run, and nobody can say why.

What Transfers No Matter Which Harness Wins

The specific tool names in this article will likely look dated within a year, because the category is moving this fast. What will stay useful is the five-step loop in Figure 1, the four mechanisms LangChain identified inside Claude Code's architecture, and the layered stack in Figure 3.

Read any new harness that shows up next month against those three references, and you'll know within an hour whether it's doing something structurally new or repackaging the same loop under a different plugin system and a louder launch post.

That's the skill worth keeping: reading architecture instead of reading marketing, the one thing this category can't make obsolete no matter how fast the tool names turn over.

Conclusion

An agent harness isn't a mysterious new category of software. It's the runtime shell that turns a model's next-token prediction into an agent that plans, acts, checks its own work, and keeps going until a task is finished.

Harnesses are built from five parts that show up in every implementation: a loop, a tool router, memory, planning, and a sandbox boundary. What changed in 2026 is scale.

Enough teams shipped competing implementations that the architectural differences between them became worth studying. The landscape now ranges from DeepSeek's plugin-everything kernel and Pi's radical minimalism to Hermes Agent's compounding skills and the four fixed mechanisms of Claude Code and Deep Agents.

The numbers from the last section back this up: 95,000 stars in two days and 91,600 stars in a year prove two different routes reach the same conclusion.

Build the 60-line version yourself. Watch it loop. After you do, every harness on the market stops looking like magic and starts looking like an engineering decision you can evaluate on its merits.

What to Explore Next

Visit my GitHub to explore the 30+ open-source software solutions and developer tools I built and shared using this agentic AI-native engineering process.

]]> <![CDATA[ How to Use Lovable Responsibly ]]> https://www.freecodecamp.org/news/how-to-use-lovable-responsibly/ 6aa2d601395968ffa3528970 Thu, 10 Sep 2026 16:08:33 +0000 Building an app used to feel like assembling furniture without instructions, while missing half the screws. Today, AI-powered tools such as Lovable can help you turn an idea into a working web application by describing what you want in plain language.

That's exciting. It's also a responsibility.

Lovable can help you move quickly, experiment with ideas, and create useful software. But speed shouldn't replace careful thinking. A generated app can contain security problems, confusing user experiences, inaccurate information, or code that works in a demonstration but falls apart in real life.

In this guide, you'll learn practical ways to use Lovable while keeping security, privacy, accessibility, and user safety in mind. We'll cover how to write clearer prompts, protect sensitive information, test authentication and authorization, validate user input, work with realistic test data, review AI-generated code, and decide when an application is ready to share.

By the end, you'll have a simple workflow for building with Lovable more responsibly without giving up the speed and creativity that make AI-powered development useful.

What We'll Cover:

What Is Lovable?

Lovable is an AI-powered app-building platform that allows you to describe an application using natural language. Instead of writing every line of code manually, you can explain what you want and let the tool generate parts of the interface, functionality, and application structure.

For example, you might write:

Create a task manager with user accounts, a dashboard, task categories, due dates, and a button for marking tasks as complete.

Lovable may then generate a starting point that you can review, test, and improve.

The key phrase is “starting point.” AI-generated software isn't automatically finished software. Think of Lovable as a very fast coding partner that needs clear instructions, thoughtful reviews, and occasional reminders not to put a banana-shaped button in the middle of your login form.

Why Responsible Use Matters

AI app builders make software development more accessible, but accessibility comes with responsibility. When you create an app, you're making decisions that can affect real people.

Your application might collect names, email addresses, messages, payment details, health information, or location data. It might make recommendations, display important information, or control access to something valuable.

A small mistake can create large problems.

Responsible development helps you:

  • Protect user information

  • Reduce security risks

  • Avoid misleading users

  • Create accessible experiences

  • Respect copyright and ownership

  • Test your application before sharing it

  • Understand the code and services your app uses

  • Make decisions that are fair and explainable

You don't need to be a security expert to use Lovable responsibly. But you do need to slow down long enough to ask good questions.

Start With a Clear and Straightforward Idea

Before asking Lovable to build an app, describe the problem you want to solve.

A vague prompt such as this:

Build a cool productivity app.

leaves a lot of room for confusion.

A clearer prompt might look like this:

Build a simple productivity app for students. Users should be able to create tasks, assign a due date, mark tasks as complete, and filter tasks by status. Use plain language, a calm color palette, and a layout that works well on phones and desktop screens.

A strong prompt usually explains:

  • Who the app is for

  • What problem it solves

  • What users should be able to do

  • What information the app stores

  • What the interface should feel like

  • What the app should not do

  • What platform or screen sizes it should support

Clear instructions make it easier to review the result. They also reduce the chance that the AI invents unnecessary features that make your project more complicated than a group project with twelve shared spreadsheets.

Don't Enter Sensitive Information Unnecessarily

When working with an AI-powered development tool, avoid including sensitive information in prompts unless it's genuinely necessary and handled through an appropriate process.

Don't paste in:

  • Passwords

  • Private API keys

  • Authentication tokens

  • Credit card numbers

  • Personal identification numbers

  • Private customer records

  • Confidential business documents

  • Unreleased product details

  • Medical records

  • Private conversations

Use placeholders instead:

Use a placeholder for the payment provider API key.

Or:

Connect to an email service using an environment variable named EMAIL\_API\_KEY. Do not hardcode the key in the source code.

A placeholder keeps your project easier to share, review, and maintain. It also prevents the classic “I accidentally published a secret to the internet” plot twist.

Protect Secrets With Environment Variables

Secrets shouldn't be placed directly in frontend code or committed to a public repository.

A safer pattern is to use environment variables:

const apiKey = process.env.API\_KEY;

For a client-side application, be especially careful. Environment variables used in browser code may be visible to users. A secret that must remain private should usually be used on a secure server or through a protected backend service.

Never use this pattern:

const apiKey = "your-real-secret-key";

Use a placeholder during development:

const apiKey = process.env.API\_KEY || "";

Then configure the real value through the appropriate secret-management system for your hosting platform.

Before deploying, search your project for common secret patterns such as:

API\_KEYSECRETTOKENPASSWORDPRIVATE\_KEY

Finding a suspicious value doesn't always mean it's a secret, but it's worth checking.

Understand What Your App Does

Don't publish an application that you can't explain at a basic level.

You should know:

  • What data the app collects

  • Where that data is stored

  • Which external services receive the data

  • Who can view or modify the data

  • How users delete their accounts or information

  • Which parts of the app require authentication

  • What happens when a request fails

  • What happens when a user enters unexpected input

You don't need to understand every line immediately. But you should understand the major building blocks.

If Lovable generates code that you don't understand, ask it to explain a specific section:

Explain how user authentication works in this project. Identify where sessions are created, how access is checked, and what could go wrong if authentication is misconfigured.

You can also ask:

List all external services used by this application and explain what data each service receives.

Explanations are useful, but they aren't proof that the code is safe. Treat them as a map, not a magical safety certificate.

Build Security Into Your Prompts

Security should be part of the original request, not an emergency patch added after someone discovers that every user can view every account.

Include security requirements in your prompts:

Only authenticated users should be able to access the dashboard. Users must only be able to view and edit their own tasks. Validate all form inputs, display safe error messages, and never expose secrets in frontend code.

For an administrative area, you might write:

Create an admin section that is available only to users with an admin role. Check authorization on the server for every admin action instead of relying only on hiding buttons in the interface.

For user-generated content:

Allow users to submit comments, but sanitize and safely render the content to reduce cross-site scripting risks. Limit comment length and reject empty submissions.

Detailed prompts help the generated application start from better assumptions.

Test Authentication and Authorization Separately

Authentication answers the question, “Who are you?”, while authorization answers the question, “What are you allowed to do?”

These are different.

A user may be successfully logged in but still not be allowed to view another user’s private records. A responsible application checks both.

Test cases should include:

  1. A logged-out visitor tries to open a private page.

  2. A regular user tries to open an administrator page.

  3. A user tries to access another user's record by changing an identifier in the URL.

  4. A user submits a request without the required session information

  5. A user logs out and them presses the browser's back button.

Don't rely only on hiding navigation links. A hidden button isn't a security system. If a user can still call a backend endpoint directly, the application may be vulnerable.

For example, suppose your application has a page at /admin that should only be available to administrators. You could test authentication and authorization separately like this:

Authentication Test

Log out of the application and then try to open /admin directly. The application should redirect you to the login page or return an appropriate unauthorized response.

Then log in with a valid account and confirm that the application recognizes the authenticated session.

Authorization Test

Log in with a normal user account that doesn't have an admin role. Then try to open /admin directly instead of using the navigation menu. The application should deny access.

Try the same test against the backend endpoint used by an admin action. Confirm that the server also rejects the request.

You can also test whether changing an identifier in a URL or request allows one user to access another user's information. The important part is to verify the behavior from the user's perspective and, where possible, confirm that the server is enforcing the permission rather than simply hiding parts of the interface.

Validate All User Input

Users will enter unexpected information. Sometimes this happens by accident. Sometimes it happens because users are testing the boundaries of your application. Occasionally, it happens because someone has decided that a username should be 4,000 characters long and contain seventeen emojis.

Validate input on the client for a better user experience and on the server for security.

Examples of validation include:

  • Required fields

  • Maximum and minimum lengths

  • Valid email formats

  • Allowed file types

  • Maximum file sizes

  • Valid dates

  • Acceptable numeric ranges

  • Safe content handling

A frontend check might look like this:

if (username.trim().length < 3) {
     showError("Username must be at least 3 characters long.");
     return;
}

But don't assume that frontend validation is enough. A user can bypass browser checks by sending requests directly to your backend.

The server should validate the data again before storing or processing it.

Server-side validation means treating everything received from the browser as untrusted input. The server should check that the submitted data has the expected type, format, length, and range before using it. It should also reject unexpected fields or values when appropriate.

For example, if an API accepts a username and age, the server could verify that the username is a non-empty string within the allowed length and that the age is a number within the application's acceptable range. If the request fails validation, the server should reject it rather than storing or processing the invalid data.

You can ask Lovable to help create these checks and generate test cases:

  • Add server-side validation for every field in this form.

  • Reject missing, incorrectly formatted, oversized, or out-of-range values before they're stored or processed.

  • Then create tests for valid input, missing fields, invalid formats, boundary values, and unexpected input.

AI-generated tests can be useful, but don't rely on them as the only verification. Run the tests yourself and manually try important edge cases as well. The goal is to use AI to speed up the work while keeping human judgment involved in checking whether the validation actually protects the application.

Be Careful With Generated Dependencies

AI-generated projects may use libraries, packages, plugins, and external services. These tools can be helpful, but each dependency adds another piece to understand and maintain.

Ask Lovable:

List the main packages used in this project and explain why each one is needed.

You can also ask:

Identify dependencies that are unnecessary for the current features and suggest a simpler alternative.

Fewer dependencies can mean:

  • Less code to maintain

  • Fewer security updates

  • Smaller application size

  • Fewer compatibility problems

  • Easier debugging

You don't need to remove every package. Just avoid collecting dependencies like digital souvenirs.

Design for Accessibility

An application isn't truly successful if many people can't use it.

Ask Lovable to include accessibility from the beginning:

Make the interface accessible. Use semantic HTML, keyboard navigation, visible focus states, descriptive labels, sufficient color contrast, and accessible error messages.

Check whether:

  • Buttons have clear names

  • Form inputs have labels

  • Keyboard users can reach every interactive element

  • Focus indicators are visible

  • Text has enough contrast

  • Images have useful alternative text

  • Error messages explain how to fix a problem

  • The layout works at different screen sizes

  • Content remains usable when text is enlarged

Avoid using color as the only way to communicate meaning. For example, don't show errors only with a red border. Add text such as:

Email address is required.

Accessibility isn't just a compliance task. It usually makes the application easier for everyone to use.

Avoid Dark Patterns

A responsible app should help users make informed choices. It shouldn't trick them into doing something they didn't intend.

Avoid:

  • Preselected marketing consent

  • Hidden cancellation links

  • Confusing double negatives

  • Misleading buttons

  • Fake countdown timers

  • Notifications that look like system warnings

  • Subscriptions that are easy to start but difficult to stop

  • Important information hidden in tiny text

Use clear labels:

Delete account

is better than:

Continue

when the action permanently deletes an account.

For destructive actions, provide a confirmation step that clearly explains what will happen:

This will permanently delete your account and all saved tasks. This action cannot be undone.

Good design respects the user’s ability to choose.

Handle Errors Effectively

Every application experiences errors. Networks fail. Services go offline. Users close tabs at inconvenient moments. Servers occasionally decide to take an unscheduled vacation.

Don't display vague or misleading messages such as:

Something went wrong.

when you can provide useful guidance.

Better:

We could not save your task because the connection was interrupted. Check your internet connection and try again.

For developers, log enough information to investigate the problem without exposing sensitive data:

try {  await saveTask(task);} catch (error) {  console.error("Task save failed", {    operation: "create\_task",    message: error.message  });
  showError("Your task could not be saved. Please try again.");}

Avoid sending passwords, tokens, private messages, or personal records into logs.

Be Honest About AI-Generated Features

If your application uses AI to generate text, recommendations, summaries, images, or decisions, users should understand that the output may be wrong.

Use clear language:

This summary was generated automatically and may contain mistakes. Review it before sharing.

Avoid presenting AI-generated information as guaranteed fact, especially in areas such as:

  • Health

  • Finance

  • Education

  • Employment

  • Legal information

  • Safety

  • Personal identity

  • News and public information

Give users ways to correct, reject, or report problematic output. If an AI feature affects important decisions, provide human review whenever possible.

Protect Personal Data

Collect only the information your app actually needs.

If a task manager only needs an email address for account recovery, it probably doesn't need a user’s home address, phone number, favorite color, and childhood nickname.

Before adding a data field, ask:

Why do we need this information?

Then ask:

What could happen if this information were exposed?

Good data practices include:

  • Collecting less information

  • Explaining why information is needed

  • Restricting access

  • Deleting information when it's no longer necessary

  • Avoiding unnecessary analytics

  • Protecting data during transmission and storage

  • Giving users meaningful control over their information

Data isn't free just because a form field is free to add.

Don't ask Lovable to copy an existing product exactly, reproduce copyrighted artwork, or imitate a brand in a way that could confuse users.

Instead, describe the qualities you want:

Create a clean project-management interface with a left sidebar, clear status labels, and a spacious layout. Use original styling and avoid copying any specific company's branding.

Be careful with:

  • Images

  • Logos

  • Icons

  • Fonts

  • Code snippets

  • Written content

  • Product names

  • Brand colors

  • User-generated material

Use assets that you created, licensed, or are allowed to use. When in doubt, choose an original design.

Test With Realistic but Fake Data

Use fictional data during development:

Name: Jordan 
ExampleEmail: jordan@example.test
testOrder ID: TEST-1001

Don't use real customer records just because they're convenient.

Create test cases for:

  • Empty states

  • Long names

  • Very long text

  • Invalid email addresses

  • Duplicate records

  • Missing images

  • Slow connections

  • Failed requests

  • Expired sessions

  • Multiple users

  • Different screen sizes

  • Keyboard-only navigation

Fake data helps you test realistic behavior without exposing real people’s information.

You can create fake data yourself by using clearly fictional names, addresses, email addresses, identifiers, and other values that can't be mistaken for real customer information. For larger datasets, you can also use a reputable fake-data generator or ask Lovable to create a dataset specifically for testing.

For example, you could ask:

Create 100 fictional user records for testing. Use clearly fake names and email addresses under `example.test`. Include different account types, missing optional fields, long names, and other edge cases. Do not use real people's information.

Review generated data before using it, especially if you obtain it from an external source. Avoid datasets containing real personal information unless you have a legitimate reason, appropriate authorization, and proper safeguards. When possible, use synthetic data designed specifically for testing so that realistic application behavior can be tested without exposing real people's information.

Test Before You Share the App

Before showing your project to others, follow a basic release checklist.

1. The app works on mobile and desktop screens.
2. Forms validate input correctly.
3. Authentication behaves as expected.
4. Users can't access data belonging to other users.
5. Secrets aren't included in frontend code.
6. Error messages are clear and safe.
7. Keyboard navigation works.
8. Important buttons have clear labels.
9. Empty states are understandable.
10. Loading states are visible.
11. Destructive actions require confirmation.
12. Test data doesn't contain real personal information.
13. External services are configured correctly.
14. The production environment uses secure settings.
15. The app has been tested after the final changes.

A checklist may feel less exciting than clicking a shiny “Publish” button, but it's much more exciting than explaining to users why the app deleted everything.

Ask Lovable to Review Its Own Work

AI tools can help with review tasks when given specific instructions.

Try prompts such as:

Review this application for authentication and authorization problems. Identify any route, database query, or API endpoint that may expose data to the wrong user.

Review the forms for missing validation, unclear error messages, and accessibility problems.

Review the project for hardcoded secrets, unsafe logging, and sensitive information that might appear in the browser.

Review the application for mobile layout problems and explain the changes you recommend.

Don't accept the review blindly. Compare the suggestions with your own testing and, for serious applications, get help from an experienced developer or security professional.

Learn From the Generated Code

Using Lovable responsibly doesn't mean avoiding AI-generated code. It means using the tool as an opportunity to learn.

When you receive a result, ask:

Explain this function in beginner-friendly language.

Show me a simpler version of this code.

What assumptions does this implementation make?

What are the possible failure cases?

How would this code behave with two users at the same time?

Try changing one small part manually. Read the error messages. Compare the before-and-after versions. Over time, the generated code will become less mysterious.

The goal isn't to memorize every programming concept immediately. The goal is to become confident enough to ask better questions and recognize risky answers.

Use Lovable for Prototyping Without Pretending It's Production-Ready

Lovable is excellent for exploring ideas quickly.

You can use it to:

  • Test a product concept

  • Build a portfolio project

  • Create a prototype for user feedback

  • Learn how web applications are structured

  • Experiment with interfaces

  • Build an internal tool

  • Turn a rough idea into something people can react to

A prototype may not have the same security, reliability, monitoring, documentation, and scalability requirements as a public production application.

Be honest about the stage of your project. Use labels such as "Prototype", "Demo", "Work in Progress", and so on.

Don't treat a prototype like a finished product simply because it has a nice gradient and a button that says “Launch.”

Create a Simple Responsible Development Workflow

A practical workflow might look like this:

  1. Define the problem

  2. Identify the users

  3. Decide what information the app needs

  4. Write a clear prompt

  5. Generate a small feature

  6. Review the result

  7. Test normal and unexpected behavior

  8. Fix security and accessibility problems

  9. Repeat for the next feature

  10. Test the complete app

  11. Remove test data and secrets

  12. Document important decisions

  13. Deploy only when the app is ready for its intended audience

This process isn't slow. It's controlled. The fastest path is often the one that avoids rebuilding the entire application after discovering that the foundation was made of optimism and unvalidated form fields.

A Responsible Prompt Template

You can use this template when asking Lovable to create a feature:

Build [feature] for [type of user].

The goal is to [explain the problem being solved].

Users should be able to:
- [action one]
- [action two]
- [action three]

The application should:
- Validate all user input.
- Protect authenticated routes.
- Ensure users can access only data they are authorized to access.
- Avoid hardcoded secrets.
- Use clear loading and error states.
- Support keyboard navigation.
- Work on mobile and desktop screens.
- Use accessible labels and sufficient color contrast.

Do not:
- Collect unnecessary personal information.
- Expose private data.
- Add unrelated features.
- Change existing authentication behavior without explaining the change.

After building the feature, explain:
- Which files changed.
- What data is stored.
- Which external services are used.
- What security risks remain.
- How I should test the feature.

This template encourages Lovable to think about more than appearance.

The Golden Rule of AI App Building

If an AI-generated feature affects another person, review it as if you will be the person affected.

Would you want your data stored there?

Would you understand what the app is doing?

Would you be able to correct a mistake?

Would you know how to delete your information?

Would you feel comfortable using the application on a phone, with a keyboard, or with a slow internet connection?

Would you trust the app if you knew how it was built?

These questions turn responsible development from an abstract idea into a practical habit.

Final Thoughts

Lovable can make app development more approachable, faster, and more fun. It can help beginners build their first projects and help experienced developers explore ideas without spending hours creating every screen from scratch.

But responsible use requires more than generating attractive interfaces. Write clear prompts. Protect secrets. Collect less data. Validate inputs. Test permissions. Design for accessibility. Respect ownership. Explain AI-generated features. Review the code. Keep people involved in important decisions.

The best AI-built applications aren't the ones created with the fewest clicks. They're the ones built with curiosity, care, and enough testing to survive contact with real users.

Use Lovable to move faster, but use your judgment to decide where you're going.

Happy coding!

]]> <![CDATA[ What a Machine Learning Model is and How to Make One ]]> https://www.freecodecamp.org/news/what-a-machine-learning-model-is-and-how-to-make-one/ 6aa1b782434f42bd4da9b37c Wed, 09 Sep 2026 19:46:10 +0000 Machine learning can sound much more complicated than it actually is. You hear words like models, training, features, datasets, predictions, and algorithms, and it can feel like you need a PhD in mathematics before you're allowed to write your first machine learning program.

But at its core, machine learning is about getting a computer to learn patterns from examples and then use those patterns to make predictions about new examples. If you've ever learned to recognize a cat after seeing lots of cats, you already understand the basic idea.

In this tutorial, we're going to build a real machine learning model in Python. We'll start with a tiny dataset, train a model to predict whether a student might pass an exam based on the number of hours they studied, and then use the trained model to make predictions about new students.

Prerequisites

You don't need any previous machine learning experience to follow this tutorial. We'll introduce each machine learning concept as we go.

But having a basic understanding of Python will make the tutorial easier to follow. You should be comfortable with:

  • Creating and using variables

  • Working with Python lists

  • Writing basic if/else statements

  • Calling functions

  • Reading and running a Python program

  • Using a terminal or command prompt to run commands

You should also have:

  • Python installed on your computer

  • A text editor or code editor, such as VS Code

  • A terminal or command prompt

  • An internet connection to install the required Python library

You do not need prior knowledge of machine learning, scikit-learn, statistics, or advanced mathematics. I'll explain the machine learning concepts and code step by step.

What You Will Learn

The goal isn't just to get the code working. We're going to understand what each important line does, why we need it, and what's actually happening behind the scenes.

By the end, you'll have a much clearer mental model of what machine learning actually is and how you can start building models yourself.

What Is a Machine Learning Model?

A machine learning model is a program that has learned a pattern from data.

That definition is intentionally simple.

Suppose you show a child several animals and tell them which ones are cats. After seeing enough examples, the child might notice that cats usually have certain characteristics: whiskers, four legs, fur, a particular face shape, and so on. When they see a new animal, they can use what they learned to make a guess about whether it is a cat.

A machine learning model works in a similar way, except instead of looking at animals, it works with numbers and data.

For example, suppose we give a model information about students:

Hours Studied Exam Result
1 Fail
2 Fail
3 Fail
4 Pass
5 Pass
6 Pass

The model can look at these examples and discover a relationship between studying time and exam results. It might learn that students who study more tend to have a higher chance of passing.

We aren't explicitly writing that rule into the program. The model learns the relationship from the examples.

That's the key idea behind machine learning.

Machine Learning vs Traditional Programming

This becomes much clearer when you compare machine learning with traditional programming.

In traditional programming, you give the computer rules and data, and it produces an answer.

For example:

Data + Rules → Answer

You might write:

hours = 5

if hours >= 4:
    print("Likely to pass")
else:
    print("Likely to fail")

Here, you explicitly created the rule:

hours >= 4

The computer isn't learning anything. You told it exactly what to do.

Machine learning flips this around. Instead of manually writing the rule, you give the computer examples:

Examples + Correct Answers → Machine Learning Model

The model figures out a useful pattern from those examples.

Then you can give the trained model new data:

New Data + Trained Model → Prediction

That difference is one of the most important concepts to understand.

What Does "Training" Mean?

Training is simply the process of teaching a machine learning model using examples.

Imagine that you're teaching someone to recognize whether a student is likely to pass an exam.

You give them examples:

1 hour → Fail
2 hours → Fail
3 hours → Fail
5 hours → Pass
6 hours → Pass

After looking at enough examples, they start noticing a pattern.

Machine learning training works similarly.

We give the algorithm data, and the algorithm adjusts the model so that its predictions become better at matching the examples it's been given.

The word training sounds fancy, but the basic idea is just to give the model examples and let it learn a useful pattern.

What Is a Dataset?

A dataset is simply a collection of data.

For our project, we can represent our dataset using Python lists.

Suppose we have:

hours = [1, 2, 3, 4, 5, 6, 7, 8]

and:

results = [0, 0, 0, 1, 1, 1, 1, 1]

Here, we're using numbers to represent the exam results.

We'll use:

0 = Fail
1 = Pass

So our data means:

1 hour → Fail
2 hours → Fail
3 hours → Fail
4 hours → Pass
5 hours → Pass
6 hours → Pass
7 hours → Pass
8 hours → Pass

The first list contains our input information. The second list contains the answers we want the model to learn from.

What Are Features and Labels?

Machine learning uses a few words that sound more complicated than they really are.

A feature is information that we use to make a prediction.

A label is the answer we want the model to predict.

In our example:

Hours studied → Feature
Pass/fail → Label

If we had more information about each student, we could have multiple features, such as:

Hours studied
Previous exam score
Homework completion rate
Attendance

Then the model could use all of those features to predict:

Pass or fail

So you can think of it like this: Features are the clues. The label is the answer.

What Kind of Machine Learning Are We Using?

Our example uses supervised learning. Supervised learning means we train the model using examples where we already know the correct answer.

For example:

Hours studied: 2
Correct answer: Fail

and:

Hours studied: 6
Correct answer: Pass

The model sees both the input and the correct output during training.

This is different from unsupervised learning, where the model receives data without being given the correct answers and tries to find patterns or groups on its own.

There are other types of machine learning too, including reinforcement learning, but supervised learning is a great place to start because the basic workflow is easy to understand.

What Are We Actually Going to Build?

We're going to create a Python program that:

  1. Creates a small dataset.

  2. Separates the inputs from the answers.

  3. Splits the data into training and testing data.

  4. Creates a machine learning model.

  5. Trains the model.

  6. Tests how well it performs.

  7. Gives the model new information.

  8. Uses the model to make a prediction.

Our final program will use a decision tree classifier from the scikit-learn library.

A decision tree is a machine learning algorithm that makes decisions by asking a series of questions about the data.

For our simple example, the model might learn a pattern similar to:

Did the student study enough hours?
        ↓
      Yes → Pass
      No  → Fail

Real decision trees can become much more complicated, but this gives you the basic idea.

Now let's get started building!

Step 1: Install Python

To follow along here, you'll need Python installed on your computer.

You can check whether Python is already installed by running:

python --version

You should see something similar to:

Python 3.12.0

The exact version doesn't have to match that example.

Step 2: Create a Project Folder

Create a folder called:

machine-learning-model

Inside that folder, create a file called:

model.py

Our project will eventually look like:

machine-learning-model/
└── model.py

Step 3: Install scikit-learn

We're going to use a Python library called scikit-learn.

scikit-learn provides many machine learning algorithms and tools, so we don't have to implement everything from mathematical equations ourselves.

Install it with:

pip install scikit-learn

We could technically build a simple machine learning algorithm ourselves, and doing that can be useful for learning the mathematics later. For our first practical model, however, using a machine learning library lets us focus on understanding the workflow.

Step 4: Import the Model

Open model.py and write:

from sklearn.tree import DecisionTreeClassifier

This line imports the DecisionTreeClassifier class from scikit-learn.

This structure:

from sklearn.tree

means we're getting something from scikit-learn's tree module.

Then:

import DecisionTreeClassifier

means we want to use the decision tree classifier.

After importing it, we can create a machine learning model with:

model = DecisionTreeClassifier()

The variable:

model

will represent our machine learning model.

At this point, the model hasn't learned anything. It's basically an empty model waiting for training data.

Step 5: Create Our Dataset

Now let's create the examples our model will learn from.

Add:

hours = [1, 2, 3, 4, 5, 6, 7, 8]

This list represents how many hours each student studied.

Then:

results = [0, 0, 0, 1, 1, 1, 1, 1]

This list represents whether each student passed.

Remember:

0 = Fail
1 = Pass

So the first student studied for one hour and failed.

The fourth student studied for four hours and passed.

The eighth student studied for eight hours and passed.

We now have examples that the model can learn from.

Step 6: Understand Why the Data Structure Matters

There's an important detail here. Machine learning libraries usually expect the input data to be structured in a particular way.

Our hours list looks like this:

[1, 2, 3, 4, 5, 6, 7, 8]

But scikit-learn expects features to be represented as a two-dimensional structure.

Why?

Because a machine learning dataset can contain multiple features.

Imagine this dataset:

Hours Studied | Attendance | Previous Score
2             | 80%        | 65
5             | 95%        | 82
7             | 98%        | 91

Each row represents one example.

Each column represents one feature.

So even though our current model only has one feature, we still need to represent it as a two-dimensional dataset.

We can do this using nested lists:

X = [
    [1],
    [2],
    [3],
    [4],
    [5],
    [6],
    [7],
    [8]
]

Each inner list represents one student.

The first student has:

[1]

meaning they studied one hour.

The second has:

[2]

and so on.

The uppercase X is a common convention for the feature data.

Now create the labels:

y = [0, 0, 0, 1, 1, 1, 1, 1]

The lowercase y is commonly used for the target or label values.

So we now have:

X = [
    [1],
    [2],
    [3],
    [4],
    [5],
    [6],
    [7],
    [8]
]

y = [0, 0, 0, 1, 1, 1, 1, 1]

You can think of X as:

Here are the clues.

And y as:

Here are the correct answers.

Step 7: Split the Data

We don't want to train and test the model using exactly the same examples.

That would be a bit like giving a student the exact questions they'll see on an exam and then saying:

“Wow, you got 100%. Great job.”

We haven't really tested whether they learned anything.

Instead, we'll separate our dataset into:

  • Training data

  • Testing data

The training data teaches the model, while the testing data checks whether the model can make predictions on examples it wasn't trained on.

Import the splitting function:

from sklearn.model_selection import train_test_split

Now we can write:

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    random_state=42
)

There is a lot happening in this one line, so let's unpack it.

train_test_split()

This function randomly divides our data into training and testing portions.

We pass it:

X

which contains our features.

Then:

y

which contains our labels.

The argument:

test_size=0.25

means we want approximately 25% of our data for testing.

The remaining 75% is used for training.

random_state=42

The data is randomly split.

If you run the program multiple times without controlling the randomness, you might get a different split each time.

Setting:

random_state=42

makes the random split reproducible.

The number 42 isn't magical. You could use another integer.

For example:

random_state=10

would also work.

We use 42 simply because it's a common example value.

The Four Variables

The function returns four pieces of data:

X_train
X_test
y_train
y_test

X_train contains the features used to train the model.

y_train contains the correct answers for those training examples.

X_test contains the features used to test the model.

y_test contains the correct answers so we can compare them with the model's predictions.

Step 8: Create the Model

Now create our decision tree:

model = DecisionTreeClassifier()

This creates the model object.

Again, nothing has been learned yet. Think of it like buying a blank notebook: the notebook exists, but it doesn't contain your notes yet.

Step 9: Train the Model

Now we get to the line that actually teaches the model:

model.fit(X_train, y_train)

This is one of the most important lines in machine learning.

The .fit() method trains the model using the data we provide.

We give it:

X_train

which contains the examples.

Then:

y_train

which contains the correct answers.

The model looks for patterns connecting the features to the labels.

In our case, it's trying to discover a relationship between:

Hours studied

and:

Pass/fail

The exact internal process depends on the algorithm. A decision tree learns decision rules that split the training data into groups that become increasingly useful for predicting the target.

The important thing to understand right now is:

model.fit(X_train, y_train)

means:

Learn from these examples and their correct answers.

Step 10: Make Predictions

After training, we can give the model new data.

Suppose a student studied for five hours.

We can write:

prediction = model.predict([[5]])

Notice that we used:

[[5]]

instead of:

[5]

The outer list represents the collection of examples. The inner list represents the features for one example.

Since our model has one feature, that example contains one value:

[5]

So:

[[5]]

means:

Predict the result for one student whose feature value is five hours.

The model returns a prediction.

We can print it:

print(prediction)

You might see:

[1]

Remember:

1 = Pass
0 = Fail

So the model predicted that the student would pass.

Step 11: Convert the Prediction Into Human-Friendly Text

A prediction of:

1

isn't particularly friendly.

We can write:

if prediction[0] == 1:
    print("The model predicts: Pass")
else:
    print("The model predicts: Fail")

Let's look at:

prediction[0]

The model returns a list containing the prediction:

[1]

The [0] gets the first item.

Python starts counting list positions at zero.

So:

prediction[0]

means:

Give me the first prediction.

Then:

if prediction[0] == 1:

checks whether the model predicted 1.

If it did, we print:

The model predicts: Pass

Otherwise, we print:

The model predicts: Fail

Step 12: Test the Model

We shouldn't just make one prediction and assume the model is good.

We need to evaluate it.

First, make predictions for the test dataset:

predictions = model.predict(X_test)

Now:

predictions

contains the model's predictions for the examples it didn't see during training.

We can compare these predictions with:

y_test

which contains the actual answers.

scikit-learn provides an accuracy function:

from sklearn.metrics import accuracy_score

Then:

accuracy = accuracy_score(y_test, predictions)

The function compares the correct answers with the model's predictions.

If the model gets:

8 out of 10

correct, the accuracy would be:

0.8

We can turn that into a percentage:

print(f"Model accuracy: {accuracy * 100:.2f}%")

The * 100 converts:

0.8

into:

80

The:

:.2f

means we want two decimal places.

So the output could look like:

Model accuracy: 80.00%

A Very Important Warning About Accuracy

Accuracy is useful, but it doesn't tell you everything about a model.

Imagine you're trying to detect a rare disease.

Suppose:

99 people are healthy
1 person is sick

A terrible model could simply predict:

Everyone is healthy.

It would be 99% accurate.

But it completely failed at the thing we actually care about: identifying the sick person.

This is why machine learning developers use other evaluation metrics depending on the problem, including precision, recall, F1 score, mean squared error, and others.

For our beginner example, accuracy is enough to understand the basic workflow.

Step 13: Put Everything Together

Our complete beginner machine learning program looks like this:

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score


# Dataset
X = [
    [1],
    [2],
    [3],
    [4],
    [5],
    [6],
    [7],
    [8]
]

y = [
    0,
    0,
    0,
    1,
    1,
    1,
    1,
    1
]


# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    random_state=42
)


# Create the machine learning model
model = DecisionTreeClassifier()


# Train the model
model.fit(X_train, y_train)


# Make predictions on the test data
predictions = model.predict(X_test)


# Calculate accuracy
accuracy = accuracy_score(y_test, predictions)


print(f"Model accuracy: {accuracy * 100:.2f}%")


# Make a prediction for a new student
hours_studied = [[5]]

prediction = model.predict(hours_studied)


# Display the prediction
if prediction[0] == 1:
    print("The model predicts: Pass")
else:
    print("The model predicts: Fail")

Reading the Complete Code From Top to Bottom

The first three lines:

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

import the tools we need.

Then:

X = [
    [1],
    [2],
    [3],
    [4],
    [5],
    [6],
    [7],
    [8]
]

creates the feature data.

Then:

y = [
    0,
    0,
    0,
    1,
    1,
    1,
    1,
    1
]

creates the labels.

Next:

X_train, X_test, y_train, y_test = train_test_split(...)

divides the dataset into training and testing data.

Then:

model = DecisionTreeClassifier()

creates the model.

Next:

model.fit(X_train, y_train)

trains it.

Then:

predictions = model.predict(X_test)

asks the trained model to make predictions about the testing examples.

Next:

accuracy = accuracy_score(y_test, predictions)

measures how many of those predictions were correct.

Finally:

prediction = model.predict([[5]])

asks the model to predict the result for a new student who studied for five hours.

That's the entire machine learning workflow.

What Is Actually Happening Inside the Model?

This is where machine learning gets more interesting.

When we run:

model.fit(X_train, y_train)

the decision tree doesn't simply memorize the phrase:

4 hours = Pass

It analyzes the training examples and looks for useful ways to split them.

For example, it might discover a rule similar to:

Is hours studied <= 3.5?

If yes:

Predict Fail

If no:

Predict Pass

The exact tree depends on the training data and algorithm settings.

If we added more features, the tree could make decisions using several pieces of information.

For example:

Is study time <= 3.5?

       Yes
        ↓
    Predict Fail

       No
        ↓
Is attendance <= 80%?

       Yes
        ↓
    Predict Fail

       No
        ↓
    Predict Pass

Again, our actual code doesn't manually create these rules.

The algorithm learns them from the training data.

What Does "Learning" Actually Mean?

This is one of the most misunderstood parts of machine learning.

The computer isn't learning in exactly the same way a human does. A machine learning algorithm uses mathematical procedures to adjust a model based on data.

Different algorithms learn in different ways. A decision tree searches for useful splits. A linear regression model learns numerical parameters that describe a relationship. A neural network adjusts many parameters using optimization algorithms. And da clustering algorithm groups similar examples together.

So "learning" is a convenient word for:

Using an algorithm to adjust a model so that it captures useful patterns in data.

What Is a Parameter?

A parameter is a value inside a machine learning model that is learned from data.

For example, in a simple linear model:

y = mx + b

the model might learn values for:

m
b

Those values determine the relationship between the input and output.

Neural networks can have millions or billions of learned parameters.

The important idea is that the model's behavior is controlled by values that are learned or adjusted during training.

Parameters vs Hyperparameters

These two terms are easy to confuse.

A parameter is generally learned from the training data, while a hyperparameter is something you configure before or during training.

For our decision tree, we could specify:

model = DecisionTreeClassifier(
    max_depth=3
)

Here:

max_depth=3

is a hyperparameter.

We're telling the algorithm:

Don't allow the decision tree to grow beyond a depth of three.

The model learns its internal decision rules from the data, while we choose the hyperparameter.

This distinction becomes increasingly important as you build more advanced models.

Why Do We Need Training and Testing Data?

Imagine you're studying for a math exam.

Your teacher gives you ten practice questions, and you memorize all ten answers.

Then the exam contains those exact ten questions, so you get everything correct.

Does that prove you understand mathematics? Not really. You might simply have memorized the examples.

Machine learning has a similar problem called overfitting. A model can become extremely good at the training data without becoming good at handling new data.

That's why we keep some examples separate. The model doesn't see the test examples during training. Then we can ask:

Can the model generalize what it learned to examples it hasn't seen before?

That ability to work on new data is one of the most important goals of machine learning.

What Is Overfitting?

Overfitting happens when a model learns the training data too specifically.

Imagine we give the model a very small dataset. Instead of learning the general pattern:

More studying tends to increase the chance of passing.

it might effectively memorize the specific examples.

That can make training performance look excellent while performance on new data is poor.

A model that performs well on training data but poorly on unseen data is often overfitting.

What Is Underfitting?

Underfitting is basically the opposite. The model is too simple to capture the important patterns in the data.

Imagine trying to predict someone's exam result using only one or two results.

That doesn't give the model enough useful information, and it might perform poorly on both training and testing data.

Good machine learning involves finding a model that's complex enough to learn useful patterns but not so complex that it simply memorizes the training examples.

Why Our Dataset Is Not a Real Machine Learning Dataset

Our eight examples are intentionally tiny.

A real machine learning project would usually use much more data.

For example, you might collect:

10,000 students

with features such as:

Hours studied
Attendance
Homework completion
Previous scores
Sleep duration

and a label such as:

Passed

Then the model could learn from thousands of examples.

Our tiny dataset is useful because we can understand every part of the process.

Step 14: Add More Features

Let's make our example slightly more realistic.

Instead of only using hours studied, suppose we have:

Hours studied
Attendance

We can represent each student like this:

X = [
    [2, 70],
    [3, 75],
    [4, 80],
    [5, 85],
    [6, 90],
    [7, 95]
]

Now each row contains two features.

For example:

[5, 85]

means:

5 hours studied
85% attendance

Our labels could still be:

y = [0, 0, 1, 1, 1, 1]

Now the model has more information to work with.

We could train it exactly the same way:

model.fit(X_train, y_train)

The difference is that the model now has two features instead of one.

Step 15: Make a Prediction With Multiple Features

Suppose we want to predict the result of a student who:

Studied for 5 hours
Had 90% attendance

We represent that as:

new_student = [[5, 90]]

Then:

prediction = model.predict(new_student)

The model uses both features to make the prediction.

This is how machine learning scales from simple examples to datasets with many columns.

What Happens When You Have Hundreds of Features?

The exact same basic concept applies.

Imagine predicting house prices using:

Number of bedrooms
Square footage
Number of bathrooms
Location
Age of house
Garage size
Lot size
Distance to school

Each one can become a feature. Then the model uses those features to predict a target:

House price

The basic structure remains:

Features → Model → Prediction

The difficult part becomes choosing useful data, selecting an appropriate algorithm, cleaning the data, evaluating the model, and making sure the model works well outside the training dataset.

What Is Regression?

So far, our model predicts categories:

Pass
Fail

This is a classification problem. Classification means predicting a category.

Examples include:

Spam / Not Spam
Cat / Dog
Fraud / Not Fraud
Pass / Fail

Regression is different. It predicts a numerical value.

For example:

House price = $425,000

or:

Temperature = 82.4°F

or:

Sales = $17,500

So a useful distinction is:

Classification → Predict a category

Regression → Predict a number

A Simple Regression Example

scikit-learn provides a model called LinearRegression.

Import it:

from sklearn.linear_model import LinearRegression

Create the model:

model = LinearRegression()

Then train it:

model.fit(X_train, y_train)

And make a prediction:

prediction = model.predict([[5]])

The workflow is almost identical.

That's one reason machine learning libraries are useful: once you understand the general workflow, learning new algorithms becomes much easier.

The General Machine Learning Workflow

Most beginner machine learning projects can be thought about using this sequence:

1. Collect Data

Get examples related to the problem you want to solve.

2. Clean the Data

Fix missing, incorrect, duplicated, or inconsistent information.

3. Select Features

Choose the information you want the model to use.

4. Choose a Model

Select an algorithm appropriate for the problem.

5. Split the Data

Separate training and testing examples.

6. Train

Use the training data to fit the model.

7. Evaluate

Measure how well the model performs.

8. Improve

Change the data, features, model, or hyperparameters.

9. Make Predictions

Use the trained model on new data.

10. Deploy

If the model is useful, integrate it into an application.

This workflow is much more important than memorizing the name of a particular algorithm.

How Machine Learning Fits Into Real Applications

A trained model is usually not the entire application.

Imagine you build a model that predicts whether an email is spam. You might eventually create:

Email
 ↓
Backend
 ↓
Machine Learning Model
 ↓
Prediction
 ↓
User Interface

The model is one component inside a larger software system.

The same idea applies to:

Recommendation systems
Fraud detection
Search engines
AI assistants
Image classification
Demand forecasting
Customer analytics

This is important for developers because machine learning engineering isn't only about training models. You also need to know how to build software around those models.

What Should You Learn After This?

Once you understand this basic project, there are several useful directions to explore.

Learn NumPy

NumPy is one of the fundamental Python libraries for numerical computing. You'll encounter arrays everywhere in machine learning.

Learn pandas

pandas is extremely useful for working with datasets.

For example:

import pandas as pd

You can load a CSV file:

data = pd.read_csv("students.csv")

and inspect it:

print(data.head())

This becomes much more useful once you start working with real datasets.

Learn Data Visualization

Libraries such as Matplotlib can help you visualize your data. For example, you might want to see whether exam scores increase as study hours increase.

Visualizing data can help you understand patterns before you even train a model.

Learn More Algorithms

Once decision trees make sense, explore:

Linear Regression
Logistic Regression
Random Forests
K-Nearest Neighbors
Support Vector Machines
Gradient Boosting
Neural Networks

You don't need to memorize all of them.

Focus on understanding what kind of problem each algorithm is designed to solve and what assumptions or tradeoffs come with it.

Learn the Mathematics

You can build useful machine learning applications without deriving every equation from scratch.

But if you want to understand machine learning deeply, mathematics becomes increasingly valuable.

Start with:

Algebra
Functions
Probability
Statistics
Linear Algebra
Calculus

Concepts such as derivatives and gradients become especially important when you start learning how neural networks train.

Here's a calculus course and a statistics handbook as well to get you started.

The Mental Model to Keep

When you're learning machine learning, don't let the terminology make everything feel more complicated than it is.

At the simplest level, think about machine learning like this:

You have examples, and each example contains information called features. Some examples also have known answers called labels.

You give those examples to a learning algorithm. The algorithm creates a model that captures patterns in the examples.

Then you give the trained model new information. The model uses the patterns it learned to make a prediction.

In code, the basic workflow looks like:

model = SomeMachineLearningModel()

model.fit(X_train, y_train)

predictions = model.predict(X_test)

That three-part structure is worth remembering.

model = ...

creates the model.

model.fit(...)

trains the model.

model.predict(...)

uses the trained model.

Everything else you learn about machine learning builds on this foundation.

Final Thoughts

A machine learning model isn't a magical brain sitting inside your computer. It's a mathematical model created by an algorithm that has learned patterns from data.

The most important shift in thinking is understanding that you don't always need to program every rule yourself.

With traditional programming, you might explicitly write:

if hours >= 4:
    result = "Pass"

With machine learning, you provide examples:

1 hour → Fail
2 hours → Fail
3 hours → Fail
4 hours → Pass
5 hours → Pass

and let the learning algorithm find a useful pattern.

Our project was intentionally small, but the same basic ideas appear in much larger systems. A recommendation engine, fraud detector, image classifier, and many other machine learning applications still have to deal with data, features, training, evaluation, and predictions.

Once you understand those fundamentals, terms like training, features, labels, classification, regression, overfitting, and models stop sounding like a collection of random AI vocabulary and start fitting into one connected idea.

You don't need to start by building the next giant AI system. Start with a tiny dataset, train one model, inspect its predictions, change something, and see what happens. That hands-on process is where machine learning starts becoming much easier to understand.

Happy coding!

]]> <![CDATA[ How to Refactor a Legacy Application Before Migrating It ]]> https://www.freecodecamp.org/news/refactor-legacy-application-before-migration/ 6a9f3503813f6309fdfcd6e3 Mon, 07 Sep 2026 22:04:51 +0000 The moment a team decides to migrate a legacy application, there's usually pressure to start moving code.

Move the database, the API, or the UI. Move the application to a new framework, runtime, cloud provider, or architecture.

That sounds reasonable, but there's a problem.

If the current system mixes business rules, persistence, infrastructure, external integrations, and orchestration inside the same modules, migration becomes much harder than it needs to be.

You're not just moving software. You're trying to move several responsibilities that have become entangled over years of development.

This is why I often prefer to refactor before migrating. Not to make the legacy system beautiful or redesign everything. And definitely not to turn the preparation phase into another rewrite.

The goal is much narrower: change the structure enough that important behavior can move independently.

In the previous steps of this workflow, we first tried to understand the codebase and then used characterization tests to protect the behavior we were about to change.

Now you'll learn how you can start changing the structure.

In this tutorial, I'll show you how to prepare a legacy application for migration by:

  • choosing a migration boundary,

  • separating business rules from infrastructure,

  • introducing seams,

  • isolating side effects,

  • creating adapters around external systems,

  • reducing dependency direction problems,

  • extracting cohesive application behavior,

  • using characterization tests throughout the refactor,

  • using AI without letting it redesign the system blindly,

  • and knowing when the application is ready to start migrating.

The examples use TypeScript, but the process applies to most languages and architectures.

The objective is not:

legacy application
↓
perfect architecture

Instead, it's:

legacy application
↓
migration-friendly structure
↓
incremental migration

That difference can save a lot of unnecessary work.

Prerequisites

You should be comfortable with:

  • reading an existing codebase

  • TypeScript or a similar language

  • unit and integration testing

  • dependency injection

  • interfaces and adapters

  • basic software architecture

  • incremental refactoring

You should also have some behavioral protection around the capability you plan to modify.

That may include:

  • characterization tests

  • integration tests

  • contract tests

or another reliable way to verify existing behavior.

Refactoring without that protection is possible. But it's also much harder to distinguish a structural improvement from an accidental behavioral change.

Table of Contents

Why Migration Problems Often Start Before the Migration

Imagine you need to migrate an order-processing application.

You inspect the main service and find something like this:

async function processOrder(orderId: string) {
  const connection = await mysql.getConnection();

  const [rows] = await connection.query(
    "SELECT * FROM orders WHERE id = ?",
    [orderId]
  );

  const order = rows[0];

  if (!order) {
    throw new Error("Order not found");
  }

  if (order.customer_type === "PREMIUM") {
    order.total = order.total * 0.9;
  }

  if (
    order.country === "AR" &&
    order.payment_method === "TRANSFER"
  ) {
    order.total -= 500;
  }

  await connection.query(
    "UPDATE orders SET total = ?, status = ? WHERE id = ?",
    [order.total, "PROCESSED", order.id]
  );

  await paymentProvider.createPayment({
    orderId: order.id,
    amount: order.total,
  });

  await eventBus.publish("order.processed", {
    id: order.id,
    total: order.total,
  });

  await emailClient.send({
    to: order.customer_email,
    template: "order-processed",
  });

  return order;
}

Suppose the migration goal is:

MySQL       → PostgreSQL
Old runtime → New runtime
Legacy API  → New service

The obvious temptation is to begin translating this function into the target stack.

But what exactly are you migrating?

The function contains:

database access
business rules
state transition
payment integration
event publication
email delivery
application orchestration

Changing the database now risks affecting pricing, while changing the payment client risks affecting persistence. And moving the function into another service means moving all of its dependencies at once.

The migration difficulty is partly caused by the current structure. So before migrating, you'll want to create enough separation that those concerns can move independently.

Choose a Migration Boundary Before Refactoring

Don't begin with:

Let's clean up the application.

Begin with:

What do we want to migrate first?

Suppose you decide the first capability will be Process Order. That gives your refactor a boundary.

Now you can map:

Input:
orderId

Business behavior:
load order
calculate adjustments
mark as processed

Side effects:
persist order
create payment
publish event
send email

Output:
processed order

This is much more useful than deciding to refactor:

src/services/

because a folder isn't necessarily a business boundary.

Migration works better when you can reason about capabilities.

For example:

Process Order
Cancel Order
Generate Invoice
Register Customer
Renew Subscription

Each can potentially become a migration unit.

Don't Refactor the Entire Application

Once you start identifying architectural problems, it becomes tempting to fix all of them.

You may notice:

circular dependencies
duplicated repositories
global configuration
large services
static helpers
direct database access
inconsistent error handling
mixed domain models

All of those may deserve attention, but the migration doesn't require all technical debt to disappear.

Suppose your target is the order-processing capability. A useful rule is to refactor only what prevents this capability from moving safely.

For example:

Problem:
Order processing calls MySQL directly.

Relevant?
Yes.

Problem:
The reporting module uses inconsistent date formatting.

Relevant?
Probably not.

Problem:
Order processing calls the payment SDK directly.

Relevant?
Yes.

Problem:
The admin UI contains duplicated CSS.

Relevant?
No.

This prevents preparation from becoming an open-ended cleanup project.

Legacy modernization needs scope discipline.

Separate Business Rules from Infrastructure

The most valuable structural change is often separating business behavior from technology-specific details.

Take this code:

async function processOrder(orderId: string) {
  const order = await mysqlOrders.find(orderId);

  if (order.customerType === "PREMIUM") {
    order.total *= 0.9;
  }

  if (
    order.country === "AR" &&
    order.paymentMethod === "TRANSFER"
  ) {
    order.total -= 500;
  }

  await mysqlOrders.update(order);

  await stripe.createPayment({
    orderId: order.id,
    amount: order.total,
  });
}

The pricing behavior itself doesn't need MySQL or Stripe.

You can extract it:

type Order = {
  id: string;
  total: number;
  customerType: "STANDARD" | "PREMIUM";
  country: string;
  paymentMethod: "CARD" | "TRANSFER";
};

function calculateOrderTotal(order: Order): number {
  let total = order.total;

  if (order.customerType === "PREMIUM") {
    total *= 0.9;
  }

  if (
    order.country === "AR" &&
    order.paymentMethod === "TRANSFER"
  ) {
    total -= 500;
  }

  return Math.max(total, 0);
}

Now:

pricing behavior

is no longer coupled to:

MySQL
Stripe

This doesn't require a complete domain-driven redesign. It's simply a useful separation.

The next migration step can replace infrastructure while leaving this behavior unchanged.

Introduce Seams Around Hard Dependencies

Legacy code often contains dependencies that can't easily be replaced in tests or migration code.

For example:

class OrderService {
  async process(orderId: string) {
    const client = new LegacyDatabaseClient();

    const order = await client.findOrder(orderId);

    // ...
  }
}

The database dependency is created inside the method.

That makes substitution difficult.

A small preparatory refactor can introduce a seam:

interface OrderRepository {
  findById(id: string): Promise<Order | null>;
  save(order: Order): Promise<void>;
}

Then:

class OrderService {
  constructor(
    private readonly orders: OrderRepository
  ) {}

  async process(orderId: string) {
    const order = await this.orders.findById(orderId);

    if (!order) {
      throw new Error("Order not found");
    }

    // existing behavior
  }
}

Now the existing MySQL implementation can satisfy the interface:

class MySqlOrderRepository implements OrderRepository {
  async findById(id: string) {
    // existing MySQL behavior
  }

  async save(order: Order) {
    // existing MySQL behavior
  }
}

Later, the migration can introduce:

class PostgresOrderRepository implements OrderRepository {
  // new implementation
}

Notice what we didn't change: we didn't change the business behavior. We changed the replaceability of a dependency.

That's exactly the kind of refactoring that helps migration.

Isolate Side Effects from Decision Logic

Another useful separation is between:

deciding

and:

performing

Suppose cancellation currently looks like this:

async function cancelOrder(order: Order) {
  if (order.status === "SHIPPED") {
    throw new Error("Cannot cancel shipped order");
  }

  order.status = "CANCELLED";

  await orders.save(order);
  await inventory.release(order.id);
  await payment.refund(order.id);
  await audit.log("ORDER_CANCELLED", order.id);
}

There are two different responsibilities here.

The business decision:

Can this order be cancelled?
What should its new state be?

And the operational effects:

persist
release inventory
refund
audit

You could first extract the decision:

function cancelOrderState(order: Order): Order {
  if (order.status === "SHIPPED") {
    throw new Error("Cannot cancel shipped order");
  }

  return {
    ...order,
    status: "CANCELLED",
  };
}

Then orchestration remains:

async function cancelOrder(order: Order) {
  const cancelled = cancelOrderState(order);

  await orders.save(cancelled);
  await inventory.release(cancelled.id);
  await payment.refund(cancelled.id);
  await audit.log("ORDER_CANCELLED", cancelled.id);

  return cancelled;
}

The behavior is still the same, but now the state transition can be tested and migrated independently.

That matters if the target architecture changes how side effects are executed.

For example, the future version might use:

transactional outbox
event-driven workflow
queue
workflow engine

You don't need to introduce those mechanisms yet. You only need to stop the current decision logic from depending directly on them.

Put External Systems Behind Adapters

External SDKs often leak deeply into legacy code.

For example:

const result = await stripe.paymentIntents.create({
  amount: order.total,
  currency: "usd",
  metadata: {
    orderId: order.id,
  },
});

If dozens of application modules depend directly on the Stripe SDK, replacing or relocating payment processing becomes difficult.

Create an application-level boundary instead:

type PaymentRequest = {
  orderId: string;
  amount: number;
};

type PaymentResult = {
  paymentId: string;
};

interface PaymentGateway {
  charge(
    request: PaymentRequest
  ): Promise<PaymentResult>;
}

The Stripe adapter contains the provider-specific details:

class StripePaymentGateway implements PaymentGateway {
  async charge(
    request: PaymentRequest
  ): Promise<PaymentResult> {
    const result =
      await stripe.paymentIntents.create({
        amount: request.amount,
        currency: "usd",
        metadata: {
          orderId: request.orderId,
        },
      });

    return {
      paymentId: result.id,
    };
  }
}

The application now knows about:

PaymentGateway

instead of:

Stripe SDK

This is useful for migration because provider-specific code is localized.

The same pattern works for:

email providers
message brokers
cloud storage
ERP integrations
CRM APIs
identity providers
search engines

The adapter isn't valuable because interfaces are fashionable. It's valuable because it creates a boundary you can move.

Improve Dependency Direction Without Rebuilding Everything

Legacy systems often have dependency relationships such as:

business logic
    ↓
database SDK
    ↓
framework utilities

That makes infrastructure difficult to replace.

You don't necessarily need to implement full Clean Architecture. You only need to improve dependency direction where migration requires it.

For example:

Before:

OrderService
   ↓
MySQL

After:

OrderService
   ↓
OrderRepository
   ↑
MySqlOrderRepository

The application depends on an abstraction. The infrastructure implements it.

The same can happen with payments:

OrderService
   ↓
PaymentGateway
   ↑
StripePaymentGateway

and messaging:

OrderService
   ↓
OrderEvents
   ↑
KafkaOrderEvents

Now replacing infrastructure no longer requires rewriting the application service. That's the important outcome.

Extract a Cohesive Application Boundary

After several small refactors, the capability may start to look like this:

interface OrderRepository {
  findById(id: string): Promise<Order | null>;
  save(order: Order): Promise<void>;
}

interface PaymentGateway {
  charge(request: {
    orderId: string;
    amount: number;
  }): Promise<void>;
}

interface OrderEvents {
  processed(order: Order): Promise<void>;
}

class ProcessOrder {
  constructor(
    private readonly orders: OrderRepository,
    private readonly payments: PaymentGateway,
    private readonly events: OrderEvents
  ) {}

  async execute(orderId: string) {
    const order = await this.orders.findById(orderId);

    if (!order) {
      throw new Error("Order not found");
    }

    const total = calculateOrderTotal(order);

    const processed: Order = {
      ...order,
      total,
      status: "PROCESSED",
    };

    await this.orders.save(processed);

    await this.payments.charge({
      orderId: processed.id,
      amount: processed.total,
    });

    await this.events.processed(processed);

    return processed;
  }
}

This isn't necessarily the final architecture. That's important.

We aren't claiming:

This is how the application should look forever.

We're just saying:

This capability now has boundaries that make migration easier.

The infrastructure can change independently.

The business rules are testable. The orchestration is visible. And the external contracts are explicit.

That's enough to start considering migration.

Keep Behavioral Tests Running During the Refactor

This is where the characterization tests from the previous step become useful.

Suppose the original behavior was protected with:

it("preserves premium order processing behavior", async () => {
  const result = await processOrder("order-1");

  expect(result.total).toBe(9000);
  expect(result.status).toBe("PROCESSED");

  expect(payment.charge).toHaveBeenCalledWith({
    orderId: "order-1",
    amount: 9000,
  });

  expect(events.processed).toHaveBeenCalled();
});

Now you can change:

direct database access

into:

repository

and run the test.

Then change:

direct payment SDK

into:

payment adapter

and run the test.

Then extract:

pricing logic

and run the test.

The rhythm becomes:

small structural change
↓
test
↓
small structural change
↓
test
↓
small structural change
↓
test

This matters because structural refactoring is much easier to reason about when behavioral changes aren't happening at the same time.

If a test fails after one small change, the possible cause is narrow.

If a test fails after a two-week rewrite, the possible cause is almost everything.

How to Use AI During Structural Refactoring

AI can help a lot during this phase.

But the useful prompts are different from:

Refactor this application using Clean Architecture.

Instead, give the model a constrained transformation.

For example:

This service currently accesses MySQL directly.

I want to introduce an OrderRepository seam without
changing observable behavior.

Tasks:

1. identify every database operation used by this service,
2. propose the smallest repository interface needed,
3. move existing database calls behind an adapter,
4. preserve return values, errors, and call order where relevant,
5. do not change business rules,
6. do not introduce additional abstractions.

Explain every structural change before generating code.

That gives AI a much narrower job.

Another useful request is:

Compare the implementation before and after this refactor.

Identify any observable behavior that may have changed.

Check specifically:

- exceptions,
- return values,
- side effects,
- ordering of side effects,
- null handling,
- transaction boundaries,
- retry behavior.

Do not assume equivalence because the code looks similar.

This is where AI can be valuable as a second reviewer.

It can inspect differences faster than you can manually scan large changes. But the tests still provide stronger evidence.

Don't Ask AI to Design the Target Architecture Too Early

AI is very good at recognizing common architecture patterns. But that can also be dangerous.

Give a model a large legacy service and ask:

How should this be modernized?

and you may receive:

microservices
event-driven architecture
CQRS
repository pattern
domain events
message broker
API gateway
distributed cache

All of those are legitimate technologies or patterns, but none of them are automatically justified.

Before choosing a target architecture, you need constraints.

For example:

deployment frequency
team size
transactional requirements
latency
failure tolerance
data ownership
integration boundaries
operational maturity
traffic
cost
regulatory requirements

A monolith with good boundaries may be a better target than microservices. A synchronous workflow may be better than event-driven processing. And a database migration may not require changing the domain model.

Architecture should follow constraints, not pattern recognition.

Use AI to evaluate options. Don't let the presence of a familiar pattern become the reason to adopt it.

How to Know When a Capability Is Ready to Migrate

At some point, you have to stop refactoring. And that decision matters.

You don't need perfect code. A capability is usually much closer to migration-ready when you can answer these questions clearly.

Can I Describe its Inputs?

For example:

orderId
customer
request payload
event

Can I Describe its Outputs?

For example:

processed order
HTTP response
event
database change

Are its Important Business Rules Visible?

They don't have to be perfect, but you should know where they live.

Are External Dependencies Explicit?

For example:

OrderRepository
PaymentGateway
OrderEvents
EmailSender

Can Infrastructure Be Substituted?

If replacing MySQL requires changing pricing logic, the boundary is probably not ready.

Are Important Behaviors Protected?

You should have enough tests to detect accidental changes.

Do You Know the Side Effects?

For example:

persist order
create payment
publish event
send email

Are Major Unknowns Documented?

Some uncertainty may remain. But it shouldn't be invisible.

If you can answer those questions, you probably have enough structure to begin migrating that capability.

A Practical Pre-Migration Refactoring Workflow

Here's the workflow I would use.

1. Choose One Capability

Don't refactor the whole application.

Pick:

Process Order
Generate Invoice
Renew Subscription

2. Confirm Behavioral Protection

Before structural changes, make sure critical behavior has tests.

Capture:

outputs
state transitions
side effects
errors
contracts

3. Identify Migration Blockers

Look for coupling such as:

direct database access
provider SDKs
global state
framework-specific objects
static dependencies
shared mutable state

4. Extract Pure Business Logic Where Possible

Move calculations and decisions away from infrastructure.

For example:

calculate price
validate transition
choose status
calculate commission

5. Introduce Seams

Create minimal boundaries around:

database
payments
events
email
storage
external APIs

Don't create abstractions without a migration reason.

6. Localize Infrastructure

Move technology-specific behavior into adapters.

For example:

MySqlOrderRepository
StripePaymentGateway
KafkaOrderEvents
SendGridEmailSender

7. Make Orchestration Visible

Aim for a capability where the sequence is understandable:

load
↓
decide
↓
persist
↓
perform side effects
↓
return

8. Run Behavioral Tests After Every Step

Don't batch ten refactors together. Keep the changes small.

9. Compare Before and After

Check:

inputs
outputs
errors
side effects
data shapes
ordering
transactions

10. Stop When Migration Becomes Possible

Don't continue refactoring because the code could still be cleaner. It always could.

The objective is migration readiness.

What Not to Refactor Before Migration

There are several things I would usually avoid changing during this phase unless they directly block migration.

Naming Everywhere

You may dislike hundreds of old names. But renaming everything produces large diffs with little migration value.

Formatting the Entire Repository

Same problem. Noise makes behavioral changes harder to review.

Replacing Every Pattern

A legacy system may contain:

singletons
service locators
static utilities
large classes

Some may remain temporarily. Fix the ones crossing your migration boundary.

Rewriting Stable Algorithms

If an old calculation is ugly but protected and isolated, it may be safer to move it first and improve it later.

Fixing Every Discovered Bug

This one is especially important.

If you discover a bug while preparing a migration, record it. Then decide whether fixing it belongs in the same change.

Mixing:

structural refactor
+
behavioral correction
+
platform migration

makes failures much harder to understand.

Sometimes the right answer is:

preserve bug
migrate
fix bug intentionally afterward

That sounds uncomfortable. But accidental behavior changes during migration can be much more dangerous.

Refactoring Is Preparation, Not the Migration

It's easy for pre-migration refactoring to become an endless architecture project.

You start with:

We need to isolate the database.

Then:

We should redesign the domain model.

Then:

Maybe we should introduce events.

Then:

If we're doing that, maybe this should become a microservice.

Months later, nothing has migrated. The refactor has become the project.

That's a failure mode, too. The objective should remain concrete.

Before:

ProcessOrder
├── MySQL
├── pricing rules
├── Stripe
├── Kafka
├── email
└── framework internals

After:

ProcessOrder
├── OrderRepository
├── pricing rules
├── PaymentGateway
├── OrderEvents
└── EmailSender

That may be enough.

Now you have choices.

You can migrate:

MySQL → PostgreSQL

without redesigning pricing.

You can replace:

Stripe adapter

without changing order orchestration.

You can move:

ProcessOrder

into another runtime while preserving its contracts.

The refactor created options. That's the value.

Conclusion

Legacy migrations become risky when several types of change happen at once.

You change:

behavior
architecture
infrastructure
runtime
data
deployment

and then try to understand which change caused the failure.

A safer approach is to reduce that uncertainty before migration begins.

First understand the capability, then characterize its behavior, and then change its structure without intentionally changing what it does.

Create boundaries around dependencies. Separate business decisions from infrastructure. Localize external systems. Keep side effects visible. Run behavioral tests after every structural change. And stop refactoring when the capability becomes movable.

The sequence becomes:

Understand
↓
Characterize
↓
Refactor
↓
Migrate

AI can make the refactoring phase dramatically faster.

It can identify dependencies, extract interfaces, move calls behind adapters, compare implementations, and review large diffs.

But faster refactoring doesn't remove the need for architectural judgment. It makes that judgment more important.

Because the goal isn't to produce the cleanest version of the legacy system. The goal is to create just enough structure to move it safely.

And once you can change the infrastructure without changing the behavior, migration stops looking like a rewrite.

It starts looking like a sequence of controlled changes.

]]> <![CDATA[ How AI Is Changing Patching and What Devs Need to Know About Exposure Management ]]> https://www.freecodecamp.org/news/how-ai-is-breaking-traditional-patch-management/ 6a9aefa26eac286787fb7078 Fri, 04 Sep 2026 16:19:46 +0000 When a vulnerability scanner reports 23 vulnerabilities in your application, of which 4 are critical, 7 are high, and the remaining 12 are medium, at first glance the answer seems clear: start patching. But which one should you fix first?

This has always been an issue in vulnerability management. While a security team might find out about the vulnerable dependency, fixing it may not always be possible at once. Developers need to ensure that the vulnerable code is in use and perform all necessary checks before releasing the fix into production.

Recently, though, there have been some solid advancements in the use of AI for discovering software vulnerabilities and exploits. This research, for example, details some of the findings and the path forward.

But how will this really help the development community? We need to fix things more quickly, but more importantly, we need to be able to figure out which vulnerabilities actually matter and which ones need attention first.

In this article, we'll examine what the classic patching process looks like, how AI is decreasing the amount of time security teams have to react, and why it's not always reasonable just to address vulnerabilities by their severity score.

We'll also discuss exposure management and the difference between it and traditional vulnerability management. Then we'll cover how developers can analyze dependencies, code reachability, and Software Bill of Materials (SBOMs) to figure out the actual vulnerabilities in their applications.

What We'll Cover:

Patching vs. Exposure Management: What's the Difference?

Before we look at how AI is changing vulnerability response, it helps to understand two important concepts: patching and exposure management.

What Is Patching?

Patching is a process of updating software to address an existing issue, including a security vulnerability, bug, or a stability problem. This may involve upgrading a library with a known vulnerability, applying a security update for your operating system, or using a new release of an application with a vulnerability fixed.

For instance, if your application uses a particular library with a known security vulnerability, you can upgrade the library once the patched version is available. After that, you'll need to test the update, ensure that the application still operates as intended, and release the upgraded version into production.

So patching isn't only about installing the latest version of the package in question. A dependency update can break an API, some functionality, or even other dependent packages. That's why teams typically use patch management strategies when identifying vulnerabilities, updating decision-making, testing, deploying, and verifying that the patches work.

What Is Exposure Management?

While vulnerability management is concerned mainly with identifying vulnerabilities, exposure management focuses more broadly on whether those vulnerabilities can actually provide a realistic route for an attack.

For example, a vulnerable library, limited in use to a development system, poses less immediate threat than a similarly vulnerable library used in an internet-facing system that's capable of accessing the database.

Some aspects to consider include accessibility via network, asset exposure, vulnerable code paths, identity and access controls, cloud environments, and the sensitivity of systems and data.

Simply put, vulnerability management is concerned with the discovery and tracking of vulnerabilities, whereas exposure management is focused on determining which of those vulnerabilities pose a true or larger risk.

The Old Patch Management Workflow Was Built Around Time

The classic process of vulnerability mitigation depended on step-by-step actions.

  1. CVE discovered

  2. Security team assesses the severity

  3. Maintainer releases an upstream patch

  4. Developer updates the dependency

  5. CI/CD pipeline runs regression tests

  6. Production deployment

  7. Remediation verified

The process wasn't flawed by nature, but it operated under the unspoken premise that the defenders were granted enough room to operate through each step.

Let's take a dependency vulnerability example for practice. If the automated scanner detects a vulnerability within a popular utility package such as lodash, engineers don't immediately bump the dependency version in the production environment. They need to confirm if the application code leverages the vulnerable function, check the presence of breaking API changes after the upgrade, and run build validation through integration tests.

Each security patch is essentially a change to the code and needs to be safely pushed through the development and deployment cycle.

AI Is Shrinking the Time Between "Found" and "Exploited"

The buffer between vulnerability discovery and exploitation that used to exist is being eliminated. This is because automated programs can scan through codebases, generate proofs of concept, and discover edge cases.

Modern AI systems help researchers and would-be attackers alike perform tasks like static binary analysis, detecting vulnerabilities, creating exploit payloads, and finding logical issues in complicated software designs.

Programs such as DARPA’s Artificial Intelligence Cyber Challenge (AIxCC) show how AI systems can be used to automatically find and patch vulnerabilities in complex open-source software. During the 2024 semifinal competition, autonomous Cyber Reasoning Systems were tested against projects based on real-world software such as Jenkins, the Linux kernel, Nginx, SQLite3, and Apache Tika. The systems discovered 22 unique synthetic vulnerabilities and successfully patched 15 of them. They also identified one real-world bug in SQLite3, which was responsibly disclosed.

In the context of a real development process, certain tasks in the patching process can be performed by AI. The AI could perform code analysis and dependency analysis in order to detect potential vulnerabilities. It could also help trace the usage of vulnerable functions, recommend changes in code and dependencies, and generate tests to make sure that the suggested patch doesn’t break the existing functionality. The security team could also use AI for pattern detection.

As AI tools get better at assessing software and detecting vulnerabilities, the window of time between vulnerability detection and its mitigation becomes smaller and more important. This change in paradigm also affects how security professionals approach AI and exposure management, especially as the exploit window gets smaller and vulnerabilities need proper prioritization.

AI can also support exposure management by connecting vulnerability information with the environment in which the vulnerable software is running. For example, an AI-assisted security system could correlate a vulnerable dependency with an internet-facing application, its network connections, cloud permissions, and the data or services it can access. This helps security teams move from simply asking whether a vulnerability exists to asking what an attacker could realistically reach through it.

Why "Patch Everything" Doesn't Work at Scale

When an organization-wide scanner generates a list of 500 vulnerabilities spread across multiple microservices, reacting to each one with urgency starts to seem impossible. Developers can suffer from alert fatigue and get overwhelmed pretty easily.

The Common Vulnerability Scoring System (CVSS) is a standardized framework used to describe the severity of a vulnerability. CVSS v3.1 uses the following severity ranges:

CVSS Score 

Severity 

0.0 

None 

0.1–3.9 

Low 

4.0–6.9 

Medium 

7.0–8.9 

High 

9.0–10.0 

Critical 

CVSS is useful because it provides both developers and security teams with a common language that describes the severity of a vulnerability. Nevertheless, the rating describes the vulnerability but not the environment where this vulnerability appears. In other words, CVSS doesn't tell you if the functionality used by the vulnerability is really used by your application or if the affected system is exposed to the Internet.

To see why CVSS alone isn't always enough, let's say we have two hypothetical vulnerabilities in an organization's environment:

  • Vulnerability A: This critical remote code execution vulnerability is part of an isolated testing harness or development-only dependency that's never included in the production environment and doesn't have any external network accessibility.

  • Vulnerability B: A high-severity input validation vulnerability is found in an internet-facing API gateway that processes malicious user input and has access to a backend database with customer information.

Looking at just the CVSS score would require the team to focus on Vulnerability A before Vulnerability B. But it's clear that Vulnerability B poses the greater threat to operations. Security studies show us that very few vulnerabilities get exploited once they're known. Telemetry data from the CISA KEV Catalog clearly indicates that attackers focus on a subset of vulnerabilities that have a real path of exploitation.

In reality, teams must consider the CVSS score in addition to many contextual factors when deciding what to fix. A particular vulnerability might have a higher priority if the following conditions are true:

  • it has an impact on an internet-facing production system,

  • there's a known exploit for the vulnerability,

  • there's sensitive information exposed,

  • it impacts an important business function,

  • or it offers an attacker a means of gaining access to other privileged systems.

But vulnerabilities that occur only in development or are inaccessible for some reason likely don't need to be fixed immediately.

The most appropriate method for determining the importance of vulnerabilities is asking some straightforward questions: Is the vulnerable system accessible? Is the vulnerable code accessible? Is there any exploit for this vulnerability? What privileges does the affected service have? What will an attacker be able to access after exploiting the vulnerability?

Assigning the same level of priority to all alerts wastes engineering efforts on vulnerabilities that might pose no or little risk at all.

Exposure Management: Moving from Flaw Counts to Contextual Risk

Exposure management shifts focus from simply cataloging static vulnerabilities to evaluating an organization's actual operational risk posture.

Instead of asking "How many CVEs exist in our repositories?", exposure management asks "Which vulnerable components, misconfigurations, and reachable network paths create exploitable risk across our running assets?"

The difference becomes easier to see when you look at what each approach focuses on:

Dimension 

Traditional Vulnerability Management 

Exposure Management 

Primary Question 

What software bugs and CVEs exist? 

What paths can an attacker exploit to access critical assets? 

Data Scope 

Isolated dependency scans and static vulnerability databases 

Code repositories, cloud runtime, network routing and IAM permissions 

Prioritization Metric 

CVSS base scores and static severity ratings 

Reachability, exploitability, asset sensitivity and environment context 

Primary Action 

Upstream package upgrades and direct software patches 

Risk-based triage: network isolation, configuration changes, or targeted patching 

When there are 10,000 cloud assets managed by an engineering environment and 1,000 vulnerable libraries found through dependency scanners, the combined numbers don't represent the actual security situation. In order to focus on the right level of risks, you should have some knowledge about the context within which each of these vulnerabilities exists.

  • Is the container exposed to the internet or is it hidden behind the internal load balancer?

  • Is the code actually invoking the risky symbol or library function?

  • What are the identity permissions, cloud roles, and databases that are accessible through the vulnerable service?

Having an understanding of this denominator (total numbers of assets that should receive a specific patch) helps teams identify the exposures that are actually threats so that engineering time is spent on solving the problems that impact production data.

The Dependency Tree as an Attack Surface

Modern software delivery depends on multi-layered packages such as npm, PyPI, Maven, NuGet, base operating system layer packages, GitHub Actions, and third-party APIs. The application logic is written by developers, but the final runtime software contains numerous levels of packages:

Your Application Logic → Direct Dependency (Declared in manifest) → Transitive Dependency (Pulled in automatically) → Underlying OS System Packages → Base Container Image / Cloud Runtime.

If a vulnerability is three levels down in the transitive dependencies and the transitive dependency is unmaintained, but can be accessed via external inputs, then that's an essential part of your application's attack surface.

That's why software development teams have started to use Software Bill of Materials (SBOMs). An SBOM is an inventory of software components that constitute the software or application. Depending on the technology used to create the SBOM, different information can be available including component name, version, dependencies and package ID.

It's helpful in cases where a new vulnerability has been identified. For example, if a vulnerability is discovered in a specific version of lodash, the security team can use its SBOMs to identify which applications or container images contain the affected version. They'll then be able to investigate if there's an exploitable exposure.

An SBOM alone doesn't provide security for the application. Its significance lies in increasing visibility to developers and security personnel regarding what is present in their applications.

Practical Takeaways for Developers

These principles will be relevant once you integrate them into your team's routine development process. There are some practical ways you and your team can employ these best practices and strategies:

Audit Transitive Dependencies

The first thing to do is to verify what dependencies are included in your application. This is very useful when it comes to transitive dependencies, because these dependencies may have been automatically added when installing some other package directly.

For Node.js applications, the command npm ls will display the dependency tree. Python programmers may use pipdeptree, while Java programs created using Maven can use the mvn dependency:tree command. These commands can help you understand the origins of packages and the direct dependency that introduced a vulnerable transitive dependency into your project.

Check Code Reachability

Finding a weak point in a dependency doesn't automatically imply that you're using it within your application. Don't take every vulnerability report as a critical production blocker. Instead, you should investigate if the impacted functionality is actually accessible from your application.

Suppose you find a vulnerability in a certain library function. In this case, you need to look through your codebase for any usage of this function and figure out if there's any possibility of passing user-controlled data to it. You may use either the search function provided by your IDE or command-line utilities such as grep.

An unused or inaccessible from the outside function reduces the urgency of the finding. But it doesn't automatically imply that you should ignore it.

Generate an SBOM in CI/CD

You can also produce a Software Bill of Materials using your CI/CD pipeline. Creating an SBOM will help you identify the components used in the software and make it easier to identify affected components once a vulnerability is found.

For instance, using Syft, you can generate an SBOM from a container image by executing the command: syft my-app:latest -o cyclonedx-json > sbom.json.

This will create a CycloneDX JSON file with information on the components within the container image. This SBOM will then be stored along with the build artifacts. Once a new vulnerability is identified in a particular package version, it becomes easy for the security team to know which applications and container images contain this particular component.

Use Compensating Controls When a Patch Isn't Ready

Sometimes there may be no patch available or it may be too risky to implement it straight away since doing so might introduce breaking changes that need further testing. In such cases, you can use compensatory controls to lessen the exposure of the application until it's patched properly.

Depending on the environment, this may involve limiting the network access to the vulnerable component, isolating the workload from sensitive resources, disabling the feature that has been compromised, or minimizing the privileges of the application.

These controls don't take the place of the security patch but only minimize the risk of exploitation until a patch is implemented.

Apply Least Privilege at Runtime

Lastly, restrict the amount of access your applications have at run time. When a compromise is made due to exploitation, it prevents the spread of that breach to other applications or systems.

When deploying container-based applications, you can leverage read-only filesystems, as well as remove capabilities that aren't required in Linux. For instance, Docker provides the option to use --read-only and --cap-drop=ALL when running containers.

Cloud applications also need to adopt the same concept by ensuring the use of IAM permissions, giving access only to what the application requires.

The goal is simple: if one element has been breached, the attacker should be able to gain access to as few components of the environment around it as possible.

The future of software security doesn't rely on how fast organizations can update their packages without knowing the underlying reasons for doing so. With vulnerability detection becoming more efficient through automation, effective mitigation relies on knowledge about the relationship between the source code, its dependencies, and infrastructure at runtime.

The purpose here isn't just finding new vulnerabilities but recognizing the ones that pose real risks to the application.

Wrap Up

While artificial intelligence is helping teams detect vulnerabilities quicker, modern applications keep becoming increasingly dependent on numerous software layers. This doesn't mean that patching becomes unnecessary. It means that all vulnerabilities don't require the same immediate attention.

Developers also still need to look at where those vulnerabilities exist, whether they're reachable by attackers, and what they could affect. As the time between vulnerability detection and exploitation continues to change, understanding exposure becomes just as important as the patch itself.

]]>