The code compiles, the tests pass, the architecture is cleaner, and the new service responds faster.

Then everybody starts asking the same question:

Can we switch traffic now?

That's where confidence becomes difficult.

A new implementation can pass its own test suite and still behave differently from the system it is replacing.

Maybe rounding changed, or null values are handled differently, or an error became a successful response.

Maybe records are sorted differently, or a side effect happens in a different order, or a business rule you never documented was lost during the migration.

This is why, during a legacy migration, I like having another source of evidence: run the old and new implementations with the same inputs and compare what they do.

That's the basic idea behind differential testing. Instead of asking only if the new system passes its tests, you also ask: given the same input, where does the new system behave differently from the old one?

Those differences become evidence.

Some are bugs, some are intentional improvements, some are harmless representation differences, and some reveal behavior nobody knew existed.

In this tutorial, I'll show you how to use differential testing during a legacy migration to:

• Compare old and new implementations

• Define what should be considered equivalent

• Normalize outputs before comparing them

• Handle timestamps and other nondeterministic values

• Compare errors and side effects

• Run differential tests automatically

• Introduce tolerances where exact equality doesn't make sense

• Analyze mismatches

• Use shadow traffic in production safely

• Use AI to classify divergences without letting it decide correctness

• Determine when the new implementation is ready for cutover

The examples use TypeScript and Vitest, but the approach applies to most languages and migration strategies.

The goal isn't to prove that two implementations are internally identical. It's to obtain evidence that they are behaviorally equivalent where equivalence matters.

Prerequisites

To follow along here, you should be comfortable with:

• TypeScript or a similar language

• unit and integration testing

• asynchronous code

• API and service boundaries

• legacy modernization

• basic observability concepts

You should also already have some understanding of the capability being migrated.

Ideally, you know its inputs, outputs, important business rules, external contracts, side effects, and known areas of uncertainty.

Differential testing works best after you've already created a boundary around the capability you want to migrate.

Table of Contents

• Prerequisites

• What Differential Testing Actually Tells You

• Start with One Observable Boundary

• Run the Legacy and New Implementations with the Same Input

• Don't Compare Raw Output Blindly

• Normalize Values Before Comparing Them

• Handle Timestamps and Other Nondeterministic Values

• Compare Business Meaning, Not Just JSON

• Compare Errors as Part of the Contract

• Compare Side Effects, Too

• Use Tolerances When Exact Equality Is Wrong

• Build a Reusable Differential Test Harness

• Generate Test Cases from Real Behavior

• Classify Every Difference

• How to Use AI to Investigate Differential Failures

• How to Use Shadow Traffic Safely

• Measure Divergence Instead of Waiting for Perfection

• How to Know When You're Ready for Cutover

• A Practical Differential Testing Workflow

• What Differential Testing Can't Prove

• Differential Testing Turns Migration Risk into Evidence

• Conclusion

What Differential Testing Actually Tells You

Imagine that your legacy application calculates the final price of an order.

The legacy implementation looks like this:

type Order = {
  subtotal: number;
  customerType: "STANDARD" | "PREMIUM";
  country: string;
};

function legacyCalculateTotal(order: Order): number {
  let total = order.subtotal;

  if (order.customerType === "PREMIUM") {
    total *= 0.9;
  }

  if (order.country === "AR") {
    total -= 500;
  }

  return Math.max(total, 0);
}

During the migration, you create a new implementation:

function newCalculateTotal(order: Order): number {
  const premiumDiscount =
    order.customerType === "PREMIUM"
      ? order.subtotal * 0.1
      : 0;

  const countryAdjustment =
    order.country === "AR"
      ? 500
      : 0;

  return Math.max(
    order.subtotal -
      premiumDiscount -
      countryAdjustment,
    0
  );
}

The implementations look different. And that's fine. What matters is whether they produce equivalent behavior.

A simple differential test can run both:

import { describe, expect, it } from "vitest";

describe("order total migration", () => {
  it("matches the legacy implementation", () => {
    const order: Order = {
      subtotal: 10000,
      customerType: "PREMIUM",
      country: "AR",
    };

    const legacy =
      legacyCalculateTotal(order);

    const migrated =
      newCalculateTotal(order);

    expect(migrated).toBe(legacy);
  });
});

For this input:

legacy → 8500
new    → 8500

Good. But one matching example proves very little.

The value comes from systematically asking:

same input
↓
legacy implementation ──→ result A

same input
↓
new implementation ─────→ result B

compare A and B

Every mismatch gives you something to investigate.

Start with One Observable Boundary

Don't begin by comparing entire applications. To start, choose one capability.

For example:

Calculate Order Total
Generate Invoice
Approve Customer
Renew Subscription
Calculate Commission
Create Shipment

Suppose the migration boundary is:

interface OrderProcessor {
  process(order: Order): Promise<ProcessedOrder>;
}

Now you have two implementations:

LegacyOrderProcessor

NewOrderProcessor

That is a useful differential boundary, because both receive the same conceptual input, and both produce the same conceptual output.

You can compare them without requiring their internal architecture to match.

That matters because migrations often change structure intentionally.

The legacy implementation might be:

controller
→ service
→ SQL
→ provider SDK

while the new implementation might be:

use case
→ repository
→ gateway
→ events

Differential testing shouldn't care. It should care about observable behavior.

Run the Legacy and New Implementations with the Same Input

Suppose both implementations expose:

interface OrderProcessor {
  process(order: Order): Promise<ProcessedOrder>;
}

You can create:

const legacyProcessor =
  new LegacyOrderProcessor();

const newProcessor =
  new NewOrderProcessor();

Then:

it("produces the same processed order", async () => {
  const input: Order = {
    id: "order-1",
    subtotal: 10000,
    customerType: "PREMIUM",
    country: "US",
  };

  const legacy =
    await legacyProcessor.process(
      structuredClone(input)
    );

  const migrated =
    await newProcessor.process(
      structuredClone(input)
    );

  expect(migrated).toEqual(legacy);
});

Notice the use of:

structuredClone(input)

That matters if either implementation mutates its input.

Without separate copies, the first execution could influence the second.

You want:

same initial state

not:

new implementation receives state modified by legacy implementation

That kind of contamination can create misleading results.

Don't Compare Raw Output Blindly

The first version of a differential test is often:

expect(newResult).toEqual(legacyResult);

Sometimes that's exactly right. But other times it's wrong.

Imagine the legacy system returns:

{
  "id": "order-1",
  "total": 9000,
  "status": "PROCESSED",
  "generatedAt": "2026-09-09T10:00:01.231Z",
  "requestId": "legacy-f93a"
}

The new system returns:

{
  "requestId": "new-b517",
  "status": "PROCESSED",
  "generatedAt": "2026-09-09T10:00:01.416Z",
  "total": 9000,
  "id": "order-1"
}

A raw object comparison may fail because:

requestId differs
timestamp differs

But the business behavior might be equivalent.

You need to decide which fields are part of the meaningful contract.

Maybe:

id
total
status

matter.

While:

generatedAt
requestId

don't need exact equivalence.

That leads to normalization.

Normalize Values Before Comparing Them

Normalization means transforming outputs into a common representation before comparing them.

The goal isn't to change the business meaning of the data. It's to remove differences that are expected and irrelevant to the comparison, such as generated request IDs or timestamps, so the test can focus on the fields that actually define the behavior you care about.

In practice, that often means creating a canonical representation: a smaller, stable shape that contains only the meaningful fields you want to compare.

For example:

type ProcessedOrder = {
  id: string;
  total: number;
  status: string;
  generatedAt: string;
  requestId: string;
};

function normalizeOrder(
  order: ProcessedOrder
) {
  return {
    id: order.id,
    total: order.total,
    status: order.status,
  };
}

Here, ProcessedOrder contains both business-relevant fields and values that may legitimately differ between executions.

The normalizeOrder() function keeps id, total, and status, while leaving out generatedAt and requestId. That means two results can still be considered equivalent even if they were generated at slightly different times or used different request identifiers.

Now compare:

expect(
  normalizeOrder(migrated)
).toEqual(
  normalizeOrder(legacy)
);

This makes your equivalence rule explicit.

You're saying:

These fields define relevant behavior for this comparison.

Normalization can also handle:

• ordering

• casing

• optional fields

• timestamps

• generated identifiers

• numeric formatting

• provider-specific metadata

But normalization must be deliberate. If you remove too much, you can hide real migration bugs.

Handle Timestamps and Other Nondeterministic Values

Legacy systems contain many nondeterministic values.

For example:

timestamps
UUIDs
random tokens
request IDs
trace IDs
database-generated IDs
unordered collections
provider-generated references

If you compare those values exactly, your differential suite may fail constantly.

One option is dependency control.

Dependency control means moving a nondeterministic source, such as the current time or an ID generator, behind an interface that you can replace during tests.

Instead of letting each implementation read the real clock independently, you inject the same controlled clock into both. That gives them the same value and removes time itself as a source of meaningless divergence.

Suppose the code uses:

new Date()

You can replace that dependency with a clock:

interface Clock {
  now(): Date;
}

Then both implementations receive:

const clock = {
  now: () =>
    new Date(
      "2026-09-09T10:00:00.000Z"
    ),
};

Now time becomes deterministic.

The same technique can work for ID generation:

interface IdGenerator {
  next(): string;
}

Then tests can provide:

const ids = {
  next: () => "fixed-id",
};

If controlling nondeterminism is impractical, normalize it out only when it's not part of the behavior you need to protect.

Compare Business Meaning, Not Just JSON

Two systems can return different representations while expressing the same business state.

Imagine you have this in your legacy system:

{
  "status": 2
}

And this in your new one:

{
  "status": "APPROVED"
}

Raw comparison says:

different

Business comparison may say:

equivalent

You can create a semantic normalizer:

function normalizeStatus(
  status: number | string
) {
  if (status === 2) {
    return "APPROVED";
  }

  return status;
}

Here, the normalizer translates the legacy numeric value 2 into the business meaning used by the new implementation: "APPROVED".

It doesn't claim that every number and string are interchangeable. It encodes one explicit equivalence rule that you've already decided is valid for this migration.

Then:

expect(
  normalizeStatus(newResult.status)
).toBe(
  normalizeStatus(legacyResult.status)
);

This is especially useful when migration intentionally changes:

database schema
API representation
enumerations
provider-specific formats
internal identifiers

The important question becomes:

Does the observable business meaning remain equivalent?

Not:

Are the bytes identical?

Compare Errors as Part of the Contract

Success responses aren't the whole behavior. Errors matter too.

Suppose the legacy implementation rejects a missing customer:

throw new Error("Customer not found");

The new implementation accidentally returns:

return null;

These two implementations behave very differently for the same invalid input.

The legacy version fails explicitly, while the new version silently returns a value that a caller may interpret as a successful result.

If your differential tests only exercise cases where a valid customer exists, both implementations may appear equivalent and this contract change will remain invisible.

That's why failure behavior has to be compared too.

Create cases that capture errors:

async function captureResult<T>(
  operation: () => Promise<T>
) {
  try {
    return {
      type: "success" as const,
      value: await operation(),
    };
  } catch (error) {
    return {
      type: "error" as const,
      error:
        error instanceof Error
          ? error.message
          : String(error),
    };
  }
}

The helper wraps an asynchronous operation and converts both possible outcomes into data.

If the operation succeeds, it returns an object with type: "success" and the returned value. If the operation throws, the catch block converts that exception into an object with type: "error" and a readable error message.

This gives both implementations the same comparison shape, so the test can compare success versus failure explicitly instead of letting an exception stop the test before the two behaviors can be evaluated.

Now:

const legacy =
  await captureResult(() =>
    legacyProcessor.process(input)
  );

const migrated =
  await captureResult(() =>
    newProcessor.process(input)
  );

expect(migrated.type).toBe(legacy.type);

If errors are contractually important, compare:

error category
HTTP status
error code
retryability
validation details

Don't necessarily compare exact wording unless clients depend on it.

Compare Side Effects, Too

One of the easiest migration mistakes is preserving the return value while losing a side effect.

Suppose both implementations return:

{
  "status": "PROCESSED"
}

But the legacy version also:

persists the order
publishes an event
creates a payment
writes an audit entry

and the new version forgets the audit entry.

Response-level differential testing won't catch that. So you'll want to capture side effects.

For example:

type Effect =
  | {
      type: "payment";
      orderId: string;
      amount: number;
    }
  | {
      type: "event";
      name: string;
      orderId: string;
    };

A test adapter can record them:

class RecordingPaymentGateway {
  effects: Effect[] = [];

  async charge(
    orderId: string,
    amount: number
  ) {
    this.effects.push({
      type: "payment",
      orderId,
      amount,
    });
  }
}

Instead of sending a real payment request, this adapter records what the application attempted to do in the effects array.

You can apply the same idea to event publication:

class RecordingEvents {
  effects: Effect[] = [];

  async publish(
    name: string,
    orderId: string
  ) {
    this.effects.push({
      type: "event",
      name,
      orderId,
    });
  }
}

The application still calls its payment and event dependencies as usual. The test doubles simply capture those calls as structured data instead of performing the real external actions.

After running the legacy and migrated implementations with their own recording adapters, you can compare the two recorded effect lists and verify that both systems attempted the same observable side effects.

Now the differential test can compare:

expect(newEffects).toEqual(legacyEffects);

Again, exact ordering should only be required if ordering matters.

Use Tolerances When Exact Equality Is Wrong

Some domains shouldn't use exact equality.

Imagine a migrated calculation produces:

legacy → 34.333333333
new    → 34.333333334

Is that a migration bug? Maybe not.

Floating-point calculations may justify a tolerance.

For example:

expect(newResult).toBeCloseTo(
  legacyResult,
  6
);

Or define an explicit comparator:

function withinTolerance(
  a: number,
  b: number,
  tolerance: number
) {
  return Math.abs(a - b) <= tolerance;
}

Then:

expect(
  withinTolerance(
    migrated.total,
    legacy.total,
    0.01
  )
).toBe(true);

But tolerances should come from domain requirements. Don't use them just to make failing tests disappear.

For financial systems, one cent can matter. For scientific calculations, a much smaller numerical difference may matter.

Equivalence is a business and engineering decision.

Build a Reusable Differential Test Harness

Once you compare more than a few cases, you can create a reusable harness.

For example:

type DifferentialResult<T> = {
  input: T;
  equivalent: boolean;
  legacy: unknown;
  migrated: unknown;
};

async function compareImplementations<
  TInput,
  TOutput
>(
  input: TInput,
  legacy: (
    input: TInput
  ) => Promise<TOutput>,
  migrated: (
    input: TInput
  ) => Promise<TOutput>,
  normalize: (
    output: TOutput
  ) => unknown
): Promise<
  DifferentialResult<TInput>
> {
  const legacyResult =
    await legacy(
      structuredClone(input)
    );

  const migratedResult =
    await migrated(
      structuredClone(input)
    );

  const normalizedLegacy =
    normalize(legacyResult);

  const normalizedMigrated =
    normalize(migratedResult);

  return {
    input,
    equivalent:
      JSON.stringify(
        normalizedLegacy
      ) ===
      JSON.stringify(
        normalizedMigrated
      ),
    legacy: normalizedLegacy,
    migrated: normalizedMigrated,
  };
}

The harness does four things.

First, it runs the legacy and migrated implementations with separate clones of the same input, so one execution can't mutate the data seen by the other.

Second, it passes both outputs through the same normalize() function. That applies the equivalence rules in one place instead of repeating them in every test.

Third, it compares the normalized results and records whether they're equivalent.

Finally, it returns the input and both normalized outputs together. That makes a failed comparison easier to inspect because the test report can show exactly which case diverged and what each implementation produced.

Then:

const result =
  await compareImplementations(
    input,
    legacyProcessor.process.bind(
      legacyProcessor
    ),
    newProcessor.process.bind(
      newProcessor
    ),
    normalizeOrder
  );

expect(result.equivalent).toBe(true);

For real systems, I would usually avoid relying on JSON.stringify() as the final equality mechanism.

The example keeps the harness readable.

In production-quality tooling, use a proper structural or domain-specific comparator.

The important part is that comparison logic becomes centralized.

Generate Test Cases from Real Behavior

Hand-written examples are useful. But migrations often fail on cases nobody thought to write manually.

Useful sources of inputs include:

existing test fixtures
historical incidents
production-safe request samples
database records
boundary values
previous bug reports
known customer scenarios

Suppose production shows these order shapes:

const cases: Order[] = [
  {
    subtotal: 0,
    customerType: "STANDARD",
    country: "US",
  },
  {
    subtotal: 500,
    customerType: "PREMIUM",
    country: "AR",
  },
  {
    subtotal: 10000,
    customerType: "STANDARD",
    country: "AR",
  },
];

The first block is the test data. It captures a small set of representative input shapes that you've observed in real usage or reconstructed safely from production behavior.

The next block is the test itself. it.each(cases) tells Vitest to run the same differential comparison once for every input in that array.

That separates two concerns: defining realistic cases and defining how every case should be evaluated.

Now:

it.each(cases)(
  "matches legacy behavior",
  async (input) => {
    const legacy =
      await legacyProcessor.process(
        structuredClone(input)
      );

    const migrated =
      await newProcessor.process(
        structuredClone(input)
      );

    expect(
      normalizeOrder(migrated)
    ).toEqual(
      normalizeOrder(legacy)
    );
  }
);

Real examples help expose assumptions that synthetic test data often misses. But production data must be handled carefully.

Remove or anonymize:

personal data
credentials
tokens
financial identifiers
confidential business data

The objective is to preserve useful behavioral shapes, not copy sensitive production information into test fixtures.

Classify Every Difference

A differential failure doesn't automatically mean that the new implementation is wrong.

Suppose you find 200 mismatches. Classify them.

I like categories such as:

migration defect
legacy defect intentionally preserved
intentional behavior change
representation difference
nondeterministic difference
test/comparator defect
unknown

For example:

Input:
subtotal = 5000

Legacy:
discount = 0

New:
discount = 500

Classification:
unknown

Investigation reveals that the new implementation changed:

amount > 5000

to:

amount >= 5000

Now you need a decision.

Was that:

accidental migration change

or:

intentional bug fix

Differential testing exposes the decision. It doesn't make the decision for you.

That's one of its greatest benefits.

How to Use AI to Investigate Differential Failures

Large migrations can produce hundreds or thousands of differences. And AI can help triage them.

Suppose you have:

{
  "input": {
    "subtotal": 5000,
    "country": "AR"
  },
  "legacy": {
    "total": 4500
  },
  "new": {
    "total": 4000
  }
}

You can give the model:

• the input

• both outputs

• relevant legacy code

• relevant migrated code

• the comparator rules

Then ask:

Analyze this differential test failure.

Identify the smallest behavioral difference that could
explain the mismatch.

Compare the legacy and migrated implementations.

Return:

1. observed difference,
2. relevant legacy branch,
3. relevant migrated branch,
4. likely cause,
5. evidence supporting the cause,
6. additional test cases that could confirm it.

Do not decide which behavior is correct.
Do not modify the code yet.

That last instruction matters. AI can be very useful for locating why two implementations diverge. It shouldn't silently turn that diagnosis into a business decision.

Don't Let AI Decide Which Behavior Is Correct

Imagine the legacy system does this:

Customer age 65 → no discount
Customer age 66 → discount

The new system does:

Customer age 65 → discount
Customer age 66 → discount

AI may look at the code and say:

The new implementation appears more logical because senior discounts typically begin at age 65.

That's irrelevant.

The business rule might be:

age > 65

for a reason. Or the legacy behavior might contain a bug.

You need evidence.

Use:

requirements
existing tests
production behavior
business owners
historical tickets
commit history
contracts

AI can help gather and summarize that evidence. It shouldn't invent the rule.

Differential testing is valuable because it tells you that there's a difference before you accidentally turn that difference into production behavior.

How to Use Shadow Traffic Safely

Once offline differential tests look good, you can sometimes compare behavior with real traffic. This is often called shadowing or traffic mirroring.

The pattern looks like:

real request
    │
    ├────────────→ legacy system
    │                  │
    │                  ↓
    │             real response
    │
    └────────────→ new system
                       │
                       ↓
                  shadow result

The user still receives:

legacy response

while the new system processes a copy of the request.

Then you compare:

legacy output
vs.
shadow output

This can reveal cases that your test suite never captured.

For example:

unexpected null combinations
rare customer states
unusual international data
old records
large values
unusual sequence patterns

But shadow execution requires careful design, especially when the operation has side effects.

How to Prevent Shadow Execution from Duplicating Side Effects

Imagine shadowing:

POST /payments

If both systems really execute the payment, you have a serious problem.

The same applies to:

send email
create shipment
charge card
modify inventory
publish event
write external record

The shadow implementation shouldn't perform destructive or externally visible effects unless they're safely isolated.

One approach is to replace real gateways with recording adapters:

class ShadowPaymentGateway
  implements PaymentGateway {
  calls: PaymentRequest[] = [];

  async charge(
    request: PaymentRequest
  ) {
    this.calls.push(request);

    return {
      paymentId: "shadow",
    };
  }
}

The new implementation still tries to execute:

payment

but instead of charging a real card, the shadow adapter records:

what would have been sent

You can then compare that intent with the legacy side effect.

This distinction is important:

compare behavior

does not mean:

duplicate production effects

Measure Divergence Instead of Waiting for Perfection

When running thousands of comparisons, a binary:

pass / fail

may not tell the whole story.

You can measure divergence.

For example:

Requests compared:     100,000
Equivalent:             99,620
Different:                 380

Divergence rate:          0.38%

Then classify those 380:

250 timestamp differences
80 known intentional changes
30 comparator problems
15 migration defects fixed
5 still unexplained

After normalization:

meaningful unresolved divergence:
5 / 100,000
= 0.005%

Now the conversation becomes much more concrete.

Instead of:

I think the migration is ready.

you can say:

We compared 100,000 representative executions and have five unresolved behavioral differences.

Whether that's acceptable depends on what those five cases are.

One incorrect financial transaction can matter more than 100 harmless formatting differences.

So don't evaluate only the percentage. Evaluate the severity.

How to Know When You're Ready for Cutover

Differential testing doesn't give you a universal threshold. But it can give you evidence.

Before cutover, I would want to answer questions such as:

Have Important Input Classes Been Compared?

Not only happy paths.

Include:

boundaries
errors
historical bugs
large values
missing values
rare states

Are Meaningful Differences Classified?

Avoid:

we have 47 unexplained mismatches

Are Critical Differences Resolved?

Especially:

money
authorization
state transitions
data integrity
external contracts
idempotency

Are Intentional Differences Documented?

If the new behavior intentionally differs, that should be explicit.

Are Side Effects Equivalent?

Not only responses.

Have Production-like Cases Been Tested?

Synthetic fixtures alone may not be enough.

Can the Migration Be Rolled Back?

Differential confidence reduces risk. It doesn't eliminate the need for rollback.

If you can answer these questions, you're much closer to a controlled cutover.

A Practical Differential Testing Workflow

Here's the workflow I would use.

1. Pick One Capability

For example:

Process Order
Calculate Invoice
Approve Customer

Don't compare the whole platform at once.

2. Define the Observable Contract

List what matters:

return value
status
error
database state
events
external calls

3. Create Legacy and New Adapters

Expose both implementations through the same conceptual interface.

4. Define Normalization Rules

Decide how to handle:

timestamps
generated IDs
ordering
representation changes
optional values

Do this before looking at lots of failures. Otherwise you may weaken the comparator simply to make results pass.

5. Compare Known Cases

Begin with:

existing tests
characterization cases
edge cases
historical bugs

6. Capture Side Effects

Use recording or fake adapters where necessary.

7. Automate the Harness

Produce structured output for every mismatch.

For example:

{
  "caseId": "case-493",
  "equivalent": false,
  "legacy": {},
  "migrated": {},
  "difference": {}
}

8. Classify Differences

Use categories:

defect
intentional change
normalization issue
nondeterminism
unknown

9. Add Representative Real-World Cases

Use anonymized or safely reconstructed production patterns.

10. Shadow Real Traffic When Appropriate

Only after controlling side effects and privacy risk.

11. Measure Divergence

Track both:

frequency
severity

12. Resolve Unknowns Before Cutover

The most dangerous category is often not:

different

It is:

different and nobody knows why

What Differential Testing Can't Prove

Differential testing has an important limitation: it compares the new system against the old one.

That means the legacy system becomes a behavioral reference. But the legacy system may already be wrong.

Suppose:

legacy output = wrong
new output    = same wrong result

The differential test passes, but that doesn't make the behavior correct.

This is why differential testing should complement:

specification tests
characterization tests
business requirements
security testing
performance testing
contract testing
domain review

It answers:

Did behavior change?

It doesn't automatically answer:

Is this the right behavior?

That distinction matters. The legacy application is evidence, it's not absolute truth.

Differential Testing Turns Migration Risk into Evidence

There's another reason I like this technique. Without differential testing, migration discussions can become subjective.

One person says:

The new implementation looks ready.

Another says:

I do not trust it yet.

Both may have reasonable instincts, but neither statement is very measurable.

Differential testing changes the conversation.

Now you can say:

12,000 cases compared
47 differences found
31 representation differences
9 intentional behavior changes
6 migration defects fixed
1 unresolved

That is a much better engineering discussion. You're converting uncertainty into observable differences. Then you can decide what to do with them.

Conclusion

A legacy migration is not complete because the new implementation passes its own tests.

The harder question is whether it preserves the behavior that matters from the system it is replacing.

Differential testing gives you another way to answer that question.

Run both implementations with the same inputs.

Compare outputs.

Compare errors.

Compare side effects.

Normalize only the differences that truly do not matter.

Investigate everything else.

And when possible, use representative production behavior to discover cases your test suite did not anticipate.

The migration sequence now becomes:

Understand
↓
Characterize
↓
Refactor
↓
Migrate
↓
Compare
↓
Cut over

AI can accelerate this process too.

It can help build comparators, analyze failures, group similar divergences, inspect code paths, and suggest additional test cases.

But it should not decide which implementation is correct.

That still requires evidence, domain knowledge, and engineering judgment.

The purpose of differential testing is not to eliminate uncertainty completely.

It is to make uncertainty visible before you switch production traffic.

Because during a migration, discovering that the new system behaves differently is useful.

Discovering it after the old system has been turned off is much more expensive.

You find a function that's difficult to understand. Or you see duplicated logic, deeply nested conditions, database calls mixed with business rules, and dependencies that make testing almost impossible.

You know the code could be better, so you start cleaning it up.

Then something breaks. Not because the new implementation is obviously wrong. It breaks because the old implementation was doing something nobody knew it was doing.

That's one of the most common risks in legacy modernization.

Before changing code, you need a way to answer a simple question:

Did I preserve the behavior that already mattered?

That is where characterization tests become useful.

A characterization test doesn't begin by asking what the software should do. It begins by documenting what the software does today.

That distinction matters.

In a greenfield application, tests usually express intended behavior. But in a legacy application, you may first need tests that capture existing behavior so you can change the implementation without accidentally changing its observable results.

In this tutorial, I'll show you how to use characterization tests as a safety net before refactoring legacy code.

We'll look at how to:

• identify behavior worth protecting,

• choose useful test boundaries,

• capture current outputs,

• deal with side effects,

• handle databases and external systems,

• use AI to accelerate test discovery,

• avoid freezing implementation details,

• decide what not to characterize,

• and turn characterization tests into a foundation for safer refactoring.

The examples use TypeScript and Vitest, but the approach applies to most languages and testing frameworks.

The goal isn't to preserve every line of legacy behavior forever. The goal is to make behavior visible before you start changing the code that produces it.

Prerequisites

You should be comfortable with:

• TypeScript or a similar programming language

• unit and integration testing

• dependency injection

• mocks and test doubles

• basic refactoring techniques

• reading an unfamiliar codebase

It also helps if you've already mapped the capability you want to change.

Before writing characterization tests, you should have some idea of:

• where the behavior starts

• what state it changes

• which external systems it touches

• which outputs may be consumed elsewhere

Table of Contents

• Prerequisites

• What Characterization Tests Actually Protect

• Start with Behavior, Not Implementation

• Choose One Capability Before Writing Tests

• Find the Smallest Useful Test Boundary

• Capture Existing Behavior Before Improving It

• Characterize Edge Cases You Don't Yet Understand

• Test Side Effects, Not Just Return Values

• How to Characterize Code That Depends on a Database

• How to Handle External Services

• How to Use AI to Discover Characterization Tests

• Don't Let AI Invent Expected Behavior

• Avoid Testing Implementation Details

• When a Characterization Test Reveals a Bug

• How Much Behavior Should You Characterize

• Use Characterization Tests During the Refactor

• A Practical Characterization Testing Workflow

• What Characterization Tests Can't Tell You

• Characterization Tests Are Temporary Knowledge Infrastructure

• Conclusion

What Characterization Tests Actually Protect

Suppose you inherit this function:

type Customer = {
  id: string;
  type: "STANDARD" | "PREMIUM";
};

type Order = {
  id: string;
  customer: Customer;
  subtotal: number;
  country: string;
  paymentMethod: "CARD" | "TRANSFER";
};

async function processOrder(order: Order) {
  let total = order.subtotal;

  if (order.customer.type === "PREMIUM") {
    total = total * 0.9;
  }

  if (order.country === "AR" && order.paymentMethod === "TRANSFER") {
    total = total - 500;
  }

  if (total < 0) {
    total = 0;
  }

  await ordersRepository.save({
    ...order,
    total,
    status: "PROCESSED",
  });

  await eventBus.publish("order.processed", {
    orderId: order.id,
    total,
  });

  return total;
}

There are several things you may want to refactor here.

For example, the pricing rules could move to another module. Persistence could be isolated. The event publisher could sit behind an interface. And the function could return an object rather than a primitive.

Those may all be good decisions, but before making them, you should ask: What behavior currently matters?

For this function, observable behavior includes at least:

• premium customers receive a 10% discount

• Argentine transfers receive another adjustment

• totals can't become negative

• the order is persisted with a specific status

• an event is published

• the event contains the calculated total

• the function returns that total

A characterization test gives you a baseline for those behaviors.

For example:

import { describe, expect, it, vi } from "vitest";

describe("processOrder", () => {
  it("applies the existing premium customer behavior", async () => {
    const save = vi.spyOn(ordersRepository, "save");
    const publish = vi.spyOn(eventBus, "publish");

    const order: Order = {
      id: "order-1",
      customer: {
        id: "customer-1",
        type: "PREMIUM",
      },
      subtotal: 10000,
      country: "US",
      paymentMethod: "CARD",
    };

    const result = await processOrder(order);

    expect(result).toBe(9000);

    expect(save).toHaveBeenCalledWith(
      expect.objectContaining({
        id: "order-1",
        total: 9000,
        status: "PROCESSED",
      })
    );

    expect(publish).toHaveBeenCalledWith("order.processed", {
      orderId: "order-1",
      total: 9000,
    });
  });
});

This test isn't saying that a 10% discount is the best pricing model.

It's saying:

This is what the system currently does.

That's the contract you need to understand before changing it.

Start with Behavior, Not Implementation

A common mistake is to write tests around the structure you're planning to create.

Suppose you want to refactor the previous code into:

OrderProcessor
PricingPolicy
OrdersRepository
OrderEventPublisher

You may be tempted to write tests for those future classes first.

But those classes don't describe the existing system. They describe your proposed design.

Characterization tests should begin at the current observable boundary.

Instead of asking:

How should PricingPolicy work?

ask:

Given this input, what does processOrder() currently produce?

That difference helps prevent your new architecture from redefining behavior accidentally.

The sequence should be:

Observe existing behavior
↓
Capture it
↓
Refactor implementation
↓
Run characterization tests
↓
Verify behavior remains stable

Not:

Design new architecture
↓
Write tests for new architecture
↓
Assume it matches the old system

The second workflow tests your design. The first protects the migration.

Choose One Capability Before Writing Tests

Don't start by trying to characterize an entire legacy application. Instead, pick one business capability.

For example:

Approve Order
Generate Invoice
Renew Subscription
Register Customer
Calculate Commission
Cancel Reservation

Then trace that capability through the system.

Suppose you choose:

Generate Invoice

You discover this path:

POST /orders/:id/invoice
        ↓
InvoiceController.generate()
        ↓
InvoiceService.generate()
        ↓
TaxCalculator.calculate()
        ↓
InvoiceRepository.save()
        ↓
PdfGenerator.create()
        ↓
EmailService.send()

That becomes the scope of your investigation.

Now ask: Which behaviors matter if I refactor this capability?

Perhaps:

tax calculation
invoice numbering
database state
PDF fields
email recipient
email attachment
error behavior

Those are candidates for characterization.

This is more useful than trying to increase test coverage across the repository indiscriminately.

Coverage isn't the goal. Behavioral confidence is.

Find the Smallest Useful Test Boundary

Characterization tests can exist at different levels.

You might test:

function
service
module
API endpoint
background job
complete workflow

The right boundary is usually the smallest one that still captures meaningful behavior.

Suppose the logic you want to refactor lives inside:

class InvoiceService {
  async generate(orderId: string) {
    // 300 lines of legacy behavior
  }
}

If generate() coordinates tax calculation, persistence, numbering, and external calls, testing a small internal helper may not protect enough behavior.

Testing the whole production stack may be too slow and difficult.

A service-level characterization test may be the useful compromise.

For example:

describe("InvoiceService.generate", () => {
  it("preserves the existing invoice calculation", async () => {
    const service = createInvoiceService();

    const invoice = await service.generate("order-123");

    expect(invoice.subtotal).toBe(10000);
    expect(invoice.tax).toBe(2100);
    expect(invoice.total).toBe(12100);
  });
});

You don't want to ask what's the smallest unit you can test. You want to ask what's the smallest boundary that gives you confidence during this refactor.

Those aren't always the same thing.

Capture Existing Behavior Before Improving It

Legacy code often contains behavior that looks suspicious.

Consider:

function calculateDiscount(amount: number) {
  if (amount > 10000) {
    return amount * 0.15;
  }

  if (amount > 5000) {
    return amount * 0.1;
  }

  return 0;
}

You run a few examples and discover:

5000  -> 0
5001  -> 500.1
10000 -> 1000
10001 -> 1500.15

You might think:

5000 should probably receive the 10% discount.

Maybe. But that's not what the current code does.

A characterization test could record:

describe("calculateDiscount", () => {
  it.each([
    [5000, 0],
    [5001, 500.1],
    [10000, 1000],
    [10001, 1500.15],
  ])(
    "returns the existing discount for amount %d",
    (amount, expected) => {
      expect(calculateDiscount(amount)).toBe(expected);
    }
  );
});

This creates a behavioral boundary around the existing implementation.

Later, if the business confirms that 5000 should receive a discount, you can intentionally change:

if (amount > 5000)

to:

if (amount >= 5000)

and update the relevant test.

The important part is that the change becomes explicit.

Without the test, it could happen accidentally during an unrelated refactor.

Characterize Edge Cases You Don't Yet Understand

The obvious cases aren't always the risky ones. Legacy systems often fail at boundaries.

Look for values such as:

0
-1
null
empty string
maximum value
minimum value
exact threshold values
unknown status
duplicate identifiers
missing related records

Suppose you find:

function normalizeBalance(balance?: number) {
  if (!balance) {
    return 0;
  }

  return Math.round(balance * 100) / 100;
}

That means:

undefined -> 0
0         -> 0

But also potentially:

NaN -> 0

because NaN is falsy.

Is that intentional? You may not know yet.

You can characterize it:

describe("normalizeBalance", () => {
  it("returns zero for undefined", () => {
    expect(normalizeBalance(undefined)).toBe(0);
  });

  it("returns zero for zero", () => {
    expect(normalizeBalance(0)).toBe(0);
  });

  it("returns zero for NaN in the current implementation", () => {
    expect(normalizeBalance(Number.NaN)).toBe(0);
  });
});

The name matters.

Notice that I wrote:

in the current implementation

I'm not pretending that behavior is correct. I'm just documenting it.

That distinction becomes important when a test describes questionable behavior.

Test Side Effects, Not Just Return Values

A return value is only one kind of behavior.

Legacy functions frequently produce side effects.

Consider:

async function cancelOrder(order: Order) {
  order.status = "CANCELLED";

  await orders.save(order);
  await inventory.release(order.id);
  await audit.log("ORDER_CANCELLED", order.id);

  return order;
}

A weak characterization test might only check:

expect(result.status).toBe("CANCELLED");

But a refactor could still accidentally remove:

inventory.release()
audit.log()

and the test would continue passing.

A stronger characterization test captures observable side effects:

it("preserves cancellation side effects", async () => {
  const save = vi.spyOn(orders, "save");
  const release = vi.spyOn(inventory, "release");
  const log = vi.spyOn(audit, "log");

  const order = {
    id: "order-1",
    status: "APPROVED",
  } as Order;

  await cancelOrder(order);

  expect(save).toHaveBeenCalled();

  expect(release).toHaveBeenCalledWith("order-1");

  expect(log).toHaveBeenCalledWith(
    "ORDER_CANCELLED",
    "order-1"
  );
});

This doesn't mean every internal call deserves an assertion.

The question is whether the call produces observable behavior that matters outside the implementation.

How to Characterize Code That Depends on a Database

Database-heavy legacy code can be difficult to test.

Suppose you have:

async function activateCustomer(customerId: string) {
  const customer = await db.customers.findById(customerId);

  if (!customer) {
    throw new Error("Customer not found");
  }

  await db.customers.update(customerId, {
    status: "ACTIVE",
    activatedAt: new Date(),
  });

  return db.customers.findById(customerId);
}

You have several options.

Use an Integration Test

If the database behavior itself matters, run against a disposable test database.

For example:

it("activates an existing customer", async () => {
  await seedCustomer({
    id: "customer-1",
    status: "PENDING",
  });

  const result = await activateCustomer("customer-1");

  expect(result?.status).toBe("ACTIVE");
  expect(result?.activatedAt).toBeTruthy();
});

This gives high confidence, but the test may be slower.

Introduce a Seam

If database access makes testing impractical, you may need a very small structural change before characterization.

For example:

type CustomerRepository = {
  findById(id: string): Promise<Customer | null>;
  update(
    id: string,
    data: Partial<Customer>
  ): Promise<void>;
};

Then:

async function activateCustomer(
  customerId: string,
  customers: CustomerRepository
) {
  // existing behavior
}

This is a useful concept from legacy-code work: create a seam, a place where behavior can be observed or replaced without rewriting the system.

The key is to keep this preparatory change mechanical.

Don't redesign the business logic while creating the test boundary.

First make it testable. Then characterize it. Then refactor.

How to Handle External Services

Legacy code frequently talks directly to:

payment providers
email services
ERPs
CRMs
message brokers
cloud storage
third-party APIs

You usually don't want characterization tests repeatedly calling those systems.

Instead, capture the interaction at the boundary.

Suppose:

async function chargeOrder(order: Order) {
  const response = await stripe.charge({
    amount: order.total,
    currency: "usd",
    customerId: order.customerId,
  });

  await orders.markPaid(order.id, response.id);

  return response.id;
}

You can characterize the request:

it("sends the existing payment payload", async () => {
  const charge = vi
    .spyOn(stripe, "charge")
    .mockResolvedValue({
      id: "payment-123",
    });

  const markPaid = vi.spyOn(orders, "markPaid");

  const order = {
    id: "order-1",
    total: 5000,
    customerId: "customer-1",
  } as Order;

  await chargeOrder(order);

  expect(charge).toHaveBeenCalledWith({
    amount: 5000,
    currency: "usd",
    customerId: "customer-1",
  });

  expect(markPaid).toHaveBeenCalledWith(
    "order-1",
    "payment-123"
  );
});

That protects the external contract without hitting the external system.

But be careful. If the provider behavior itself matters, mocks alone may not be enough.

You might also need:

• provider sandbox tests

• contract tests

• integration tests

• schema validation

Characterization testing doesn't eliminate the need for those layers.

How to Use AI to Discover Characterization Tests

AI is particularly useful when you are staring at a large legacy function and trying to understand what deserves a test.

Suppose you have a 400-line service.

Instead of asking:

Write unit tests for this class.

use a more investigative prompt:

Analyze this class without changing it.

Identify observable behaviors that could change during refactoring.

Group them into:

1. returned values,
2. state changes,
3. persistence effects,
4. external calls,
5. emitted events,
6. exceptions,
7. boundary conditions.

For every proposed characterization test:

- reference the relevant source code,
- explain what behavior the test would protect,
- distinguish observed behavior from inferred behavior.

Do not invent expected values.

That final instruction matters: you want AI to help identify what to observe. You don't want it inventing what the software should do.

Another useful prompt is:

Review the existing test suite for this capability.

Compare the behaviors covered by tests with the
observable behaviors in the implementation.

List behavior that appears unprotected.

Do not generate tests yet.

This is often more valuable than immediately asking for test code.

First identify the gaps, and then decide which gaps matter.

Don't Let AI Invent Expected Behavior

This is probably the most important rule when combining AI with characterization testing, and it's worth talking a bit more about.

Suppose AI reads:

if (customer.age > 65) {
  discount = 0.2;
}

It may generate:

expect(calculateDiscount(65)).toBe(0.2);

because it assumes the intended business rule is:

Customers aged 65 or older receive a discount.

But that's not what the code says.

The existing behavior is:

65 -> no discount
66 -> discount

The expected values in characterization tests should come from evidence.

Useful evidence includes:

• running the current system

• existing tests

• fixtures

• production-safe observations

• documented examples

• database state

• historical behavior

Don't derive expectations solely from what seems reasonable.

A better AI instruction is:

For each candidate test, tell me how I can obtain
the expected result from the current implementation.

Do not propose the expected result yourself unless it
can be directly derived from executable behavior
or an existing test.

This turns AI into an assistant for experiment design rather than an authority on business rules.

Avoid Testing Implementation Details

Characterization tests can become harmful if they freeze the current code structure.

Suppose the implementation is:

async function processOrder(order: Order) {
  validateOrder(order);
  calculatePrice(order);
  reserveInventory(order);
  saveOrder(order);
}

A brittle test might assert:

expect(validateOrder).toHaveBeenCalledBefore(calculatePrice);
expect(calculatePrice).toHaveBeenCalledBefore(reserveInventory);
expect(reserveInventory).toHaveBeenCalledBefore(saveOrder);

Maybe that order matters. Maybe it doesn't.

If consumers only care about:

correct total
inventory reserved
order persisted

then asserting the exact sequence unnecessarily constrains the refactor.

Prefer protecting externally meaningful behavior.

For example:

expect(savedOrder.total).toBe(9000);
expect(inventory.reserve).toHaveBeenCalledWith(
  "product-1",
  2
);
expect(repository.save).toHaveBeenCalled();

Characterization tests should create a safety net. They shouldn't turn the legacy implementation into a specification of every internal decision.

When a Characterization Test Reveals a Bug

Eventually you'll encounter behavior that appears clearly wrong.

For example:

function calculateFee(amount: number) {
  if (amount === 0) {
    return 100;
  }

  return amount * 0.02;
}

You confirm that zero-value transactions are charged a fixed fee. Everyone agrees this looks suspicious.

What should the characterization test do?

First, separate two questions:

1. What does the system do today?

2. What should the system do?

The characterization test answers the first.

it("currently charges 100 for a zero-value transaction", () => {
  expect(calculateFee(0)).toBe(100);
});

Then investigate whether this is intentional business behavior, a historical workaround, or an actual defect.

If the business confirms it is a bug, create a separate change.

For example:

it("does not charge a fee for a zero-value transaction", () => {
  expect(calculateFee(0)).toBe(0);
});

Then modify the production code.

This may sound overly formal for a small condition. But it creates a clean distinction between:

behavior discovered during refactoring

and:

behavior intentionally changed

That distinction becomes extremely valuable in large migrations.

How Much Behavior Should You Characterize

You don't need to characterize everything. Trying to preserve every observed detail can create another form of paralysis.

Prioritize behavior with high change risk or high business impact.

I usually look first at:

• financial calculations

• state transitions

• authentication and authorization

• external contracts

• queue and event payloads

• data transformations

• retry behavior

• idempotency

• regulatory rules

• critical error handling

You may care less about:

• internal helper naming

• private method structure

• log wording that nobody consumes

• temporary object shapes

• implementation-specific call sequences

A useful question is: If this behavior changed during refactoring, could somebody outside this function notice?

If the answer is yes, it's probably worth considering.

Use Characterization Tests During the Refactor

Once the characterization suite exists, keep the refactor small.

Suppose you begin with:

async function processOrder(order: Order) {
  // validation
  // pricing
  // inventory
  // persistence
  // event publishing
}

You might first extract pricing:

function calculateOrderTotal(order: Order) {
  let total = order.subtotal;

  if (order.customer.type === "PREMIUM") {
    total *= 0.9;
  }

  if (
    order.country === "AR" &&
    order.paymentMethod === "TRANSFER"
  ) {
    total -= 500;
  }

  return Math.max(total, 0);
}

Run the characterization suite. If everything still passes, continue.

Next isolate inventory and run it again.

Then persistence. Run it again.

This gives you a migration rhythm:

small structural change
↓
run tests
↓
observe
↓
continue

If something fails, the search space is small.

Compare that with rewriting 2,000 lines and then discovering 47 broken tests.

Small changes turn failures into useful feedback, while Large changes turn failures into archaeology.

Again.

A Practical Characterization Testing Workflow

Here is the workflow I would use on an unfamiliar legacy capability.

1. Map the Capability

Identify:

entry point
business logic
state changes
side effects
external contracts
outputs

Don't refactor yet.

2. Find Existing Tests

Search for tests that already describe the capability.

Look for:

happy paths
boundary cases
errors
historical bugs
integration behavior

Don't duplicate useful tests unnecessarily.

3. List Observable Behaviors

Create a table such as:

Now you know where the risk is.

4. Pick the Test Boundary

Decide whether the useful boundary is:

function
service
module
endpoint
job
workflow

Choose based on confidence, not test ideology.

5. Capture Current Behavior

Run the existing system.

Use real observable outputs when possible.

Don't guess expectations.

6. Add Critical Edge Cases

Test:

thresholds
empty values
nulls
errors
duplicate operations
retry scenarios

especially around logic you intend to change.

7. Capture Side Effects

Protect meaningful:

writes
events
messages
external calls
state transitions

not only function return values.

8. Mark Uncertain Behavior

Use test names or documentation that clearly distinguishes:

confirmed business rule

from:

current observed behavior

9. Refactor Incrementally

Make one structural change.

Run the suite.

Repeat.

10. Replace Characterization with Intent Where Appropriate

As understanding improves, some characterization tests can evolve into true specification tests.

Instead of:

currently returns 0 for this input

you may eventually be able to say:

does not apply a discount below the premium threshold

That transition is useful. It means the system is becoming understood rather than merely preserved.

What Characterization Tests Can't Tell You

Characterization tests are powerful, but they have an important limitation.

They tell you what happened for the cases you observed. They don't automatically tell you why.

Suppose the test says:

Argentine transfer orders receive a 500-unit adjustment.

The test can protect that behavior.

It can't tell you whether the adjustment exists because of:

• a tax rule

• a banking fee

• an old promotion

• a customer-specific workaround

• a bug nobody removed

For that, you still need other evidence:

• documentation

• Git history

• production telemetry

• domain experts

• incident records

• external system contracts

This is why characterization testing belongs after codebase understanding, not instead of it.

You first discover the behavior. Then you protect it. Then you continue investigating what it means.

Characterization Tests Are Temporary Knowledge Infrastructure

There's another way I think about these tests.

Legacy systems contain knowledge that's often trapped inside implementation details. A characterization test moves some of that knowledge into an executable form.

Before:

Nobody knows what changing this condition will break.

After:

Changing this condition causes these four observable behaviors to change.

That's already progress.

The test suite becomes part of your understanding of the system. It creates a bridge between:

what the code currently does

and:

what we eventually want the system to do

You don't have to keep every characterization test forever.

Some will become proper specification tests.

Some will disappear when obsolete behavior is intentionally removed.

Some will remain as regression tests.

Their first job is simpler: make change safer while understanding is still incomplete.

Conclusion

AI makes refactoring legacy code faster.

It can explain functions, generate candidate abstractions, extract interfaces, suggest module boundaries, and rewrite large sections of code in seconds.

That makes characterization testing more important, not less.

When the cost of producing a new implementation decreases, the risk shifts toward verifying that the new implementation still preserves the behavior that matters.

Before asking:

How should I refactor this?

ask:

What does this do today?

Then:

Which of those behaviors matter?

Then:

How can I prove they still work after the change?

That is what characterization tests give you.

They don't tell you that legacy behavior is correct. They give you evidence that it exists.

And once that evidence is executable, you can refactor with much more confidence.

The sequence becomes:

Understand
↓
Characterize
↓
Refactor
↓
Verify

AI can accelerate every step in that workflow. But the engineering judgment remains in deciding what behavior deserves to survive, what behavior should change, and when you have enough evidence to safely make that distinction.

Coming from traditional software testing, I was used to a familiar pattern.

A requirement tells us what the system should do. We create a test case, provide an input, define an expected result, execute the test, and compare the actual result with what we expected.

For example:

Then I started learning conversational AI. Suddenly, the same approach didn't fit quite as neatly.

If I ask an AI agent "How can I reset my password?", it might answer, "You can reset your password using the Forgot Password option on the login page."

Ask the same question again and it might say: "Select Forgot Password from the sign-in screen and follow the instructions sent to your registered email."

The wording is different, but both responses may be perfectly acceptable. So how do we test something when the exact response can change?

That question changed the way I approached conversational AI testing.

In this article, I'll walk through the testing areas I found most important while learning how conversational systems behave, and show how traditional QA techniques can be adapted for AI agents.

Table of Contents

• 1. Start With Intent, Not Exact Wording

• 2. Don't Use Exact Text Matching for Every Response

• 3. Evaluate Response Quality Across Multiple Dimensions

• 4. Test the Conversation, Not Just the Response

• 5. Test Whether the AI Can Handle Corrections

• 6. Test Ambiguity

• 7. Test the Knowledge Behind the Answer

• 8. Test for Hallucinations

• 9. Test Fallback Behavior

• 10. Test Human Escalation

• 11. Test Integrations Like You Would in Any Other Application

• 12. Build a Golden Dataset

• 13. Don't Only Measure Pass Rate

• 14. Create Risk-Based Conversational Tests

• 15. A Practical Conversational AI Test Strategy

• What Traditional QA Engineers Already Bring to AI Testing

• Wrapping Up

1. Start With Intent, Not Exact Wording

Consider these three messages:

1. "How do I reset my password?"

2. "I can't access my account."

3. "Forgot password."

They look different. But depending on the application, they may all represent the same underlying user goal:

PASSWORD_RESET

In conversational AI, the sentence a user types is often called an utterance, while the goal behind that message can be represented as an intent.

This creates an important testing question: Can the system understand the same intent when users express it differently?

A simple test set could look like this:

Let's look at a small example:

test_cases = [ 
          { "message": "I forgot my password", 
            "expected_intent": "PASSWORD_RESET", 
          }, 
          { "message": "How do I change my password?", 
            "expected_intent": "PASSWORD_RESET", 
          }, 
          { "message": "Can't get into my account",  
            "expected_intent": "PASSWORD_RESET", 
          }, 
]

for test in test_cases: 
    response = ai_agent.send(test["message"])
    assert response.intent == test["expected_intent"]

Before running this test, you need to know the expected intent. It usually comes from the application's approved intent definitions or a reviewed test dataset. The automation isn't deciding what the correct intent should be. It's checking whether the AI agent classified the user's message according to the behavior the team already defined:

Input: "Can't get into my account"

Expected intent: PASSWORD_RESET Actual intent: PASSWORD_RESET

PASS

Don't stop with clean sentences.

Real users make spelling mistakes, use abbreviations, provide incomplete information, and sometimes type only a few words.

So I would also test:

"forgot pwd"

"cant login"

"password help"

"locked out"

This is where conversational AI testing starts becoming interesting. We're not testing whether the system recognizes one predefined sentence. We're testing whether it understands variations of a user's goal.

2. Don't Use Exact Text Matching for Every Response

One of the first habits I had to reconsider was comparing actual and expected responses word for word.

Suppose the expected answer is: "You can reset your password using the Forgot Password link."

But the AI responds: "Select Forgot Password on the login screen to begin resetting your password."

An exact string comparison fails. But from a user's perspective, the response may be completely correct.

Instead of defining the expected result as one sentence, define the properties a good response must contain.

For example, the response should:

• explain how to start the password-reset process

• provide an actionable next step

• not ask the user to reveal their password

• avoid inventing account information

• stay relevant to password recovery

Now multiple responses can pass without being identical. This was one of the biggest changes for me.

For deterministic applications, expected output is often a value. For conversational AI, the expected result may need to be a set of evaluation criteria.

3. Evaluate Response Quality Across Multiple Dimensions

Correctness is important, but it shouldn't be the only thing you evaluate. I find it useful to break response quality into several dimensions.

1. Accuracy: is the information correct? If the AI says customers can reset passwords through email when the actual process requires contacting support, the response fails even if it sounds convincing.

2. Relevance: did the AI answer what the user actually asked? A response can contain accurate information and still be irrelevant.

3. Completeness: did the response include the information required for the user to move forward?

4. Clarity: can the user easily understand the answer?

5. Helpfulness: does the response actually help the user accomplish their goal?

A simple scoring rubric could look like this:

You can then define a threshold appropriate for your application.

The exact scoring system isn't the important part. What's important is making the evaluation criteria explicit instead of relying on: "This answer looks good to me."

4. Test the Conversation, Not Just the Response

Single-turn testing is useful, but users rarely interact with an AI agent using perfectly isolated questions.

Consider this conversation:

User: I need to update my address.

The AI explains the process.

Then:

User: Can I do that online?

What does "that" mean? The second message depends entirely on the first.

Now imagine:

User: I need to update my address.

AI: Sure. I can help with that.

User: Actually, before that, can you tell me when my next payment is due?

AI: Your next payment is due on September 15.

User: Thanks. Now back to the address.

Can the system return to the original topic?

That's a different type of test.

Your multi-turn test suite should include scenarios such as:

• follow-up questions

• references to earlier messages

• topic switching

• returning to a previous topic

• user corrections

• incomplete information

• ambiguous questions

• repeated questions

This changed the unit of testing for me. Sometimes you're testing a response, while other times you're testing the entire conversation.

Here's an example:

ai_agent.send("My order number is A10245")
response = ai_agent.send("When will it arrive?")

assert response.order_id == "A10245"

The second message doesn't contain the order number. This check verifies that the AI agent retained information from the earlier turn instead of treating “When will it arrive?” as an unrelated question.

5. Test Whether the AI Can Handle Corrections

People change their minds and make mistakes.

For example:

User: My account number ends in 4567.

Then:

User: Sorry, I meant 4576.

What happens next?

The system should ideally use the corrected information rather than continuing with the original value.

The same principle applies to other conversational details.

"I'm traveling to Boston."

followed by:

"Actually, make that Chicago."

Or:

"I need the report for June."

followed by:

"Sorry, July."

These are useful tests because they expose whether the agent is genuinely maintaining conversational context or simply accumulating information without understanding which information is current.

6. Test Ambiguity

Users don't always provide enough information.

Imagine someone types: "I want to change it."

Change what? Their address? Password? Payment method? Notification preference?

A poor conversational system may guess. A better one may ask: "What would you like to change?"

This gives us another important test category: clarification behavior.

Create intentionally ambiguous utterances such as:

• "How do I update it?"

• "It's not working."

• "Can you change that?"

• "I need help with my account."

Then evaluate whether the AI recognizes that information is missing, avoids making unsupported assumptions, and asks an appropriate clarification question.

Sometimes the best AI response isn't an answer, it's another question.

7. Test the Knowledge Behind the Answer

At first, I focused almost entirely on what the AI said. Then I realized that a poor answer doesn't automatically mean the language model itself is the problem.

The system may be using a knowledge base, retrieval system, documentation, APIs, or other enterprise data sources.

If that information is wrong, incomplete, conflicting, or outdated, the AI may produce a poor response even when the model is behaving as designed.

Suppose the official policy says: Customers have 30 days to return a product. But an outdated knowledge article says that customers have 60 days.

If the AI retrieves the outdated article and confidently answers "60 days," the response is wrong.

But the investigation shouldn't end with: "The AI hallucinated." The tester needs to determine where the wrong information came from.

Questions I would investigate include:

• What source did the response use?

• Was the source approved?

• Is the information current?

• Were multiple sources contradictory?

• Did retrieval return the correct document?

• Did the final response accurately represent the retrieved information?

This becomes particularly important in systems using retrieval-augmented generation, or RAG.

8. Test for Hallucinations

One of the most important conversational AI tests is surprisingly simple: Ask about something the system doesn't know.

Suppose an internal support assistant contains documentation for Products A, B, and C. Ask: "What is the cancellation policy for Product Z?"

Product Z doesn't exist.

So what should happen? The worst outcome is for the system to confidently invent a cancellation policy.

Depending on the application, better behavior might be:

"I don't have information about Product Z."

or:

"I couldn't find that information. Would you like me to connect you with support?"

A useful hallucination test suite should include:

• nonexistent products

• fake policy names

• unsupported features

• deliberately incorrect assumptions

• questions outside the knowledge domain

• requests for information unavailable to the system

You're testing whether the AI knows when not to answer.

9. Test Fallback Behavior

Every conversational system will eventually receive something it doesn't understand. But that isn't necessarily a failure.

The important question is what happens next?

Imagine a scenario like this:

User: I need help with my ZXP adjustment.

The system doesn't recognize "ZXP."

A poor fallback might repeatedly say: "Sorry, I don't understand." A better fallback might ask: "Could you tell me a little more about what you mean by ZXP adjustment?"

If the system still can't understand the request, it may need to offer another path.

Fallback tests should cover:

• unknown intents

• misspellings

• incomplete requests

• unsupported topics

• conflicting requests

• repeated misunderstanding

Also test what happens after multiple failures. An AI agent shouldn't trap the user in an endless loop of: "Sorry, I didn't understand that."

10. Test Human Escalation

Sometimes the AI agent should recognize that it can no longer handle the conversation reliably. This might happen when the user explicitly asks for a person, the request falls outside the agent's capabilities, or the situation requires human judgment. In those cases, continuing to generate answers may be worse than handing the conversation over

Consider situations involving:

• repeated misunderstanding

• unsupported account problems

• user requests for a human

• sensitive workflows

• exceptions the automated process cannot handle

If escalation is part of the product design, test the entire transition.

For example:

User: I want to speak to someone.

Does the AI recognize the request? Does it transfer the conversation correctly? Does the human agent receive the relevant conversation history? Does the user have to explain everything again? Does the AI continue trying to answer after escalation should have occurred?

A technically successful transfer can still create a poor experience if all the context is lost.

11. Test Integrations Like You Would in Any Other Application

Conversational interfaces can make complex systems look simple.

The user sees:

"What's the status of my order?"

But behind that sentence, the agent might:

1. identify the user's intent

2. authenticate the customer

3. call an order API

4. retrieve the order

5. interpret the response

6. generate a natural-language answer

Traditional testing skills become extremely valuable here.

If the API returns:

{
  "order_id": "A10245",
  "status": "SHIPPED"
}

the AI shouldn't tell the user: "Your order is still processing."

To check these integrations, make sure you test:

• correct API mapping

• authentication failures

• timeouts

• empty responses

• malformed responses

• unavailable services

• incorrect status codes

• partial data

AI doesn't eliminate conventional integration testing. It adds another layer on top of it.

12. Build a Golden Dataset

Manual exploratory testing is useful when you're learning how an AI behaves, but eventually you need repeatability. This is where a golden dataset becomes useful.

A golden dataset is a curated collection of representative test inputs and expected behaviors that can be rerun as the system changes.

For example:

Then expand each important intent with paraphrases, edge cases, negative cases, and multi-turn scenarios.

Whenever prompts, knowledge, models, integrations, or conversation logic change, rerun the dataset.

Now you have something much closer to traditional regression testing.

13. Don't Only Measure Pass Rate

Suppose you execute 1,000 conversational tests and 950 pass. A 95% pass rate sounds good.

But what failed? Five hundred harmless FAQ questions? Or five critical account-security scenarios?

Aggregate pass rate alone doesn't tell the whole story. Depending on the application, useful metrics might include:

• Intent recognition accuracy: How often was the user's goal understood correctly?

• Fallback rate: How often did the system fail to understand the user?

• Task completion rate: How often did users successfully accomplish the intended task?

• Escalation success rate: When human help was required, did the transition succeed?

• Grounding failures: How often did responses conflict with approved knowledge?

• Context failures: How often did the system lose important information during multi-turn conversations?

• Critical hallucinations: How often did the system confidently provide unsupported information?

The right metrics depend on the product and its risks. A customer-service FAQ bot and an AI system supporting financial decisions shouldn't necessarily have the same quality thresholds.

14. Create Risk-Based Conversational Tests

This is another traditional QA principle that transfers very well.

Not every AI failure has the same impact. If an AI responds awkwardly to: "What are your business hours?", that's inconvenient.

If it gives incorrect information about a payment, account security, healthcare instruction, or financial policy, the impact could be much greater.

So categorize scenarios by risk.

For example:

Then concentrate regression coverage on the scenarios where incorrect AI behavior would cause the greatest harm.

15. A Practical Conversational AI Test Strategy

If I were starting a conversational AI QA effort today, I would organize it into these layers:

Layer 1: Intent Testing

Can the system understand what the user wants despite variations in language?

Layer 2: Response Evaluation

Are responses accurate, relevant, complete, clear, and useful?

Layer 3: Conversation Testing

Can the system maintain context across multiple turns?

Layer 4: Knowledge and Grounding

Are answers supported by approved and current information?

Layer 5: Negative and Hallucination Testing

Does the system avoid confidently answering when it doesn't have enough information?

Layer 6: Fallback and Escalation

Can the system recover when it doesn't understand, and can it hand off to a human when necessary?

Layer 7: Integration Testing

Are APIs, authentication, databases, and downstream systems behaving correctly?

Layer 8: Regression Testing

Can important behaviors be rerun after model, prompt, knowledge, or application changes?

That gives QA teams a much more structured starting point than simply opening a chatbot and asking random questions.

What Traditional QA Engineers Already Bring to AI Testing

When I first started learning conversational AI, I thought I needed to forget everything I knew about traditional testing. But I don't believe that anymore.

A lot of our existing skills transfer extremely well. We already know how to:

• question assumptions

• explore edge cases

• design negative tests

• trace failures through multiple systems

• validate integrations

• prioritize by risk

• build regression suites

• investigate unexpected behavior

What changes is the definition of the expected result.

For some AI scenarios, the expected result isn't:

Response = X

It's closer to:

The response must satisfy X, Y, and Z while avoiding A and B.

Once I understood that distinction, conversational AI testing started making much more sense to me.

Wrapping Up

My first instinct when learning conversational AI was to search for the test cases.

Now I think that's the wrong place to start. Start with the user.

• What are they trying to accomplish?

• What are the different ways they might ask for it?

• What information does the AI need?

• What would a useful answer contain?

• What should the system never say?

• What happens if the AI doesn't know?

• What happens when the conversation changes direction?

• What evidence would make you confident enough to release that experience to real users?

Those questions eventually become your test cases.

Conversational AI may be less deterministic than the applications many QA engineers are used to testing. But that doesn't make it untestable. It simply means we need to move beyond asking:

"Did I get the exact output I expected?"

and start asking:

"Did the system behave correctly, safely, and usefully across the different ways a real person might interact with it?"

For me, that was the biggest shift. The tools are changing, and so is the interface. Even the definition of an expected result is changing. But the fundamental responsibility of quality engineering hasn't changed very much at all: understand how the system can fail before the user has to discover it for you.

It's a common perception, and to be fair, finding bugs is an important part of the job. But if you spend even a few weeks working on a modern software team, you'll quickly realize that's only a small part of what QA engineers actually do.

Software development has changed dramatically over the last decade. Teams no longer wait months to release new features. Many organizations deploy updates every week, every day, or even several times a day. Applications have become more complex, with cloud services, APIs, microservices, mobile apps, and third-party integrations all working together behind the scenes.

As software has evolved, the role of QA has evolved with it.

Today's QA engineers are involved long before a feature reaches testing. They help review requirements, identify risks, clarify business expectations, verify APIs and databases, automate repetitive tests, investigate production issues, and work closely with developers throughout the entire development lifecycle.

In other words, QA isn't just about finding problems after software has been built. It's about helping prevent those problems from happening in the first place.

Whether you're thinking about becoming a QA engineer, transitioning from manual testing into automation, or simply curious about what quality engineering looks like today, understanding how the role has changed is an important first step.

In this article, we'll explore what QA engineers actually do, why the profession has evolved, and the skills that have become essential for building reliable software in today's fast-moving development environments.

Table of Contents

• How the QA Role Has Changed

• Good Testing Starts Before the First Test Case

• Real-world Example

• Modern QA Is About More Than Automation

• Where Different Types of Testing Fit

• Skills Every QA Engineer Should Develop

• Common Misconceptions About QA

• If You're Considering a Career in QA

• Wrapping Up

How the QA Role Has Changed

Many software usually teams followed a simple workflow. Business analysts gathered requirements, developers built the application, and once development was complete, the software was handed over to QA for testing. If defects were found, the application went back to the development team before eventually being released.

In that model, QA was often viewed as the final checkpoint before production. Testing happened after most of the important technical decisions had already been made.

That approach worked reasonably well when software releases happened only a few times a year. Teams had enough time to finish development, perform weeks of manual testing, fix defects, and prepare for a scheduled release.

Todays software development looks very different.

Many organizations use Agile methodologies, Continuous Integration (CI), and Continuous Delivery (CD). Instead of delivering software every few months, teams continuously add features, fix bugs, and release improvements in short development cycles.

Because development moves much faster, quality can no longer be treated as the final phase of a project.

Instead, QA engineers work alongside developers, product owners, business analysts, UX designers, and DevOps engineers from the beginning of the development process. They participate in sprint planning, review user stories, discuss acceptance criteria, identify potential risks, and help ensure new features are designed with testing in mind before development even starts.

This shift has changed the role of QA from testing completed software to helping teams build quality into the software from the start.

For example, imagine a team is building an online banking application that allows customers to transfer money between accounts. A traditional testing approach might focus on verifying whether the transfer succeeds after the feature has been developed.

An experienced QA engineer starts much earlier by asking questions such as:

• What happens if the network connection is interrupted during the transfer?

• What should happen if the customer has insufficient funds?

• Can the same transfer request be submitted twice accidentally?

• How should the application respond if the receiving bank is temporarily unavailable?

Questions like these help uncover potential problems before developers spend time writing code. Addressing these scenarios early is often much less expensive than discovering them during testing—or worse, after the application has been released to customers.

This is one of the biggest reasons QA has evolved from a role focused primarily on testing into one that contributes throughout the entire software development lifecycle.

Quality is no longer something that's checked at the end of a project.

It's something the entire team builds together, one decision at a time.

Traditional Development

Requirements -> Development -> QA Testing -> Production

Modern Development

Requirement -> DEV+QA+Product Owner -> Continuous Development -> Continuous Testing -> Production

Good Testing Starts Before the First Test Case

One of the biggest surprises for people entering software testing is discovering that a significant part of a QA engineer's work happens before a single test case is written.

Many people picture testing as something that begins only after developers finish building a feature. In reality, experienced QA engineers become involved much earlier because that's often where they can have the greatest impact.

Consider a simple requirement:

A user must create a password containing at least eight characters.

At first glance, the requirement seems complete. A developer can implement it, and a tester can verify that passwords shorter than eight characters are rejected.

But software requirements are rarely that straightforward.

A QA engineer naturally starts looking beyond the obvious by asking questions such as:

• Should spaces or count as characters? Or Should leading or trailing spaces be removed automatically?

• Is there a maximum password length?

• Are special or Unicode characters required?

• Will the same validation rules apply on both the web application and the mobile app?

• What error message should users see if the password is invalid?

None of these questions are about trying to catch developers making mistakes.

They're about making sure the entire team has the same understanding of how the feature should work before development moves forward.

This process is often called requirement clarification, and it's one of the most valuable contributions a QA engineer can make.

Without these discussions, developers may implement one interpretation of the requirement while testers validate another. Business stakeholders may expect something entirely different. Even when everyone is working hard, unclear requirements can lead to unnecessary defects, rework, and frustration.

Asking thoughtful questions early helps avoid those situations.

It also saves time.

Finding an unclear requirement during a planning meeting usually takes a few minutes to resolve. Discovering the same issue after development, testing, and deployment can take hours—or even days—to investigate and fix.

Good QA isn't only about verifying that software works correctly.

It's also about helping ensure the team is building the right software in the first place.

Real-world Example

Imagine an online shopping website that offers discount coupons.

The requirement says:

"Users can apply one coupon during checkout."

At first, it sounds simple.

A QA engineer might ask:

• What happens if the coupon has expired?

• Can two browser tabs apply the same coupon simultaneously?

• What if the customer removes an item after applying the coupon?

• Can multiple coupons be combined?

• What happens if the payment fails after the coupon has been applied?

These questions often uncover situations that weren't considered during the initial discussion.

Instead of becoming production defects, they become design decisions that the team can address before writing code.

That's one of the reasons experienced QA engineers spend so much time asking questions. They're not slowing development down. They're helping the team avoid expensive surprises later.

Modern QA Is About More Than Automation

If you browse job postings for QA engineers, you'll probably notice one thing almost immediately.

Many of them mention automation tools such as Selenium, Plawright, Cypress, Appium, Javascript, Java, Python, and so on - the list goes on

Because of that, it's easy to assume that QA engineers spend their entire day writing automated tests. Automation is certainly an important part of the job, but it isn't the job itself.

Think about a new feature being added to a food delivery app. Customers can now save multiple delivery addresses and choose one during checkout. Before any automation script is written, a QA engineer is already involved.

They review the requirement to understand how the feature is expected to work. They discuss different scenarios with developers and product owners, such as what should happen if a customer deletes their default address or enters an invalid ZIP code. They identify edge cases that may not have been considered during planning and think about how the feature interacts with existing functionality.

Only after those discussions do they begin designing test scenarios.

Some of those tests may eventually become automated, especially if the feature will be used in future regression testing. Others may be better suited for exploratory testing because they require human observation and judgment.

QA engineers choose the right testing approach based on the problem they're trying to solve rather than trying to automate everything.

A typical week for a QA engineer might include a variety of responsibilities, such as:

• Reviewing new requirements before development begins.

• Designing functional and edge-case test scenarios.

• Verifying REST API responses using tools like Postman or Bruno.

• Validating backend data with SQL queries.

• Building or maintaining automated regression tests.

• Investigating issues reported from production.

• Working with developers to reproduce and understand defects.

• Confirming bug fixes before a release.

Some weeks involve more automation than others. Some involve more investigation, collaboration, or exploratory testing.

That's one of the reasons QA engineering is such a diverse field. No two projects are exactly the same, and the work changes depending on the product, the team, and the stage of development.

Automation simply helps QA engineers spend less time repeating predictable tasks so they can focus on the work that requires critical thinking.

It's a tool that supports quality engineering—not the definition of it.

Where Different Types of Testing Fit

Another common misconception is that QA engineers only test the user interface. In reality, software can be tested at many different levels.

For example:

QA engineers don't necessarily perform all of these types of testing every day, but understanding when and why each approach is useful helps them choose the right strategy for different situations.

Quality isn't achieved by relying on a single testing technique.

It's achieved by combining multiple testing approaches to build confidence that the software behaves correctly under real-world conditions.

Skills Every QA Engineer Should Develop

If you're just starting your QA journey, it's easy to focus on learning specific tools. While tools are important, they're only part of the picture. The most successful QA engineers build a combination of technical skills, problem-solving abilities, and effective communication.

Here are some of the core skills that will help you grow in QA role.

We don't need to masters all of these skills overnight. The important thing is to build a strong foundation, stay curious, and keep learning as the industry evolves.

Common Misconceptions About QA

If you're new to software testing, you've probably heard some of these statements before. While they may sound reasonable, they don't reflect how modern QA teams actually work.

Myth: QA engineers Only Find Bugs

Reality: Finding bugs is only one part of the job. QA engineers also review requirements, identify risks, design test strategies, automate repetitive testing, validate APIs and databases, and help teams prevent defects before software reaches users.

Myth: Automation Will Replace QA Engineers.

Reality: Automation is a tool, not a replacement for human thinking. Automated tests can execute repetitive tasks, but they can't decide what should be tested, identify unclear requirements, or evaluate whether a feature delivers a good user experience.

Myth: QA is Responsible for Quality.

Reality: Quality is a shared responsibility. Developers write reliable code, product owners define clear requirements, designers focus on usability, DevOps engineers build dependable deployment pipelines, and QA engineers help ensure everything works together as expected.

Myth: Manual Testing is No Longer Useful.

Reality: Automation is excellent for repetitive regression testing, but manual testing remains valuable for exploratory testing, usability checks, and investigating unexpected behavior. The two approaches complement each other rather than compete.

Myth: QA is Easier than Software Development.

Reality: Modern QA requires strong analytical thinking, technical knowledge, communication skills, and a solid understanding of how software systems work. While the responsibilities differ from software development, both roles play an equally important part in delivering high-quality software.

If You're Considering a Career in QA

If you're thinking about becoming a QA engineer, don't worry if you don't know every testing tool or automation framework yet. Every experienced QA professional started by learning the fundamentals.

Begin by understanding how software works. Learn how web applications communicate with APIs, how data is stored in databases, and how different components interact to deliver a feature. These concepts will help you understand why you're testing something, not just how to test it.

At the same time, practice thinking like a user. Ask questions, explore different scenarios, and look beyond the "happy path." Some of the most valuable defects are discovered simply because someone asked, "What happens if this doesn't go as expected?"

As you grow, gradually build technical skills such as SQL, API testing, automation, version control, and CI/CD. Focus on continuous learning and improving one skill at a time.

Most importantly, remember that software quality isn't created by one person or one team. The best QA engineers work collaboratively, communicate effectively, and help everyone build better software together.

Wrapping Up

Software testing has come a long way from being viewed as the final step before a release.

Today's QA engineers contribute throughout the software development lifecycle. They ask thoughtful questions, clarify requirements, automate repetitive testing, validate APIs and databases, investigate production issues, and help teams deliver reliable software with confidence.

Finding bugs will always be an important part of the role, but it's no longer what defines a successful QA engineer.

What truly makes a difference is the ability to prevent problems before they happen, think critically about how software is used, and collaborate with the entire team to deliver a better product.

If you're just starting your journey in software quality, don't measure your progress by how many testing tools you've learned. Focus on building strong fundamentals, staying curious, and continuously improving your problem-solving skills.

Technology will continue to evolve, but those qualities will always be valuable.

Go takes a deliberately minimal approach to testing. There are no built-in assertions, no annotations, and no special syntax. Instead, tests are written as regular Go code using a small standard library package, and run with a single command. This can feel unusual at first if you're coming from ecosystems with richer testing frameworks, but it quickly becomes predictable and easy to reason about.

In this article, we'll look at how unit testing works in Go in practice. We'll write a few small tests, run them from the command line, and cover the most common patterns you'll see in real Go codebases, such as table-driven tests and testing functions that return errors. We'll focus on the essentials and won't cover more advanced topics like mocks or external frameworks.

The goal is to show how familiar testing concepts translate into idiomatic Go. By the end, you should feel comfortable reading and writing basic unit tests and integrating them into your regular Go workflow.

What We'll Cover:

1. Prerequisites

2. Writing Your First Test

   • Running Your Test

   • Divide by Zero

   • t.Errorf vs t.Fatalf

3. Table-Driven Tests

   • Table-Driven Add Test

   • Table-Driven Divide Test

   • Exercise

4. Testing Functions That Return Errors

   • Safe Divide Function

   • Writing Tests for SafeDivide()

   • Exercise

5. Best Practices and Tips

   • Name Tests Clearly

   • Keep Tests Small and Focused

   • Use Table-Driven Tests for Repetitive Cases

   • Check Errors Explicitly

   • Avoid Panics When Possible

   • Run Tests Frequently

   • Keep Tests in the Same Package

   • Use t.Fatalf vs t.Errorf Appropriately

6. Conclusion

7. Solutions to Exercises

   • Subtract Function and Tests

   • SafeSubtract Function and Tests

Prerequisites

Before you start, you should be comfortable with:

• Writing and running basic Go programs

• Defining and calling functions in Go

• Understanding basic Go types (int, string, bool, and so on)

• Using the Go command-line tool (go run, go build)

• Basic understanding of unit tests: what a test is and why it's useful

• Familiarity with Test-Driven Development concepts like testing before or alongside writing code

• Awareness of common testing ideas such as assertions, test coverage, and checking error conditions

You don't need prior experience with Go's testing package or Go-specific test patterns, as this guide will cover all of that.

Writing Your First Test

Let's start with a simple function to test. Imagine you have a small calc package with an Add function:

// calc.go
package calc

// Add returns the sum of two integers
func Add(a, b int) int {
    return a + b
}

To test this function, create a new file named calc_test.go in the same package. In Go, test files must end with _test.go to be recognized by the testing tool.

Inside calc_test.go, you write a test function:

// calc_test.go
package calc

import "testing"

func TestAdd(t *testing.T) {
    got := Add(2, 3)
    want := 5
    if got != want {
        t.Errorf("Add(2, 3) = %d; want %d", got, want)
    }
}

Here's what's happening:

• The function name starts with Test and takes a single *testing.T parameter. Go automatically discovers and runs any function that follows this convention.

• The t.Errorf call reports a test failure. Unlike some frameworks, Go doesn't provide special assertions – you simply check a condition and call t.Errorf or t.Fatalf if it fails.

• Each test is a standalone function. You can write as many as you like, and Go will run them all.

Running Your Test

Once the file is saved, you can run your test with:

go test

This runs tests for the current package (files ending with _test.go). If you want to run tests recursively in all subdirectories of your project, use:

go test ./...

The ./... pattern is shorthand for "run tests in this directory and all subdirectories". This is especially useful in larger projects where your code is spread across multiple packages.

If everything is working, you should see output indicating that the test passed:

$ go test
PASS
ok      _/C_/projects/Articles/Go_Testing       0.334s

You can add the -v flag for verbose output:

go test -v

This will show you the names of the tests as they run:

$ go test -v
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
PASS
ok      _/C_/projects/Articles/Go_Testing       0.356s

Not much difference for a single test, but it becomes useful as you add more tests.

Now let's see what happens if the test fails. Change the expected value in calc_test.go to an incorrect one:

  ...
    want := 6 // Incorrect expected value
  ...

Run the tests again:

$ go test
--- FAIL: TestAdd (0.00s)
    calc_test.go:9: Add(2, 3) = 5; want 6
FAIL
exit status 1
FAIL    _/C_/projects/Articles/Go_Testing       0.340s

or with verbose output:

$ go test -v
=== RUN   TestAdd
    calc_test.go:9: Add(2, 3) = 5; want 6
--- FAIL: TestAdd (0.00s)
FAIL
exit status 1
FAIL    _/C_/projects/Articles/Go_Testing       0.337s

Of course, your tests should always check for the correct expected values! A failing (but correct) test is a sign that your code needs to be fixed.

We only created one test file and one test function with one assertion here, but Go's testing tool can handle many files and functions at once. Behind the scenes, Go will automatically:

• Find all _test.go files in the specified packages (for example, current directory for go test, or recursively in all subdirectories with go test ./...).

• Identify functions that start with Test and have the correct signature.

• Compile them together with your package into a temporary test binary.

• Execute each test function and report the results.

To prove this, let's quickly add a Divide function to our package:

// calc.go
...
// Divide returns the result of dividing a by b
func Divide(a, b int) int {
    return a / b
}

(Note that this is an integer division, so fractional parts are discarded. Divide(5, 2) would return 2.)

And another test file with a corresponding test:

// calc_2_test.go
package calc

import "testing"

func TestDivide(t *testing.T) {
    got := Divide(10, 2)
    want := 5    
    if got != want {
        t.Errorf("Divide(10, 2) = %d; want %d", got, want)
    }    
}

Now when you run go test, both TestAdd and TestDivide will be executed:

$ go test
PASS
ok      _/C_/projects/Articles/Go_Testing       0.325s

Or:

$ go test -v
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
=== RUN   TestDivide
--- PASS: TestDivide (0.00s)
PASS
ok      _/C_/projects/Articles/Go_Testing       0.323s

Divide by Zero

What happens if we try to Divide by zero? Let's add another test case for that:

// calc_test.go
...
func TestDivideByZero(t *testing.T) {
    defer func() {
        if r := recover(); r == nil { // Check if a panic occurred
            t.Errorf("Divide did not panic on division by zero")
        }
    }()
    Divide(10, 0) // This should cause a panic
}

This test checks that the Divide function panics when dividing by zero. When you run the tests again, you'll see that this new test also passes:

$ go test -v
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
=== RUN   TestDivide
--- PASS: TestDivide (0.00s)
=== RUN   TestDivideByZero
--- PASS: TestDivideByZero (0.00s)
PASS
ok      _/C_/projects/Articles/Go_Testing       0.312s

(Note that in real-world Go code, it's better to return (int, error) for unsafe operations instead of panicking.)

Feel free to experiment by adding more test cases, changing expected values, and exploring how Go's testing framework handles different scenarios.

t.Errorf vs t.Fatalf

In the examples above, we used t.Errorf to report test failures. This function logs the error but allows the test to continue running. This is useful when you want to check multiple conditions in a single test function.

In contrast, t.Fatalf logs the error and immediately stops the execution of the current test. Use t.Fatalf when continuing the test after a failure doesn't make sense or could cause misleading results.

For example, in the TestDivideByZero test, if the Divide function does not panic, we use t.Errorf to report the failure but continue to the end of the test. But if we had additional checks after the division, we might want to use t.Fatalf to stop execution immediately upon failure.

While t.Errorf and t.Fatalf use fmt-style formatting, for simple messages without formatting, you can also use t.Error and t.Fatal, respectively.

In the next section, we'll look at table-driven tests, a common Go pattern for testing multiple cases efficiently.

Table-Driven Tests

In Go, it's common to want to run the same test logic for multiple inputs and expected outputs. Rather than writing a separate test function for each case, Go developers often use table-driven tests. This pattern keeps your tests concise, readable, and easy to extend.

Table-Driven Add Test

Let's rewrite our Add test using a table-driven approach (and delete calc_2_test.go for clarity):

// calc_test.go
package calc

import "testing"

func TestAddTableDriven(t *testing.T) {
    tests := []struct {// Define a struct for each test case and create a slice of them
        name string
        a, b int
        want int
    }{
        {"both positive", 2, 3, 5},
        {"positive + zero", 5, 0, 5},
        {"negative + positive", -1, 4, 3},
        {"both negative", -2, -3, -5},
    }

    for _, tt := range tests {// Loop over each test case
        t.Run(tt.name, func(t *testing.T) {// Run each case as a subtest
            got := Add(tt.a, tt.b)
            if got != tt.want {// Check the result
                t.Errorf("Add(%d, %d) = %d; want %d", tt.a, tt.b, got, tt.want) // Report failure if it doesn't match
            }
        })
    }
}

Here's how it works:

• We define a slice of structs, each representing a test case.

• Each struct contains the test name, input values, and the expected result.

• We loop over the slice and call t.Run(tt.name, func(t *testing.T) { ... }) to run each test as a subtest.

• If a subtest fails, you can see which one by its name in the output.

$ go test
PASS
ok      _/C_/projects/Articles/Go_Testing       0.452s

Or to see detailed output:

$ go test -v
=== RUN   TestAddTableDriven
=== RUN   TestAddTableDriven/both_positive
=== RUN   TestAddTableDriven/positive_+_zero
=== RUN   TestAddTableDriven/negative_+_positive
=== RUN   TestAddTableDriven/both_negative
--- PASS: TestAddTableDriven (0.00s)
    --- PASS: TestAddTableDriven/both_positive (0.00s)
    --- PASS: TestAddTableDriven/positive_+_zero (0.00s)
    --- PASS: TestAddTableDriven/negative_+_positive (0.00s)
    --- PASS: TestAddTableDriven/both_negative (0.00s)
PASS
ok      _/C_/projects/Articles/Go_Testing       0.385s

Table-Driven Divide Test

We can apply the same pattern to Divide, including checking for divide-by-zero:

// calc_test.go
...
func TestDivideTableDriven(t *testing.T) {
    tests := []struct { // Define test cases
        name     string
        a, b     int
        want     int
        wantPanic bool
    }{
        {"normal division", 10, 2, 5, false},
        {"division by zero", 10, 0, 0, true},
    }

    for _, tt := range tests { // Loop over
        t.Run(tt.name, func(t *testing.T) { // Run subtest
            if tt.wantPanic { // Check for expected panic
                defer func() { // Recover from panic
                    if r := recover(); r == nil {
                        t.Errorf("Divide(%d, %d) did not panic", tt.a, tt.b)
                    }
                }()
            }
            got := Divide(tt.a, tt.b) // Tests that do not panic
            if !tt.wantPanic && got != tt.want {
                t.Errorf("Divide(%d, %d) = %d; want %d", tt.a, tt.b, got, tt.want)
            }
        })
    }
}

This example shows how to handle both normal and panic cases in a single table-driven test:

• The wantPanic field tells the test whether we expect a panic.

• We use defer and recover to check for a panic when needed.

• Normal test cases still check the result as usual.

Run all tests as before:

$ go test -v
=== RUN   TestAddTableDriven
=== RUN   TestAddTableDriven/both_positive
=== RUN   TestAddTableDriven/positive_+_zero
=== RUN   TestAddTableDriven/negative_+_positive
=== RUN   TestAddTableDriven/both_negative
--- PASS: TestAddTableDriven (0.00s)
    --- PASS: TestAddTableDriven/both_positive (0.00s)
    --- PASS: TestAddTableDriven/positive_+_zero (0.00s)
    --- PASS: TestAddTableDriven/negative_+_positive (0.00s)
    --- PASS: TestAddTableDriven/both_negative (0.00s)
=== RUN   TestDivideTableDriven
=== RUN   TestDivideTableDriven/normal_division
=== RUN   TestDivideTableDriven/division_by_zero
--- PASS: TestDivideTableDriven (0.00s)
    --- PASS: TestDivideTableDriven/normal_division (0.00s)
    --- PASS: TestDivideTableDriven/division_by_zero (0.00s)
PASS
ok      _/C_/projects/Articles/Go_Testing       0.321s

Subtest names make it easy to see which case passed or failed.

Exercise

Try creating your own table-driven test for a new function, Subtract(a, b int) int. Include at least four test cases:

• Both positive numbers

• Positive minus zero

• Negative minus positive

• Both negative

Then run your tests and verify the output.

Testing Functions That Return Errors

Many Go functions return an error as the last return value. Writing tests for these functions is slightly different from testing pure functions like our Add or Divide, because you need to check both the result and whether an error occurred.

Safe Divide Function

Let's add a SafeDivide function to return an error instead of panicking:

// calc.go
...
import "fmt"
...
// SafeDivide returns the result of dividing a by b.
// It returns an error if b is zero.
func SafeDivide(a, b int) (int, error) {
    if b == 0 {
        return 0, fmt.Errorf("cannot divide by zero")
    }
    return a / b, nil
}

Writing Tests for SafeDivide()

We can use a table-driven test again:

// calc_test.go
func TestSafeDivide(t *testing.T) {
    tests := []struct {
        name      string
        a, b      int
        want      int
        wantError bool
    }{
        {"normal division", 10, 2, 5, false},
        {"division by zero", 10, 0, 0, true},
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := SafeDivide(tt.a, tt.b)
            if tt.wantError {
                if err == nil {
                    t.Errorf("SafeDivide(%d, %d) expected error, got nil", tt.a, tt.b)
                }
                return // stop here, no need to check `got`
            }
            if err != nil {
                t.Errorf("SafeDivide(%d, %d) unexpected error: %v", tt.a, tt.b, err)
            }
            if got != tt.want {
                t.Errorf("SafeDivide(%d, %d) = %d; want %d", tt.a, tt.b, got, tt.want)
            }
        })
    }
}

What's happening here:

• We added a wantError field to indicate whether the test expects an error.

• If an error is expected, we check that err != nil. If not (that is, err == nil), we fail the test.

• If no error is expected, we check both the returned value (got) and that err == nil.

• Using t.Run subtests keeps everything organized and readable.

Running the tests again:

$ go test -v
...
=== RUN   TestSafeDivide
=== RUN   TestSafeDivide/normal_division
=== RUN   TestSafeDivide/division_by_zero
--- PASS: TestSafeDivide (0.00s)
    --- PASS: TestSafeDivide/normal_division (0.00s)
    --- PASS: TestSafeDivide/division_by_zero (0.00s)
PASS
ok      _/C_/projects/Articles/Go_Testing       0.323s

Showing that both normal and error cases are handled correctly.

Exercise

Update your Subtract(a, b int) int function to a SafeSubtract(a, b int) (int, error) variant that returns an error if the result would be negative. Then write a table-driven test that covers:

• A positive result

• Zero result

• A negative result (should return an error)

Best Practices and Tips

Writing tests in Go is straightforward, but there are a few conventions and tips that make your tests more readable, maintainable, and idiomatic:

Name Tests Clearly

First, make sure you use descriptive names for test functions and subtests. A good name explains what you're testing and under what conditions.

Here’s an example:

t.Run("Divide positive numbers", func(t *testing.T) { ... })
t.Run("Divide by zero returns error", func(t *testing.T) { ... })

Keep Tests Small and Focused

Each subtest should verify one thing, and each test function should cover a single function or method.

Try to avoid combining multiple unrelated checks in the same test function, and use table-driven tests help keep multiple similar checks concise without losing clarity.

Use Table-Driven Tests for Repetitive Cases

If you find yourself writing multiple similar test functions, switch to a table-driven pattern. It makes it easier to add new cases, reduces duplicated code, and keeps output organized with t.Run.

Check Errors Explicitly

In Go, functions often return error. So make sure you always check for errors in tests, even if you expect nil.

You can use the wantError pattern in table-driven tests for clarity.

if tt.wantError {
    if err == nil {
        t.Errorf("expected error, got nil")
    }
}

Avoid Panics When Possible

Panics are fine for some internal checks, but in production code, prefer returning an error.

Your tests can check for panics using defer and recover, but this should be the exception rather than the norm.

Run Tests Frequently

Try to make running tests a habit: go test -v ./.... Frequent testing helps catch mistakes early and reinforces TDD practices.

Keep Tests in the Same Package

By convention, tests live in the same package as the code they test. You can create _test.go files for testing, and Go automatically recognizes them.

Only use a separate package calc_test if you want to test your code from the outside, like a consumer. External test packages (just like every other external package) cannot access unexported identifiers.

Use t.Fatalf vs t.Errorf Appropriately

• t.Errorf reports a failure but continues running the test.

• t.Fatalf stops the test immediately, which is useful if subsequent code depends on successful setup.

These tips will help you write clean, maintainable, and idiomatic Go tests that are easy to read and extend. Following these practices early in your Go journey will make testing less intimidating and more effective.

Conclusion

Unit testing in Go may feel different at first, especially if you're coming from ecosystems with heavy frameworks and assertions. But the simplicity of Go's testing tools is one of its strengths: once you understand the conventions, writing, running, and organizing tests becomes predictable and intuitive.

In this guide, you've seen how to:

• Write basic test functions with the testing package

• Run tests from the command line and interpret the results

• Use table-driven tests to cover multiple cases efficiently

• Handle functions that return errors and check for expected failures

Beyond these fundamentals, testing is not just about verifying correctness, it's also about confidence. Well-tested code allows you to refactor, experiment, and add new features with less fear of breaking existing functionality.

As you continue writing Go code, try to integrate testing early, follow the idiomatic patterns you've learned, and explore more advanced topics such as:

• Using mocks or interfaces to isolate dependencies

• Benchmark tests with testing.B

• Coverage analysis with go test -cover

The key takeaway is that testing in Go is accessible, flexible, and powerful, even without fancy frameworks. By building these habits now, you'll write code that's more reliable, maintainable, and enjoyable to work with.

Solutions to Exercises

Subtract Function and Tests

// calc.go
package calc

func Subtract(a, b int) int {
    return a - b
}

// calc_test.go
package calc

import "testing"

func TestSubtractTableDriven(t *testing.T) {
    tests := []struct {
        name string
        a, b int
        want int
    }{
        {"both positive", 5, 3, 2},
        {"positive minus zero", 5, 0, 5},
        {"negative minus positive", -1, 4, -5},
        {"both negative", -3, -2, -1},
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got := Subtract(tt.a, tt.b)
            if got != tt.want {
                t.Errorf("Subtract(%d, %d) = %d; want %d", tt.a, tt.b, got, tt.want)
            }
        })
    }
}

SafeSubtract Function and Tests

// calc.go
package calc

import "fmt"

func SafeSubtract(a, b int) (int, error) {
    result := a - b
    if result < 0 {
        return 0, fmt.Errorf("result would be negative")
    }
    return result, nil
}

// calc_test.go
package calc

import "testing"

func TestSafeSubtract(t *testing.T) {
    tests := []struct {
        name      string
        a, b      int
        want      int
        wantError bool
    }{
        {"positive result", 5, 3, 2, false},
        {"zero result", 3, 3, 0, false},
        {"negative result", 2, 5, 0, true},
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := SafeSubtract(tt.a, tt.b)
            if tt.wantError {
                if err == nil {
                    t.Errorf("SafeSubtract(%d, %d) expected error, got nil", tt.a, tt.b)
                }
                return
            }
            if err != nil {
                t.Errorf("SafeSubtract(%d, %d) unexpected error: %v", tt.a, tt.b, err)
            }
            if got != tt.want {
                t.Errorf("SafeSubtract(%d, %d) = %d; want %d", tt.a, tt.b, got, tt.want)
            }
        })
    }
}

That is where TestOps comes in. It blends testing with operational efficiency, creating a streamlined approach to quality assurance. Scaling TestOps for global teams means setting up processes that work smoothly across continents, delivering speed and consistency without compromising on quality.

The challenges are real: communication gaps, tool compatibility issues, and cultural differences. But the payoff is worth it. A well-structured TestOps framework helps teams collaborate easily, automate testing, and produce software that meets global expectations.

This guide will walk you through overcoming these challenges, adopting practical strategies, and turning your global TestOps into a hub for innovation and quality.

Here’s what we’ll cover:

1. Understanding TestOps

2. Limitations of Scaling TestOps

3. Strategies for Scaling TestOps

4. How to Integrate TestOps into Global DevOps Pipelines

5. How to Use AI and Analytics in TestOps

6. Future of TestOps in Global Development

Understanding TestOps

TestOps is all about using automation to make software testing smoother and more efficient. It brings together scattered teams and processes into a unified system, helping you deliver better software faster and with fewer bugs. But what does it actually do?

TestOps makes testing easier to manage, run, and review. It keeps the testing process organized, consistent, and team-friendly. By using automation and central tools, TestOps helps you avoid mistakes, save time, and deliver better-quality software.

Here are the four central components of TestOps:

• Planning: This step focuses on deciding what needs to be tested, how it will be tested (including the test environment), when testing will happen, and who will handle it.

• Management: This ensures testing is efficient and scalable by using tools that improve teamwork and visibility.

• Execution: This is the actual process of running tests on the software.

• Analysis: This step involves reviewing test performance, diagnosing issues, and finding ways to improve the overall testing process.

At scale, TestOps focuses on:

• Standardization: Setting up consistent testing methods and tools that everyone can use across teams and projects.

• Automation: Increasing the use of automated tests to handle more tasks quickly and accurately.

• Collaboration: Improving how teams work together, even if they are spread out in different locations.

• Scalability: Making sure testing systems and processes can grow as needs increase.

• Insights: Using data from large-scale testing to make better decisions and improve how things work.

Limitations of Scaling TestOps

Scaling TestOps for global software teams comes with its fair share of challenges. While the advantages of smooth, integrated testing are clear, getting there requires careful planning.

Here are some key obstacles:

• Communication barriers: With teams spread across different time zones, keeping communication clear and timely can be tough. Delays or misunderstandings can slow progress and affect the quality of testing.

• Tool compatibility: Teams may use different testing tools, leading to inefficiencies and fragmentation. It's important to make sure all tools can work together and are compatible across different environments.

• Cultural and organizational differences: Teams from various regions may have different work cultures, priorities, and expectations. Finding common ground without creating friction is essential for smooth collaboration.

• Time zone management: Coordinating meetings or ensuring real-time review of test results becomes difficult with global teams in different time zones.

• Quality consistency: Ensuring consistent testing standards across multiple locations can be tricky. Without centralized control, practices can vary, which may lead to missed defects and unreliable releases.

Overcoming these challenges requires a well-thought-out strategy, effective communication, and the right tools to align teams and processes across the globe.

Strategies for Scaling TestOps

Scaling TestOps for global teams requires smart strategies to address communication issues, tool mismatches, and operational challenges. Here are some key approaches to make scaling work:

Standardize Testing Processes

Set up clear, consistent testing protocols and tools across all teams to ensure everyone is on the same page.

For example, you can standardize testing using frameworks like Jest to ensure consistency across teams.

bashCopy codenpm install --save-dev jest

In your package.json:

jsonCopy code{
  "scripts": {
    "test": "jest"
  }
}

Use Cloud-Based Tools

Choose cloud tools that allow teams to collaborate smoothly, provide real-time feedback, and access testing environments from anywhere.

For example, cloud tools like LambdaTest enable remote testing across browsers and devices.

javascriptCopy codeconst { remote } = require('webdriverio');

async function runTest() {
  const browser = await remote({
    capabilities: {
      browserName: 'chrome',
      platform: 'Windows 10',
      version: 'latest',
      'build': 'TestOps Scaling Build',
      'name': 'Test Parallel Execution',
    },
    host: 'hub.lambdatest.com',
    port: 80,
    user: 'your_username',
    key: 'your_access_key'
  });

  await browser.url('https://www.yoursite.com');
  console.log(await browser.getTitle());
  await browser.deleteSession();
}

runTest();

Automate Testing

Integrate automated tests into CI/CD pipelines to reduce manual work, speed up feedback, and improve test coverage.

For example, you can use GitHub Actions for CI/CD test automation.

yamlCopy codename: Run Tests

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: Run tests
        run: npm test

Use Centralized Reporting Tools

Use dashboards to give everyone real-time updates on testing progress, keeping all teams and stakeholders in the loop.

Here’s an example of integrating with TestRail for centralized reporting.

javascriptCopy codeconst axios = require('axios');

const result = {
  "status": "passed", 
  "test_case_id": 123,
  "run_id": 456
};

axios.post('https://your-testrail-instance/api/v2/add_result_for_case/1/123', result, {
  auth: { username: 'your_email', password: 'your_password' }
})
.then(response => console.log('Test result posted successfully'))
.catch(error => console.error('Error:', error));

Encourage Cross-Regional Collaboration

Use collaboration tools and hold regular meetings to bridge time zone and cultural differences between teams.

You can use Slack or similar tools for real-time communication and alerts.

javascriptCopy codeconst slackMessage = { text: "Test Execution Completed: All tests have passed successfully!" };

axios.post('https://hooks.slack.com/services/your-webhook-url', slackMessage)
  .then(response => console.log('Slack message sent'))
  .catch(error => console.error('Error:', error));

Create a Continuous Feedback Loop

Set up systems that provide immediate feedback and allow for quick action, ensuring quality isn’t delayed.

For example, you can trigger feedback loops with Slack for an immediate response.

javascriptCopy codeconst slackMessage = { text: "Alert: Test failure detected!" };

axios.post('https://hooks.slack.com/services/your-webhook-url', slackMessage)
  .then(response => console.log('Alert sent to Slack'))
  .catch(error => console.error('Error:', error));

Upskill Teams

Offer training so all team members know how to use TestOps tools effectively.

Try providing training through GitHub repositories with testing best practices.

markdownCopy code# Automated Testing Guide

## Steps:
1. Clone repo
2. Install dependencies: `npm install`
3. Run tests: `npm test`
4. Review TestRail dashboard

Adapt to Time Zones

Organize workflows and shifts that allow for continuous testing, helping teams overcome time zone challenges.

You can schedule tests using Jenkins, for example, to accommodate global teams.

groovyCopy codepipeline {
    agent any
    triggers {
        cron('H 0 * * *')
    }
    stages {
        stage('Run Tests') {
            steps {
                sh 'npm test'
            }
        }
    }
}

How to Integrate TestOps into Global DevOps Pipelines

Integrating TestOps into global DevOps pipelines is crucial for maintaining software quality across distributed teams. This integration makes testing a seamless and automated part of the software delivery process, helping improve and release software quickly.

Tools like containerization and orchestration platforms play a big role in scaling TestOps across global pipelines. Here’s how to do it effectively:

Test Early and Continuously

When you start testing early in the development cycle, you catch issues before they reach production. This early approach allows developers to fix bugs while the changes are still fresh. It also prevents those issues from becoming bigger later.

Continuous testing means tests run automatically whenever code changes. These are usually triggered during the Continuous Integration (CI) process. Since the tests run right after a change is made, feedback is quick.

This quick feedback helps reduce debugging time. It also supports teams working from different regions, since they can move ahead without waiting on others. And because tests fail fast, blockers are identified early and resolved quickly.

Example:
A global logistics company uses GitHub Actions to run unit and integration tests whenever a developer submits a pull request. The setup alerts developers immediately if any test fails. Since the teams are based in India, the US, and Germany, this system helps them work independently. It also avoids delays that often happen due to time zone differences.

Automate Test Execution

Using test automation frameworks lets you run tests automatically across different stages of development. Tools like TestNG, Playwright, and Cypress can help you do this easily. These tools are great for saving time and reducing human error.

By automating the test process, you avoid the need for manual execution. This also makes regression testing more manageable, especially in large applications. It gives your team more confidence to release code frequently.

As tests run on every code change, any new issues are quickly caught. And because automation supports repeatability, it keeps testing consistent across teams.

Example:
A healthcare software company uses Cypress for automating UI tests. These tests are connected with GitLab CI and run whenever someone updates a feature branch. The tests run in parallel containers, which helps speed up the process. This setup ensures key features are always verified before merging code. Even when several features are being developed at once, their system keeps everything on track.

Sample Cypress Test:

javascriptCopy codedescribe('Login Functionality', () => {
  it('should log in with valid credentials', () => {
    cy.visit('https://app.healthcare-demo.com/login')
    cy.get('input[name=email]').type('testuser@demo.com')
    cy.get('input[name=password]').type('securePassword123')
    cy.get('button[type=submit]').click()
    cy.url().should('include', '/dashboard')
    cy.contains('Welcome back')
  })
})

GitLab CI Configuration (gitlab-ci.yml):

yamlCopy codestages:
  - test

cypress_tests:
  stage: test
  image: cypress/browsers:node-18.12.0-chrome-106
  script:
    - npm ci
    - npx cypress run
  artifacts:
    when: always
    paths:
      - cypress/videos/
      - cypress/screenshots/
  only:
    - merge_requests
    - branches

This code demonstrates how Cypress runs UI tests, and how GitLab CI automatically triggers those tests when a new branch is pushed or a merge request is created. It reflects the kind of scalable, repeatable test execution process that supports global software teams.

Use Containerization for Environment Consistency

When you use Docker, you create containers that include your application and everything it needs to run. These containers can be shared and used anywhere. That means every developer and tester uses the same environment.

This removes the “it works on my machine” issue. It also helps create identical setups across development, staging, QA, and production. Since everyone works with the same tools and settings, there are fewer environment-related bugs.

With containerization, it becomes easier to test on different systems without needing to reconfigure anything. And it helps teams in different locations stay in sync.

Example:
A fintech startup packages its API and testing framework using Docker. These containers are then used inside Azure DevOps pipelines. The same tests run across staging, QA, and production environments. Since the containers do not change, the results are always reliable and consistent.

Sample Dockerfile for API Testing:

DockerfileCopy code# Use official Node.js image
FROM node:18

# Set working directory
WORKDIR /usr/src/app

# Copy package files
COPY package*.json ./

# Install dependencies
RUN npm install

# Copy test files and configuration
COPY . .

# Run tests (can be overridden in CI/CD)
CMD ["npm", "test"]

Example docker-compose.yml:

yamlCopy codeversion: '3.8'

services:
  api:
    image: fintech-api:latest
    build:
      context: ./api
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=test

  tests:
    build:
      context: ./tests
    depends_on:
      - api
    command: npm run test

This setup builds two containers – one for the API and one for the test runner. It mirrors how the startup runs its tests inside Azure DevOps pipelines. Developers, QA, and staging environments all use the same containers, reducing variability and giving more predictable test results.

Enable Scalable Test Orchestration

When tests are run in parallel across different environments, they finish faster. Tools like Selenium Grid, and LambdaTest help you do this. These platforms allow you to test across various browsers, operating systems, and devices.

By running tests this way, you save time and cover more scenarios at once. This is especially useful when your product needs to work globally. It ensures users across different regions have the same experience.

Parallel testing also helps teams working in different time zones. While one team sleeps, another team can pick up where they left off.

Example:
A retail company uses LambdaTest to run regression tests every night. These tests cover Chrome, Firefox, and Safari, both on desktop and mobile. Since the tests run in parallel, the UK team finishes validation before the US team starts their workday. This keeps their pipeline flowing without delays.

Sample Setup for Parallel Execution with Selenium Grid (Docker-based):

yamlCopy code# docker-compose.yml
version: "3"
services:
  selenium-hub:
    image: selenium/hub:4.18.1
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"

  chrome:
    image: selenium/node-chrome:4.18.1
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443

  firefox:
    image: selenium/node-firefox:4.18.1
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443

This creates a Selenium Grid with Chrome and Firefox nodes that connect to the central hub. It allows you to distribute your tests in parallel across browsers.

Sample Java Test (TestNG) for Running in Parallel:

javaCopy codepublic class ParallelTest {
    WebDriver driver;

    @Parameters("browser")
    @BeforeMethod
    public void setup(String browser) throws MalformedURLException {
        DesiredCapabilities capabilities = new DesiredCapabilities();

        if (browser.equalsIgnoreCase("chrome")) {
            capabilities.setBrowserName("chrome");
        } else if (browser.equalsIgnoreCase("firefox")) {
            capabilities.setBrowserName("firefox");
        }

        driver = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), capabilities);
    }

    @Test
    public void runTest() {
        driver.get("https://retail-demo.com");
        Assert.assertTrue(driver.getTitle().contains("Retail"));
    }

    @AfterMethod
    public void tearDown() {
        if (driver != null) {
            driver.quit();
        }
    }
}

TestNG Parallel Configuration (testng.xml):

xmlCopy code<suite name="ParallelTests" parallel="tests" thread-count="2">
  <test name="ChromeTest">
    <parameter name="browser" value="chrome"/>
    <classes>
      <class name="ParallelTest"/>
    </classes>
  </test>
  <test name="FirefoxTest">
    <parameter name="browser" value="firefox"/>
    <classes>
      <class name="ParallelTest"/>
    </classes>
  </test>
</suite>

This setup allows tests to run in parallel on both Chrome and Firefox, using Selenium Grid hosted locally with Docker. It mirrors how global teams can scale their test execution and speed up feedback loops.

Role of Containerization and Orchestration Tools

Tools like Docker and Kubernetes are essential for integrating TestOps into global DevOps pipelines.

• Docker creates lightweight, repeatable test environments that can be quickly set up or shut down. By packaging apps and their dependencies into containers, Docker ensures tests run consistently, whether on local machines or in the cloud.

• Kubernetes manages the deployment and scaling of containerized apps. In TestOps, Kubernetes automates test execution across multiple containers, speeding up testing in global pipelines. It helps scale testing to handle large volumes of tests in different environments and locations.

Containerization with Docker and orchestration with Kubernetes can streamline the testing process, especially for global DevOps pipelines.

Step 1: Docker for Repeatable Test Environments

You can use Docker to package your application along with its dependencies into a container, making it easy to run tests consistently across various environments.

Example Dockerfile for Test Environment:

dockerfileCopy codeFROM node:14

WORKDIR /app

# Install dependencies
COPY package*.json ./
RUN npm install

# Copy application code
COPY . .

# Run tests
CMD ["npm", "test"]

This Dockerfile sets up a container to install dependencies and run the tests using npm test.

Step 2: Kubernetes for Orchestration

You can use Kubernetes to scale the test execution across multiple containers. Kubernetes can manage the deployment of containers and automatically distribute them across nodes, enabling parallel testing.

Example Kubernetes Deployment YAML:

yamlCopy codeapiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment
spec:
  replicas: 3  # Number of containers to run in parallel
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
    spec:
      containers:
        - name: test-container
          image: my-app-test-env:latest  # Docker image built earlier
          ports:
            - containerPort: 80

This Kubernetes deployment configuration specifies that 3 replicas (containers) of the test environment will be created and run in parallel. This helps speed up the testing process by distributing the workload.

Step 3: Running Tests in Kubernetes

Once you have the containers and Kubernetes deployment configured, you can integrate this setup into your CI/CD pipeline. Kubernetes can handle scaling the test execution, making it ideal for global pipelines where tests need to run across different environments.

Example Kubernetes Command to Deploy:

bashCopy codekubectl apply -f test-deployment.yaml

This command deploys the test containers to the Kubernetes cluster, ensuring that your tests run across multiple containers, in parallel, at scale.

Continuous Monitoring and Feedback

TestOps relies on continuous monitoring to provide real-time insights into test results, performance, and system health.

Kubernetes helps manage testing resources and spot problems quickly. Real-time feedback from automated tests lets developers fix issues immediately, improving software quality.

Cross-Tool Integration

TestOps works well with different DevOps tools, creating a smooth feedback loop. It connects test management platforms (like TestRail) with CI/CD tools (like Jenkins or GitLab CI) and uses containerized environments to run tests consistently. Kubernetes ensures testing resources scale automatically to meet the needs of global teams.

Shift-Left Testing

TestOps follows a shift-left approach, which means integrating testing earlier in the pipeline to catch issues right away. Running tests in containerized environments speeds up testing and allows teams to find problems earlier in the development process, reducing risks and improving quality.

Shift-Left Testing with Docker and CI

Shift-left testing integrates tests early in the development pipeline to catch issues sooner. Using Docker in a CI pipeline automates test execution in a consistent environment.

Dockerfile Example:

dockerfileCopy codeFROM node:14
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
CMD ["npm", "test"]

Jenkinsfile Example:

groovyCopy codepipeline {
    agent any
    environment {
        DOCKER_IMAGE = 'my-app-test-env'
    }
    stages {
        stage('Checkout') {
            steps {
                git 'https://github.com/your-repository/my-app.git'
            }
        }
        stage('Build Docker Image') {
            steps {
                script {
                    sh 'docker build -t $DOCKER_IMAGE .'
                }
            }
        }
        stage('Run Tests') {
            steps {
                script {
                    sh 'docker run --rm $DOCKER_IMAGE'
                }
            }
        }
    }
    post {
        always {
            sh 'docker rmi $DOCKER_IMAGE'
        }
    }
}

Scalability and Flexibility

Global teams need to manage large test environments. Containerization and Kubernetes provide the scalability needed to run thousands of tests across different regions at once. Containers package tests into small, isolated environments, while Kubernetes automates their scaling and management, keeping testing efficient as the pipeline grows.

How to Use AI and Analytics in TestOps

When you bring AI and analytics into TestOps, it helps simplify complex testing tasks. It reduces manual work, improves accuracy, and supports better decision-making. Since DevOps teams often work across regions, this becomes even more important.

AI helps reduce repetitive testing tasks, while analytics turns test data into clear insights. Together, they create smarter and faster testing pipelines. And because these pipelines are shared globally, consistency is key.

What Tools Can You Use?

There are different tools that support AI and analytics in TestOps. Some focus on automation with intelligence, while others give you clear visibility into your test data.

AI-powered testing tools like Mabl, Testim, and Functionize use machine learning. These tools help create, run, and even repair test cases when the app changes. Since applications change frequently, these tools help keep your tests up to date.

They also save time on maintenance, since the tests adjust themselves when needed. And because the tools learn from patterns, they help teams catch issues faster.

Analytics and observability platforms such as TestRail Analytics, Xray, Grafana, and Kibana focus on trends. They turn raw test results into visual dashboards and alerts.

These platforms connect with CI/CD tools, so you get real-time updates on test quality. This makes it easier for teams to stay on top of what matters, even when they are spread across locations.

Example – Running a Working Test on LambdaTest

LambdaTest lets you execute real browser tests in the cloud, making it easy to scale your testing across browsers and OS combinations. Here is a working example using Python and Selenium, which opens a page, checks the title, and closes the browser:

pythonCopy codefrom selenium import webdriver
from selenium.webdriver.common.by import By

# Define LambdaTest capabilities
capabilities = {
  "browserName": "Chrome",
  "browserVersion": "latest",
  "LT:Options": {
    "platformName": "Windows 11",
    "build": "TestOps Working Demo",
    "name": "Title Verification Test",
    "selenium_version": "4.8.0",
    "w3c": True
  }
}

# Replace with your LambdaTest username and access key
USERNAME = "your_username"
ACCESS_KEY = "your_access_key"

# Connect to LambdaTest cloud grid
driver = webdriver.Remote(
    command_executor=f"https://{USERNAME}:{ACCESS_KEY}@hub.lambdatest.com/wd/hub",
    desired_capabilities=capabilities
)

try:
    # Step 1: Navigate to the app under test
    driver.get("https://www.lambdatest.com/selenium-playground/")

    # Step 2: Interact with the page (click a link)
    driver.find_element(By.LINK_TEXT, "Simple Form Demo").click()

    # Step 3: Enter message and verify output
    message_box = driver.find_element(By.ID, "user-message")
    message_box.send_keys("TestOps in action!")
    driver.find_element(By.ID, "showInput").click()

    output = driver.find_element(By.ID, "message").text
    assert output == "TestOps in action!", "Message output did not match input."

    print("✅ Test Passed: Message displayed correctly.")
except Exception as e:
    print("❌ Test Failed:", e)
finally:
    driver.quit()

What this test does:

• Launches a browser on LambdaTest cloud

• Navigates to their Selenium Playground

• Fills out a form and clicks a button

• Verifies that the output matches the input

• Logs the result and closes the session

Once the test finishes, you can view detailed logs, screenshots, and video recordings on the LambdaTest Automation Dashboard, which also includes AI-based debugging info.

What Types of Problems Can ML Help Solve — and How?

Machine learning can solve several pain points in testing. It looks at patterns in your data and helps identify things that manual checks might miss.

Flaky Tests:

ML helps detect tests that pass and fail randomly across different builds. It finds patterns in those failures and flags the ones that are unstable. And by doing this early, it prevents teams from wasting time chasing false bugs.

Test Prioritization:

ML studies your past test results and recent code changes. It then ranks your tests based on risk and importance. So, the most critical ones run first. This way, your pipeline moves faster without skipping key checks.

Failure Prediction:

ML uses logs, crash reports, and previous outcomes to predict where failures may happen. If it finds something risky, it warns the team in advance. This gives them time to fix problems before they grow bigger.

Root Cause Clustering:

When many tests fail at once, ML groups them by shared failure reasons. It helps you understand whether the issue is with one module or across several. That means your team can solve the actual root problem quicker.

Anomaly Detection:

ML tracks things like test duration and system behavior. If something suddenly changes, like a test taking too long or using too much memory, it flags it. These alerts help teams spot performance dips early.

What Types of Analytics Tools Can You Use?

Analytics tools help you turn your test results into useful information. They highlight patterns, gaps, and areas that need your attention. And because these insights are visual, they are easier to act on.

These tools can show how your pass/fail rates have changed over time. They also help you check which parts of your app are not covered by tests. If some tests are being skipped or are too flaky, the tools will highlight that too.

They also measure how long tests take to run and where your pipeline slows down. This helps teams reduce bottlenecks and improve efficiency.

Some platforms include dashboards that link test quality to deployment status. This gives you a clear picture of whether your product is ready for release.

They also track failures by environment—like which browsers or regions face more issues. This helps teams debug faster and improve global reliability.

All of these insights help QA and DevOps teams improve their strategies. They let you remove unnecessary tests, fix flaky ones, and focus where testing matters the most.

Example – Using Grafana and Kibana for Test Analytics

Analytics tools help your teams understand test trends, flakiness, coverage gaps, and slowdowns in the CI/CD pipeline. Here is how you can set them up to actually work with your test data.

Example 1: Visualizing Test Results in Grafana using InfluxDB

Grafana is commonly paired with InfluxDB to display metrics such as pass/fail rates, test durations, and failure frequencies. Here’s how you can push your test results into InfluxDB and visualize them in Grafana.

Step-by-step setup:

1. Push test results to InfluxDB after each test run. This can be done from Jenkins, GitHub Actions, or any test automation framework that generates test results.

2. Query and visualize data in Grafana using InfluxDB as the data source.

Python script to send test metrics to InfluxDB:

pythonCopy codefrom influxdb import InfluxDBClient
import time

# Create an InfluxDB client to send data
client = InfluxDBClient(host='localhost', port=8086)
client.switch_database('test_metrics')  # Switch to your specific database

# Example test result data
json_body = [
    {
        "measurement": "test_results",
        "tags": {
            "test_suite": "login_tests",  # Name of the test suite
            "environment": "staging"  # Environment like 'production', 'staging', etc.
        },
        "time": time.strftime('%Y-%m-%dT%H:%M:%SZ'),  # Timestamp for the test execution
        "fields": {
            "pass": 10,        # Number of tests that passed
            "fail": 2,         # Number of tests that failed
            "skipped": 1,      # Number of tests that were skipped
            "duration": 12.5   # Duration of the test run in seconds
        }
    }
]

# Write the data points to InfluxDB
client.write_points(json_body)

Grafana setup:

• Data Source: In Grafana, connect to your InfluxDB instance.

• Dashboard: Create a dashboard that queries the test_results measurement and display:

  • Line charts for pass/fail trends over time.

  • Pie charts for distribution of test results.

  • Table showing tests and their durations.

This approach helps you track key metrics and trends for each test suite and environment.

Example 2: Debugging Failures by Environment with Kibana and Elasticsearch

If your testing framework logs results into Elasticsearch, you can use Kibana to analyze and visualize those logs. For example, you can track which browsers or regions are facing more issues and display the results in Kibana.

Elasticsearch Data Model:

First, let’s assume that test results are logged in Elasticsearch with the following format:

jsonCopy code{
  "timestamp": "2025-04-16T14:00:00Z",
  "test_name": "checkout_flow_mobile",  # Name of the test
  "status": "fail",  # Pass or fail status
  "browser": "Safari",  # Browser used for the test
  "region": "EU-West",  # Region where the test was run
  "error": "Element not visible",  # Error message in case of failure
  "duration": 9.8,  # Duration of the test in seconds
  "env": "QA"  # Environment where the test ran
}

Kibana setup:

1. Data Ingestion: Your CI pipeline or test scripts push results to Elasticsearch after each run.

2. Create Visualizations: In Kibana, create visualizations like:

   • Pie Chart: Show failure rates by browser type (e.g., Chrome, Firefox, Safari).

   • Line Chart: Track test durations over time for a specific test suite.

   • Table: Display flaky tests that fail repeatedly by region or environment.

Example of a Kibana query that you might use to filter failures by browser:

jsonCopy code{
  "query": {
    "bool": {
      "must": [
        { "match": { "status": "fail" }},
        { "match": { "browser": "Safari" }}
      ]
    }
  }
}

This will show all test failures in Safari, helping you identify browser-specific issues.

Why This Matters

Using Grafana and Kibana with your test results helps your team gain valuable insights:

• Identify flaky tests and prioritize them for maintenance.

• Track performance trends, including test duration and failure rates.

• Debug faster by identifying failure patterns tied to specific browsers, environments, or regions.

With these analytics in place, teams can make data-driven decisions to improve test coverage, reduce bottlenecks, and ensure better product quality.

Future of TestOps in Global Development

As global software delivery keeps growing more complex, TestOps is changing fast. It is no longer just a supporting function. Instead, it is becoming a central part of DevOps strategies. And as development moves faster, this shift is only expected to continue.

Here are some of the key changes that are shaping the future of TestOps. You will need to watch for them and prepare ahead.

AI-Driven Decision-Making in TestOps

AI in TestOps is moving beyond just automating test cases or running scripts. It is starting to take on decision-making roles within the testing process. For instance, orchestration tools powered by AI will soon go further.

They will decide which tests should run, based on recent code changes and how those changes affect the business. They will also predict which parts of the system are more likely to break in the next release.

And as these tools learn from real-time usage patterns, they will suggest ways to improve your test strategy. That means TestOps professionals will not only need to know how to use these tools. They will also need to understand how to read the insights and make smart choices with them.

Cloud-Native and Edge Testing

While teams are already embracing cloud-based testing, the next step is even more distributed. This is where edge-aware TestOps comes in. It focuses on testing software where it is used, not just where it is built.

That means tests will run closer to the user, in specific networks or regional setups. It will also mean checking how systems perform in places with different latency or unreliable connectivity.

And as data laws vary across regions, teams will need to manage test data carefully and securely. Because TestOps now stretches across countries and cloud platforms, it must adapt to decentralized architectures.

TestOps as a Unifier of Observability and Automation

TestOps will not only be about running tests anymore. It will play a larger role in bringing together testing, monitoring, and automation across the pipeline. This will create a more complete view of system health and product quality.

Tools under TestOps will start using production monitoring data to improve test design. If something fails in production, it can guide what should be tested next.

Real-time behavior from users can even trigger specific regression tests. This helps teams fix issues faster and smarter. As a result, TestOps will create a feedback loop between pre-release and post-release stages.

That means teams will not only rely on TestOps for test automation. They will also use it to see how everything connects—from development to operations.

Skill Shifts in TestOps Teams

As testing tools become more advanced, the skills that TestOps teams need will change too. There will be less demand for manual testing roles. But there will be more need for engineers who think strategically about quality.

You will see more roles focused on site reliability, automation frameworks, and test strategy. These roles will require knowledge of cloud infrastructure and continuous delivery.

And instead of working in silos, testers will work closely with developers and ops teams. This shift will require people who can think across functions and understand how everything fits together.

Conclusion

Scaling TestOps for global software development teams is essential in today’s fast-moving, distributed work environment. By using best practices like standardizing tools, automating tests, promoting collaboration, and taking advantage of cloud and AI solutions, teams can ensure smooth, high-quality software delivery across different regions and time zones.

As TestOps evolves with advances in automation, AI, and cloud technology, it will make the testing process even more efficient. Teams will be able to respond faster, predict problems before they happen, and maintain high-quality standards.

The future of TestOps looks even more promising with smarter tools, better collaboration, and more automation, driving success for global development teams and improving the entire software development process.

Effective unit tests not only catch errors but also help you be confident that your code can be maintained and is dependable. But it takes time and resources to manually create an extensive suite of unit tests.

There have been some recent developments in artificial intelligence that promise to help automate the unit test development processes. In February, researchers at Meta released a paper on Automated Unit Test Improvement using Large Language Models. This introduced an innovative method for automating unit testing.

Their research focuses on a new tool called TestGen-LLM, which explores the possibilities of using LLMs to analyze already-existing unit tests and improving on them to increase code coverage.

Although the code for the TestGen-LLM was not released, I will introduce an open-source alternative inspired by their research in this article. You'll learn how it generates test suites, why it's better than most LLMs, and where to get your hands on this technology and start trying it out.

Table of Contents

• Meta’s TestGen-LLM

• How does TestGen-LLM work?

• Open-Source Implementation (Cover-Agent)

• How does Cover-Agent work?

• How to Use Cover-Agent

• Benefits of the Open-Source Cover-Agent

• How can you Contribute to this Technology?

• Conclusion

Meta’s TestGen-LLM

Meta's TestGen-LLM tackles the time-consuming task of unit test writing by leveraging the power of Large Language Models (LLMs). General-purpose LLMs like Gemini or ChatGPT might struggle with the specific domain of unit test code, testing syntax, and generating tests that don't add value. But TestGen-LLM is specifically tailored for unit testing.

This specialization allows it to understand the intricacies of code structure and test logic, leading to more targeted test suites and generate tests that actually add value and increase code coverage.

TestGen-LLM is able to evaluate unit tests and identify areas for improvements. It achieves this through its understanding of common testing patterns which it has been trained with. But generating tests alone is insufficient for proper code coverage.

Meta's researchers implemented safeguards within TestGen-LLM to ensure the effectiveness of the tests it writes. These safeguards, referred to as filters, act as a quality control mechanism. They eliminate suggestions that:

• wouldn't compile

• fail consistently, or

• fail to actually improve code coverage (suggestions that are already covered by other tests).

How Does TestGen-LLM Work?

TestGen-LLM uses an approach called "Assured LLM-based Software Engineering" (Assured LLMSE). TestGen-LLM simply augments an existing test class with additional test cases, retaining all existing test cases and thereby guaranteeing that there will be no regression.

Test generation workflow(From TestGen_LLM paper)

The TestGen-LLM generates a bunch of tests, then filters out the tests that don’t run and drops any that don’t pass. Finally, it discards those that don't increase the code coverage.

After using the TestGen-LLM to automate a test suite, Meta used a human reviewer to accept or reject tests where the generated tests had an acceptance rate of 73% in their best reported cases.

According to the paper, TestGen-LLM generates a single test on each run that is then added to an existing test suite which was written previously by a developer. But it doesn’t necessarily generate tests for any given test suite.

The effectiveness of TestGen-LLM was demonstrated at Meta's internal test-a-thons. Here, the tool was used to analyze existing test suites and suggest improvements. The results were promising:

“75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. During Meta's Instagram and Facebook test-a-thons, it improved 11.5% of all classes to which it was applied, with 73% of its recommendations being accepted for production deployment by Meta software engineers”.

Also, the recommendations from TestGen-LLM were deemed useful and relevant by the developers who took part in the test-a-thons.

Open-Source Implementation (Cover-Agent)

The TestGen-LLM research from Meta has a lot of potential to change unit testing and automated test generation. The tool will likely help improve code coverage and speed up test creation by utilizing LLMs particularly trained on code. But this technology is not available for use by just anyone, since the code for the TestGen-LLM was not released.

Developers who have taken an interest in this technology are probably frustrated by the lack of publicly available code. After all, Meta's TestGen-LLM study provides a glimpse into the future of what automated testing can be.

It's quite appealing to be able to dive into the newest technology’s internal workings, comprehend its decision-making procedures, and maybe even help shape its evolution. But while the absence of Meta's code is a hindrance, there's an open-source implementation called Cover-Agent which can serve as a helpful alternative.

CodiumAI's Cover-Agent is the first open-source implementation of an automated testing tool based on TestGen-LLM. Inspired by Meta's research, Cover-Agent is now at the forefront of developments in open-source AI-driven unit testing as a result.

Why are specific testing-focused LLMs necessary?

Since most LLMs (like ChatGPT and Gemini) are capable of generating tests, then why bother with a new technology?

Well, Cover-Agent and TestGen-LLM were created to be the next step in the evolution of efficient unit testing. Their aim is to avoid common pitfalls that developers run into when generating tests with LLMs such as:

• LLM Hallucination

• Generating tests that do not add value

• Generating tests that omit some parts of the code, resulting in low code coverage

To overcome such challenges (specifically for regression unit tests) the TestGen-LLM researchers came up with the following criteria which the generated tests must meet before the test can be accepted:

• Does the generated test compile and run properly?

• Does the test increase code coverage?

• Does it add value?

• Does it meet any additional requirements that we may have?

These are fundamental questions and issues that the generated test must solve before it is deemed an upgrade on the existing technology. Cover-Agent provides tests that answer these questions to an amazingly high degree.

How Does Cover-Agent Work?

Cover-Agent is part of a broader suite of utilities designed to automate the creation of unit tests for software projects. Utilizing the TestGen-LLM Generative AI model, it aims to simplify and expedite the testing process, ensuring high-quality software development.

The system is made up of several components:

• Test Runner: Executes the command or scripts to run the test suite and generate code coverage reports.

• Coverage Parser: Validates that code coverage increases as tests are added, ensuring that new tests contribute to the overall test effectiveness.

• Prompt Builder: Gathers necessary data from the codebase and constructs the prompt to be passed to the Large Language Model (LLM).

• AI Caller: Interacts with the LLM to generate tests based on the prompt provided.

These components work together with TestGen-LLM to generate only tests that are guaranteed to improve the existing code base.

How to Use Cover-Agent

Requirements

You need to have the following requirements before you can begin using Cover-Agent:

• OPENAI_API_KEY set in your environment variables, which is required for calling the OpenAI API.

• Code Coverage tool: A Cobertura XML code coverage report is required for the tool to function correctly. For example, in Python you could use pytest-cov. Add the --cov-report=xml option when running Pytest.

Installation

If you're running Cover-Agent directly from the repository, you will also need:

• Python installed on your system.

• Poetry installed for managing Python package dependencies. You can find installation instructions for Poetry here.

Standalone Runtime

You can install Cover-Agent as a Python Pip package or run it as a standalone executable.

Python Pip

To install the Python Pip package directly via GitHub, run the following command:

pip install git+https://github.com/Codium-ai/cover-agent.git

Binary

You can run the binary without any Python environment installed on your system (for example, within a Docker container that does not contain Python). You can download the release for your system by navigating to the project's release page.

Repository Setup

Run the following command to install all the dependencies and run the project from source:

poetry install

Running the Code

After downloading the executable or installing the Pip package, you can now run Cover-Agent to generate and validate unit tests.

Execute it from the command line by using the following command:

cover-agent \
--source-file-path "path_to_source_file" \
--test-file-path "path_to_test_file" \
--code-coverage-report-path "path_to_coverage_report.xml" \
--test-command "test_command_to_run" \
--test-command-dir "directory_to_run_test_command/" \
--coverage-type "type_of_coverage_report" \
--desired-coverage "desired_coverage_between_0_and_100" \
--max-iterations "max_number_of_llm_iterations" \
 --included-files "<optional_list_of_files_to_include>"

You can use the example projects within this repository to run this code as a test.

Command Arguments

• source-file-path: Path of the file containing the functions or block of code we want to test for.

• test-file-path: Path of the file where the tests will be written by the agent. It’s best to create a skeleton of this file with at least one test and the necessary import statements.

• code-coverage-report-path: Path where the code coverage report is saved.

• test-command: Command to run the tests (for example pytest).

• test-command-dir: Directory where the test command should run. Set this to the root or the location of your main file to avoid issues with relative imports.

• coverage-type: Type of coverage to use. Cobertura is a good default.

• desired-coverage: Coverage goal. Higher is better, though 100% is often impractical.

• max-iterations: Number of times the agent should retry to generate test code. More iterations may lead to higher OpenAI token usage.

• additional-instructions: Prompts to ensure the code is written in a specific way. For example, here we specify that the code should be formatted to work within a test class.

On running the command, the agent starts writing and iterating on the tests.

How to Use Cover-Agent

It's time test out Cover-Agent. We will use a simple calculator.py app to compare the code coverage for manual and automated testing.

Manual Testing

def add(a, b):
    return a + b

def subtract(a, b):
    return a - b

def multiply(a, b):
    return a * b

def divide(a, b):
    if b == 0:
        raise ValueError("Cannot divide by zero")
    return a / b

This is the test_calculator.py placed in the test folder.

# tests/test_calculator.py
from calculator import add, subtract, multiply, divide

class TestCalculator:

    def test_add(self):
        assert add(2, 3) == 5

To see the test coverage, we need to install pytest-cov, a pytest extension for coverage reporting stated earlier.

pip install pytest-cov

Run the coverage analysis with:

pytest --cov=calculator

The output shows:

Name            Stmts   Miss  Cover
-----------------------------------
calculator.py      10      5    50%
-----------------------------------
TOTAL              10      5    50%

The output above shows that 5 of the 10 statements in calculator.py are not executed, resulting in just 50% code coverage. For a larger code base, this will become a serious issue and lead to setbacks.

Now let's see if cover-agent can do better.

Automated Testing with Cover-Agent

To set up Codium's Cover-Agent, follow these steps:

First, install Cover-Agent:

pip install git+https://github.com/Codium-ai/cover-agent.git

Make sure that your OPENAI_API_KEY is set in your environment variables, as it is required for the OpenAI API.

Next, write the commands to start generating tests in the terminal:

cover-agent \
--source-file-path "calculator.py" \
--test-file-path "tests/test_calculator.py" \
--code-coverage-report-path "coverage.xml" \
--test-command "pytest --cov=. --cov-report=xml --cov-report=term" \
--test-command-dir "./" \
--coverage-type "cobertura" \
--desired-coverage 80 \
--max-iterations 3 \
--openai-model "gpt-4o" \
--additional-instructions "Since I am using a test class, each line of code (including the first line) needs to be prepended with 4 whitespaces. This is extremely important to ensure that every line returned contains that 4 whitespace indent; otherwise, my code will not run."

This generates the following code:

import pytest
from calculator import add, subtract, multiply, divide

class TestCalculator:

    def test_add(self):
        assert(add(2, 3), 5

    def test_subtract(self):
        """
        Test subtracting two numbers.
        """
        assert subtract(5, 3) == 2
        assert subtract(3, 5) == -2

    def test_multiply(self):
        """
        Test multiplying two numbers.
        """
        assert multiply(2, 3) == 6
        assert multiply(-2, 3) == -6
        assert multiply(2, -3) == -6
        assert multiply(-2, -3) == 6

    def test_divide(self):
        """
        Test dividing two numbers.
        """
        assert divide(6, 3) == 2
        assert divide(-6, 3) == -2
        assert divide(6, -3) == -2
        assert divide(-6, -3) == 2

    def test_divide_by_zero(self):
        """
        Test dividing by zero, should raise ValueError.
        """
        with pytest.raises(ValueError, match="Cannot divide by zero"):
            divide(5, 0)

You can see that the agent also wrote tests that check for errors for any edge cases.

Now it's time to test the coverage again:

pytest --cov=calculator

Output:

Name            Stmts   Miss  Cover
-----------------------------------
calculator.py      10      0   100%
-----------------------------------
TOTAL              10      0   100%

In this example we reached 100% code coverage. For larger code bases, the procedure is relatively the same. You can read through this guide for a walkthrough on a larger code base.

While Cover-Agent represents a significant step forward, it's important to note that this technology is still in its early stages. Continued research and development are crucial for further refinement and broader adoption and codiumAI invites you to make your contributions to this open source tool.

Benefits of the Open Source Cover-Agent

The open source nature of Cover-Agent offers several advantages that should help propel the technology forward. Among them are:

• Accessibility: Its open source nature enables LLM-based testing experimentation and it is accessible to developers with varying backgroundsThis will increase the number of users and lead to the development of a better technology and more applications..

• Cooperation: Developers are able to make contributions, suggest enhancements, propose new features and report issues. Cover-Agent will grow and develop quickly into a project perfect for developers..

• Transparency: Information about the internal operations are available and this promotes trust and will ultimately increase the potential for the technology.

In addition to it’s open-source advantages, Cover-Agent provides developers with a strong set of benefits of its own:

• Simple Access: Developers can easily install and experiment with LLM-based testing. This allows for firsthand and immediate exploration of the technology's capabilities and with little to no disruption in their workflow.

• Customization for Specific Needs: Cover-Agent's open-source nature allows developers to adapt the tool to their specific project requirements. This could involve modifying the LLM model used, adjusting training data to better reflect their codebase, or integrating Cover-Agent with existing testing frameworks. This level of customization empowers developers to leverage the power of LLM-based testing in a way that aligns with their project needs.

• Easy Integration: It is easily integrated with VSCode (a popular code editor) which makes integration with existing workflows a breeze. You can also easily integrate it with existing human-written tests.

How Can You Contribute to Cover-Agent?

Cover-Agent source code is publicly available through this GitHub repo. They encourage developers of all backgrounds to test their product and make contributions to further improve and grow this new technology.

Conclusion

LLM-based test improvement tools hold immense potential for revolutionizing the way developers approach unit testing. By leveraging the power of large language models specifically trained on code, these tools can streamline test creation, improve code coverage, and ultimately enhance software quality.

While Meta's research with TestGen-LLM offers valuable insights, the lack of publicly available code hinders wider adoption and ongoing development. Fortunately, Cover-Agent has provided a readily accessible and customizable solution. It empowers developers to experiment with LLM-based testing and contribute to its evolution.

The potential for the TestGen-LLM and Cover-Agent is immense, and further development through contributions by developers will lead to a revolutionary tool that will transform automated test generation forever.

• Meta’s Research Paper

• Cover-Agent Blog for further study

• Article for installation process and use cases

Connect with me on LinkedIn and Twitter if you found this helpful.

But it can be difficult to make sure your React application functions as intended in various different scenarios.

Cypress is an end-to-end testing framework that you can use to test your React apps. And in this tutorial, I'll explain how to create efficient end-to-end tests for your web application using Cypress and React.

Table of Contents

1. What is Cypress?

2. How to Set Up Your Environment

3. How to Write Your First Cypress Test

4. How to Interact with React Components

5. Cypress Queries

6. Cypress Assertions

7. Cypress Actions

8. How to Organize and Run Your Cypress Tests

9. Plugins

10. Describe Blocks

11. Custom Commands

12. Cypress Dashboard

13. How to Debug Your Code with Cypress

14. Continuous Integration

15. Conclusion

What is Cypress?

Cypress is a straightforward open-source end-to-end testing framework designed for modern web development. It's based on JavaScript.

The tool operates within the browser, which sets it apart from other testing tools like Selenium. The concise, uncomplicated API that Cypress provides for interacting with your application makes it easy to create and manage tests.

You can create and manage tests that mimic user interactions with Cypress. You can also examine your web application for expected behavior and troubleshoot any concerns that arise.

How to Set Up Your Environment

Before we begin writing tests, you'll want to set up your environment. You must have Node.js and npm (Node Package Manager) installed in order to use Cypress with your React application.

You can visit https://nodejs.org/en/ to download to install Node.js on your computer if it isn't already installed. To get started using Cyprus after installing Node, you can follow these steps:

1) Create a React Application

We'll utilize Vite to develop our React application, as it helps us quickly and efficiently create web apps.

To get started, create a React app on Vite by entering the following code in your console:

npm create vite@latest

Vite will ask you to give your project a name. Choose whatever name you like.

Following that, Vite will give you the choice of framework. Just make sure you select React for this lesson.

It'll then ask you to select between TypeScript and JavaScript when you select React. Pick JavaScript.

Then go to your project directory and choose your project.

Finally, type npm install after that, and then npm start dev to launch your project.

Your project will be hosted by Vite at http://localhost:5173/. To view your project in a browser, click the link that's provided.

2) Install cypress

You can install Cypress in your project by using this code install cypress –-save-dev in your terminal. This code installs Cypress in your React application.

3) Initialize Cypress

Run npx Cypress open from the directory of your React project. This command should launch Cypress Test Runner, a graphical tool for creating and managing tests. You will now have two alternatives for automated testing:

• End-to-End Testing: End-to-end testing involves a browser using an automated tool like Cypress, so you can make tests consisting of functions like pressing buttons, typing in inputs, and so on. This helps make sure everything’s working perfectly.

• Component Testing: This involves testing individual components instead of the entire React application.

What is the Cypress Test Runner?

The test runner opens up once you pass in npx cypress open. It is a graphical tool for creating and managing tests, it also allows you to write and execute your test cases in Cypress.

After selecting what type of testing you want to do, the next section of the test runner helps you specify what script you want to execute, and gives you the option to select the browser you want to use and validate your tests as expected.

4) Folder Structure

Cypress will automatically save its test files and configurations inside of your project in a subfolder named cypress.

How to Write Your First Cypress Test

We'll use JavaScript to write our Cypress tests which are executed within the browser. Here's how to create your React application's first test.

Suppose we wish to check to see if the Document Object Model (DOM) displays our h2 element. To accomplish this using Cypress, take the following actions:

Step 1: In your Cypress folder, go to the e2e file and there you’ll see a line of code similar to this:

describe('template spec', () => {
    it('passes', () => {
        cy.visit('https://example.cypress.io')
    })
    })

Step 2: Now, in cy.visit, update the default http address to your React app's address (http://localhost:5173/, if you recall).

Step 3: We wish to check to see if the DOM has our h2 element. Navigate to the element you wish to test and add data-testid as an attribute to accomplish this. Next, assign a name to the attribute. Example: <h2 data-testid="cypress-h1"> Leo Website </h2>.

Step 4: Make a space and add this piece of code to your cypress folder to see if the attribute exists.

cy.get(‘[data-testid=”cypress-header”]’).should(“exists”):

Since Cypress is configured to execute tests upon save, as soon as you save anything in your code editor, the test ought to instantly start in Cypress.

How to Interact With React Components

You'll want to be familiar with the vast array of commands used to imitate user interactions in order to write more insightful tests.

With its extensive command set for interacting with and testing online elements, Cypress is designed specifically for webpages and web applications.

To imitate user interactions and assert certain behaviors of the application, test scripts employ these instructions. Because Cypress maintains a promise chain on your behalf, you can chain commands together. Until the chain breaks and an error occurs, each command yields a "subject" to the following command.

Cypress commands can be classified into the following groups: queries, assertions, actions, and others.

Cypress Queries

These are Cypress commands that retrieve your application's current status. They retry as necessary to ensure that the DOM element or other data they offer is constantly current. They return a subject for additional commands to act upon or assert.

Here are some ways you can organize and run tests:

Using the .children() method

To choose and work with a parent element's direct child elements in Cypress, use the .children() method. It enables you to access particular child elements of a given parent element by navigating the DOM tree.

Here are some examples of how to use .children() in Cypress:

Example 1: Assuming you have this sort HTML structure in your jsx file:

<div class="parent-element">
  <div class="child">Child 1</div>
  <div class="child">Child 2</div>
  <div class="child">Child 3</div>
</div>

To choose and work with the child elements inside the .parent element, use the .children() function like this:

cy.get('.parent').children('.child').should('have.length', 3);

// Click on the second child element
cy.get('.parent').children('.child').eq(1).click();

Example 2: you can perform a whole range of tasks using .children() combined with other Cypress functions. You can, for instance, recognize certain child objects and define their properties:

// Ensure that the first child element contains the text "Child 1"
cy.get('.parent').children('.child').first().should('contain', 'Child 1');

// Verify that all child elements have a specific class
cy.get('.parent').children('.child').each(($child) => {
  cy.wrap($child).should('have.class', 'child');
});

Example 3: Many Cypress instructions may be chained together to perform more complex interactions. Using .children(), you can select the parent element, select its child elements, and then operate with the child elements as demonstrated by this example:

cy.get('.parent')
  .children('.child')
  .first()
  .click()
  .next()
  .type('I'm Leo');

To interact with the next child element in this example, we first pick the .parent element, then use .children('.child') to select its child elements, click on the first child element, and then chain more instructions to type "I'm Leo."

These examples show you how to pick and work with child items inside of a parent element in Cypress using the .children() function. Depending on your unique HTML structure, you can modify the CSS selector inside .children() to match the child elements you wish to target.

Now you should have an have an idea of how .children() works and how you can use queries as a type of Cypress command. There are more query commands but you can always refer to the documentation on cypress.io to learn more on them.

Using the cy.get() command

cy.get() is a fundamental command in the Cypress framework. You use it to choose and perform actions on DOM elements inside a web application, including those created using React.

To choose one or more DOM elements from your React application for testing, use the cy.get() command. It lets you pick elements with CSS selectors or other Cypress-compatible methods.

The fundamental syntax of cy.get() is cy.get(selector).

selector is an instance of a CSS selector. There are alternative ways for choosing components based on attributes like ID, class name, or data.

Suppose you have a React component like this:

<button data-testid="Leo-button">Click</button>

You can use cy.get() to select the button element:

cy.get(‘[data-testid=”Leo-button”]’);

cy.get('#Leo-button') in this case chooses the button element with ID "Leo-button."

Cypress Assertions

These are Cypress commands that make assertions about the state of your application. They stop your test when they reach the time limit or the specified condition is fulfilled.

There are only two types of assertion commands: .and() and .should(). I'll discuss .should() here.

In Cypress, statements concerning the states of particular elements are made using the .should() command. It enables you to specify requirements that components must fulfill. In the event that the requirements are not satisfied, the test will fail.

Fundamental Use of .should()

The basic syntax of .should() is as follows:

cy.get('selector').should('condition', expectedValue);

• The DOM element(s) that you wish to make assertions against are chosen by cy.get('selector').

• .should('condition', expectedValue) indicates what conditions and values the chosen element(s) must satisfy.

You can also chain .should() assertions. Chained .should() statements are used to verify different conditions on the same element:

cy.get('input#username')
  .should('be.visible')
  .should('have.attr', 'placeholder', 'Enter your username')

In this example, we verify that the username input element is there and that the placeholder content is appropriate.

.should() also allows you to use custom assertions by passing a callback function as the expected value:

cy.get('.my-element').should(($element) => {
  expect($element).to.have.class('active');
  expect($element).to.have.css('color', 'rgb(255, 0, 0)');
});

In this code, we're using custom assertions within .should() to check if the element has the class "active" and if its text color is red.

Negating assertions with .should()

.should('not.condition', expectedValue) can be used to refute the following claims:

cy.get('#error-message').should('not.exist');

This code verifies that there isn't an element with the ID "error-message".

Combining .should() with Other Commands

To create testing scenarios with greater complexity, you can combine .should() with other Cypress commands. For instance, you can use it to assert the element's final state following a .click() or .type() command.

cy.get('button').click().should('be.disabled');
cy.get(‘[data-testid=”search-input”]’).type('Leonard').should('have.value', 'Leonard');

In these instances, we interact with the elements first (by clicking a button or entering text into an input box), and then we use the resultant states of the elements' states as determined by using .should().

When constructing relevant assertions to confirm the behavior and state of items in your web application during end-to-end testing, Cypress's .should() command is a potent tool.

Cypress Actions

Cypress commands that interact with your program like a user would are called actions. They do not interact with the page until components or elements are actionable. There are numerous examples, but for now I'll use the .click() command.

To mimic a click on a DOM element in Cypress, use the .click() command. It's among the most typical and basic things you'll do in your end-to-end tests.

Let's look at an example of some code to demonstrate how to use .click():

Basic .Click() Usage

The following is the syntax of .click():

cy.get('selector').click();

To choose the DOM element you want to click, use cy.get('selector'). .click() replicates a click event on the element that is chosen.

Managing Clicks During User Interaction

Sometimes, you may need to interact with things before clicking. For instance, you may want to complete a form field and press the submit button afterward. You can use .type() to communicate with input fields before continuing on to the submit button, .click():

cy.get(‘[data-testid=”username”]’).type('okosaleo');
cy.get(‘[data-testid=”password”]’).type('password');
cy.get(‘[data-testid=”Leo-button”]’).click();

First, in this case we employ .type() to input values in the password and username fields, and then click .click() to click the login button and submit the form.

Managing Dynamic Elements

You can utilize .click() while working with dynamic elements, or elements that emerge following an action. In order to make sure the element is there and prepared for interaction, you can use .click() in conjunction with other commands.

For instance, clicking a dynamic element. Assume you have a list of items and that you wish to click on a certain item that pops up following a certain amount of user interaction:

cy.get(‘[data-testid=”Leo-button”]’).click();
cy.get('.dynamic-item').should('have.length.gt', 5);
cy.get('.dynamic-item').eq(4).click();

In this example, to load new things dynamically, we first click the "Leo-button" button. After making sure that at least six things are shown using .should('have.length.gt', 5), we click the fifth item using .eq(4).

Other commands

Cypress has many other commands which we haven't discussed here but that you can use to write other tests. You can check them out in the Cypress docs here.

How to Organize and Run Your Cypress Tests

Your application may accumulate a number of tests over time. Cypress offers several methods for setting them up and managing them effectively.

Plugins

To assist with tasks such as creating code coverage reports and simulating APIs, Cypress offers a number of extensions and plugins.

Plugins are specially designed JavaScript modules that let you modify and add new features to the Cypress testing framework. With the use of plugins, you can increase Cypress's capabilities and customize it to suit your unique requirements.

Cypress plugins include:

1. Integrations: Cypress may be integrated with other programs, services, and systems by the creation of plugins. Examples of these include deployment pipelines, version control systems, and platforms for continuous integration (CI).

2. Custom Commands: To encapsulate and reuse routine test procedures or user interactions with your application, you may define custom commands using plugins. This improves the organization and maintainability of your test code.

There are many more plugin features that Cypress offers. Plugins are a flexible way to customize your testing framework to the specific needs of your project, increase the power of your testing, and organize and maintain your test code better.

Describe Blocks

The Mocha testing framework, which is integrated with Cypress to help you organize and structure your test suites, includes describe blocks. These describe blocks provide your tests a clear structure and help you group related test cases.

You can use describe blocks to categorize and explain the functionality and behavior of the program you are evaluating.

Purpose of Describe Blocks in Cypress

• Organize Test Cases: You can organize relevant test cases together by using the describe blocks. A description block can be made for a particular application feature, part, or functionality.

• Enhance Test framework: They give your tests a hierarchical framework that makes the test suite easier to explore and comprehend. Developers and testers can find certain tests more quickly with the aid of this structure.

• Clarity and Documentation: You can give titles for your tests that are both human-readable and descriptive by using describe blocks. This functions as a type of test documentation.

A describe block is structured like this:

describe('Description of the Test Suite or Feature', function () {
  // Your test cases (it blocks) go here
});

• "Description of the Test Suite or Feature": This is a string that contains a description of the test suite or feature. It acts as the group of test cases title.

• function () { /*... */ }: This function holds the various test cases that are related to this describe block and are written within it blocks.

How to Use Describe Blocks Custom Commands

Let's say you are evaluating a web application's login functionality. The following is how you could structure your tests using describe blocks:

describe('Login Feature', function () {
  // This is the outermost describe block for the Login Feature

  it('should display the login form', function () {
    // Test case logic to check if the login form is displayed
  });

  it('should show an error message on invalid login', function () {
    // Test case logic to check error message display
  });

  it('should log in successfully with valid credentials', function () {
    // Test case logic to check successful login
  });

  describe('Password Reset', function () {
    // Nested describe block for Password Reset functionality

    it('should allow users to request a password reset', function () {
      // Test case logic for password reset request
    });

    it('should reset the password with a valid reset token', function () {
      // Test case logic for password reset with a valid token
    });
  });
});

In this instance:

1. All of the test cases for the login feature are grouped together in the outermost describe block.

2. Tests pertaining to the password reset feature are grouped together in a describe block nestled within it.

You can easily identify related tests thanks to this hierarchical structure, which also aids in clarifying the goals of each set of test cases.

Effective use of describe blocks in Cypress allows you to write well-structured and easily accessible test suites, which will facilitate test management and maintenance as your project expands.

Custom Commands

To simplify repetitive tasks and improve the readability of your tests, you can create custom Cypress commands.

To encapsulate and reuse common test processes or interactions with the application being tested, you may develop and add custom commands—user-defined JavaScript functions—to your Cypress test suite. These can help augment the built-in Cypress commands and improve the readability, organization, and maintainability of your test code.

You need to define your custom command in one of your test files, usually the commands.js file. Often, you may find this file in the cypress/support directory.

On the other hand, you can arrange your customized commands in different files or folders according to the features of your program.

Here's an example:

Cypress.Commands.add('customCommandName', (arg0, arg1, ...) => {
  // Define the custom command logic here
});

• customCommandName: This is the name that you'll use in your test scripts to identify your custom command.

• (arg0, arg1,...): These are the parameters or arguments that you can give to the custom command in order to alter how the program behaves.

• // Custom command logic }: Here you provide the precise operations you wish to be carried out by your custom command.

How to Use Custom Commands

Once your custom command is defined, you can use it in your test scripts in the same way as any other Cypress command that is built-in.

Using the following syntax, you can invoke a custom command:

cy.customCommandName(arg0, arg1, ...);

Advantages of Custom Commands

Cypress custom commands have a number of advantages:

1. Reusability: You can encapsulate intricate or commonly used interactions with your application using custom commands. This lessens redundancy in your test scripts and encourages the reuse of code.

2. Readability: Your test scripts become more accessible and understandable when typical tasks are abstracted into custom commands. Your test code will be more self-explanatory with custom commands.

3. Maintenance: Rather than having to look for and update each instance of an action in several test scripts, you just need to change the custom command once when you need to update a common action or interaction.

4. Consistency: Using custom commands guarantees that your test suite's common interactions are carried out in a consistent manner. This keeps the testing process consistent.

This is a basic illustration of a custom command that allows a user to log in by filling out a login form with their username and password:

Cypress.Commands.add('login', (username, password) => {
  cy.get(‘[username-input”]’).type(username);
  cy.get(‘[data-testid=”password-input”]’).type(password);
  cy.get(‘[data-testid=”Leo-button”]’).click();
});

Cypress Dashboard

This tool lets you monitor films of text executors, organize test runs, and discuss outcomes with your team. You can find the Cypress dashboard in the test runner window.

To see the history of your test runs, click the "Runs" tab. The Runs tab will include a "View Dashboard" link at the top. When you click the "View Dashboard" link, your web browser will launch the Cypress Dashboard.

Cypress offers the dashboard as a web-based service. The goal of the Cypress Dashboard is to improve and streamline the web application testing and test management process. It provides a single platform to monitor, analyze, and manage your test runs using Cypress.

The Cypress Dashboard's main attributes and capabilities are as follows:

1. Real-Time Test Execution: You can watch the test execution process in real time when you use the Cypress Dashboard service to execute Cypress tests. It offers a real-time view of your testing, including with videos and live browser screenshots. This tool is useful for tracking test progress and troubleshooting problems.

2. Collaboration and Sharing: The development and QA teams can work together thanks to the dashboard. Test results may be accessed and reviewed by developers, testers, and other stakeholders with ease when test runs are shared with team members.

3. Centralized Test Management: You can store and manage your Cypress test results centrally using the Cypress Dashboard. Test runs can be accessed and arranged from a single location.

4. The Cypress Dashboard facilitates the concurrent running of tests across many computers and web browsers. This can speed up the test execution process considerably and provide you the results you need more quickly.

5. Test Insights and Analytics: By tracking test results over time, you can learn more about your test runs. Comprehensive analytics, including execution times, pass/fail statuses, and more, are available in the Cypress Dashboard. You can use this data to find trends and patterns in your test results.

6. Test Automation and Scheduling: You can automate the testing process by scheduling tests to run at predetermined times or intervals. Running regression tests and maintaining ongoing test coverage can both benefit from this.

These are some of the functions and attributes the Cypress dashboard offers.

Here's an example of what the cypress dashboard looks like

How to Debug Your Code With Cypress

It's simple to debug your tests with Cypress. To halt test execution and view an application's status in the browser, add cy.debug(). Let's look at an example of how to use cy.debug in Cypress:

describe('Cypress Debug Example', () => {
  it('performs a login and checks the user profile', () => {
    cy.visit('/login');
    cy.get('#username-input').type('testuser');

    // Pause the test execution to inspect the application state
    cy.pause();

    cy.get('#password-input').type('password123');
    cy.get('#login-button').click();

    // Continue with assertions
    cy.url().should('include', '/profile');
    cy.get('.user-info').should('contain', 'Welcome, testuser');
  });
});

In this example, the test will halt when it reaches the cy.pause() command, and the Cypress Test Runner will prompt you to view the DevTools console.

After that, you may interact with the program, examine the DOM, verify variables, and execute more Cypress commands interactively for debugging reasons via the console.

Time Travel Debugging

Cypress also has a powerful capability known as "Time Travel Debugging." This feature allows you to see the current state of your application at any time while it is being tested. This greatly simplifies the diagnosis process.

Using this feature, you can record and replay test execution stages to troubleshoot your Cypress tests. By giving you a visual depiction of the test execution process step-by-step, it assists you in identifying and resolving problems with your tests and application. Using Test Driven Development (TTD), you may see your application's state interactively at various stages of the test run.

This is how Cypress's Time Travel Debugging works:

You must first record a test run in order to use TTD. You may do this using the Cypress Dashboard service or by running your tests locally with the --record parameter.

This is an example of how to conduct testing while recording:

npx cypress run --record

The test is launched in the Electron browser by default.

Then you can use the Cypress Dashboard web interface once the test run is finished and the results are submitted to the Cypress Dashboard.

Next, find the test run you wish to debug in the Cypress Dashboard and click on it. This will direct you to the page with test specifics.

You may see a timeline representation of your application's state at each Cypress command execution on the test details page.

To see the state of the application at a certain moment in time, click on a snapshot on the timeline. You may browse console logs, work with the DOM, and even run Cypress commands in the context of that snapshot.

To examine the condition of the application at various test phases, you can either go to particular points in the timeline or walk through each command in the timeline one at a time.

This is an illustration of how to apply TTD in a Cypress test:

describe('Time Travel Debugging Example', () => {
  it('performs a login and checks the user profile', () => {
    cy.visit('/login');
    cy.get(‘[username-input”]’).type('testuser');
    cy.get(‘[password-input”]’).type('password');
    cy.get(‘[login-input”]’).click();

    // At this point, we may want to perform TTD to inspect the application's state
    // and ensure the user is returned to the profile page.

    cy.url().should('include', '/profile');
    cy.get('.user-info').should('contain', 'Welcome, testuser');
  });
});

In this scenario, you may run into a problem that you wish to diagnose after entering your login information and hitting the login button.

To make sure the user is on the profile page and that their name is shown appropriately, you may use TTD to halt at the cy.url() assertion and verify the status of the application.

With TTD, you can examine the status of the application at different times to interactively debug and troubleshoot your Cypress tests, which will help you find and address problems more efficiently.

Continuous Integration

Cypress tests can be integrated into your continuous integration (CI) workflow using solutions like as Travis CI, GitHub actions, Jenkins, or CircleCI. Let's talk about continuous integration using Github actions:

Cypress Continuous Integration Using GitHub Actions

You can automate the testing of your web apps with Cypress as part of your development workflow with Continuous Integration (CI) using GitHub Actions.

You can create workflows using GitHub Actions that automate a variety of processes, such as Cypress test runs, whenever changes are made to your source. Let's walk through how to get it set up.

Prerequisites:

1. GitHub Repository: The Cypress tests and application code should be located in a GitHub repository.

2. Cypress Setup: Verify that Cypress is installed in your project and that your Cypress test configurations are accurate. Cypress Open may be used to set up and execute your tests locally.

Step 1: Establish a Workflow Configuration:

Create a file in your GitHub repository called workflow configuration. The CI workflow is defined in this file, which is usually named .github/workflows/cypress.yml. It also provides instructions on when and how to execute Cypress tests.

This is an example of a simple configuration file:

name: Cypress Tests

on:
  push:
    branches:
      - main # Align to your repository's default branch

jobs:
  cypress:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v2

    - name: Install dependencies
      run: npm install

    - name: Run Cypress tests
      run: npm run cypress:run

Every time a push is made to the main branch, a process to execute Cypress tests is established by this configuration file. This may be altered to meet your needs.

Step 2: Describe the npm Scripts:

Define npm scripts in your package.json file so that Cypress tests may be executed. For instance:

"scripts": {
  "cypress:run": "cypress run",
  "cypress:open": "cypress open"
}

While cypress:open is used to run tests interactively during development, cypress:run is used to execute Cypress tests in headless mode.

Step 3: Add to GitHub Repo

Add your application code and the GitHub Actions configuration file (cypress.yml) to your GitHub repository. Send the modifications to GitHub.

Step 4: Switch on GitHub Actions:

If you haven't already, go to your GitHub repository, choose the "Actions" tab, and activate GitHub Actions for your repository.

Step 5: Manage Your Process:

GitHub Actions will launch the specified process, which includes your Cypress tests, automatically each time you post changes to your repository.

Step 6: Examine the Results:

The "Actions" page on your GitHub repository is where you can view the status and outcomes of your Cypress tests. Test results and logs will be displayed to you, allowing you to diagnose and look for any issues.

Headless Option

Cypress offers a headless option for continuous integration (CI) testing. Without the Cypress test runner, you can still run Cypress headlessly. By using the command npx cypress run this simply means there is no graphical user interface involved so the test runner isn't being deployed, here we mostly write scripts to use Cypress.

Web browsers that function without a graphical user interface, or headless browsers, are appropriate for automated testing in server contexts such as continuous integration/continuous delivery systems.

When you run Cypress tests in headless mode, no visible browser window is displayed during test execution. This is very helpful when testing without a user interface in an automated continuous integration and delivery pipeline.

To use this put the command npx cypress run in your terminal.

1. Cypress will launch your test in an electron browser when you execute this command.

2. Upon completion, Cypress will automatically provide a video clip of the test run and screenshots in the event that a test fails, facilitating easy debugging.

3. The user can adjust screenshots and videos. The folder labeled "screenshots and videos" contains both screenshots and videos cypress produces.

Another way of running tests is through adding scripts to your package.json file. You can add scripts which consist of the key and value pair that work as objects. with key being the name of the command and value being the command that's being run.

Let's run a test headlessly using a key value pair:

"script": {
    "cypress:run": "cypress run",
    "cypress:open": "cypress open"
},

In this code "cypress:run" and "cypress:open" are the keys and "cypress open" and "cypress open" are the commands. To run the command we need the npm run followed by the key used example: npm cypress:run.

Hopefully you now understand how to run tests headlessly in Cypress.

Conclusion

Cypress is an effective tool you can use to test your web apps. You can use it to write end-to-end tests that make sure your React projects function as intended and identify any issues.

Cypress is a useful addition to your testing toolkit because of its interactive Test Runner, easy-to-use API, and robust debugging features. Now go ahead and use Cypress to test your React applications to make sure the user experience is seamless and error-free.

If you want to learn about Cypress in more detail, you can check out their documentation at cypress.io. Cheers to your testing!

In this tutorial, I am going to show you some methods to simulate user interactions with different interactive elements.

Before getting started, make sure you know how to structure a test in React and how to write simple tests. You can brush up on that knowledge by reading my previous post about how to write unit tests in React.

I also have a challenge for you, so make sure you read the complete article.

How to Set Up the Project

Create your React app using the command create-react-app testing-user-interactions. Next, install the library user-event.

npm i user-event

I am assuming you know how to use this library. If not, I explained how it works in my previous tutorial.

Basically, this library has a few functions you can use to simulate a user. We’ll go through all these functions with examples.

Before writing your tests, you'll need to make the following imports at the start of your test suite (the file in which you write all your tests).

import {render, screen} from '@testing-library/react'
import userEvent from '@testing-library/user-event'

You'll also need to import the component you’ll be testing.

For every test, we'll be simulating user interactions and test how our component behaves.

How to Test select Element Interactions

In your application, you may have select elements that allow a user to select something from a list of options. Let’s take an example:

<select id='selectCity'>    
    <option> Mumbai</option>    
    <option> Bangalore</option>    
    <option> Chennai</option>
</select>

Normally, when a user sees a select element, they would click it and select an option they want. We are going to simulate the exact same behaviour and test if our component reacts accordingly. For instance, if a user selects Chennai, the value shown must be the same.

First, let’s see how to query the element. Let's use getByRole to query the element. You can use the role combobox for the above element.

screen.getByRole('combobox')

What is a role? How do you know which role to use for an element? In React, for testing purposes, there's an attribute called role defined for every HTML element. This makes it easier for you to query an element while testing.

Now, in order to know the value of the role attribute for each element, you can refer to the list of roles here.

A Snapshot of the List of Roles

If you want to query an anchor element containing an href attribute, you can use the role link. Same is true for other elements in the list.

However, do not define an explicit role attribute for your HTML element, as it's already defined for you.

Let's see another method of querying the element. If there are is more than one select element, you can use getByLabelText(). For this, add a label, <label htmlFor=”selectCity”>Select City</label> for the element and do screen.getByLabelText(/select city/i). You’ll get the element.

Now, use the [selectOptions()](https://testing-library.com/docs/ecosystem-user-event/#selectoptionselement-values-options) method of the library to simulate a user selecting an option. It takes three arguments: the element, the value, and any options.

userEvent.selectOptions(    
    screen.getByRole('combobox'),    
    'Chennai'
)

Then, add an assertion at the end of your test to check if the element has the same value that the user selected.

expect(screen.getByRole('combobox')).toHaveValue('Chennai')

The user selected Chennai and our select element has the same value, so the test passes.

How to Test State Change

Now, I’ll make a state update on changing the select element’s value. First, I’ll have a state called cityName to display the option selected.

const [cityName, setCityName] = useState('Mumbai');

<h2> City Name:  {cityName} </h2>

Then, I’ll add an onChange attribute with a method that updates the state.

onChange={(e) => { setCityName(e.target.value)}}

To write the test for this, follow the same process:

test("City Name header rendering", () => {    
    render(<SelectElement/>)    
    userEvent.selectOptions(        
        screen.getByRole('combobox'),        
        'Bangalore'    )    
    expect(screen.getByRole('heading',               
                     { name: /bangalore/i})).toBeInTheDocument();
})

To query the h2 element, use [getByRole()](https://testing-library.com/docs/queries/byrole/) with the name option. The name option adds another filter to the query – in case there are multiple h2 elements.

How to Test a Hover Event

Your application may have functionalities where a user can hover over an element to see some information as a tooltip. So, to learn how to test this, let’s implement a basic tooltip functionality.

<div className="tooltip">    
    <p> Hover over me</p>    
    <span className="tooltiptext">Tooltip text</span>
</div>

There’s also some CSS styles for the classes defined above, but I won’t show them here. Basically, when a user hovers over the text Hover over me, the tooltip text is visible.

To simulate a user hovering over the text element, use the hover() method. Add an assertion at the end to test if the tooltip is visible.

test("Tooltip visible", () => {    
    render(<HoverEvent/>)
    userEvent.hover(screen.getByText(/hover over me/i))          
    expect(screen.getByText(/tooltip text/i)).toBeInTheDocument()
})

The test will pass if the tooltip is visible on hover. If it doesn't, you'll need to refactor your code.

How to Test a File Upload Event

Another important event is a user uploading files to your website. This is a pretty common functionality in any UI application.

Let’s have an input of type file.

<label htmlFor="singleFile">Upload File</label>
<input type='file' id='singleFile' />

In the test, you’ll need to simulate the user uploading a file. Use the upload() method to do that:

test("Test single file upload", () => {    
    render(<FileUpload/>)    
    const file = new File(['hello'], 'hello.png', {type: 'image/png'})   
    userEvent.upload(screen.getByLabelText(/upload file/i), file)  
    expect(screen.getByLabelText(/upload file/i).files[0]).toEqual(file)
})

First, create a mock file and upload the same. Then, make an assertion that the file present in the input matches our uploaded file. It does, so the test passes.

Now, let’s take an example of multiple file uploads.

<label htmlFor='multipleFiles'>Upload multiple files</label>
<input type='file' id='multipleFiles' multiple />

In our test, upload multiple files.

test("Test multiple file uploads", () => {    
    render(<FileUpload/>)    
    const files = [        
           new File(['file1'], 'file1.png', {type: 'image/png'}),       
           new File(['file2'], 'file2.png', {type: 'application/pdf'})   
    ]    
    userEvent.upload(screen.getByLabelText(/upload multiple files/i), 
                        files)    
    const fileInput = screen.getByLabelText(/upload multiple files/i)  

    expect(fileInput.files.length).toBe(2);   
    expect(fileInput.files[0]).toEqual(files[0]);   
    expect(fileInput.files[1]).toEqual(files[1]);
})

In the assertions, check the number of files and whether both the files match the uploaded ones. Thus, all our tests pass with 100% coverage.

Now, time for the challenge. Go through my posts on Muliple File Uploads and Different Examples of the useState Hook and write unit tests for all the functionalities. Upload your code on GitHub and share it with me.

How to Test Form Behaviour

An important feature of most web apps is form handling. When writing tests for a React app, it’s highly likely you will need to test your component’s behaviour with forms.

In our example, let’s have a form with two input fields, one select field and a submit button.

First, let’s define the states for the respective fields and an additional state for indicating whether the form is submitted.

const [name, setName] = useState('')
const [password, setPassword] = useState('')
const [country, setCountry] = useState('')
const [formSubmitted, setFormSubmitted] = useState(false)

<form>    
    <input onChange={(e) => {setName(e.target.value)}}             
           placeholder='Enter name'/>    
    <input onChange={(e) => {setPassword(e.target.value)}}                          placeholder='Enter password'/>    
    <label htmlFor='selectCountry'> Select Country </label>    
    <select id='selectCountry'             
            onChange={(e) => {setCountry(e.target.value)}}>       
        <option>India</option>        <option>Japan</option>       
        <option>China</option>        <option>USA</option>       
        <option>England</option>    
    </select>    
    <button onClick={handleSubmit}> Submit </button>
</form>

In the onChange attribute, make the state updates. In the end, display a text on successful submission of the form.

{formSubmitted && 'Form Submitted'}

Also, let’s add some validation in the handleSubmit method. When a validation fails, a list of errors will be shown. Create another state to hold errors.

const [errors, setErrors] = useState([])

Now, implement the handleSubmit method.

const handleSubmit = (event) => {    
    event.preventDefault();    
    const errorList=[];    
    if(name == '' || password == '' || country == '') {        
        errorList.push('Please fill all the details')    
    }        
    const regex = /^[a-z]*$/i;    
    if(!name.match(regex)) {        
        errorList.push('Please enter a valid name')    
    }    
    if(errorList.length != 0) {        
        setErrors(errorList);        
        setFormSubmitted(false);        
        return;      
    }     
    setErrors([]);    
    setFormSubmitted(true)
}

First, we’ll do event.preventDefault() which cancels out any events that prevent submitting the form. Then, we’ll add few error checks and add any errors to the state errors. In the end, if there are no errors then set errors to an empty array and set formSubmitted as true.

We’ll also display the list of all the errors which will be visible on clicking the submit button.

errors.map(err => (    
    <p>        
        {err}   
    </p>
))

Let’s write the tests for this example. There are a few different things you should test. First, write a test simulating a user interacting with the form and submitting it with correct values.

test("Form getting submitted with correct input values", () => {
    render(<FormBehaviour/>)

    userEvent.type(screen.getByPlaceholderText(/enter name/i), 'kunal')
    userEvent.type(screen.getByPlaceholderText(/enter password/i),                                   'pass')
    userEvent.selectOptions(screen.getByLabelText(/select country/i),                               'India')
    userEvent.click(screen.getByText(/submit/i))

    expect(screen.getByText(/form submitted/i)).toBeInTheDocument();
})

Use the type and click methods of the userEvent library to simulate the respective actions. In the end, add an assertion to see if the text matching the regex /form submitted/i is visible in the document since our logic displays the text as soon as the form is submitted.

Test Coverage for the Form Behaviour Component

Our test passes, but there are still some uncovered lines. We have tested regular form behaviour. However, we also have error handling mechanisms in place. So, you also need to write tests that cover different error scenarios.

First, simulate a user clicking the submit button without entering anything.

test("Empty fields error shown", () => {    
    render(<FormBehaviour/>)    
    userEvent.click(screen.getByText(/submit/i))    
    expect(screen.getByText(/please fill all the details/i))                         .toBeInTheDocument()
})

On clicking the submit button, an error message should be visible prompting the user to fill all the details. Make an assertion for the same.

Not all lines are covered yet. We still have one more error scenario. What if the user enters an invalid name?

test("Invalid name error shown", () => {    
    render(<FormBehaviour/>)    
    userEvent.type(screen.getByPlaceholderText(/enter name/i), '@###')       userEvent.type(screen.getByPlaceholderText(/enter password/i),                                    'pass')    
    userEvent.selectOptions(screen.getByLabelText(/select country/i),                                  'India')    
    userEvent.click(screen.getByText(/submit/i))    
    expect(screen.getByText(/Please enter a valid name/i))
          .toBeInTheDocument();
})

This test is similar to the first test, except here, the user enters an invalid name. So, assert whether the corresponding error is shown.

100% Coverage for the Form Behaviour Component

Now, all our tests pass with 100% coverage.

You can find the complete code on GitHub.

Conclusion

Testing user interactions helps you make sure that users can interact efficiently and effectively with your application. You need to cover all possible scenarios a user may encounter. In this article, I have shown you how you can simulate user interactions with different elements.

I have also demonstrated testing form behaviour covering many scenarios. I hope you now know how to use the userEvent library to test your component’s behaviour with user events.

If you are unable to understand the content or find the explanation unsatisfactory, get in touch with me and let me know. New ideas are always appreciated! Feel free to connect with me on Twitter. Till then, Goodbye!

Now, you can’t do that just by going through the website yourself – you need to check if each unit of your code is working as you want. For that, you need to write unit tests. And they can be quite a hassle when you actually get down to writing them.

So, in this post, I am going to show you how you can get started with writing unit tests in React. I will explain the process through several examples to help you understand better.

For this post, I assume you have a basic knowledge of React. If not, then head over to the docs to get started.

How Is a Test Structured?

Testing involves checking if your code is functioning as it's supposed to by comparing the expected output with the actual output.

What to Test

In general, your tests should cover the following aspects of your code:

1. If a component renders with or without props

2. How a component renders with state changes

3. How a component reacts to user interactions

What Not to Test

Testing most of your code is important, but here are some things you do not need to test:

1. Actual Implementation: You do not need to test the actual implementation of a functionality. Just test if the component is behaving correctly.
   Let’s say you want to sort an array on the click of a button. There’s no need to test the actual sorting logic. You only test if the function was called and if the state changes are rendering correctly.

2. Third Party libraries: If you are using any third party libraries like Material UI, no need to test those – they should already be tried and tested.

This might seem a little complicated at the moment, but you should understand better through examples.

Any test in React, no matter how complicated, follows this structure:

1. Render the component

2. Get an element from the component and simulate any user interactions

3. Write an assertion.

How to Set Up Our Project

First, create the app with create-react-app. We’ll be using Jest and React Testing Library. Both of them come pre - installed with create-react-app. You can see what it looks like here:

"dependencies": {
    "@testing-library/jest-dom": "^5.16.5",
    "@testing-library/react": "^13.4.0",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-scripts": "5.0.1",
    "web-vitals": "^2.1.4"
},

As you can see, Jest is not visible here. But, if you go to the node_modules folder, you will see Jest there. So, no need to install it separately.

Also, make sure you have the following lines in your setupTests.js file:

import '@testing-library/jest-dom';
import '@testing-library/jest-dom/extend-expect';

Also, in your package.json file, modify your scripts like this:

"scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test --watchAll --coverage",
    "eject": "react-scripts eject"
},

This will run your tests in watch mode and also show you the coverage (that is, the portion of the code covered by your tests). You can also define the coverage threshold by adding another property jest. For now, keep the threshold at 80%.

"jest": {
    "coverageThreshold": {
        "global": {
            "lines": 80
        }
    }
}

That’s it, now you are ready to start testing. Let’s first start with a basic test.

How to Write Your First Test for a React App

Let’s write a test for the following component:

const FirstTest = () => {
  return (
    <div>
        <h2> First test </h2>
    </div>
  )
}

Here, we just need to test if the h2 element renders. Now, where should we write the tests? We can write them inside a __tests__ folder anywhere in the src folder. The test file just needs to have a .test.js/jsx extension and the test runner will pick it up.

This is how the test looks in our FirstTest.test.jsx file:

import { render, screen } from '@testing-library/react'
import FirstTest from '../components/FirstTest';

test("Example 1 renders successfully", () => {
    render(<FirstTest/>);

    const element = screen.getByText(/first test/i);

    expect(element).toBeInTheDocument();
})

First, import the required methods. The test() method contains an individual test. It takes the name of the test and a callback function as the two arguments.

Now, following the structure mentioned above, render the component you're testing using the render method. Then, use the screen object to make a query for an element. In this case, it’s the h2 element. Our query gets an element containing text that matches the regex /first test/i (i means ignore case).

Lastly, make the assertion using the expect method. We expect the element to be in the document and it is, so the test passes.

There are lots of other assertions you can make in your tests. You can read more about them here. Also, you can find a list of ways to query an element here. We’ll use some of them in our further examples.

How to Test With Mock Data in React

Here, we have a component with a prop data that displays a list of items. Let’s assume this data comes from the backend and your component is displaying this data.

import React from 'react'

const TestWithMockData = ({data}) => {
  return (
    <div>
        <ul>
            {data.map(item => (
                <li key={item.id}>
                    {item.id}
                    {item.first_name},
                    {item.last_name},
                    {item.email}

                </li>
            ))}
        </ul>
    </div>
  )
}

export default TestWithMockData

While writing tests for a component with props, you need to pass some mock data while rendering this component that pertains to your functionality. Here, one object in our data contains four fields, so we pass some mock data here.

const mockData = [
    {
        "id": 1,
        "first_name": "Fletcher",
        "last_name": "McVanamy",
        "email": "mmcvanamy0@e-recht24.de",
        "age": 30
      }, {
        "id": 2,
        "first_name": "Clarice",
        "last_name": "Harrild",
        "email": "charrild1@dion.ne.jp",
        "age": 35
      }, 
]

Note that the actual data could contain thousands of records, but the mock data only needs to be relevant to what you want to test. Now, let’s write the test and check if the list renders.

test("List renders successfully", () => {
    render(<TestWithMockData data={mockData} />)
    expect(screen.getByText(/fletcher/i)).toBeInTheDocument();
})

How to Test With Mock Data Covering All Branches in React

Let’s introduce some branching in the above component. We’ll have another prop, displayUnorderedList, that determines whether to display an ordered or unordered list. We’ll also render Senior for age > 50 and Not Senior otherwise.

import React from 'react'

const TestWithMockData = ({data, displayUnorderedList, handleClick}) => {
  return (
    <div>
        {displayUnorderedList ? 
            <ul>
                {data.map(item => (
                    <li key={item.id}>
                        {item.id}
                        {item.first_name},
                        {item.last_name},
                        <a onClick={() => {
                            console.log("email link clicked")
                            handleClick()
                        }}>{item.email}</a>

                        {item.age > 50 ? 'Senior' : 'Not senior'}

                    </li>
                ))}
            </ul>
        : 
            <ol>
                {data.map(item => (
                    <li key={item.id}>
                        Last name: {item.last_name}
                    </li>
                ))}
            </ol>
        }
    </div>
  )
}

export default TestWithMockData

Here's the coverage report so far:

Failed Test: Unable to find the element

You can see that Line 9 (that is, the unordered list part) is not covered by our existing tests. And our previous test has also failed since it is unable to find fletcher in the component.

Why is this so? In our previous test, we have not passed the displayUnorderedList prop to the component so it is null. So, the component renders the ordered list and it does not contain the first_name.

So, let’s write another test covering the ordered list part.

test("Ordered list renders", () => {
    render(<TestWithMockData data={mockData} displayUnorderedList={false} />)

    expect(screen.getByText(/McVanamy/i)).toBeInTheDocument()
})

Here, pass the prop’s value as false to render the ordered list. Also, add the displayUnorderedList prop to the failing test and pass the value true.

Now, all our tests pass with 100% coverage.

Uncovered Branch

One line is still not covered by the test cases, which is the branching logic for age. So, add another record to the mock data that has an age greater than 50.

{
    "id": 3,
    "first_name": "Amby",
    "last_name": "Emmer",
    "email": "aemmer2@buzzfeed.com",
    "age": 67
}

Now, all our tests should pass with 100% coverage.

How to Test User Interactions in React

The most important part of testing any UI app is testing its behaviour with various user interactions. Almost every functionality in a UI app involves user interactions.

You can use the user-event library to simulate user interactions. It has methods for simulating various user events such as click, type, hover, and so on.

First, we need to install the library:

npm install --save-dev @testing-library/user-event

We can use this library to simulate user events. In our examples, we’ll interact with different elements, mostly input and button elements.

How to Test a Function Call on Click of an Element

In our above component, we need toTestWithMockData, make the email field clickable and call a function handleClick on it. This will be passed as props to the component. There, replace {item.email} with:

<a onClick={() => {
    console.log("email link clicked")
    handleClick()
}}>{item.email}</a>

Now, our test coverage takes a hit. To cover this scenario, write the following test:

test("Email link click handler called", async () => {
    const mockHandleClick = jest.fn();
    render(<TestWithMockData 
                data={mockData} 
                displayUnorderedList={true}
                handleClick = {mockHandleClick}
          />)
    await userEvent.click(screen.getByText(/mmcvanamy0@e-recht24.de/i));
    expect(mockHandleClick).toHaveBeenCalled();
})

First, create the mock of the handleClick function using jest.fn(). We do not need the actual implementation of the method, as we only want to test the behaviour of the component. So, we create an empty mock and pass the same as props. Read more about mocking functions here.

Now, query the <a> element by text (any email from the mock data). Use the click() method to simulate a click event. Use await as simulating a user event is an asynchronous operation.

Write an assertion in the end to check if the method was called. The method was called, so our test passes with 100% coverage.

How to Query Input Fields and Buttons

So far, we have used only one method of querying elements – getByText(). Now, let’s see how you can query input fields and buttons.

<input placeholder='Enter name'/>
<button> Submit </button>

To query these elements, you can do the following:

const inputElement = screen.getByRole('textbox')

getByRole finds an element by the given role. In this case, the textbox role means the input element.

How is role determined? Each element has a defined role, so you do not need to specify an explicit role attribute. You can see a list of roles for different elements here. Whatever element you want, just do getByRole and refer to the list.

For button, the default role is ‘button’ as you can see here:

const button = screen.getByRole('button')

What if we add another input element, <input placeholder=’Enter description’/>? The test runner will now throw an error saying that there are two elements with the same role. What should we do in such a scenario? Use another query getByPlaceholderText().

const nameInput = screen.getByPlaceholderText(/enter name/i);
const descrInput = screen.getByPlaceholderText(/enter description/i);

You can also use getByLabelText() if your input has a label.

<label htmlFor='password'> Enter password</label>
<input type='password' id='password'/>

const passwordInput = screen.getByLabelText(/enter password/i);

For querying buttons, we can do screen.getByRole('button').

<button> Submit </button>
<button> Apply</button>

Since we have two buttons here, just doing getByRole will throw an error. So, we use a name option.

const submitButton = screen.getByRole('button', { name: /submit/i });
const applyButton = screen.getByRole('button', { name: /apply/i });

The name option can contain the label of a form element, text of a button, or the value of the aria-label attribute of any element. We can also do getByText() for a button.

const submitButton = screen.getByText(/submit/i);

How to Test State Updates in React

We have seen how to query form elements like input and button. Now, let’s simulate some user interactions and test state updates. What do I mean by testing state updates?

State updates cause a component to re-render. So, when your functionality performs a state update, you should test how the component is behaving due to the state change.

First, let’s take a simple example where we set the state as soon as the component is loaded – that is, in the useEffect block.

const TestingStateChange = () => {
    const [loaded, setLoaded] = useState(false)
    useEffect(() => {
        setLoaded(true)
    }, [])
  return (
    <div>
        {loaded && <h3> Page Loaded </h3>}
    </div>
  )
}

Here, your entire component, starting from the useEffect statement to the end, is uncovered. When you write the following test, not only does the HTML part get covered but also the useEffect part where you set the state.

test("Testing page load", () => {
    render(<TestingStateChange/>)
    expect(screen.getByText(/page loaded/i)).toBeInTheDocument();
})

This is how you test state updates. You test whether the component behaves as you expect with a state change.

How to Test a State Change on Click of a Button

Let’s have a button and a text that toggles on the click of the button.

const [toggleTextVisible, setToggleTextVisible] = useState(false)`

{toggleTextVisible && <p> Text visible </p> }

<button onClick={() => { setToggleTextVisible(!toggleTextVisible) }}> 
    Toggle text 
</button>

We’ll write the test for this:

test("Toggle text visible", async () => {
    render(<TestingStateChange/>);
    await userEvent.click(screen.getByText(/toggle text/i));
    expect(screen.getByText(/text visible/i)).toBeInTheDocument();
})

With userEvent, we have simulated the click of a button and asserted if the text is visible.

Now, let’s test another scenario where we check if the above button is disabled on the click of another button.

const [btnDisabled, setBtnDisabled] = useState(false);

Add a disabled={btnDisabled} attribute to the above button and create another button that controls its value.

<button onClick={() => { setBtnDisabled(!btnDisabled) }}> 
      Toggle button disabled 
</button>

Use the toBeDisabled() method to test if the button is disabled.

test("Button disabled on click", async () => {
    render(<TestingStateChange/>)
    await userEvent.click(screen.getByText(/toggle button disabled/i));
    expect(screen.getByText(/toggle text/i)).toBeDisabled();
})

How to Test if an Element Was Added to a List

Let’s perform another assertion. Here, we’ll check if an element was added to a list. We’ll have some data and create a state with the data as the initial value. We'll display it while also having a button to add an element to it.

const [elements, setElements] = useState(data);

<h3> List </h3>
{elements.map(item => (
    <div key= {item.id}>
       {item.id}: { item.name }
    </div>
))}
<button onClick={() => {
    setElements([...elements, {
        id: 4,
        name: 'abhinav'
    }])
}}> Add to list </button>

Add an attribute data-testid='record' to each record to help query the element. Let’s write the test for this:

test("Element added to the list", async () => {
    render(<TestingStateChange/>)
    expect(screen.getAllByTestId('record').length).toBe(3);

    await userEvent.click(screen.getByText(/add to list/i));
    expect(screen.getAllByTestId('record').length).toBe(4);
})

To get multiple elements with the same query, use the getAllBy… method. In this case, there are multiple elements with the test-id record, so using the getAllByTestId() method gets a list of all such elements.

Here, we have checked the length of the list. We can add another assertion to check if the new element is visible.

expect(screen.getByText(/abhinav/i)).toBeInTheDocument();

How to Test if an Element Was Removed From a List

Now, let’s add another button that removes an element from the list.

<button onClick={() => {
    setElements(elements.filter(item => item.id != 2))
}}> Remove from list </button>

The test for this:

test("Element removed from list", async () => {
    render(<TestingStateChange/>)
    await userEvent.click(screen.getByText(/remove from list/i));
    expect(screen.getAllByTestId('record').length).toBe(2);
})

Here, we test whether the length of the list was reduced. All our tests pass with 100% coverage.

Coverage Report for Testing State Change: Tests Passed

How to Test API Calls in React

API calls are an important part of any UI application. Let’s understand how to test an API call.

In this example, we have a file data.json that has some data. We’ll make an API call to fetch this data.

For demonstration, we’ll save this file on the local server only (inside the public folder), but the procedure remains the same when fetching a file from a remote server.

We’ll use fetch to make the API call.

fetch("http://localhost:3000/data.json")
  .then(res => {
      return res.json();
  })
  .then(data => {
      // Store the data in state
  })

Now, let’s have a state to store the data and render it.

const [data, setData] = useState([])

<div>
    {data.map(item => (
        <div>
            {item.name}
        </div>
    ))}
</div>

How do we write the test for this? One way is to mock the fetch method and provide our own mock implementation. But what if there are multiple Fetch API calls in the component? You cannot have the same mock implementation for everyone.

So, we put the fetch call in a separate function and have a mock implementation of that function. We put our API call in a FetchData method that returns the promise returned by fetch.

We put the method in a separate Services.js file and export it from there.

export const FetchData = () => {
    return fetch("http://localhost:3000/data.json")
        .then(res => {
            return res.json();
        }) 
}

Now, when we call this method inside our component, it looks like this:

const [data, setData] = useState([])

useEffect(() => {
    FetchData().then(data => {
        setData(data);
    })
})

return (
  <div>
      {data.map(item => (
          <div>
              {item.name}
          </div>
      ))}
  </div>
)

Now, let’s write the test for this. The main thing is to mock the FetchData method. First, import all methods from Services.js as a module.

import * as services from '../utils/Services'

Let’s mock our function using jest.spyOn(). The method takes two arguments, the object and the method name as a string.

const mockFetchData = jest.spyOn(services, 'FetchData')
        .mockImplementation(async () => {
            return [{
                name: 'kunal'
            }];
        })

spyOn creates an empty mock of the method. Now, we provide our mock implementation where we return some mock data. This will be called when our component renders.

render(<TestingAPICalls/>)
expect(mockFetchData).toHaveBeenCalled();

We use toHaveBeenCalled() to test if the method was called. It was – so the test passes. Now, to test the behaviour of our component, let’s test if the name was rendered.

expect(screen.getByText(/kunal/i)).toBeInTheDocument();

In this case, the test fails as it is unable to find the element. Why is this so?

Failed Test: Unable to find the element

Since API calls and state updates are asynchronous, the text hasn’t been rendered yet. To test asynchronous behaviour, wrap the query in a waitFor() block.

await waitFor(() => {
    expect(screen.getByText(/kunal/i)).toBeInTheDocument();
})

waitFor does exactly what the name suggests. It waits for the asynchronous behaviour to finish before making the query. Now, our test passes with 100% coverage.

100% Coverage for Testing API Calls

You can find the complete code on GitHub.

Conclusion

Phew! That was a long one. But it needed to be. I have explained, through basic examples, how you can start writing tests in React.

This article started from a basic test with a simple assertion and explained how to use mock data to serve the purpose of your test.

I also explained how to simulate user interactions, how to test a component’s behaviour on state updates, and lastly how to test API calls. Hopefully, all these examples help you write tests for your upcoming project.

If you are unable to understand the content or find the explanation unsatisfactory, please let me know. New ideas are always appreciated! Feel free to DM me on LinkedIn if you want to discuss anything. Until then, Goodbye!

But many users are often unaware of the fact that they are putting their trust in an API and not a person.

This is why it's important to leverage API testing techniques to ensure that every aspect of your website or application works as expected.

Why API Testing is Important

My first interaction with APIs was while working on a personal project, a VirtualAssistant. It required me to fetch real data from third party apps.

As a beginner, the knowledge I acquired was mainly from tutorials I had watched which basically covered how to GET, POST and DELETE data. I didn't know that there were rules that I should follow while working with APIs.

But as time progressed I came to understand the importance of APIs and the major role they play in the daily applications of a developer's job.

Mastering the rules is an essential step in learning about APIs. Most applications today are made up of pieces of software, each of which needs to be tested in its own way.

There are always new and exciting ways to test an application – and we're not just talking about checking for bugs here. It's more about the app functionality.

As technology continues to advance, the trends related to how to test these applications will also change.

In this guide, we will be discussing the trends that apply to the testing of APIs, how they differ from other types of testing, tools we can leverage when it comes to testing APIs, and how you can stay on top of your game when it comes to API testing.

First, let's get an understanding of what we mean by APIs.

What is an API?

APIs are designed for developers to use. They are, in essence, a coding tool that allows your application to communicate with other applications. APIs allow you to integrate third-party applications into your work or use your own data and processes in the cloud.

The functionality of any application is defined by the connection it has with the outside world. For this reason, connecting your application with APIs becomes very helpful.

In today's development process, APIs have become an essential part in web and mobile applications. They enable communication of different components of these systems and help you access data and services.

A very important thing to keep in mind is that when dealing with APIs, you need to be sure everything works properly and that you get the correct feedback before integrating them into your applications. That's why testing them is essential.

For this reason, a vital part of testing any type of software requires evaluating whether or not there's a way the app can be broken. In order to ensure a certain level of continuity when using an API in multiple ways simultaneously, you need to conduct API testing.

Now that you have an understanding of what an API and why testing it is important, let's clarify what exactly might we mean by this – is there more to the term testing?

What is API Testing?

API Testing is a process that checks that an API adheres to the given requirements. Achieving this goal is not an easy task and there are many ways in which a tester can go about doing it effectively.

You can conduct API testing manually or automatically, and it is often considered a part of integration testing.

• Manual API testing involves directly interacting with the API using tools like Postman, API Tester or any other available online tools.
• Automated API testing uses specialized software to send requests to the API and then compares the results to the expected behavior.

API testing is important because it helps ensure that the various components of a system are working together as expected. By verifying the functionality of the API, you can be confident that the system as a whole will function as intended.

API Testing Tools

You can use API testing tools to test the functionality of an API. They can help you test the performance of an API, as well as to check for security vulnerabilities.

There are a number of different API testing tools available, each with its own advantages and disadvantages. Some of the most popular API testing tools include SoapUI, Postman, Runscope, and API Tester.

API Testing Principles

Having a set of standard rules is the best way to ensure the quality of your APIs and their implementations. You can apply these rules during testing, coding, and development, as well as in production.

Here are some principles that you should keep in mind:

1. API testing should be a part of your continuous integration and delivery pipeline.
2. API tests should be easy to maintain and write.
3. A well-designed API will make your tests easier to write.
4. You should test at the boundary of your system.
5. Keep your tests small and focused.
6. Make sure your tests are deterministic.
7. Run your tests in parallel for speed.
8. Use the available and freely accessible tools to simplify API testing.

How to Start API Testing

There are many different types of tests you can perform on your APIs:

• Functionality tests focus on ensuring that the API is able to perform its intended functions.
• Performance tests measure the response times of the API and check for bottlenecks.
• Security tests assess the vulnerability of the API to attack vectors such as SQL injection and cross-site scripting (XSS).

To get started with API testing, you will need to have access to an application with an exposed API. You will also need to choose a method for sending requests to the API (manual or automated), and select a tool or framework for writing your tests (if using automated testing).

Once you have these things set up, you can begin writing your test cases and running them against the API.

API Testing Tips

API testing can be a challenge, but regardless of the tools you decide to use here are some tips that can help:

1. Make sure you have a clear understanding of the API before you start testing. Read the documentation and any other available materials. This will help you know what to expect and how the API should work.
2. Use API testing tools: This will give you a better understanding of how the API works and make it easier to find any issues.
3. Test all aspects of the API, including input validation, error handling, and security. These are all important factors in ensuring that your API is working correctly.
4. Keep your tests up-to-date as the API changes. This will help ensure that you catch any new issues that may arise.
5. Use mobile apps for API testing. Things are changing all the time these days, where many people are working remotely, and from their phones too. By using mobile apps you can increase your productivity, become more mobile, and work from anywhere in the world. A good example of such a tool is the API Tester.

By following these tips, you can help make sure that your API testing is effective and efficient.

Conclusion

API testing is a key part of ensuring the quality of your software. By following these highlighted rules you will be able to ensure your tests are functional, secure and have a reliable performance.

This will in turn help you improve the overall quality of your software applications and provide a better user experience.

🔐 Here's What We'll Cover

• You'll see how easy it is to set up your first unit tests in React.
• You'll improve your general React knowledge.
• You'll get the hang of why Test Driven Development (TDD) is helpful for your coding workflow.

📝 Prerequisites

• You should be familiar with the basic React workflow structure (including functional components and hooks).
• You should have a basic knowledge of Redux (I'm using Redux Toolkit for this guide).
• You don't need any prior knowledge about testing.
• I'm using the npm installation approach, not the yarn one.

🎯 The Objective

While learning advanced concepts of React, you'll probably stumble across the topic of testing. Being able to work with automatic tests is also quite handy for any upcoming frontend developers.

However, as I myself was learning React, I had a hard time finding information about how to implement tests for libraries like Redux (even though it's a library I work with all the time).

Beyond that, I found that doing any component testing in React is basically unfeasible when you don't know how to work with the Redux library.

So I took some time to read the Redux documentation and went back and forth with it a bit. Then I decided to write a practical starter guide for unit testing in React, including Redux, to share what I learned.

Since I would like to take a modern approach, I'm also going to use the Redux Toolkit. We'll cover the Redux implementation in this guide.

What we'll cover:

To start off, I will provide some general information about tests before I directly go into creating the first general unit tests.

Next I'll give a quick overview of how to implement some Redux Toolkit logic.

Then we will work on some unit tests within an application which uses Redux Toolkit. For this step we will adjust our previously created tests to the new Redux environment.

This is a step-by-step guide. If you are new to tests, I recommend following this guide in order from top to bottom.

I have also created a public GitHub repository for this guide with some commentary. You can use that if you want to look something up without scrolling through this guide in its full length again.

Table of Contents

1. What Different Kinds of Tests Are There?
2. How to Set Up Your React Testing Environment
3. Check Out Your Created React Application
4. How to Create Your First Unit Test
5. How to Create a Failing Test on Purpose
6. How to Create Some Additional Tests
7. How to Perform Testing with the React Redux Toolkit
8. Outlook for Advanced Testing

📋 What Different Kinds of Tests Are There?

This quick guide won't provide you with detailed theoretical knowledge about all the different kinds of testing out there. At this point, you should just understand that there are generally three kinds of tests:

• Unit Tests
• Integration Tests
• End-To-End Tests

To put it in simple words: You can see these three types of tests as generally increasing in their complexity.

While unit tests cover single functions and components, integration tests typically focus on multiple functions and their connections to each other. End-to-end tests are even more complex and give insights about multiple function and component structures.

There are other test concepts, but these three are the most important ones for web developers, for example.

Again, this is really put in simple words. But in this case it's sufficient to know that unit tests are basically the least complex tests out of these three.

It's also quite easy to work with unit tests as soon as you have a basic understanding of how testing in general works.

I would also like to quickly emphasize that there are mainly two ways of testing your application.

• Manual Testing
• Automatic Testing

Manual Testing is pretty much what you probably already do for all of your application which you create. When manually testing your application, you basically start your React application with npm run start and actually click on buttons to check if the corresponding function works.

Automatic Tests, on the other hand, are pretty much functions you create which automatically check your application to see if the respective steps work that you defined within these tests. This automatic kind of testing is extremely important for larger projects.

With this automatic approach, it's also way easier to scale your tests. In the end, you have a lot of tests which automatically test your whole application in a relatively short amount of time. These test can help you spot any potential errors which might have occurred during development. This would take much more time if you were to go back constantly to manually test your application.

Being able to work with automatic tests is also typically a big plus for your résumé as a frontend developer.

🔧 How to Set Up Your React Testing Environment

In order to get a practical start, we are going to directly dive into our React application.

You will see that the setup of a testing environment is relatively easy in React – or, to be more precise, React does it all for you during the regular install setup.

Therefore, I'm creating a React application with the following line:

npx create-react-app <name of your application>

After this step, we need everything that should be added for using Redux in our React application:

• React Redux: npm install react-redux (provides some mandatory hooks, for example)
• React Redux Toolkit: npm install @reduxjs/toolkit (provides logic for creating a store or slices, for example)

It's worth mentioning that there also is the Redux core (npm install redux). But this is already part of the React Redux Toolkit installation, so we don't have to install it here too.

If you wanted to use React without the React Redux Toolkit, then you would have to separately reach out to the Redux core installation.

You can also create a new React application from scratch with npx create-react-app my-app --template redux which includes the React Redux Toolkit, the React core, React Redux, as well as a template from the React Redux Toolkit.

Choose this approach if you don't have any existing React applications, since it's probably more convenient.

Under the hood, you now have an application that uses the React Testing library combined with Jest (a testing framework). Together, they have pretty much everything that you will need for testing your React application.

You don't have to install anything else for this purpose. These tools come out of the box with a standard React installation.

🔎 Check Out Your Created React Application

As you go into your newly created React application, you will find the folder and file structure you are likely familiar with. Besides others, there is the App.js file, which is created like this:

import logo from './logo.svg';
import './App.css';

function App() {
  return (
    <div className="App">
      <header className="App-header">
        <img src={logo} className="App-logo" alt="logo" />
        <p>
          Edit <code>src/App.js</code> and save to reload.
        </p>
        <a
          className="App-link"
          href="https://reactjs.org"
          target="_blank"
          rel="noopener noreferrer"
        >
          Learn React
        </a>
      </header>
    </div>
  );
}

export default App;

Within the src folder, you also have the file App.test.js. This file is actually a first test that came out of the box with React installation. This file is structured like this:

import { render, screen } from '@testing-library/react';
import App from './App';

test('renders learn react link', () => {
  render(<App />);
  const linkElement = screen.getByText(/learn react/i);
  expect(linkElement).toBeInTheDocument();
});

Even without fully understanding what render or screen is, for example, we can see that something is going on with our App component in there. In fact, this is a unit test that is focusing on a specific part of the App component.

While this first template for a test is a handy representation of what a test looks like, I would like to create a test file from scratch.

Generally speaking, tests are separated into different test suites. These test suites are typically a group of tests that focus on the same component, for example. Tests within the same test suite basically have the same superordinate topic.

To check this, try to enter npm run test in your terminal when you are within your React application.

It could say something like "There are no new tests or changes since the last commit" – in this case, just enter a in the terminal to run all tests regardless.

In the end, you should be able to see this within the terminal:

Result of npm run test

At the top, you can see that the App.test.js file passed. Basically, all tests within this file were successful.

Below that, you can see renders learn react link: This is the description for this particular test, which we can define individually. We will get back to this later.

Further down, we can finally see the test suites and tests. As you can see, we have one test suite and one test. To be more precise, we have one test suite that includes this one test.

Later on, you will recognize that we will use like 1-3 test suites while there will be around 5+ tests, for example. Again, test suites basically provide a structure that groups single tests together.

The stuff with the snapshots is not important for your specific case.

Snapshots are an advanced concept for testing. So a reference snapshot (like an image that was taken) is being compared with the version after some actions took place. This can help to check whether the UI stays the same after some actions or if some changes happened all of a sudden.

I won't focus on testing with snapshots in this article. This is a topic that you might want to look up after understanding some unit testing basics.

🔨 How to Create Your First Unit Test

Now that we've looked at a unit test, lets dive into the first test which we'll build from scratch on our own.

For that, I would like to create a new folder called __tests__. This is common when you are working with tests or checking out other applications.

I'm also dragging the already-available App.test.js file into this folder. This doesn't change anything about the result.

Our folder structure now looks like this:

general folder structure with __tests__

Within __tests__, we create the file myFirstTesting.test.js. We need this file structure of <test name>.test.js. You can also create a test file with <test name>.spec.js – both approaches will work the same.

Our first step is to import the App.js component: import App from "../App";.

To create our first test, we have to make use of the test() function. You could also use it(). Both will achieve the same result.

The first parameter of this function has to be a string, which describes what we are going to test (remember the stuff with "renders learn react link" within the test file we viewed?). This is going to help you have a more precise overview after running all the tests.

In this case, I will use the description "renders logo in App component". The second parameter is another function for which we are using an anonymous arrow function. Our myFirstTesting.test.js file now looks like this:

import App from "../App";

test("renders logo in App component", () => {

})

Even though there is not much going on, let's try entering npm run test again. We will find the following result in our terminal:

Therefore, we now have two test files, resulting in two test suites and two tests.

Now we would actually like to test something. Since we added the description "renders logo in App component", we are going to test exactly that.

In order to do that, we need the render() function, which we'll use whenever we actually want to render a component from our application.

In order to add the render() function, we have to import it from the React Testing library, which is already part of our React application without any other installations.

While we're at this step of importing, let's also import screen (also part of the React Testing library). It provides access to different functions that will look through the current screen after something gets rendered and find specific elements, for example.

After adding these two imports, our myFirstTesting.test.js file now looks like this:

import App from "../App";
import { render, screen } from "@testing-library/react";

test("renders logo in App component", () => {

})

Now that we have all that, let's actually start working on our test.

First of all, we need to render our component. Remember that tests are self-contained and don't know that we have an App.js with the corresponding content in our React application. We have to individually tell the specific test that a component exists by rendering it with render() at the top of the test. This is going to look like this: render(<App />);.

Now that we have rendered the App.js component in this test, we should try to check if a specific content part can be found by the test. This way, we can actually test if App.js was rendered like it was supposed to be.

Assuming something went wrong, we would not be able to find the React logo, for example, which is currently part of the App.js component.

So we will try to find this logo, which is an img element. To do this, we can make use of the getByAltText() function that finds an element by its specific alt attribute, which is commonly utilized for images. We have access to this function with screen that we imported earlier.

We now have this expression: screen.getByAltText("logo"). So the test looks at the screen where we rendered the App.js component beforehand, and then gets an element, which has an alt attribute of "logo". We will connect all this to a variable.

Our test file now looks like this:

import App from "../App";
import { render, screen } from "@testing-library/react";

test("renders logo in App component", () => {
  render(<App />);
  const image = screen.getByAltText("logo");
})

There are a bunch of different functions like getByAltText() that you can use to look for elements with a specific text content, a specific role like a button, or even with a test id that you can add to the actual element.

You also have the opportunity to look for multiple elements. Besides that, you don't have to use a string as parameter. A regular expression with /logo/i is also feasible, for example. We will use different ways to find elements throughout this starter guide.

For the last step, we have to utilize expect(), which we use to see what behavior we can expect. In this case, we expect that our image variable is part of the component and therefore exists.

For this approach, our file would look like this:

import App from "../App";
import { render, screen } from "@testing-library/react";

test("renders logo in App component", () => {
  render(<App />);
  const image = screen.getByAltText("logo");
  expect(image).toBeInTheDocument();
})

❗ How to Create a Failing Test on Purpose

If we now run our tests with npm run test, everything will pass. Now, let's try to reverse this logic so that we actually create a failing test. This way, we can check if this test actually has some impact or not.

To do this, we can go into our App.js file and change the alt attribute for the logo image. If you change it to alt="loo", the test will fail and it'll give you some information.

In our case, though, I would like to change something on the test itself to make it fail and show you another expression that is handy to know. Instead of expect(image).toBeInTheDocument(); we can also type expect(image).not.toBeInTheDocument();. So here we added a not. This basically reversed the logic, and now the test expects that no such image element exists.

If we now try to run the test, we will find the following error message in our terminal:

The error message we get

You can see that the test expected that there was no such element as image. However, it found something and therefore answered with an error message.

You don't have to make all your tests purposely fail to check if they work or not. I just wanted to show you what an actual failing test would look like.

✏️ How to Create Some Additional Tests

Now we have finished with our first test and have some basic knowledge about what to expect when working with unit tests. Next, we will check out some other test examples.

In order to create a more realistic scenario, I will add one additional component, which we'll insert into the App.js component.

For this step, first we create a folder called components in our src folder. This is not a must, but it is common to structure your files like that.

Within the components folder, we create List.js. Our folder structure now looks like this:

current folder structure

Now, let's try to follow more of a test driven development (TDD) workflow, which is quite modern. I'm not necessarily sayint that this is always recommended. But a TDD approach is considered best practice by more and more people nowadays.

Of course, in this tutorial we'lre "only" talking about unit tests and not integration or end-to-end tests, but the general TDD workflow is similar for all three test categories.

So using this test driven development approach, we basically add tests and work on our application simultaneously.

To be more precise, we even create tests for single components and function parts before you even implement this tested logic in your application.

So there is a lot of going back and forth instead of creating all the tests at once at the end.

How to Start the Setup for List.js

In our example, we have added the List.js component. Within this component, I would like to add a list with a button. When a user clicks on the button, it adds something to the list (an object with multiple keys and values).

In order to have some sort of frame, I will first add some div elements and similar stuff to our List.js component before we dive into the actual logic.

The List.js component now looks like this:

const List = () => {
  return (
    <div
      style={{ marginLeft: "auto", marginTop: "500px", marginBottom: "500px" }}
    >
      <h1>This is a list</h1>
      <ul style={{ listStyleType: "none" }}>
        <li>This is the first list entry</li>
      </ul>
      <button>This button can add a new entry to the list</button>
    </div>
  );
};

export default List;

I also added the List.js component as a child to App.js (below all the other stuff in App.js) so it will be visible without changing anything else.

The result looks like this:

How it looks

This won't win you any style competitions but it's sufficient for our case.

Setup for the test for List.js

Since we want to test while we are working on our component, I will now jump directly to the testing part even if nothing really happened in our List.js component in terms of click functions, for example.

We could create a new test file, but I would like to show you a new function we can use for our test suites specifically. This function is called describe() and can be handy for further structuring our tests.

To use describe(), we jump to myFirstTesting.test.js within __tests__. Right now, this file basically serves as one test suite for the test we specifically created for the App.js component. But I would like to have two test suites within this test file: one for the App.js tests and one for the List.js tests.

For this step, I'm using the describe() function, which basically works like the test() function in terms of parameters.

The first parameter will be a string, describing the respective test suite. The second parameter is a function, which then includes our test() functions with their stuff.

It will look like this in our case:

import App from "../App";
import { render, screen } from "@testing-library/react";

describe("App.js component", () => {
  test("renders logo in App component", () => {
    render(<App />);
    const image = screen.getByAltText("logo");
    expect(image).toBeInTheDocument();
  });
});

describe("List.js component", () => {
  test("example", () => {});
});

Before we jump into this new test, I would actually like to add something for the App.js testing. Since we have this describe() block, we could just add a new test() function – and this is what I'm going to do.

See the following newly added test described with "renders List.js component in App.js":

import App from "../App";
import { render, screen } from "@testing-library/react";

describe("App.js component", () => {
  test("renders logo in App component", () => {
    render(<App />);
    const image = screen.getByAltText("logo");
    expect(image).toBeInTheDocument();
  });

   test("renders List.js component in App.js", () => {
    render(<App />);
    const textInListJS = screen.getByText(/This is a list/i);
    expect(textInListJS).toBeInTheDocument();
  });
});

describe("List.js component", () => {
  test("example", () => {});
});

So I'm rendering the App.js component and looking for text via a regular expression, which is part of the List.js component. This test can basically be understood as a render test for List.js. If List.js had not been able to be rendered within App.js, this test would not pass.

If you are confused that this works without separately using render() on the List.js component, remember that List.js is part of App.js and everything inside App.js will be rendered under typical conditions. If you tried to look for a text phrase that doesn't exist in List.js, this new test would fail. Right now, in our case, it passes.

I would also like to emphasize that you can have multiple expect() functions within the same test. Therefore, we also could have structured this new test like this:

import App from "../App";
import { render, screen } from "@testing-library/react";

describe("App.js component", () => {
  test("renders logo in App component", () => {
    render(<App />);
    const image = screen.getByAltText("logo");
    const textInListJS = screen.getByText(/This is a list/i);

    expect(image).toBeInTheDocument();
    expect(textInListJS).toBeInTheDocument();
  });
});

describe("List.js component", () => {
  test("example", () => {});
});

This would also work perfectly fine in our case. And this can be handy in situations where you test for some small elements which are directly connected to each other and have the same requirements to be rendered.

But keep in mind that in our case, we should have adjusted the description for this test. That's because "renders logo in App component" is not correct anymore if we are testing more than that in this test. So let's head back to the structure with two separate tests for now. But have in mind that you are able to work like this.

Back to the test for List.js

Now I would like to work with the second describe() block that we created a few moments ago, where we want to work with tests specifically for the List.js component.

Since we are aiming for a test-driven development approach, we should think about what we are going to build, write a test, and then implement that logic in our component.

We want to create a simple list in our List.js component. So there will be an array, which we will go through with map().

For this approach, we will utilize the useState() hook so we have a state that can dynamically adjust (the array of list items). Our first test will be to check if the length of this array in its initial state is equal to 1.

To find the items within this state, we will make use of the getAllByTestId() method, which allows us to search for specific elements we marked with a data-testid in the frontend.

The test with the description "renders initial state of listData state" that I created is now included:

import App from "../App";
import { render, screen } from "@testing-library/react";

describe("App.js component", () => {
  test("renders logo in App component", () => {
    render(<App />);
    const image = screen.getByAltText("logo");
    const textInListJS = screen.getByText(/This is a list/i);

    expect(image).toBeInTheDocument();
    expect(textInListJS).toBeInTheDocument();
  });
});

describe("List.js component", () => {
   test("renders initial state of listData state", () => {
    render(<List />);
    const list = screen.getAllByTestId("list-item");
    expect(list.length).toEqual(1);
  });
});

Right now this test will fail, of course, because we haven't added any of this logic to the component yet.

So I adjusted the List.js component. It now looks like this:

import { useState } from "react";

const List = () => {
  const initialState = [
    {
      id: `${new Date().getSeconds()}`,
      description: "This is something",
      significance: 7,
    },
  ];
  const [listData, setListData] = useState(initialState);

  return (
    <div
      style={{ marginLeft: "auto", marginTop: "500px", marginBottom: "500px" }}
    >
      <h1>This is a list</h1>
      <ul style={{ listStyleType: "none" }}>
        {listData.map((listItem) => {
          return (
            <li key={listItem.id} data-testid="list-item">
              {listItem.description}
            </li>
          );
        })}
      </ul>
      <button>This button can add a new entry to the list</button>
    </div>
  );
};

export default List;

Newly added was the listData state array via a useState() hook as well as an initialState, which I initialized with one object at the very top. I also made use of the map() function to go through this listData to create a list.

For each <li> element, I'm adding a key and a data-testid. This data-testid is the id we need for our test to find the respective elements.

On our actual application, we can see the listItem.description for this initial state:

So by manually testing (actually looking at our application in the browser), we can see that this should work. If we now run our tests, we will also see that the test we created passed.

How to add an object to the state

Now let's test something more exciting: the logic to add a new object to this listData state. Again, we will start by working on our test first before actually implementing the required logic within the React component.

With this newly added test described by "adds a new data entry to listData after button click", our test file now looks like this:

import App from "../App";
import List from "../components/List";
import { render, screen } from "@testing-library/react";

import userEvent from "@testing-library/user-event";

describe("App.js component", () => {
  test("renders logo in App component", () => {
    render(<App />);
    const image = screen.getByAltText("logo");
    expect(image).toBeInTheDocument();
  });

  test("renders List.js component in App.js", () => {
    render(<App />);
    const textInListJS = screen.getByText(/This is a list/i);
    expect(textInListJS).toBeInTheDocument();
  });
});

describe("List.js component", () => {
  test("renders initial state of listData state", () => {
    render(<List />);
    const list = screen.getAllByTestId("list-item");
    expect(list.length).toEqual(1);
  });

  test("adds a new data entry to listData after button click", () => {
    render(<List />);
    let listItems = screen.getAllByTestId("list-item");
    const button = screen.getByRole("button", {
      name: /This button can add a new entry to the list/i,
    });

    expect(list.length).toEqual(1);
    userEvent.click(button);
    list = screen.getAllByTestId("list-item");
    expect(list.length).toEqual(2);
  });
});

At the bottom, you can see this test. Therefore, we are first rendering the List.js component before looking for all available list items we assigned a test id to. You will see exactly where we put the test id in a few moments.

We also have to look for the button that we want to test to see if clicking on it adds something to the list. We do this with getByRole() which expects a role like "button" or "table" as a first parameter, for example (there are a bunch of different roles you can target). The second parameter is optional and is an object that can receive a value for the name key.

name is pretty much the text content we have specifically for the button in this case. This optional second parameter is handy when you have multiple elements of type "button" in your component and want to get a specific button out of these.

After getting the listItems as well as the button, we start off with a first expect() to basically test the initial state. In this initial state, we expect to have only one list item.

Then, with the help of userEvent, we are going to click on the button. You could also use fireEvent for this situation (userEvent is still pretty new compared to the fireEvent approach). Both will work, and both are helpful for any action where you want to interact with specific elements. In this case, I want to simulate clicking on a button.

Tests generally follow a "arrange -> act -> assert" pattern that you can follow to structure them. Within the "arrange" part, you initialize and get all required elements. With the "act" part, you would simulate a mouse click (as in our case), for example. With "assert," you are checking if it all behaves like you expected it to.

In another case, you could simulate changing the value of an input field with fireEvent.change(inputField, { target: { value: someValueVariable } }), for example. Maybe you want to focus an input field or even drag an element - such actions can be simulated via fireEvent and userEvent.

After the button click, it again looked for all listItems since the current value of this variable would still be 1 from the previous initialization. As soon as this step is completed, it uses another expect() function to check if the length of the listItems array is now equal to 2 and not 1.

Now that we have our test logic, let's jump back to the List.js component and implement the corresponding logic:

import { useState } from "react";

const List = () => {
  const initialState = [
    {
      id: `${new Date().getSeconds()}`,
      description: "This is something",
      significance: 7,
    },
  ];
  const [listData, setListData] = useState(initialState);

  return (
    <div
      style={{ marginLeft: "auto", marginTop: "500px", marginBottom: "500px" }}
    >
      <h1>This is a list</h1>
      <ul style={{ listStyleType: "none" }}>
        {listData.map((listItem) => {
          return (
            <li key={listItem.id} data-testid="list-item">
              {listItem.description}
            </li>
          );
        })}
      </ul>
      <button
        onClick={() =>
          setListData([
            ...listData,
            { id: 999, description: "999", significance: 100 },
          ])
        }
      >
        This button can add a new entry to the list
      </button>
    </div>
  );
};

export default List;

The only part that changed is the button at the bottom of this file. So I added a function that gets invoked when clicking on this button. The function then adjusts the current state of listData which is responsible for rendering our list. I copied the current state with a spread operator and then added another hard-coded object as the new entry for this list.

Of course, there are more creative ways to fill in the values for the id, description, and significance keys.

I would also like to emphasize that you have the opportunity to create a separate function outside of the return() and access this function like this: onClick={separateFunctionToAddObjectToState} on the same button element. This would also work without having to render something additional within the test.

If we now run our test, we will see that it passes. If you try to still expect a length of 1 after clicking on the button, the test will fail like this:

error alert for length of 1

So it actually does what it is supposed to do.

🔧 Setup for Redux

After working with local states via the useState() hook, I would like to work on the same files and adjust them for Redux (or the Redux Toolkit, to be precise).

I'm not going to dive deep into what Redux actually is and what every term like action, store, or reducer means in detail – since this would be worthy of a whole new guide. If you want that, you can read this guide to Redux basics.

Instead, I will give just a quick rundown and show which files I'm adding and editing. Then I'll talk about how to handle the render() method, including the Redux store provider, which can cause a lot of frustration when testing if you don't know about it.

Overall folder structure:

current overall folder structure with the React Redux Toolkit

You can see that I added an app (for the store) as well as a features (for the slice) folder.

Updated index.js file:

import React from "react";
import ReactDOM from "react-dom/client";
import "./index.css";
import App from "./App";
import reportWebVitals from "./reportWebVitals";
import { Provider } from "react-redux";
import store from "./app/store";

const root = ReactDOM.createRoot(document.getElementById("root"));
root.render(
  <React.StrictMode>
    <Provider store={store}>
      <App />
    </Provider>
  </React.StrictMode>
);

You can see that I added a provider and wrapped it around the application so we have access to the store from anywhere.

Created store.js file:

import { configureStore } from "@reduxjs/toolkit";
import { ListSlice } from "../features/ListSlice";

const store = configureStore({
  reducer: {
    listReducers: listSlice.reducer,
  },
});

export default store;

In this file, we have created the required store for the Redux implementation.

Created ListSlice.js file in features folder:

import { createSlice } from "@reduxjs/toolkit";

export const initialState = {
  value: [
    {
      id: `${new Date().getSeconds()}`,
      description: "This is something",
      significance: 7,
    },
  ],
};

export const ListSlice = createSlice({
  name: "listReducers",
  initialState,
  reducers: {},
});

export const { } = ListSlice.actions;
export default ListSlice.reducer;

Here we have created the slice that we added to the store. Notice that I haven't added any reducer yet. This slice just contains the current corresponding state.

Updated List.js file in components folder:

import { useSelector, useDispatch } from "react-redux";

const List = () => {
  const listState = useSelector((state) => state.listReducers.value);
  const dispatch = useDispatch(); // not used right now

  return (
    <div
      style={{ marginLeft: "auto", marginTop: "500px", marginBottom: "500px" }}
    >
      <h1>This is a list</h1>
      <ul style={{ listStyleType: "none" }}>
        {listState.map((listItem) => {
          return (
            <li key={listItem.id} data-testid="list-item">
              {listItem.description}
            </li>
          );
        })}
      </ul>
      <button>This button can add a new entry to the list</button>
    </div>
  );
};

export default List;

On the frontend, we swapped the local state (using the useState hook) with the Redux state (using the useSelctor hook). You'll also see that I adjusted the button. There is no click function anymore (we will get back to that later on).

🔎 How to Perform Testing with the React Redux Toolkit

Now that we have updated and created all the necessary files for the React Redux Toolkit logic, I would like to run a quick test of all the tests we previously created.

The result is that all tests have failed:

Keep in mind that I adjusted the button in List.js, for example, so the corresponding test was expected to fail. However, not all tests should have failed.

The test environments are working in their own world. They don't know if you wrap a provider somewhere in index.js and enable Redux logic. So the tests are still trying to make the rendering work without Redux. But our application now depends on Redux to manage our main state.

This means that we have to adjust the render() function so that this function is actually aligned with the Redux logic.

A method to make this work is to introduce a helper function, which we will store in a new folder called utils. The file will be called utils-for-tests.jsx. The content will look like this:

import React from "react";
import { render } from "@testing-library/react";
import { configureStore } from "@reduxjs/toolkit";
import { Provider } from "react-redux";
// As a basic setup, import your same slice reducers
import { ListSlice } from "../features/ListSlice";

export function renderWithProviders(
  ui,
  {
    preloadedState = {},
    // Automatically create a store instance if no store was passed in
    store = configureStore({
      reducer: { listReducers: ListSlice.reducer },
      preloadedState,
    }),
    ...renderOptions
  } = {}
) {
  function Wrapper({ children }) {
    return <Provider store={store}>{children}</Provider>;
  }

  // Return an object with the store and all of RTL's query functions
  return { store, ...render(ui, { wrapper: Wrapper, ...renderOptions }) };
}

This code information can be found in the Redux documentation. You can almost copy and paste it all for your application.

But you have to adjust the slices that are used in there. Since in our application there is only the ListSlice we don't have much to add. Just import that and update the content of the configureStore() function, like we managed it in our store.js file.

This step is necessary to basically mock the entire Redux logic and put it together into one new render() function.

With that, we can import this new function into our test files (App.test.js and myFirstTesting.test.js) and then replace all render() functions with renderWithProviders(). The App.test.js file, for example, now looks like this:

import { screen } from "@testing-library/react";
import App from "../App";
import { renderWithProviders } from "../utils/utils-for-tests";

test("renders learn react link", () => {
  renderWithProviders(<App />);
  const linkElement = screen.getByText(/learn react/i);
  expect(linkElement).toBeInTheDocument();
});

There is not much more to do! If we now run our tests again (and comment out this one test, which is going to fail regardless because the button logic is not active anymore), it will work again.

Slice testing

Another exciting part about testing with Redux is testing the slices. If you created your application with the React Redux Toolkit template, then you will be provided with some corresponding tests.

For our case, I also want to implement a new test file where we will specifically test ListSlice.js and its corresponding Redux logic.

For this slice, we have to import the slice and the corresponding reducers we want to test. To start, I will import the slice and test if it gets initialized with the initialState.

This is actually not the TDD approach since we already manually tested this part. Netherless, I would like to implement an automatic test as well:

import ListSlice, { initialState } from "../features/ListSlice";

describe("tests for ListSlice", () => {
  test("initialize slice with initialValue", () => {
    const listSliceInit = ListSlice(initialState, { type: "unknown" });
    expect(listSliceInit).toBe(initialState);
  });
});

Notice that I'm using .spec instead of .test. This doesn't matter. You can choose either. In this case, I'm going with .spec to remind you that this is also a viable option.

Also remember that we exported the initialState within our slice (see above). So we are able to import it here.

Other than that, we are already familiar with the describe() environment, which includes one test(). Within this test, I'm initializing a variable listSliceInit, which will hold the value we are receiving after the slice operation took place.

For this operation, we use ListSlice as a function and include the initial state as the first argument (in this case initialState). The second argument will be a reducer in most cases.

But in this case, we don't need to enter a reducer. Instead, we are using an object with type: "unknown". This is basically telling the function that we don't want to perform any additional operations.

Therefore, listSliceInit should now include our state value, which includes an array with one entry. The corresponding test will pass.

To force a failure, I'm entering expect(listSliceInit).toBe({ value: [] }); instead of the previous expect() function. So instead of our initialState we are expecting it to have an empty array. Now our test environment will tell us the following:

failing test

So it actually tells us what exactly it expected – in this case, it expected the initialState.

Next, I would like to test a reducer. However, we haven't added one yet. So I will adjust ListSlice in the ListSlice.js file like this:

export const ListSlice = createSlice({
  name: "listReducers",
  initialState,
  reducers: {
    testAddReducer: (state, action) => {
      state.value.push(action.payload);
    },
  },
});

Thus, I added testAddReducer(), which is responsible for pushing one additional element to the current state value, which it receives via an input from the dispatch (through action.payload).

If we now jump back to the listSlice.spec.js file, I'm adding another unit test:

import ListSlice, { initialState, testAddReducer } from "../features/ListSlice";

describe("tests for ListSlice", () => {
  test("initialize slice with initialValue", () => {
    const listSliceInit = ListSlice(initialState, { type: "unknown" });
    expect(listSliceInit).toBe(initialState);
  });

  test("testAddReducer", () => {
    const testData = {
      id: `${new Date().getSeconds()}`,
      description: "This is for the test section",
      significance: 5,
    };

    const afterReducerOperation = ListSlice(
      initialState,
      testAddReducer(testData)
    );

    expect(afterReducerOperation).toStrictEqual({
      value: [initialState.value.at(0), testData],
    });
  });
});

I added the test for testAddReducer. You can see that I imported the reducer as well.

Firstly, I'm initializing a new variable, testData, which contains the data I would like to push to the current state.

After that, we follow the same structure as before with afterReducerOperation. But instead of this type: "unknown" stuff, we add the reducer as the second argument. This receives the testData as a parameter – basically like you would see it in a dispatch.

Then, we expect this afterReducerOperation variable to be strictly equal to the value of an array, which has two entries: initialState.value.at(0) (the first entry of our initialState) and testData. And this test will pass like we actually expected it.

If we are trying to enter some other entries or change the current ones, you would be able to see this test failing:

forced error: I added a third entry to the array

How to make the button click function work again

Remember the button click within the List.js component (for adding something to the listData state) that wasn't working anymore after we changed to the Redux setup? Let's quickly update that to make that logic work within a Redux environment for the sake of completeness. Since we have the required reducer now, this will be an easy step.

To make the test work again, which added a new element to the state, we have to adjust it a little on the frontend to implement the Redux logic. (Previously we used the useState hook for a local state.)

For this step, we are making use of the dispatch() function in order to reach out to the testAddReducer:

import { useSelector, useDispatch } from "react-redux";
import { testAddReducer } from "../features/ListSlice";

const List = () => {
  const listState = useSelector((state) => state.listReducers.value);
  const dispatch = useDispatch();

  return (
    <div
      style={{ marginLeft: "auto", marginTop: "500px", marginBottom: "500px" }}
    >
      <h1>This is a list</h1>
      <ul style={{ listStyleType: "none" }}>
        {listState.map((listItem) => {
          return (
            <li key={listItem.id} data-testid="list-item">
              {listItem.description}
            </li>
          );
        })}
      </ul>
      <button
        onClick={() =>
          dispatch(
            testAddReducer({
              id: `${new Date().getSeconds()}1`,
              description: "This is added",
              significance: 5,
            })
          )
        }
      >
        This button can add a new entry to the list
      </button>
    </div>
  );
};

export default List;

Besides the button logic, nothing else has changed in this file.

In the corresponding test (within myFirstTesting.test.js nothing has changed), if we now test everything – including this updated test – we will see that everything works fine:

final test run

And that's pretty much it for fundamental slice and general Redux unit testing!

🔭 Outlook for Advanced Testing

There are different topics like thunks (or RTK Query as an alternative) which could also be tested. But I'm considering this as an advanced topic, and it would take some more time to explain these processes.

If you are not aiming to be an expert in testing at this point, the topics we discussed for unit tests in Redux in this tutorial should be sufficient for you.

Generally speaking, I would recommend diving deeper into so-called mocks, spies, and also snapshots. These will be helpful if you are working on some other more advanced tests.

The stuff with renderWithProvider() is basically based on such a mock – there, we artificially created a store with reducers and a provider to create this new render() function. So mocks are especially helpful for any third-party libraries, for example.

As I said, though, mocks, spies, and snapshots are more of an advanced topic to wrap your head around.

📣 Further Learning Opportunities

I recently started to work on my first free Udemy course. While this first free course covers the basics of the React Redux Toolkit with German audio and manually added English subtitles, I'm also planning to publish other Udemy courses completely in English in the future.

I would really appreciate it if you would check out this cost-free course in order to provide me with some feedback.

Testing is a vital part of software development. The sooner you start testing, the better.

In this article, I'll show you how to write tests for your NodeJs/ExpressJS and MongoDB/Mongoose applications with Jest and Supertest.

Let's get started

First let's set up a demo Express.js app.

Let's say we are building a backend REST API for an eCommerce application.

This app should:

• Get all the products
• Get a product by id
• Add product(s) to the database
• Delete product(s) from the database
• Update product information

Express.js App Set Up

Step 1: Project set up

First, create a folder and start a blank application with npm.

npm init

Fill all the details it asks for.

Then, install express, mongoose, axios and dotenv with the following command:

npm i express mongoose axios dotenv

Here's a link to the package.json on my GitHub.

Step 2: Create the boilerplate

Let's create all the folders and files and then fill them with some boilerplate code.

This is how your folder hierarchy should look:

.
├── controllers
│   └── product.controller.js
├── models
│   └── product.model.js
├── routes
│   └── product.route.js
├── package-lock.json
├── package.json
├── .env
├── app.js
└── server.js

Use these files' code by copying and pasting. Analyze the code and flow as best you can.

• [product.controller.js](https://github.com/itsrakeshhq/jest-tests-demo/blob/main/controllers/product.controller.js)
• [product.model.js](https://github.com/itsrakeshhq/jest-tests-demo/blob/main/models/product.model.js)
• [product.route.js](https://github.com/itsrakeshhq/jest-tests-demo/blob/main/routes/product.route.js)
• [app.js](https://github.com/itsrakeshhq/jest-tests-demo/blob/main/app.js)
• [server.js](https://github.com/itsrakeshhq/jest-tests-demo/blob/main/server.js)

Step 3: Database setup

I advise using two databases for a project—one for testing, the other for development. But just one database will be sufficient for learning purposes.

First, create a MongoDB account or log in.

Then create a new project. Give it a name and press the Next button.

Naming the project

Then click Create Project after that.

We must create a database in the following window by selecting a cloud provider, a location, and specs. So press Build a Database to get going.

Build a database

Choose "Shared" because it is sufficient for learning purposes. And then click Create.

Choose a deployment option

Next, select "aws" as your cloud provider and the region that is closest to you. Following your selection, click Create Cluster.

The cluster's formation will take some time. Create a user to access your database in the meanwhile.

Create Superuser

Choose "My Local Environment" because we are developing our application. You can then add an IP addresses. To conclude, click Close.

Add IP addresses

You will receive a URI string after the database is set up, which we'll use to connect to the database. The string appears as follows:

mongodb+srv://<YOUR_USERNAME>:<YOUR_PASSWORD>@<YOUR_CLUSTER_URL>/<DATABASE_NAME>?retryWrites=true&w=majority

Put this string in the .env file.

MONGODB_URI=your database string

Now we're ready to start testing our app.

How to Write Tests with Jest and SuperTest

Step 1: Install packages

You need three npm packages to begin writing tests: jest, supertest, and cross-env. You can install them like this:

npm i jest supertest cross-env

• jest: Jest is a framework for testing JavaScript code. Unit testing is the main usage of it.
• supertest: Using Supertest, we can test endpoints and routes on HTTP servers.
• cross-env: You can set environmental variables inline within a command using cross-env.

Step 2: Add test script

Open your package.json file and add the test script to the scripts.

"scripts": {
    "test": "cross-env NODE_ENV=test jest --testTimeout=5000",
    "start": "node server.js",
    "dev": "nodemon server.js"
},

In this case, we're using cross-env to set environment variables, jest to execute test suites, and testTimeout is set to 5000 because certain requests might take a while to finish.

Step 3: Start writing tests

First, create a folder called tests at the application's root, and then create a file there called product.test.js. Jest searches for the folder tests at the project's root when you do npm run test. As a result, you must place your test files in the tests folder.

Next, import the supertest and mongoose packages into the test file.

const mongoose = require("mongoose");
const request = require("supertest");

Import dotenv to load environment variables, and import app.js as that is where our application starts.

const mongoose = require("mongoose");
const request = require("supertest");
const app = require("../app");

require("dotenv").config();

You'll need to connect and disconnect the database before and after each test (because we don't require the database once testing is complete).

/* Connecting to the database before each test. */
beforeEach(async () => {
  await mongoose.connect(process.env.MONGODB_URI);
});

/* Closing database connection after each test. */
afterEach(async () => {
  await mongoose.connection.close();
});

Now you can write your first unit test.

describe("GET /api/products", () => {
  it("should return all products", async () => {
    const res = await request(app).get("/api/products");
    expect(res.statusCode).toBe(200);
    expect(res.body.length).toBeGreaterThan(0);
  });
});

In the above code,

• We use describe to describe the unit test. Even though it is not required, it will be useful to identify tests in test results.
• In it, we write the actual test code. Tell what the test performs in the first argument, and then in the second argument, write a callback function that contains the test code.
• In the callback function, the request is sent to the endpoint first, and the expected and actual responses are then compared. The test passes if both answers match, else, it fails. ✨ As simple as that ✨.

You can write tests for all the endpoints in the same manner.

describe("GET /api/products/:id", () => {
  it("should return a product", async () => {
    const res = await request(app).get(
      "/api/products/6331abc9e9ececcc2d449e44"
    );
    expect(res.statusCode).toBe(200);
    expect(res.body.name).toBe("Product 1");
  });
});

describe("POST /api/products", () => {
  it("should create a product", async () => {
    const res = await request(app).post("/api/products").send({
      name: "Product 2",
      price: 1009,
      description: "Description 2",
    });
    expect(res.statusCode).toBe(201);
    expect(res.body.name).toBe("Product 2");
  });
});

describe("PUT /api/products/:id", () => {
  it("should update a product", async () => {
    const res = await request(app)
      .patch("/api/products/6331abc9e9ececcc2d449e44")
      .send({
        name: "Product 4",
        price: 104,
        description: "Description 4",
      });
    expect(res.statusCode).toBe(200);
    expect(res.body.price).toBe(104);
  });
});

describe("DELETE /api/products/:id", () => {
  it("should delete a product", async () => {
    const res = await request(app).delete(
      "/api/products/6331abc9e9ececcc2d449e44"
    );
    expect(res.statusCode).toBe(200);
  });
});

Then run npm run test to run the test suites (suite - test file).

Test results

And that's it! You now know how to test your Express/Mongoose apps with Jest and SuperTest.

Now go forth and create new tests for your apps. :)

If you have any questions, feel free to message me on Twitter.

But software testing is a complex topic and can be a bit difficult to understand.

In this article, I'll explain the major topics in software testing and how this practice can help you.

Table of Contents:

• What is Software Testing?
• Types of Software Testing
• Different Types of Functional Software Testing
• Software Testing Principles
• Why is Software Testing needed?
• Conclusion

What is Software Testing?

Software testing is the process of making sure your software/app works as it should. There are various methods you can use to test your code, and each testing method has different requirements.

For instance, unit testing involves writing test cases to ensure the code works as it should, and Beta testing consists of testing the preview version of the software or app to make sure users can use the product.

Software testing is integral to the process of building good software that works as it should. It also helps improve productivity and performance. Testing is an important part of the Software Development Life Cycle (SDLC).

Other benefits of testing your code include preventing bugs, reducing cost, and reducing time of development.

Types of Software Testing

There are two general types of software testing:

Functional Testing:

Functional Testing is a software testing method that validates the system against the customer's requirements or specifications.

This type of testing aims to test each function of the software by providing the correct input and ensuring the output is right.

For examples, let's say you write a test case to test creating a user. The test case provides the correct input (email, first name, last name and password) and ensures the output (success message) is accurate, as well.

Functional testing checks that everything is functioning properly by emulating business scenarios based on applicable requirements.

Non-functional Testing:

Non-functional testing is a software testing method that tests for end-user experiences, such as performance and reliability under load. This could either make or break a user experience.

When your code fails at non-functional testing, it may not cause an issue that user would note but it can flag a problem in the system.

Non-functional testing is just about testing the software to know how it responds to load on the system.

In this guide, we will focus on Functional Software Testing.

Different Types of Functional Software Testing

There are different types of software testing, and each has a specific aim. We'll look at each one quickly now.

Unit Testing:

Unit testing is a type of software testing that validates how each software unit performs and whether that specific piece of code does what it should. A unit is the smallest testable component of an application.

The aim is to confirm that each unit of software code works as expected. You do unit testing during the coding (development) stage or phase. Developers write these tests as they go.

Unit tests isolate possible bugs in your code and help you correct them. A unit could be a single function, method, procedure, module, or object.

Code Snippet of a unit test case in Python

Code Snippet of a unit test case in Java

Integration Testing:

Integration Testing is software testing which helps ensure that software components or functions work together properly. This is the second phase of the software testing process that comes after unit testing.

In this type of testing, units or individual software components are tested in groups. This testing method mainly focuses on exposing defects in interactions between integrated components and units.

System Testing:

System testing involves the process of testing integrated software. The aim is to evaluate the system's compliance with specify requirements.

In system testing, the quality assurance team evaluates how each component of the application or software work together in a full, integrated environment.

Acceptance Testing:

Acceptance testing is a software testing method where a system is tested or checked for acceptability. It evaluates the system's compatibility with the business requirements and assesses whether it is acceptable for delivery.

It is also known as formal testing performed to fit user needs, requirements, and business processes. It determines if a system satisfies the standard business criteria and if users or customers will be able to accept it.

Acceptance testing is the last stage of software testing done after system testing and before making the system available for public use.

Regression Testing:

Regression testing ensures that a component continues working as it should, after including additional components in the program. You perform regression testing when something changes, such as adding a new module to the program.

This type of testing represents the complete testing of executed test cases that are re-executed to ensure the current functionalities still work just fine.

Alpha Testing and Beta Testing:

Alpha testing is also known as initial validation testing. It is an aspect of acceptance testing done before the product is given to the consumers or users. QA (Quality Assurance) testers usually do this. Alpha testing is done internally by the QA team.

Beta testing is also known as second phase of validation testing. But this type of testing is done externally, which means the public does it.

The version of the code/software for this phase of testing is released to a limited number of users for testing in a real-time scenario. For instance, freeCodeCamp's math curriculum is available for beta testing here.

Software Testing Principles

Everything in tech has principles. These are guidelines to help you build better software and avoid errors.

Here are some software testing principles you should follow when writing tests for your code:

Testing aims to show the presence of defects, not the absence:

Software testing aims to spot software failures. This reduces the presence of faults and errors.

Software testing ensures defects are visible to the developer but doesn't guarantee defect-free software. Multiple types of testing can't even ensure error-free software. Testing can only decrease the number of errors.

Exhaustive testing is not possible:

Exhaustive Testing is the process of testing software for all valid and invalid inputs and pre-conditions.

This method of testing is not realistic because test cases presume that the software is correct and it produces the correct output in every test case. If you truly try to test every aspect and test case in your software, it will take too much time and effort, and it's not practical.

Perform early testing:

Testing your software at an early phase helps avoid minor bugs or errors. When you can spot errors at an early stage of the Software Development Life Cycle(SDLC), it's always less expensive. It is best to start software testing from the beginning of the project.

Defect clustering:

Defect clustering refers to when most of the problems you find occur in just a few parts of the application or software. If you can identify the modules or areas where these defects occur, you can focus most of your testing efforts on them.

Keep the Pareto Principle in mind when testing your code: 80% of software defects tend to come from 20% of the modules.

Beware of the Pesticide paradox:

This principle is based on a theory – "the more you use pesticide on a crop, the more immune the crop will eventually grow, and the pesticide will not be effective."

When you repeat particular test cases over and over, you will see fewer and fewer new bugs. So to find new bugs, update your test cases and run them once you add new test cases.

Testing is context-dependent:

Testing is context-dependent, which means that you should test your software based on its needs, functionalities, and requirements.

Your test approach should depend on what your software does. Not every software needs the same type/method of testing because every application has its unique functionalities.

For instance, when testing an eCommerce web app, you will focus on its functionality to display products, so you will test how it shows products to end-users. When dealing with an API, you will focus on the response the API returns when an endpoint is called.

You wouldn't necessarily use the same test cases for both – that is what it means that testing is context-dependent.

The absence of errors is a fallacy:

If you build software that is 99% bug-free, but it doesn't follow user requirements, it is not usable for end-users.

Know that it is very much necessary that your 99% bug-free software still meets or fulfills your user requirements. It is important to write test cases to find errors in the code, but you also need to test your software for your end-users (with them and how they'll use it in mind). The best way to do this is to carry out beta testing.

Why is Software Testing Needed?

Besides making sure your software is bug-free and meets user requirements, software testing has other advantages.

Software testing improves security:

When building software, security is a crucial part of your planning. This is because vulnerable software could jeopardize you users and their information, as hackers can use stolen info for malicious purposes.

As a product undergoes testing, the end-user can count on the fact that they will be getting a reliable product and their details will be secured and safe. So users are more likely to get a product that is free from vulnerabilities with the help of software testing.

Software testing improves product quality:

You want your software or product to be bug-free, low-risk, and effective at what it should do. And you can achieve this by including test cases and other testing methods when building out the code.

In addition, you won't know how good your product is until you test it. This helps you provide the best product version before it gets released (and discover any inconsistencies or pain points along the way – so you can improve them).

Software testing improves customer satisfaction:

For instance, let's say you download a new app and try to use some of its functionality – but it shows an error. This will probably frustrate you, and you might not want to use the app again, right?

This is exactly why software testing is important. It can help you discover such errors and detect them before you release the product to the user, and gives the developers a chance to prevent the error.

By investing in software testing early in the development stage, you are letting the users know that you care about their experience. It could also help you create a solid long-term customer relationship.

Software testing saves money:

Software testing can save you a lot of money – but how?

Each stage of development involves many things, such as clear communication and coordination between multiple teams, and each step has a laundry list of things that could go awry.

Catching those errors when the product is live is a horrible experience because you may have to handle PR, retasking fixes, and trying to sort the problem in real time.

In addition, your users won't be able to access the app while you're fixing it, which defeats the app's purpose and provides a bad user experience in the meantime. Software testing helps resolve this stress, and once live, your user can enjoy your app/product to the fullest.

Conclusion

In conclusion, software testing is a crucial part of development. It can help save your team a lot of trouble, and it feels great to create a usable, bug-free product that users enjoy and recommend.

If software testing interests you, you can check freeCodeCamp's QA certificate course here to learn more about QA testing. QA testers are techies that focus on testing softwares and apps for errors.

These Docker containers are lightweight, and once the tests are finished the containers get destroyed.

In the article we are going to understand what the TestContainers is and how it helps you write more reliable Tests.

We are also going to understand the important components (Annotations and Methods) of the library which help you write the Tests.

Finally, we will also learn to write a proper Integration Test in SpringBoot using the TestContainers library and its components.

Limitations of Testing with an H2 In-Memory Database

The most common approach to Integration Testing today is to use an H2 in-memory database. But there are certain limitations to this method.

First of all, say we are using version 8.0 of MySQL in production, but our integration tests are using H2. We can never execute our tests for the database version running on Production.

SpringBoot app with MySQL DB and H2

‌Secondly, the test cases are less reliable because in production we are using an altogether different database and the tests are pointing to H2. The application may run into issues in production, but the integration tests may succeed.

I was trying to access my RESTful service on local and faced this error:

“Caused by: org.postgresql.util.PSQLException: FATAL: database "example_db" does not exist”.

It happened because of a permission issue, but the tests on local worked fine.

And finally, as documented here, H2 is compatible with other databases only up to a certain point. There are few areas where H2 is incompatible. If you need to use “nativeQueries” in a SpringBoot application, for example, then using H2 may cause problems.

Enter the TestContainers Library

By using TestContainers we can overcome the limitations of H2.

• Integration tests will point to the same version of the database as it’s in production. So we can tie our TestContainer Database Image to the same version running on production.

• Integration tests are lot more reliable because both application and tests are using the same database type and version and there won't be any compatibility issues in Testcases.

What is TestContainers?

The TestContainers library is a wrapper API over Docker. When we write code to create a container behind the scenes it may be translated to some Docker command, for example‌:

MySQLContainer Creation

This code may be translated to something like the following:

docker run -d --env MYSQL_DATABASE=example_db --env MYSQL_USER=test --env MYSQL_PASSWORD=test ‘mysql:latest’

TestContainers has a method name “withCommand”. You use it to set the command that should be run inside the Docker container which confirms that TestContainers is a wrapper API over Docker.

TestContainers downloads the MySQL, Postgres, Kafka, Redis images and runs in a container. The MySQLContainer will run a MySQL Database in a container and the Testcases can connect to it on the local machine. Once the execution is over the Database will be gone – it just deletes it from the machine. In the Testcases we can start as many container images as we want.

TestContainers supports JUnit 4, JUnit 5 and Spock. If you go to the TestContainers.org website, just visit the QuickStart section that explains how to use it: