You've probably seen the routine. The agent changes some code, runs the formatter, the type checker, and the tests, everything comes back green, and you get some version of "Implemented and verified."

But verified how, exactly? A passing test suite is useful evidence, but it isn't the same as opening the app and using the thing that just changed.

I've had plenty of changes where the code made sense and the tests passed, and the UI still had the kind of problem a person spots in about three seconds. A control overlapping something else. A selected state that exists in the store but isn't highlighted on screen. Reset working for one path and not another. A filter that updates state but leaves the old data on the page. Save looking fine until you refresh. A map layer drawn in the wrong order. A modal that technically opens but doesn't fit on the screen.

The agent had tested its code, but nobody had tested the application. So I started making browser review part of the definition of done for anything user-facing, and a few weeks in, that habit turned into a tool. I'll get to that near the end.

In this article, you'll learn how to give a coding agent a real browser with Playwright MCP, and how to write the instructions that make it test the change it just made instead of confirming that a page loads.

You'll see how to have the agent start your app the way your project already starts it, how to split those instructions between AGENTS.md and a reusable skill, why a separate reviewer agent catches bugs the author misses, and how to make the agent clean up only what it started.

In the last part, I'll walk through katalepsis, the open source tool this turned into, including how it decides whether a check really passed.

Table of Contents

• Prerequisites

• The Loop Agents Are Missing

• Why a Browser on Its Own Falls Short

• How to Start the Real App

• Where the Instructions Live

• Why the Reviewer Should Work Independently

• Clean Up After Yourself and Only Yourself

• When a Skill Stopped Being Enough

• How to Try katalepsis

• How katalepsis Works Under the Hood

• What Happened When I Used It

• Where I Landed

Prerequisites

This isn't a beginner tutorial, so it helps to have:

• Some experience with a coding agent that supports MCP (Model Context Protocol), such as Claude Code, Codex, or Cursor, and an idea of how you give it instructions.

• A web app you can run locally with a dev script, written in JavaScript or TypeScript. The examples use a Node project with pnpm, but nothing depends on a specific framework.

• Basic Playwright knowledge: what a test, a locator, and an assertion look like. You don't need an existing test suite to follow the first half.

• Node.js 20 or newer for Playwright MCP, and Node.js 22.12 or newer plus Git if you want to install katalepsis.

• Comfort with the terminal and Git, since the agent will be starting processes and checking git status.

The Loop Agents Are Missing

Coding agents got useful quickly in large part because they already have a decent feedback loop for code. They read, edit, and run the formatter and tests, read the failure, and try again. Nobody has to get it right in one heroic attempt.

For frontend work, the clear missing input is the frontend itself. The loop I want goes a bit further: edit the code, run the tests, start the app, use the changed feature, look at what happened, fix whatever's broken, and then use it again.

That last step matters more than you might think. Agents are very comfortable telling you a fix "should now work", and I'd rather they just open the page again and find out.

Playwright MCP is a good way to give them that. I already used Playwright for E2E tests, but the MCP server is useful for a different reason: it hands the agent a browser directly. It can open localhost, read the page, find controls, click and type, check what changed, and take screenshots.

That isn't the same as asking the agent to run pnpm playwright test. The test suite tells you whether the scenarios you already wrote down still pass. Browser review asks what this particular change could have affected and goes to look.

If an agent refactors some shared map-layer utilities, the tests might cover a few helpers. But from the diff, it can also work out that the code runs during initial render, selection, filtering, and reset, and try each of those. That's a lot closer to how I review my own frontend work.

Setup is small. You need Node 20 or newer and an MCP-capable client:

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@latest"]
    }
  }
}

A quick sanity check is to ask the agent to open the Playwright TodoMVC demo, add a todo called "Browser QA," and confirm it shows up. If it manages that, it has a browser. What it does with one is the harder part.

Why a Browser on Its Own Falls Short

Giving an agent a browser only gives it the ability to look. It doesn't tell the agent what to look at, how far to go, or what counts as proof, and without that it falls back to the cheapest thing that technically involves a browser. An agent with Playwright can still do hilariously weak QA. I've had reports like this:

Opened /route.
Page loaded successfully.
Browser verification complete.

That's a report about opening a page, not about testing a change. Nothing in it would have caught any of the bugs from the start of this article.

So the browser needed instructions to go with it. What follows are the rules I wrote for how an agent should review its own frontend work: what to test, what to check, and what evidence to bring back. They started as a few lines I kept repeating in prompts, and they later became the browser-review skill you'll see in the next sections.

The first rule I settled on is that the review has to come from the change. If it changed a filter, use the filter. If it changed reset logic, change some state first and then reset it. If it touched persistence, refresh the page. If it changed a modal, open and close the modal, and if it changed responsive layout, resize the viewport.

That all sounds obvious when a person says it, but it isn't obvious at all if your agent instructions just say "verify that the app works".

In practice, this means the agent reads the diff and turns it into a short test plan before it touches the browser. For a change to a filter bar, the plan I'd expect looks something like this:

Changed: src/features/projects/FilterBar.tsx, src/features/projects/useFilters.ts

Plan:
1. Open /projects and note how many rows are showing.
2. Filter by Status: Active. Every row should say Active.
3. Add Owner: me on top. The list should narrow again.
4. Clear filters. The original row count should come back.
5. Apply a filter and refresh. The filter lives in the URL, so it should survive.

Then it carries out that plan through the browser tools Playwright MCP exposes. A trimmed version of the calls for the first few steps looks like this:

browser_navigate   { "url": "http://localhost:5173/projects" }
browser_snapshot   {}                      → table with 40 rows
browser_click      { "element": "Status filter", "ref": "e14" }
browser_click      { "element": "Active option", "ref": "e31" }
browser_snapshot   {}                      → table with 12 rows, all "Active"
browser_click      { "element": "Clear filters", "ref": "e15" }
browser_snapshot   {}                      → table with 40 rows

The ref values come from the accessibility snapshot, which is how the agent finds a control without guessing at CSS selectors.

This paid off on a PR that fixed a cursor getting stuck in its hover state on an interactive map. The PR's new unit tests passed against the old code too, so they didn't prove the fix did anything. What did prove it was the agent opening the app, moving on and off the interactive layers, and watching the cursor change and reset exactly when it should.

The second rule is to check the end result rather than the interaction. "I clicked Reset and nothing threw" doesn't tell you much. What you actually care about is that reset puts things back the way they were. Same with saving: the real claim is that you can change a setting, save, refresh, and it's still there.

Written as a Playwright test, that claim looks like this. The important line is the last one, which runs after the reload:

import { test, expect } from "@playwright/test";

test("weekly digest setting survives a reload", async ({ page }) => {
  await page.goto("/settings/notifications");
  await page.getByRole("switch", { name: "Weekly digest" }).click();
  await page.getByRole("button", { name: "Save" }).click();

  await page.reload();

  await expect(page.getByRole("switch", { name: "Weekly digest" })).toBeChecked();
});

The agent doesn't need to write this test to follow the rule. Through MCP it does the same thing by hand: click, save, browser_navigate to the same URL, and read the switch's state from a fresh snapshot.

I also ask for different evidence depending on the question. Whether the URL changed, a button is disabled, or a dialog opened is a question for the DOM and browser state, not a screenshot. Whether that dialog is clipped, two things overlap, or text wraps badly on a phone is a visual question, and that's where screenshots are useful. Sometimes you need both, since a modal can sit happily in the DOM while being unusable on screen.

For a layout change, that means resizing and capturing what a person would actually see:

browser_resize            { "width": 375, "height": 812 }
browser_click             { "element": "Export button", "ref": "e22" }
browser_take_screenshot   { "filename": "export-dialog-375.png" }
browser_resize            { "width": 1440, "height": 900 }
browser_take_screenshot   { "filename": "export-dialog-1440.png" }

While it's in there, the agent may as well look at the console and, when the feature depends on an API call, the network. I only care about errors the change introduced, though. A warning that's been on main for six months shouldn't derail every review, and a client-only transition doesn't need a network audit. If a save that "worked" actually got a 500, I want to hear about it.

browser_console_messages and browser_network_requests return what the page logged and requested, so a failed save shows up as a line like this even when the UI looked fine:

[PUT] http://localhost:5173/api/settings => [500] Internal Server Error

Put together, the rules turn the weak report from the start of this section into something I can actually check:

Route: /settings/notifications
Did: turned on "Weekly digest", clicked Save, reloaded the page.
Checked: switch still on after reload (aria-checked="true");
  PUT /api/settings returned 200; no new console errors.
Screenshots: settings-375.png, settings-1440.png (label wraps cleanly on mobile)
Tests: pnpm test settings (14 passed)
Found and fixed: Save stayed disabled after the first save. Fixed in
  SettingsForm.tsx, then went through the flow again.

How to Start the Real App

The agent can't test anything until the app is running, and this is where it's most tempting for it to take shortcuts.

I didn't want to write "run pnpm dev" into every repository, so the procedure tells the agent to work out how the project starts. It checks the dev script in package.json first, then whatever the README documents, and only falls back to an existing Docker setup if the app can't reasonably run otherwise.

The procedure here is the "Start the application" section of the browser-review skill, which you'll see in full in the next section. In a typical Vite or Next.js project, following it looks something like this:

# 1. How does this project start?
jq .scripts package.json
# {
#   "dev": "vite",
#   "build": "vite build",
#   "test": "vitest run",
#   "e2e": "playwright test"
# }

# 2. Start the dev server in the background, in a temp folder the agent owns,
#    and record the PID so it can stop exactly this process later.
QA_DIR=$(mktemp -d)
pnpm dev --port 5173 > "$QA_DIR/dev.log" 2>&1 &
echo $! > "$QA_DIR/dev.pid"

# 3. Wait until the server answers before opening a browser.
until curl -sf http://localhost:5173 > /dev/null; do sleep 1; done

If there's no dev script, the README is next, and if the README says the app needs Postgres and Redis from docker compose up, the agent uses that compose file and records the containers it started. It doesn't write a new one.

What I really don't want is the agent changing the app to make it easier to test. No temporary auth bypass, no fake server, no edits to .env, and no throwaway route that exists only because it couldn't work out the real setup. The point is to exercise the software I'm actually shipping.

This is the kind of change I mean, and agents really do reach for it when a login page is in the way:

- if (!session) redirect("/login");
+ if (!session && process.env.NODE_ENV !== "development") redirect("/login");

If the agent can't get past authentication with the project's documented test account or seed data, the right move is to stop and report that browser review is blocked and why. A blocked review is honest. A review of an app with its auth switched off isn't a review of my app.

Where the Instructions Live

My first instinct was to keep adding to AGENTS.md. It already held the rules every agent in the repo should know: structure, conventions, testing philosophy, keep changes surgical, and no pointless comments.

I was about to add another hundred lines about starting the app, browser setup, what to click, screenshots, console errors and teardown...but at some point that file stops being instructions and turns into a small religious text.

It helps to know how each of these files is actually used. AGENTS.md is a plain Markdown file that coding agents such as Codex read automatically at the start of every session in that repository, and Claude Code does the same with CLAUDE.md. Everything in it lands in the agent's context for every task, whether the task is a CSS fix or a database migration. So it costs context on every run, and it applies everywhere.

A skill works differently. It's a folder with a SKILL.md file that starts with a short name and description. At the start of a session the agent only sees that name and description, a line or two per skill. When a task matches the description, or another instruction points at the skill, the agent reads the rest of the file and follows it. So you can have a long, detailed procedure that costs almost nothing until it's needed.

What I landed on is treating AGENTS.md as the constitution and skills as the playbooks. A constitution is short, applies to everything, and says what must be true without explaining how to do it. A playbook is a step-by-step procedure for one kind of situation, and you only pull it out when you're in that situation.

In this setup, AGENTS.md says that user-visible work isn't done until it's been checked in a browser, and the browser-review skill says how to do the checking:

/
├── AGENTS.md
└── .agents/
    └── skills/
        ├── browser-review/
        │   └── SKILL.md
        ├── pr-review/
        │   └── SKILL.md
        └── grill-me/
            └── SKILL.md

The rule in AGENTS.md stays short and always visible:

## Browser Verification

User-visible work is not complete until the affected behavior has been
verified in the running application.

For user-visible changes, follow `.agents/skills/browser-review/SKILL.md`.

A successful build, type check, unit test suite, or E2E test suite does
not replace browser verification.

Do not commit browser-verification artifacts. Stop anything you started
for verification and clean up before hand-off.

The skill holds the procedure and only gets loaded when it's needed. I use Codex and Claude Code, so there's one global AGENTS.md that Claude's CLAUDE.md imports, and one shared skills folder both of them read. Whichever agent I'm in that day, it follows the same playbook.

The import is a single line, because Claude Code lets CLAUDE.md pull in another file with @:

@~/.codex/AGENTS.md

Here's a shortened version of the one I first wrote:

---
name: browser-review
description: Verify user-visible changes in the running application with Playwright MCP.
---

# Browser Review

## Before verification

Run `git status --short` and remember what was already there. Don't touch
pre-existing work.

Read the task and the diff. Work out the affected routes, the changed flow,
the loading, empty, error, selected and disabled states that matter,
whether anything needs to survive a refresh, and what nearby behavior
could regress.

## Start the application

Use the repository's dev script, then its documented local setup, then its
existing Docker setup. Don't invent a new startup config and don't change
app behavior to make QA easier. Record every process and container you start.

## Exercise the change

Don't stop once the page loads. Go through the whole changed flow and check
the final result. Use browser or DOM state for functional claims and
screenshots for visual ones. Report new relevant console errors and failed
requests, not old warnings.

If something is broken, fix the smallest cause, rerun the affected tests
and go through the flow again. Don't assume the fix works.

## Cleanup

Teardown is required even when verification fails. Close browsers, stop
servers and containers, and delete screenshots, traces and other artifacts,
but only the ones you created. Run `git status --short` again before
hand-off. Never run `git add`, `git commit` or `git push`.

## Report

Say which routes you tested, what you did, which states you checked, what
tests you ran and what you found and fixed. If you couldn't finish, say why.

None of that is clever. The value is that it's the default expectation, so I'm not relying on the agent to decide by itself that it should open a browser.

Why the Reviewer Should Work Independently

The clear weakness is that the agent deciding whether the work is done is the same one that wrote it. That's still worth something, since I test my own code too, but it starts with a theory of how its implementation works.

By theory I mean the author's mental model of its own change: which code it touched, what it believes that code affects, and, just as important, what it believes it didn't affect. Its test plan comes straight from that model. So it tests the paths it meant to change, they work, and it never looks at the path it didn't realize was involved. That's the same blind spot people have when they test their own code, and it's exactly where the bugs that survive review tend to live.

So when a second agent reviews a branch, I tell it to work out the test scope for itself. My global rules now require a fresh reviewer with no conversation history: it gets the original request, the diff, and the review skill, and nothing about why the author thinks it works. It decides what the code could affect and tests that, without leaning on the author's screenshots, QA notes, or green test run.

The prompt for that reviewer is short on purpose:

You are reviewing a branch you did not write.

Original request:
  Make the E2E app URL configurable through E2E_BASE_URL so the
  tests can run against any port.

Run `git diff main...HEAD` to see the change. Follow the browser-review
skill. Work out for yourself what this change could affect, test that,
and report what you checked with evidence.

The best catch so far came from a PR that made an end-to-end test setup's app URL configurable. The reviewer noticed the seeded test data still hardcoded image URLs to the old port, and running the app on a different port confirmed it. Three image requests went to the wrong port and failed, and the page quietly rendered fallback cards where the images should have been. Nothing threw.

Simplified, the global setup looked like this after the change:

// e2e/global-setup.ts
const baseURL = process.env.E2E_BASE_URL ?? "http://localhost:8080";

await seedDatabase({
  media: [
    { id: "m1", url: "http://localhost:8080/test-media/cover-1.jpg" },
    { id: "m2", url: "http://localhost:8080/test-media/cover-2.jpg" },
    { id: "m3", url: "http://localhost:8080/test-media/cover-3.jpg" },
  ],
});

The author's theory was "the app URL now comes from E2E_BASE_URL", and every test it ran used the default port, where the hardcoded URLs happened to be right. The reviewer had no such theory. It read the whole setup file the diff touched, ran the app on port 8081, and came back with this:

Found: seeded media URLs in e2e/global-setup.ts still point at :8080.
Ran the app with E2E_BASE_URL=http://localhost:8081.
Network: 3 requests to http://localhost:8080/test-media/... failed
  (net::ERR_CONNECTION_REFUSED).
Screen: the gallery rendered 3 fallback cards instead of images
  (gallery-8081.png).
Suggested fix: build the seeded URLs from E2E_BASE_URL, or make them
  relative.

The fix was one line per URL:

{ id: "m1", url: new URL("/test-media/cover-1.jpg", baseURL).href },

A test that checks the cards exist passes either way, because a fallback card is still a card. You only see it by looking at the page and the network tab, which is the whole argument of this article in one bug.

Clean Up After Yourself and Only Yourself

Once agents can start servers and browsers, cleanup is part of the design. Otherwise every session leaves a layer behind: a stray node process, a Chromium window, screenshots, a Playwright trace, a temporary auth file, or a Docker container.

"Clean everything" is the wrong fix, though. pkill node isn't cleanup, it's an agent going on a small rampage, and it'll happily kill the dev server you started in another terminal. git clean -fd will delete an untracked file that has nothing to do with the agent.

The rule is to clean up what you created and leave everything else alone, and the git status before and after is there so browser QA doesn't quietly end up in my next commit.

This is why the startup step records a PID in a temp folder. Cleanup can then be precise:

# Not cleanup
pkill node
git clean -fd

# Cleanup
kill "$(cat "$QA_DIR/dev.pid")"
rm -rf "$QA_DIR"

And this is roughly what I expect to see at the end of a review:

Cleanup
- Stopped the dev server I started (pid 48213, pnpm dev --port 5173).
- Closed the Playwright MCP browser.
- Deleted export-dialog-375.png and export-dialog-1440.png.
- Left running: pid 39102 (pnpm dev on :3000). It was already running
  before I started, so it isn't mine.

git status --short before review:
 M src/features/export/ExportDialog.tsx
?? notes/todo.md

git status --short after cleanup:
 M src/features/export/ExportDialog.tsx
?? notes/todo.md

The two git status blocks match, so nothing from the review leaked into the working tree, and the untracked notes file that was already there is still there.

I didn't make this rule up in the abstract, either. While building the tool I'm about to describe, the agent testing it ran a cleanup command that closed every Playwright browser on my machine, not just its own. To be fair, it owned up to it in its summary.

When a Skill Stopped Being Enough

The skill worked, in the sense that agents started actually opening the app. But what I got back was still prose. The agent would tell me it opened the page, clicked through the flow and everything looked right, and I had to take its word for it.

It's the same problem I started with, just one level up. I didn't want a nicer sentence about the browser. I wanted the evidence: what was checked, what passed, what failed, and the screenshots and traces to back it up, in a form I could drop into a PR for whoever reviews it.

So I wrote a build plan and spent a Saturday building it, pairing with Claude Code through milestones and independent review rounds. It's called katalepsis, the Stoic word for grasping something so clearly you can't be wrong about it, which felt like a fair bar for a pull request.

You point it at a preview URL and it writes a report: HTML, Markdown you can paste into the PR, and a JSON manifest.

It does two things. katalepsis run runs your existing Playwright tests against the preview, and Playwright's result is final. katalepsis sorts, redacts and bundles it, and never reinterprets an assertion.

katalepsis verify takes plain-English checks for behavior nobody has written a test for, and a model drives a real browser with your own Anthropic, OpenAI, or OpenRouter key. There's also an MCP server, so the agent that just made a change can run either one itself and read the report back.

How to Try katalepsis

katalepsis isn't on npm yet, so the installer clones the repository and builds it. It needs Node 22.12 or newer, Git, and pnpm 10 (or corepack, which ships with Node):

curl -fsSL https://www.katalepsis.glamboyosa.xyz/install.sh | sh

That puts two launchers, katalepsis and katalepsis-mcp, in ~/.local/bin.

To run your existing tests against a preview, your Playwright config has to use the URL katalepsis passes in. It's also worth turning on failure evidence so the report has something to show:

// playwright.config.ts
use: {
  baseURL: process.env.KATALEPSIS_BASE_URL ?? "http://localhost:3000",
  screenshot: "only-on-failure",
  video: "retain-on-failure",
  trace: "retain-on-failure",
},

The repository includes a small demo app, a signup form with a deliberate bug: submitting the empty form does nothing instead of showing an error. It has three Playwright tests. Here's a real run against it:

$ katalepsis run --cwd apps/demo-form --url http://127.0.0.1:4173
katalepsis: FAILED
  2 passed, 1 failed, 0 blocked, 0 not tested
  Preview commit: not reported, so unconfirmed
  Report: artifacts/katalepsis/2026-10-19T08-21-25-288Z-3dba5a/report.html
  Markdown: artifacts/katalepsis/2026-10-19T08-21-25-288Z-3dba5a/report.md

The Markdown report is meant to be pasted into a PR. The top of it looks like this:

# katalepsis report: Failed

**2 passed, 1 failed, 0 blocked** out of 3 checks run.

- Preview: http://127.0.0.1:4173/
- Preview commit: not reported, so unconfirmed
- Runner: katalepsis 0.1.0, Playwright 1.63.0, Node 22.22.0

## Checks

| Status | Check | Project | Location | Duration |
| --- | --- | --- | --- | --- |
| Failed | signup form › empty submission shows a required-field error | chromium | signup.spec.ts:8 | 3.2 s |
| Passed | signup form › valid submission shows a confirmation | chromium | signup.spec.ts:13 | 418 ms |
| Passed | signup form › malformed email shows an email error and does not submit | chromium | signup.spec.ts:20 | 449 ms |

### Failed: signup form › empty submission shows a required-field error

- Failure type: assertion
- Evidence: screenshot, video, log, trace

Below that, it includes Playwright's own failure message (getByRole('alert') never appeared) and a list of limitations, such as the preview not reporting which commit it was built from.

For behavior without a test, you write checks in plain English. Each one has an intent, which is what to do, and a pass condition, which is what must be true at the end:

{
  "checks": [
    {
      "id": "empty-form",
      "name": "Empty form validation",
      "intent": "Submit the signup form without filling it",
      "passCondition": "A required-field error is visible"
    },
    {
      "id": "invalid-email",
      "name": "Invalid email",
      "intent": "Fill the name, enter an invalid email address and submit",
      "passCondition": "An email validation error is visible and the success confirmation is not shown"
    }
  ]
}

Then you give katalepsis a key and run them. The key is stored in your OS keychain or read from an environment variable, and it's never passed on the command line:

$ katalepsis auth set openrouter
$ katalepsis verify --cwd apps/demo-form --url http://127.0.0.1:4173 --checks checks.json \
    --provider openrouter --model anthropic/claude-haiku-5.5
katalepsis: FAILED (openrouter/anthropic/claude-haiku-5.5)
  2 passed, 1 failed, 0 blocked
  Preview commit: not reported, so unconfirmed
  Report: artifacts/katalepsis/2026-10-10T12-45-05-343Z-b51a0a/report.html

The demo's full checks.json has three checks, and the agent checks find the same empty-form bug the Playwright test did, without anyone having written that test.

Finally, so the coding agent can do all of this itself, you register the MCP server. With Codex, that's:

codex mcp add katalepsis -- katalepsis-mcp --root /path/to/your-app

The server exposes three tools, verification_run_tests, verification_check_browser, and verification_get_report, and only works inside the folder you give it. In practice the agent's instruction becomes something like "after changing the signup form, run verification_check_browser with a check for the change and paste the Markdown report into your summary."

How katalepsis Works Under the Hood

A few of the decisions came straight out of the problems in this article. Here's how they work in the code.

katalepsis is a TypeScript monorepo with five packages:

• core (the result schema, the Playwright runner, and the report renderers)

• agent (the browser tools and evidence rules for verify)

• credentials (keys and preferences)

• mcp (the MCP server)

• cli

It uses Effect for the parts where errors, cancellation, and dependencies matter, such as running Playwright and probing the preview, and Effect Schema for the report format. The model loop in verify uses the Vercel AI SDK, so the same code works with Anthropic, OpenAI, and OpenRouter.

The Report is a Schema, Not a Log

Every run, whether it came from Playwright or from a model, decodes into the same VerificationRun structure before anything is written. A trimmed version of a single check looks like this:

export const CheckStatus = Schema.Literals(["passed", "failed", "blocked"]);

export const VerificationCheck = Schema.Struct({
  id: Schema.String,
  source: Schema.Literals(["playwright-test", "agent"]),
  title: Schema.String,
  status: CheckStatus,
  flaky: Schema.Boolean,
  failure: Schema.NullOr(Schema.Struct({ code: FailureCode, message: Schema.String })),
  attempts: Schema.Array(Attempt),
  evidence: Schema.Array(BundlePath),
});

Tests that never ran go into a separate notTested list with a reason, which is how you end up with four outcomes: passed, failed, blocked, or not tested. Every evidence file is stored with its SHA-256 hash, and BundlePath rejects absolute paths and .., so a report can't point outside its own folder.

Blocked isn't Failed

If the preview is down, you get one blocked run, not three failed tests that send you hunting for bugs that don't exist. Before starting Playwright, katalepsis requests the preview URL once. This is the core of that probe:

const UNAVAILABLE_STATUSES = new Set([502, 503, 504]);

const response = await fetch(url, {
  redirect: "manual",
  signal: AbortSignal.any([signal, AbortSignal.timeout(30_000)]),
});
if (UNAVAILABLE_STATUSES.has(response.status)) {
  return { reachable: false, reason: `the preview answered with HTTP ${response.status}` };
}

A network error or a gateway status blocks every selected test, and Playwright never starts. Pointing the demo at a port with nothing on it gives you this:

$ katalepsis run --cwd apps/demo-form --url http://127.0.0.1:4999
katalepsis: BLOCKED
  0 passed, 0 failed, 3 blocked, 0 not tested
  preview-unavailable: http://127.0.0.1:4999/ did not respond (connect ECONNREFUSED 127.0.0.1:4999). Playwright was not run.

Playwright's Verdict is Final

run starts the project's own Playwright with --reporter=json and reads the result file, rather than reimplementing anything. Playwright already marks a test that passed on retry as flaky, so katalepsis keeps that flag and lists every attempt.

What it does add is classification. A failure whose message names the preview's own origin as unreachable becomes blocked, but a dead third-party API stays failed, because that's a real problem with your app. A browser that couldn't launch is blocked too, which is exactly what I got the first time I ran the demo for this article on a machine with the wrong browser build installed.

The Model Doesn't Get to Call Something a Pass

The model in verify can click and type and read the page, but it doesn't get to call something a pass. A check only passes if the model points to a browser assertion that katalepsis ran, that passed, and that shows something that wasn't on the page before the model acted. Otherwise it comes back blocked.

Here's how that works. The model only gets a small set of tools: navigate, snapshot, click, fill, press, assert, and finish. assert doesn't return the model's opinion. katalepsis runs the check in the browser itself, takes a screenshot, and records the result with an ID like a3. When the model calls finish, it has to cite those IDs:

finish: tool({
  description: "Report the verdict once, citing assertion IDs such as a1.",
  inputSchema: schema(
    Schema.Struct({
      verdict: Schema.Literals(["passed", "failed", "blocked"]),
      summary: Schema.String,
      assertionIds: Schema.Array(Schema.String),
    }),
  ),
  execute: (verdict) => {
    onFinish(verdict);
    return "Recorded.";
  },
}),

Then a plain function, decideOutcome, compares the claim with the evidence. A trimmed version of the pass rule:

if (!cited.every((assertion) => assertion.passed)) {
  return "The agent reported a pass, but a cited assertion failed.";
}
const shown = cited.filter((assertion) => assertion.expectation === "visible");
if (shown.length === 0) {
  return "The agent reported a pass citing only hidden or URL assertions.";
}
if (!observeOnly && shown.every((assertion) => assertion.trueAtStart === true)) {
  return "The agent reported a pass, but everything it cited was already on the page before it acted.";
}

Any of those messages turns the check into blocked with the code no-evidence. Citing an assertion ID that never ran is blocked too. The function also blocks a pass if an uncited assertion failed after the last cited one, because the latest evidence disagrees with the verdict.

That trueAtStart condition exists because one of the models I tested tried to prove a form worked by citing the label of the email field, which had been there all along. T

o catch it, each check opens two copies of the start page in separate browser contexts. The model works in one, and the other stays untouched. When the model asserts that something is visible, katalepsis looks for the same element on the untouched copy, and if it's there too, the assertion gets trueAtStart: true and can't carry a pass on its own.

One Action at a Time

The first live runs also caught parallel tool calls stepping on each other, and a name ending up typed into the email field. Models like to send several tool calls at once, and two fill calls racing each other type into whichever field has focus. The fix is a promise queue that every browser action goes through:

private exclusive<A>(action: () => Promise<A>): Promise<A> {
  const next = this.queue.then(action, action);
  this.queue = next.catch(() => undefined);
  return next;
}

Each action waits for the previous one to settle, whether it succeeded or failed, and a failure doesn't poison the queue for the next action.

Budgets and Boundaries

Each agent check runs in its own browser context with tracing on, so the report always has a trace to replay. Navigation is limited to the preview's origin plus anything you allow with --allow-origin. The model has a step limit (--max-steps, 15 by default), a token limit (--max-tokens, 200,000), and a time limit (--check-timeout, 120 seconds), and running out of any of them is reported as blocked with the reason, not as a failure. Query strings are redacted from URLs in the report.

How it Was Built

Before writing code, I wrote a build plan split into milestones. M0 was a review of how two of my earlier projects handled credentials and providers, plus the first architecture decision record and the monorepo tooling. M1 was the Playwright runner and the report, M2 was agent checks with your own key, and M3 was the MCP server. Each milestone ended with a fresh agent with no history reviewing the work against the plan, the same independent-reviewer rule from earlier in this article.

Those reviews shaped the tool more than the plan did. The first one came back with 15 findings, including the rule that only an unreachable preview counts as blocked while other hosts stay failed. The M2 review found that a model could get a pass by citing an assertion that proved nothing, which is where the stricter pass rule and the untouched copy of the page came from. The M3 review found that a client quitting mid-run left Playwright and its browsers running, and that a crafted symlink could let the report tool read files outside the project. All of those are fixed and covered by tests now.

What Happened When I Used It

Before telling anyone about it, I ran it against two of my own projects. Conclave's suite came back with 95 passed. A larger app came back with 137 passed and one failure, an assertion on a zoom level that was slightly off.

That was a test I already had and hadn't noticed was failing, so katalepsis didn't find the bug so much as make it impossible to miss. Both projects also hardcoded their baseURL, which meant their tests never touched the URL I passed in. katalepsis now warns about that, because a green run against the wrong server is worse than no run.

After I posted it on LinkedIn, someone in the comments pointed out that the commit you have checked out isn't necessarily the commit the preview was built from. They were right, so the report now records both and marks them confirmed, mismatched, or unknown, and a mismatch blocks a run that would otherwise pass, since a green run against a stale preview tells you nothing about your diff.

The preview reports its commit with an x-commit-sha response header or a <meta name="commit-sha"> tag, which the same probe that checks availability reads. If your host can't do either, you can pass --preview-commit yourself.

I'm dogfooding it in the same instructions this article describes. The browser-review skill now tells my agents to run katalepsis first, and to fall back to plain browser automation for what it can't check yet: field values, persistence, console and network, and layout across viewports.

The first Codex session to pick that up went looking for a "katalepsis skill" because of how I'd worded the line in AGENTS.md, didn't find one, and fell back to its old browser review. Wrong for a silly reason, but the fallback did exactly what it was supposed to, and the fix was saying plainly that katalepsis is a CLI and browser-review is the skill.

A few days later I used it on a real work branch, a change that made GeoJSON the main download button and moved CSV and KML into a dropdown. The agent stopped before running anything, because verify sends page content to the model provider and my rules say it needs my OK before doing that for an app. That's the behavior I want

With seeded synthetic data and an explicit OK, the check passed: one visible primary button, and the other formats showing up once the dropdown opened, with the screenshots and trace in the report. The skill still keeps pages with real data on the fallback path.

Where I Landed

Not every task needs any of this. I don't need Playwright to confirm a Markdown typo got fixed, and a backend refactor that can't change what renders doesn't need Chromium. I do want it for components, forms, navigation, dialogs, client state, maps, filtering, responsive layout, persistence, and any user-visible bug fix.

My rule of thumb is that if you're making a claim about how the UI behaves at runtime, the runtime UI should be part of the evidence.

None of this replaces good tests, code review, or me looking at changes that matter. It just adds the feedback loop that was missing. The agent already has the repository and the shell, and it knows what it changed. With a browser it can start answering whether the thing it built actually works, and with a report it can show me instead of telling me. If it can write the button, it can click it and show me what happened before it says it's done.

katalepsis is MIT licensed and on GitHub. If an agent check ever gives you a result you don't believe, I'd like to hear about it.

But they don't check whether the numbers on those elements are correct. A data-pipeline bug that shows Malaysia's population as 3.4 million instead of the real 34 million slips past every selector test in the suite.

The element still exists. A number still renders. The page still looks right. But the bug ships and sits there until a human notices.

I work as a full-stack engineer. Writing end-to-end (E2E) tests with Playwright and unit tests with Jest is part of my day job. I also use Playwright MCP, the bridge between AI assistants like Claude and a running browser, when I need to generate first-draft test code or debug a flow.

None of that tooling closes the maintenance tax on selector-based suites. Every E2E suite I keep alive at work accumulates data-testid selectors, waitForSelector calls, and tests that break because someone renamed a button.

Bug0's Breaking Apps Hackathon gave me a pretext to try something different. Over a weekend, I built an automated regression suite for Malaysia's three public open data portals, data.gov.my, OpenDOSM, and KKMNow, using Passmark, Bug0's open-source AI-driven Playwright library.

The tests are written in plain English. Two AI models verify each assertion. A third arbitrates disagreements.

What You'll Find Below:

• How to write an E2E test that checks whether a dashboard's numbers are correct, not only whether the page renders

• A specific assertion pattern (range-bounded KPIs) that catches an entire class of data-pipeline bug that selector tests miss, with working examples ready to copy

• A cross-field math assertion that takes one sentence in Passmark and around a hundred lines of code without it

• How Passmark's own failure explanations became my debugging loop (the single biggest shift in how I'll write E2E tests going forward)

• The real limits: a 14% cache-hit rate, a dependency on OpenRouter, and what two-model voting fails to catch

Table of Contents

• Why Malaysia's Open Data Portals?

• What Is Passmark?

• The Hero Spec: Range-Bounded Assertions

  • What Two-Model Voting Doesn't Catch

• Going Further: Cross-Field Math

• What I Found Across Three Runs

  • The Debugging Loop

  • The Two Specs That Still Fail Are the Most Interesting

• What It Cost, and Why Cache Rate Is Cost Rate

• The Pattern Worth Stealing

• Honest Verdict

• Resources

Why Malaysia's Open Data Portals?

The hackathon suggested targets like Vercel Commerce, Cal.com, and Hashnode. These all would've been solid picks.

But I wanted to test something local and closer to my day-to-day work instead. I also wanted a data-heavy site where the numbers on screen have to be accurate, as I work with numbers too on a daily basis.

Malaysia has three public open-data portals:

• data.gov.my, run by MAMPU, the government's digital transformation agency

• OpenDOSM, run by the Department of Statistics

• KKMNow, run by the Ministry of Health

They're public, no authentication required, with documented APIs. Seemed like a good fit for an automated test suite. The data on them is what Malaysians use every day, so accuracy isn't optional.

What Is Passmark?

Passmark is a Playwright library where the tests read like specs. Here's an example:

await runSteps({
  page,
  userFlow: "population dashboard smoke",
  steps: [
    { description: "Navigate to https://data.gov.my/dashboard/kawasanku" },
    {
      description: "Wait for the country-level Malaysia view to render",
      waitUntil: "A headline population number is visible",
    },
  ],
  assertions: [
    {
      assertion:
        "The page shows Malaysia's total population as a number greater than 20 million and less than 40 million",
    },
  ],
  test,
  expect,
});

There are no selectors, no data-testid, and no page.locator(). The assertion expresses what I care about, in the words I would use with a colleague.

On the first run, an AI agent drives the page and caches the resolved Playwright action to Redis. Every run after that replays at native Playwright speed with zero model calls.

When the UI changes and a cached action fails, the AI re-engages only for that step. Two assertion models (Claude and Gemini) vote. A third model arbitrates disagreements.

The Hero Spec: Range-Bounded Assertions

Range-bounded assertions were the first shape of test I wrote, and the one I came back to most across the suite.

The idea is straightforward: check that a number on the page falls inside a sensible range, not that a specific element exists.

The image below is the Playwright report from the population spec, with all four range-bounded assertions passing.

The range-bounded population test is the one that shows Passmark's real value.

Traditional Playwright asserts DOM structure. It confirms that an element with class kpi-total contains the text 34.2 million. That tells you the page rendered, not whether the number makes sense.

A bug that shows Malaysia's population as 3.42 million sails past any selector test. The DOM is correct. The number renders. Nothing breaks in the conventional sense.

Passmark reads the page, evaluates the claim, and fails because 3.42 million falls outside the sane range. Two models vote. A hallucination by one model alone produces no false pass.

What Two-Model Voting Doesn't Catch

Voting defends against one model misreading the page. It doesn't defend against both models misreading the page the same way. If Claude and Gemini both parse "32.4 million" as "3.24 million" because of the same unusual spacing in the DOM, they agree, they vote pass, and the bug ships.

The mitigation is assertion design. Write assertions that are hard to misread. A range check ("between 20 million and 40 million") is harder for a model to get wrong than a prose check ("roughly 34 million"). Numerical bounds leave less room for interpretation than adjectives. The more your assertion looks like a unit test written in English, the less room the models have to disagree.

Going Further: Cross-Field Math

Range-bounded assertions are a good first step. They catch "is this number in the right ballpark?" But they don't catch "do these numbers agree with each other?"

For that, you need cross-field math. If a dashboard shows a total population and a breakdown by gender, those two things are supposed to agree. Male plus female should equal total. Ethnicity breakdown percentages should sum to 100.

test("Cross-field math: sex breakdown sums to total population", async ({ page }) => {
  test.setTimeout(180_000);
  await runSteps({
    page,
    userFlow: "population sex breakdown consistency",
    steps: [
      { description: "Navigate to https://data.gov.my/dashboard/kawasanku" },
      {
        description: "Wait for the Malaysia country-level view with breakdown data",
        waitUntil:
          "A headline total population figure is visible and a breakdown by sex is shown on the page",
      },
    ],
    assertions: [
      {
        assertion:
          "The male and female population values shown on the page add up to approximately the headline total population, within a 5% margin",
      },
      {
        assertion:
          "Any percentage-based breakdowns visible on the page (by sex, age, or ethnicity) sum to approximately 100% within a 2 percentage-point margin",
      },
      {
        assertion: "No breakdown value is negative or greater than the headline total",
      },
    ],
    test,
    expect,
  });
});

Try writing that in vanilla Playwright. You need selectors for the headline number, selectors for the breakdown components, number parsing with a comma-aware regex, and a margin calculation. Seventy to a hundred lines of code to verify three invariants a primary school student would call obvious.

The Passmark version is one spec. I ran it against Kawasanku's live country view. All three assertions passed in 1.4 minutes. Passmark's annotation, verbatim:

"The headline total population figure 'Malaysia has a population of 32,447,385 people.' is visible, and 'Gender And Age Distribution' is shown, which implies a breakdown by sex (male, female) will be available."

Two models read the page, extract the numbers, do the arithmetic, and agree. When the dashboard changes layout in three months, the same assertion still works, because it never named a selector.

This is the class of test I want running against every dashboard product that I touch. Financial totals matching their line items. Percentages that sum to 100. Inventory counts equal to the sum of warehouse locations. This rarely gets checked today, because writing the check by hand outweighs the perceived value of running it.

What I Found Across Three Runs

Every passing spec after run 1 came from Passmark telling me, in plain English, why my assertion didn't match the page.

Here are three examples from run 1:

For dataset-detail.spec.ts, I asserted "an API usage snippet (curl or JS) is shown." knowingly that the page is using Python and I wanted to see what the result was. Passmark replied:

"The page contains API usage snippets, but they are specifically for Python using the requests library. There are no snippets provided in curl or JavaScript formats."

The page had snippets. I asked for the wrong languages. Fix: accept any language.

For dashboard-population.spec.ts, I asserted "a chart visualizing population by age or ethnicity is rendered." Passmark replied:

"The current page displays charts for vital statistics such as Live Births, Deaths, and Natural Increase over time, but there is no chart visualizing population specifically by age groups or ethnicity."

The charts are there. Not the slice I guessed. Fix: accept any chart about population.

For kkmnow/hospital-utilisation.spec.ts, I asserted a "headline bed-utilisation percentage." Passmark replied:

"While there are multiple bed-utilisation percentages listed in tables and rankings further down the page, there is no prominent, top-level headline KPI figure displaying the overall bed-utilisation percentage."

The numbers are there. I had asked for a layout the designers didn't build.

This is the killer feature: Passmark's failure messages aren't stack traces. They're explanations. The AI read the page, compared it against my words, and pointed me at the fix. Nothing like a selector-based test throwing TimeoutError: waiting for locator.

The Debugging Loop

Once I saw the pattern, the loop became my main technique. Here's the procedure:

1. Read the failure message word for word. Don't skim.

2. Trust it as a description of what is on the page. The AI has read the page. Your assertion has not.

3. Rewrite the assertion so it matches what's on the page. Broaden, narrow, or restate.

4. Run it again.

The discipline is to not argue with the tool. The page is what the page is. Your assertion is what is wrong. Every time I tried to "fix" the page (convinced my assertion was right and the site was broken), I lost some time. Every time I took the failure message at face value and rewrote, the test passed on the next run.

This is the one of the changes in how I'll write E2E tests going forward. The feedback loop is the tool. Every failed assertion is a draft of the correct one.

The Two Specs That Still Fail Are the Most Interesting

1. The two models disagreed and the arbiter call failed.

On catalogue-search.spec.ts, Claude voted fail (72% confidence) and Gemini voted pass (100% confidence) on the same assertion. I had written the assertion in a way that read two ways.

Passmark escalated to an arbiter model through OpenRouter. The call came back with a 504 from Cloudflare. The arbiter never ran. The suite failed the spec.

This is an honest limit, not a fluke. Any CI that runs Passmark depends on OpenRouter's availability. External gateway errors happen. My fix for the final run was a global retry wrapper around the OpenRouter call, and the 504 stopped being a problem in practice.

If you bring this to production CI, plan for retries and treat OpenRouter outages as a first-class failure mode in your runbook.

This failure taught me something about assertion design: my wording was ambiguous. Claude's reading was reasonable. Gemini's reading was reasonable. When you write tests in English, being precise about what you mean is part of writing a good test.

2. The wait condition fired too early.

On the KKMNow spec, I had waitUntil: "A utilisation metric is visible". The page showed the section label "Hospital Bed Utilisation (%)" before the numbers finished loading. The wait step saw the label, decided the condition was met, and moved on. By the time the numbers rendered, the test had run out of time. Once the page was fully loaded, the range assertions would have passed on content.

"The page displays multiple bed-utilisation percentages within the specified range (0% to 120%). For example, the ranked list shows Perlis at 93.1% and Melaka at 88.2%."

The lesson: your waitUntil wording needs the same care as your assertion wording. Both are read by AI. A vague wait is as bad as a vague assertion.

What It Cost, and Why Cache Rate Is Cost Rate

Each of the three runs took about 20 minutes on 13 specs with a single worker. The hackathon's pooled OpenRouter key covered the AI costs, so I have no personal dollar figure to report.

The more useful cost finding is what gets cached.

$ docker exec passmark-redis redis-cli DBSIZE
5

Five steps out of roughly 35 were cached across three runs. A 14% cache-hit rate. The Passmark README explains why:

Only steps that produced a single tool call get cached. Multi-step sequences are considered non-deterministic.

Most of my steps described multi-tool sequences. "Open the area selector and choose Selangor, then wait for navigation" becomes click, wait, verify. Those don't cache by design.

This matters for your budget. An 86% miss rate means 86% of your steps call a model on every run. The cost model is per-tool-call via OpenRouter.

To estimate your own bill: count non-atomic steps in your suite, multiply by your chosen model's per-call price at current OpenRouter rates, and the product is your recurring cost per run. Cache rate is cost rate.

The fix is authoring discipline. Split compound descriptions into atomic steps. Treat cache fill rate as a metric you track, not an implementation detail to ignore. A suite with 80% atomic steps costs a fifth of a suite with 14%.

The Pattern Worth Stealing

The idea here is bigger than Passmark.

Check that the numbers on your dashboards make sense. Most teams don't. They should.

A one-line assertion like "the headline number is between 20 million and 40 million" catches several classes of bug regular tests miss.

Here are four common ones:

• The data pipeline divided by the wrong thing, so the number on screen is ten times too small.

• A timezone bug made yesterday's total show up under tomorrow's date.

• The data never refreshed, so users are looking at last week's numbers.

• A locale flip swapped commas and decimals, so 1,234,567 is now reading as 1.234567.

Civic portals were my target. The pattern applies anywhere a dashboard shows numbers. Fintech reports, SaaS analytics, healthcare metrics, e-commerce admin panels. Any screen where a number is supposed to mean something.

Most of these numbers never get tested. Writing the check by hand is tedious. You need a selector to find the number, code to parse it, code to handle units, and a margin calculation. Fifty lines for one check. Nobody bothers.

You don't need Passmark to steal the idea. The same check works in plain Playwright with page.evaluate and number parsing. The Passmark version is just more efficient to write and readable by anyone on the team, not only engineers.

Honest Verdict

Passmark works. Across three runs I went from 4 of 13 passing to 12 of 13 without touching a selector, guided by the tool's own feedback.

Still, the caveats are real:

• On a cold cache, every step waits for a model. Budget more wall-clock time than a selector suite.

• In my suite only 14% of steps cached. The other 86% pays model cost on every run. Authoring discipline (atomic steps) is the difference between cents and dollars per run.

• Two-model voting doesn't protect against both models misreading the same way. Write assertions that are hard to misread.

• Every assertion depends on OpenRouter's availability. External gateway errors need a retry strategy before this runs in CI.

What stuck with me: Passmark didn't make me better at Playwright. It made me write tests I would have skipped otherwise.

What I imagine myself doing at work:

• Run a small nightly Passmark suite against the critical dashboards, focused on range and freshness checks.

• Keep traditional Playwright and Jest for everything that has to be fast and deterministic.

• Treat every Passmark failure message as a specification of the page, not an error to argue with.

Try this, even if you never touch Passmark. Pick a number on a dashboard you work with. Write a test that fails if the number is outside a sane range. See what breaks. That is the whole pattern and purpose of this article.

Resources

• Repo: github.com/LeeRenJie/passmark-hackathon

• Passmark: github.com/bug0inc/passmark

• Breaking Apps Hackathon: hashnode.com/hackathons/breaking-things

• Test targets: data.gov.my, OpenDOSM, KKMNow

In order to make our app a bit more performant, we will utilize the browser’s local storage to store already scraped data so that we don’t trigger new scraping requests every time the browser is refreshed.

Here’s what it will look like:

Table of Contents

• How to Spin Up the App with Vite

• How to Build the Server

• How to Build the Front End

• How to Deploy to render.com

• Conclusion

If you want to jump straight into the code, here 💁 is the GitHub repository with a detailed README 🙌, and here you can see the live demo.

Now if you’re ready, let’s go step by step and see how to build and the deploy the app.

First, let’s get everything set up and ready to go.

How to Spin Up the App with Vite

We will use the Vite build tool to quickly spin up a bare bones React application, equipped with TailwindCSS for styling. In order to do that, run this in your terminal app:

npm create vite@latest web-scraper -- --template react

And then install and configure TailwindCSS as follows:

npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init -p

Add the paths to all of your template files in your tailwind.config.js file like this:

/** @type {import('tailwindcss').Config} */
export default {
  content: [
    "./index.html",
    "./src/**/*.{js,ts,jsx,tsx}",
  ],
  theme: {
    extend: {},
  },
  plugins: [],
}

Now you should have a brand new React application with Tailwind installed and configured.

Let’s start our work with the server.

How to Build the Server

Since we are building a full stack application, the bare minimum we need to have in place is a server, a client, and an API. The API will live in the server world and the client app will call the endpoints it exposes in order to fetch the data we need to display on the front end.

Set Up the HTTP Server with Express.js

We are going to use the Express.js library to spin up an HTTP server that will handle our API requests. To do so, follow these steps:

First, install the necessary packages with this command:

npm install express cors playwright

Then create an empty server.js file in the project's root folder and add the following code:

import express from "express";
import { chromium } from "playwright";
import cors from "cors";
import { scrapeListings } from "./utils/scraper.js";

const app = express();
const PORT = 5001;

app.use(cors());

app.get("/scrape", async (req, res) => {
  let browser;
  try {
    browser = await chromium.launch();
    const listings = await scrapeListings({ browser, retryCount: 3 });
    res.json(listings);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

app.listen(PORT, () => {
  console.log(`Scraper server running on http://localhost:${PORT}`);
});

Before we continue with the scraper, let me first explain what we are doing here.

This is a pretty simple setup of an Express server that is exposing an endpoint called “scrape“. Our client-side application (the front end) can send GET requests to this endpoint and receive the data returned as a result.

What’s important here is the async callback function that we pass to the app.get method. This is where we call our scraping function within a try/catch block. It will return the scraped data or an error if something goes wrong.

The last few lines indicate that our server will listen on the specified PORT, which is set to 5001 here, and display a message in the terminal to show that the server is running.

What is Web Scraping?

Before diving into the code, I want to briefly explain web scraping if you’re unfamiliar with it. Web scraping involves automatically reading content from websites using a piece of software. This software is called a “web scraper“. In our case, the scraper is what’s in the scrapeListing function.

An essential part of the scraping process is finding something in the DOM tree of the target website that you can use to select the data we want to scrape. This something is known as a selector. Selectors can be different HTML elements, such as tags (h3, p, table) or attributes, like class names or IDs.

You can use various programming techniques or features of the programming language you’re using to create the scraper, aiming for better results when implementing the selecting part of the scraper.

In our case, we’re using [itemprop="itemListElement"] as the selector. But you might wonder, how did we figure this out and decide to use that? How do you know which selector to use?

This is where it gets tricky. You have to manually inspect the DOM tree of the target website and determine what would work best. That’s the case unless the site provides an API specifically designed for scrapers.

Here is how this looks like in practice. This is a screenshot from the Airbnb website:

Usually, you’ll need the information you’re scraping for some particular purpose, which means you’ll need to store it somewhere and then process it. This processing often involve some kind of visualization of the data. This is where our client application comes into play.

We will store the results of our scraping in the browser's local storage. Then, we will easily display those results in a grid layout using React and TailwindCSS. But before we get to all this, let’s go back to the code to understand how the scraping process is done.

Set Up Playwright

For the scraping functionality, we will use another library that has gotten pretty famous over the last few years: Playwright. It mainly serves as an e2e testing solution, but, as you’ll see now, we can use it for scraping the web as well.

We will put the scraping function in a separate file so that we have it all in order and keep the separation of concerns in place.

Create a new folder in the root directory and name it utils. Inside this folder, add a new file named scraper.js and include the following code:

const MAX_RETRIES = 3;

const validateListing = (listing) => {
  return (
    typeof listing.title === "string" &&
    typeof listing.price === "string" &&
    typeof listing.link === "string"
  );
};

export const scrapeListings = async ({ browser, retryCount }) => {
  try {
    const page = await browser.newPage();

    try {
      await page.goto("https://www.airbnb.com/", { waitUntil: "load" });

      await page.waitForSelector('[itemprop="itemListElement"]', {
        timeout: 10000,
      });

      const listings = await page.$$eval(
        '[itemprop="itemListElement"]',
        (elements) => {
          return elements.slice(0, 10).map((element) => {
            const title =
              element.querySelector(".t1jojoys")?.innerText || "N/A";
            const price =
              element.querySelector("._11jcbg2")?.innerText || "N/A";
            const link = element.querySelector("a")?.href || "N/A";
            return { title, price, link };
          });
        }
      );

      const validListings = listings.filter(validateListing);

      if (validListings.length === 0) {
        throw new Error("No listings found");
      }

      return validListings;
    } catch (pageError) {
      if (retryCount < MAX_RETRIES) {
        console.log(`Retrying... (${retryCount + 1}/${MAX_RETRIES})`);
        return await scrapeListings(retryCount + 1);
      } else {
        throw new Error(
          `Failed to scrape data after ${MAX_RETRIES} attempts: ${pageError.message}`
        );
      }
    } finally {
      await page.close();
    }
  } catch (browserError) {
    throw new Error(`Failed to launch browser: ${browserError.message}`);
  } finally {
    if (browser) {
      await browser.close();
    }
  }
};

Retry Mechanism

At the top of the file, there's a constant called MAX_RETRIES used to implement a retry mechanism. This tactic is often used by web scrapers to bypass or overcome protections that some websites have against scraping. We will see how to use it below.

It's important to mention the legal aspect here as well. Always respect the terms and conditions along with the privacy policy of the website you plan to scrape. Use these techniques only to handle technical challenges, not to break the law.

A small helper function follows that you can use to validate the received data. Nothing interesting here.

Next is the main scraping function. We’re passing the browser object, provided by Playwright, and the number of retry attempts as arguments to the function.

There are two try/catch blocks to handle possible failures: one for launching the browser (in headless mode, meaning you won't see anything) and one for the scraping process. In the latter, we’ll use Playwright's features to request the website, wait until the page is fully loaded, and then locate the selector we defined.

In the callback function we pass to $$eval, we access the elements returned by the scraping, allowing us to process them and get the data we want. In this case, I’m using three selectors to fetch the title, price, and link of the property. The first two are class names, and the last is the HTML tag <a>.

Then we return an object, { title, price, link }, with the fetched data, that is the values of the three properties. And in the end of the try part, we validate the results before returning them to the front end.

What follows in the catch part is the implementation of the retry mechanism we talked about a minute ago:

 } catch (pageError) {
      if (retryCount < MAX_RETRIES) {
        console.log(`Retrying... (${retryCount + 1}/${MAX_RETRIES})`);
        return await scrapeListings(retryCount + 1);
      } else {
        throw new Error(
          `Failed to scrape data after ${MAX_RETRIES} attempts: ${pageError.message}`
        );
      }
    }

If an error occurs during the reading process, we enter the catch phase and check if the retry count is below the maximum limit we set. If it is, we try again by running the function recursively. Otherwise, we throw an error indicating that the scraping failed and the maximum retry attempts have been reached.

That's all you need to set up basic web scraping of the Airbnb homepage.

You can see all this in the Github repo of the project so don’t worry if you missed something here.

How to Build the Front End

Now it's time to put the scraped data to use.

Let's display the last 10 properties in a grid layout, allowing you (or anyone) to open them by clicking on their links. We will also add a Refresh feature that lets you perform a new scrape to get the most up-to-date data.

This is how the structure of the front end part of the project looks:

We have a simple app structure: one main container (App.jsx) that holds all the components and includes some logic for making requests to the API and storing the data in local storage.

import { useEffect, useState } from "react";
import { useLocalStorage } from "@uidotdev/usehooks";
import axios from "axios";
import Footer from "./components/Footer";
import Header from "./components/Header";
import RefreshButton from "./components/RefreshButton";
import Grid from "./components/Grid";
import Loader from "./components/Loader";

function App() {
  const [listings, setListings] = useLocalStorage("properties", []);
  const [loading, setLoading] = useState(false);
  const [error, setError] = useState("");

  const fetchListings = async () => {
    setLoading(true);
    setError("");
    setListings([]);

    try {
      const response = await axios.get("http://localhost:5001/scrape");
      if (response.data.length === 0) {
        throw new Error("No listings found");
      }
      setListings(response.data);
    } catch (err) {
      setError(
        err.response?.data?.error ||
          "Failed to fetch listings. Please try again."
      );
    } finally {
      setLoading(false);
    }
  };

  useEffect(() => {
    if (listings.length === 0) {
      fetchListings();
    }
  }, []);

  return (
    <div className="flex flex-col items-center justify-center min-h-screen bg-gray-100">
      <Header />
      <RefreshButton callback={fetchListings} loading={loading} />
      <main className="flex flex-col items-center justify-center flex-1 w-full px-4 relative">
        {error && <p className="text-red-500">{error}</p>}
        {loading ? <Loader /> : <Grid listings={listings} />}
      </main>
      <Footer />
    </div>
  );
}

export default App;

All components are placed in the components directory (this is what I call a surprise, ah:)). Most of the components are quite simple, and I included them to give the app a more complete appearance.

The Header displays the top bar. The RefreshButton is used to send a new request and get the latest data. In the <main> section, we either show an error message if fetching fails, or we display a Loader component and a Grid component.

The loading part is straightforward. The Grid component is the interesting one. We pass the scraping results to it using a prop called 'listings'. Inside, we use a simple map() function to go through them and display the properties. We use Tailwind to style the grid, ensuring the properties are neatly listed and look good on both desktop and mobile screens.

And in the end with we have the Footer component that renders a simple bar with text. Again, I’ve added it just for completeness.

How to Deploy to render.com

Maybe a little over a year ago, I needed a place to deploy full-stack applications, ideally for free, since they were just for educational purposes.

After some research, I found a platform called Render and managed to deploy an app with both client and server parts, getting it to work online. I left it there until now. Since our scraper requires both parts to function properly, we will deploy it there and have it working online, as you can see here.

To do this, you need to create an account with Render and use their dashboard application. The process is simple, but I'll include a few screenshots below for clarity.

This is the Overview page where you can see all your projects:

Here is the Project page where you can view and manage your projects. In our case, we can see both the server and the client app as separate services.

You can click on each service to open its page, where you can view the deployments and the commits that triggered them. You can find even more details if you explore further.

You should be able to manage the deployment process on your own, as everything is clearly explained. But if you need help, feel free to reach out.

I should mention that I am not affiliated with Render in any way and I am not receiving any benefits for mentioning them here. I just found it to be a useful tool and wanted to share it with you – so I’ve used it here.

Conclusion

A web scraper app can be a powerful tool for gathering data, but there are several areas for improvement and important considerations to keep in mind here.

Firstly, you can enhance the app's performance and efficiency by optimizing the scraping process and ensuring that the data is stored and processed effectively. You can also implement more robust error handling and retry mechanisms to improve the reliability of the scraper.

Also, keep in mind that ethical scraping is crucial, and it's important to always respect the terms of service and privacy policies of the websites you are scraping. This includes not overloading the website with requests and ensuring that the data is used responsibly. Always seek permission from the site if required and consider using APIs provided by the website as a more ethical and reliable alternative.

Lastly, respecting the law is paramount. Ensure that your scraping activities comply with legal regulations and guidelines to avoid any potential legal issues. By focusing on these aspects, you can build a more effective, ethical, and legally compliant web scraper app.