But they don't check whether the numbers on those elements are correct. A data-pipeline bug that shows Malaysia's population as 3.4 million instead of the real 34 million slips past every selector test in the suite.

The element still exists. A number still renders. The page still looks right. But the bug ships and sits there until a human notices.

I work as a full-stack engineer. Writing end-to-end (E2E) tests with Playwright and unit tests with Jest is part of my day job. I also use Playwright MCP, the bridge between AI assistants like Claude and a running browser, when I need to generate first-draft test code or debug a flow.

None of that tooling closes the maintenance tax on selector-based suites. Every E2E suite I keep alive at work accumulates data-testid selectors, waitForSelector calls, and tests that break because someone renamed a button.

Bug0's Breaking Apps Hackathon gave me a pretext to try something different. Over a weekend, I built an automated regression suite for Malaysia's three public open data portals, data.gov.my, OpenDOSM, and KKMNow, using Passmark, Bug0's open-source AI-driven Playwright library.

The tests are written in plain English. Two AI models verify each assertion. A third arbitrates disagreements.

What You'll Find Below:

• How to write an E2E test that checks whether a dashboard's numbers are correct, not only whether the page renders

• A specific assertion pattern (range-bounded KPIs) that catches an entire class of data-pipeline bug that selector tests miss, with working examples ready to copy

• A cross-field math assertion that takes one sentence in Passmark and around a hundred lines of code without it

• How Passmark's own failure explanations became my debugging loop (the single biggest shift in how I'll write E2E tests going forward)

• The real limits: a 14% cache-hit rate, a dependency on OpenRouter, and what two-model voting fails to catch

Table of Contents

• Why Malaysia's Open Data Portals?

• What Is Passmark?

• The Hero Spec: Range-Bounded Assertions

  • What Two-Model Voting Doesn't Catch

• Going Further: Cross-Field Math

• What I Found Across Three Runs

  • The Debugging Loop

  • The Two Specs That Still Fail Are the Most Interesting

• What It Cost, and Why Cache Rate Is Cost Rate

• The Pattern Worth Stealing

• Honest Verdict

• Resources

Why Malaysia's Open Data Portals?

The hackathon suggested targets like Vercel Commerce, Cal.com, and Hashnode. These all would've been solid picks.

But I wanted to test something local and closer to my day-to-day work instead. I also wanted a data-heavy site where the numbers on screen have to be accurate, as I work with numbers too on a daily basis.

Malaysia has three public open-data portals:

• data.gov.my, run by MAMPU, the government's digital transformation agency

• OpenDOSM, run by the Department of Statistics

• KKMNow, run by the Ministry of Health

They're public, no authentication required, with documented APIs. Seemed like a good fit for an automated test suite. The data on them is what Malaysians use every day, so accuracy isn't optional.

What Is Passmark?

Passmark is a Playwright library where the tests read like specs. Here's an example:

await runSteps({
  page,
  userFlow: "population dashboard smoke",
  steps: [
    { description: "Navigate to https://data.gov.my/dashboard/kawasanku" },
    {
      description: "Wait for the country-level Malaysia view to render",
      waitUntil: "A headline population number is visible",
    },
  ],
  assertions: [
    {
      assertion:
        "The page shows Malaysia's total population as a number greater than 20 million and less than 40 million",
    },
  ],
  test,
  expect,
});

There are no selectors, no data-testid, and no page.locator(). The assertion expresses what I care about, in the words I would use with a colleague.

On the first run, an AI agent drives the page and caches the resolved Playwright action to Redis. Every run after that replays at native Playwright speed with zero model calls.

When the UI changes and a cached action fails, the AI re-engages only for that step. Two assertion models (Claude and Gemini) vote. A third model arbitrates disagreements.

The Hero Spec: Range-Bounded Assertions

Range-bounded assertions were the first shape of test I wrote, and the one I came back to most across the suite.

The idea is straightforward: check that a number on the page falls inside a sensible range, not that a specific element exists.

The image below is the Playwright report from the population spec, with all four range-bounded assertions passing.

The range-bounded population test is the one that shows Passmark's real value.

Traditional Playwright asserts DOM structure. It confirms that an element with class kpi-total contains the text 34.2 million. That tells you the page rendered, not whether the number makes sense.

A bug that shows Malaysia's population as 3.42 million sails past any selector test. The DOM is correct. The number renders. Nothing breaks in the conventional sense.

Passmark reads the page, evaluates the claim, and fails because 3.42 million falls outside the sane range. Two models vote. A hallucination by one model alone produces no false pass.

What Two-Model Voting Doesn't Catch

Voting defends against one model misreading the page. It doesn't defend against both models misreading the page the same way. If Claude and Gemini both parse "32.4 million" as "3.24 million" because of the same unusual spacing in the DOM, they agree, they vote pass, and the bug ships.

The mitigation is assertion design. Write assertions that are hard to misread. A range check ("between 20 million and 40 million") is harder for a model to get wrong than a prose check ("roughly 34 million"). Numerical bounds leave less room for interpretation than adjectives. The more your assertion looks like a unit test written in English, the less room the models have to disagree.

Going Further: Cross-Field Math

Range-bounded assertions are a good first step. They catch "is this number in the right ballpark?" But they don't catch "do these numbers agree with each other?"

For that, you need cross-field math. If a dashboard shows a total population and a breakdown by gender, those two things are supposed to agree. Male plus female should equal total. Ethnicity breakdown percentages should sum to 100.

test("Cross-field math: sex breakdown sums to total population", async ({ page }) => {
  test.setTimeout(180_000);
  await runSteps({
    page,
    userFlow: "population sex breakdown consistency",
    steps: [
      { description: "Navigate to https://data.gov.my/dashboard/kawasanku" },
      {
        description: "Wait for the Malaysia country-level view with breakdown data",
        waitUntil:
          "A headline total population figure is visible and a breakdown by sex is shown on the page",
      },
    ],
    assertions: [
      {
        assertion:
          "The male and female population values shown on the page add up to approximately the headline total population, within a 5% margin",
      },
      {
        assertion:
          "Any percentage-based breakdowns visible on the page (by sex, age, or ethnicity) sum to approximately 100% within a 2 percentage-point margin",
      },
      {
        assertion: "No breakdown value is negative or greater than the headline total",
      },
    ],
    test,
    expect,
  });
});

Try writing that in vanilla Playwright. You need selectors for the headline number, selectors for the breakdown components, number parsing with a comma-aware regex, and a margin calculation. Seventy to a hundred lines of code to verify three invariants a primary school student would call obvious.

The Passmark version is one spec. I ran it against Kawasanku's live country view. All three assertions passed in 1.4 minutes. Passmark's annotation, verbatim:

"The headline total population figure 'Malaysia has a population of 32,447,385 people.' is visible, and 'Gender And Age Distribution' is shown, which implies a breakdown by sex (male, female) will be available."

Two models read the page, extract the numbers, do the arithmetic, and agree. When the dashboard changes layout in three months, the same assertion still works, because it never named a selector.

This is the class of test I want running against every dashboard product that I touch. Financial totals matching their line items. Percentages that sum to 100. Inventory counts equal to the sum of warehouse locations. This rarely gets checked today, because writing the check by hand outweighs the perceived value of running it.

What I Found Across Three Runs

Every passing spec after run 1 came from Passmark telling me, in plain English, why my assertion didn't match the page.

Here are three examples from run 1:

For dataset-detail.spec.ts, I asserted "an API usage snippet (curl or JS) is shown." knowingly that the page is using Python and I wanted to see what the result was. Passmark replied:

"The page contains API usage snippets, but they are specifically for Python using the requests library. There are no snippets provided in curl or JavaScript formats."

The page had snippets. I asked for the wrong languages. Fix: accept any language.

For dashboard-population.spec.ts, I asserted "a chart visualizing population by age or ethnicity is rendered." Passmark replied:

"The current page displays charts for vital statistics such as Live Births, Deaths, and Natural Increase over time, but there is no chart visualizing population specifically by age groups or ethnicity."

The charts are there. Not the slice I guessed. Fix: accept any chart about population.

For kkmnow/hospital-utilisation.spec.ts, I asserted a "headline bed-utilisation percentage." Passmark replied:

"While there are multiple bed-utilisation percentages listed in tables and rankings further down the page, there is no prominent, top-level headline KPI figure displaying the overall bed-utilisation percentage."

The numbers are there. I had asked for a layout the designers didn't build.

This is the killer feature: Passmark's failure messages aren't stack traces. They're explanations. The AI read the page, compared it against my words, and pointed me at the fix. Nothing like a selector-based test throwing TimeoutError: waiting for locator.

The Debugging Loop

Once I saw the pattern, the loop became my main technique. Here's the procedure:

1. Read the failure message word for word. Don't skim.

2. Trust it as a description of what is on the page. The AI has read the page. Your assertion has not.

3. Rewrite the assertion so it matches what's on the page. Broaden, narrow, or restate.

4. Run it again.

The discipline is to not argue with the tool. The page is what the page is. Your assertion is what is wrong. Every time I tried to "fix" the page (convinced my assertion was right and the site was broken), I lost some time. Every time I took the failure message at face value and rewrote, the test passed on the next run.

This is the one of the changes in how I'll write E2E tests going forward. The feedback loop is the tool. Every failed assertion is a draft of the correct one.

The Two Specs That Still Fail Are the Most Interesting

1. The two models disagreed and the arbiter call failed.

On catalogue-search.spec.ts, Claude voted fail (72% confidence) and Gemini voted pass (100% confidence) on the same assertion. I had written the assertion in a way that read two ways.

Passmark escalated to an arbiter model through OpenRouter. The call came back with a 504 from Cloudflare. The arbiter never ran. The suite failed the spec.

This is an honest limit, not a fluke. Any CI that runs Passmark depends on OpenRouter's availability. External gateway errors happen. My fix for the final run was a global retry wrapper around the OpenRouter call, and the 504 stopped being a problem in practice.

If you bring this to production CI, plan for retries and treat OpenRouter outages as a first-class failure mode in your runbook.

This failure taught me something about assertion design: my wording was ambiguous. Claude's reading was reasonable. Gemini's reading was reasonable. When you write tests in English, being precise about what you mean is part of writing a good test.

2. The wait condition fired too early.

On the KKMNow spec, I had waitUntil: "A utilisation metric is visible". The page showed the section label "Hospital Bed Utilisation (%)" before the numbers finished loading. The wait step saw the label, decided the condition was met, and moved on. By the time the numbers rendered, the test had run out of time. Once the page was fully loaded, the range assertions would have passed on content.

"The page displays multiple bed-utilisation percentages within the specified range (0% to 120%). For example, the ranked list shows Perlis at 93.1% and Melaka at 88.2%."

The lesson: your waitUntil wording needs the same care as your assertion wording. Both are read by AI. A vague wait is as bad as a vague assertion.

What It Cost, and Why Cache Rate Is Cost Rate

Each of the three runs took about 20 minutes on 13 specs with a single worker. The hackathon's pooled OpenRouter key covered the AI costs, so I have no personal dollar figure to report.

The more useful cost finding is what gets cached.

$ docker exec passmark-redis redis-cli DBSIZE
5

Five steps out of roughly 35 were cached across three runs. A 14% cache-hit rate. The Passmark README explains why:

Only steps that produced a single tool call get cached. Multi-step sequences are considered non-deterministic.

Most of my steps described multi-tool sequences. "Open the area selector and choose Selangor, then wait for navigation" becomes click, wait, verify. Those don't cache by design.

This matters for your budget. An 86% miss rate means 86% of your steps call a model on every run. The cost model is per-tool-call via OpenRouter.

To estimate your own bill: count non-atomic steps in your suite, multiply by your chosen model's per-call price at current OpenRouter rates, and the product is your recurring cost per run. Cache rate is cost rate.

The fix is authoring discipline. Split compound descriptions into atomic steps. Treat cache fill rate as a metric you track, not an implementation detail to ignore. A suite with 80% atomic steps costs a fifth of a suite with 14%.

The Pattern Worth Stealing

The idea here is bigger than Passmark.

Check that the numbers on your dashboards make sense. Most teams don't. They should.

A one-line assertion like "the headline number is between 20 million and 40 million" catches several classes of bug regular tests miss.

Here are four common ones:

• The data pipeline divided by the wrong thing, so the number on screen is ten times too small.

• A timezone bug made yesterday's total show up under tomorrow's date.

• The data never refreshed, so users are looking at last week's numbers.

• A locale flip swapped commas and decimals, so 1,234,567 is now reading as 1.234567.

Civic portals were my target. The pattern applies anywhere a dashboard shows numbers. Fintech reports, SaaS analytics, healthcare metrics, e-commerce admin panels. Any screen where a number is supposed to mean something.

Most of these numbers never get tested. Writing the check by hand is tedious. You need a selector to find the number, code to parse it, code to handle units, and a margin calculation. Fifty lines for one check. Nobody bothers.

You don't need Passmark to steal the idea. The same check works in plain Playwright with page.evaluate and number parsing. The Passmark version is just more efficient to write and readable by anyone on the team, not only engineers.

Honest Verdict

Passmark works. Across three runs I went from 4 of 13 passing to 12 of 13 without touching a selector, guided by the tool's own feedback.

Still, the caveats are real:

• On a cold cache, every step waits for a model. Budget more wall-clock time than a selector suite.

• In my suite only 14% of steps cached. The other 86% pays model cost on every run. Authoring discipline (atomic steps) is the difference between cents and dollars per run.

• Two-model voting doesn't protect against both models misreading the same way. Write assertions that are hard to misread.

• Every assertion depends on OpenRouter's availability. External gateway errors need a retry strategy before this runs in CI.

What stuck with me: Passmark didn't make me better at Playwright. It made me write tests I would have skipped otherwise.

What I imagine myself doing at work:

• Run a small nightly Passmark suite against the critical dashboards, focused on range and freshness checks.

• Keep traditional Playwright and Jest for everything that has to be fast and deterministic.

• Treat every Passmark failure message as a specification of the page, not an error to argue with.

Try this, even if you never touch Passmark. Pick a number on a dashboard you work with. Write a test that fails if the number is outside a sane range. See what breaks. That is the whole pattern and purpose of this article.

Resources

• Repo: github.com/LeeRenJie/passmark-hackathon

• Passmark: github.com/bug0inc/passmark

• Breaking Apps Hackathon: hashnode.com/hackathons/breaking-things

• Test targets: data.gov.my, OpenDOSM, KKMNow

In order to make our app a bit more performant, we will utilize the browser’s local storage to store already scraped data so that we don’t trigger new scraping requests every time the browser is refreshed.

Here’s what it will look like:

Table of Contents

• How to Spin Up the App with Vite

• How to Build the Server

• How to Build the Front End

• How to Deploy to render.com

• Conclusion

If you want to jump straight into the code, here 💁 is the GitHub repository with a detailed README 🙌, and here you can see the live demo.

Now if you’re ready, let’s go step by step and see how to build and the deploy the app.

First, let’s get everything set up and ready to go.

How to Spin Up the App with Vite

We will use the Vite build tool to quickly spin up a bare bones React application, equipped with TailwindCSS for styling. In order to do that, run this in your terminal app:

npm create vite@latest web-scraper -- --template react

And then install and configure TailwindCSS as follows:

npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init -p

Add the paths to all of your template files in your tailwind.config.js file like this:

/** @type {import('tailwindcss').Config} */
export default {
  content: [
    "./index.html",
    "./src/**/*.{js,ts,jsx,tsx}",
  ],
  theme: {
    extend: {},
  },
  plugins: [],
}

Now you should have a brand new React application with Tailwind installed and configured.

Let’s start our work with the server.

How to Build the Server

Since we are building a full stack application, the bare minimum we need to have in place is a server, a client, and an API. The API will live in the server world and the client app will call the endpoints it exposes in order to fetch the data we need to display on the front end.

Set Up the HTTP Server with Express.js

We are going to use the Express.js library to spin up an HTTP server that will handle our API requests. To do so, follow these steps:

First, install the necessary packages with this command:

npm install express cors playwright

Then create an empty server.js file in the project's root folder and add the following code:

import express from "express";
import { chromium } from "playwright";
import cors from "cors";
import { scrapeListings } from "./utils/scraper.js";

const app = express();
const PORT = 5001;

app.use(cors());

app.get("/scrape", async (req, res) => {
  let browser;
  try {
    browser = await chromium.launch();
    const listings = await scrapeListings({ browser, retryCount: 3 });
    res.json(listings);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

app.listen(PORT, () => {
  console.log(`Scraper server running on http://localhost:${PORT}`);
});

Before we continue with the scraper, let me first explain what we are doing here.

This is a pretty simple setup of an Express server that is exposing an endpoint called “scrape“. Our client-side application (the front end) can send GET requests to this endpoint and receive the data returned as a result.

What’s important here is the async callback function that we pass to the app.get method. This is where we call our scraping function within a try/catch block. It will return the scraped data or an error if something goes wrong.

The last few lines indicate that our server will listen on the specified PORT, which is set to 5001 here, and display a message in the terminal to show that the server is running.

What is Web Scraping?

Before diving into the code, I want to briefly explain web scraping if you’re unfamiliar with it. Web scraping involves automatically reading content from websites using a piece of software. This software is called a “web scraper“. In our case, the scraper is what’s in the scrapeListing function.

An essential part of the scraping process is finding something in the DOM tree of the target website that you can use to select the data we want to scrape. This something is known as a selector. Selectors can be different HTML elements, such as tags (h3, p, table) or attributes, like class names or IDs.

You can use various programming techniques or features of the programming language you’re using to create the scraper, aiming for better results when implementing the selecting part of the scraper.

In our case, we’re using [itemprop="itemListElement"] as the selector. But you might wonder, how did we figure this out and decide to use that? How do you know which selector to use?

This is where it gets tricky. You have to manually inspect the DOM tree of the target website and determine what would work best. That’s the case unless the site provides an API specifically designed for scrapers.

Here is how this looks like in practice. This is a screenshot from the Airbnb website:

Usually, you’ll need the information you’re scraping for some particular purpose, which means you’ll need to store it somewhere and then process it. This processing often involve some kind of visualization of the data. This is where our client application comes into play.

We will store the results of our scraping in the browser's local storage. Then, we will easily display those results in a grid layout using React and TailwindCSS. But before we get to all this, let’s go back to the code to understand how the scraping process is done.

Set Up Playwright

For the scraping functionality, we will use another library that has gotten pretty famous over the last few years: Playwright. It mainly serves as an e2e testing solution, but, as you’ll see now, we can use it for scraping the web as well.

We will put the scraping function in a separate file so that we have it all in order and keep the separation of concerns in place.

Create a new folder in the root directory and name it utils. Inside this folder, add a new file named scraper.js and include the following code:

const MAX_RETRIES = 3;

const validateListing = (listing) => {
  return (
    typeof listing.title === "string" &&
    typeof listing.price === "string" &&
    typeof listing.link === "string"
  );
};

export const scrapeListings = async ({ browser, retryCount }) => {
  try {
    const page = await browser.newPage();

    try {
      await page.goto("https://www.airbnb.com/", { waitUntil: "load" });

      await page.waitForSelector('[itemprop="itemListElement"]', {
        timeout: 10000,
      });

      const listings = await page.$$eval(
        '[itemprop="itemListElement"]',
        (elements) => {
          return elements.slice(0, 10).map((element) => {
            const title =
              element.querySelector(".t1jojoys")?.innerText || "N/A";
            const price =
              element.querySelector("._11jcbg2")?.innerText || "N/A";
            const link = element.querySelector("a")?.href || "N/A";
            return { title, price, link };
          });
        }
      );

      const validListings = listings.filter(validateListing);

      if (validListings.length === 0) {
        throw new Error("No listings found");
      }

      return validListings;
    } catch (pageError) {
      if (retryCount < MAX_RETRIES) {
        console.log(`Retrying... (${retryCount + 1}/${MAX_RETRIES})`);
        return await scrapeListings(retryCount + 1);
      } else {
        throw new Error(
          `Failed to scrape data after ${MAX_RETRIES} attempts: ${pageError.message}`
        );
      }
    } finally {
      await page.close();
    }
  } catch (browserError) {
    throw new Error(`Failed to launch browser: ${browserError.message}`);
  } finally {
    if (browser) {
      await browser.close();
    }
  }
};

Retry Mechanism

At the top of the file, there's a constant called MAX_RETRIES used to implement a retry mechanism. This tactic is often used by web scrapers to bypass or overcome protections that some websites have against scraping. We will see how to use it below.

It's important to mention the legal aspect here as well. Always respect the terms and conditions along with the privacy policy of the website you plan to scrape. Use these techniques only to handle technical challenges, not to break the law.

A small helper function follows that you can use to validate the received data. Nothing interesting here.

Next is the main scraping function. We’re passing the browser object, provided by Playwright, and the number of retry attempts as arguments to the function.

There are two try/catch blocks to handle possible failures: one for launching the browser (in headless mode, meaning you won't see anything) and one for the scraping process. In the latter, we’ll use Playwright's features to request the website, wait until the page is fully loaded, and then locate the selector we defined.

In the callback function we pass to $$eval, we access the elements returned by the scraping, allowing us to process them and get the data we want. In this case, I’m using three selectors to fetch the title, price, and link of the property. The first two are class names, and the last is the HTML tag <a>.

Then we return an object, { title, price, link }, with the fetched data, that is the values of the three properties. And in the end of the try part, we validate the results before returning them to the front end.

What follows in the catch part is the implementation of the retry mechanism we talked about a minute ago:

 } catch (pageError) {
      if (retryCount < MAX_RETRIES) {
        console.log(`Retrying... (${retryCount + 1}/${MAX_RETRIES})`);
        return await scrapeListings(retryCount + 1);
      } else {
        throw new Error(
          `Failed to scrape data after ${MAX_RETRIES} attempts: ${pageError.message}`
        );
      }
    }

If an error occurs during the reading process, we enter the catch phase and check if the retry count is below the maximum limit we set. If it is, we try again by running the function recursively. Otherwise, we throw an error indicating that the scraping failed and the maximum retry attempts have been reached.

That's all you need to set up basic web scraping of the Airbnb homepage.

You can see all this in the Github repo of the project so don’t worry if you missed something here.

How to Build the Front End

Now it's time to put the scraped data to use.

Let's display the last 10 properties in a grid layout, allowing you (or anyone) to open them by clicking on their links. We will also add a Refresh feature that lets you perform a new scrape to get the most up-to-date data.

This is how the structure of the front end part of the project looks:

We have a simple app structure: one main container (App.jsx) that holds all the components and includes some logic for making requests to the API and storing the data in local storage.

import { useEffect, useState } from "react";
import { useLocalStorage } from "@uidotdev/usehooks";
import axios from "axios";
import Footer from "./components/Footer";
import Header from "./components/Header";
import RefreshButton from "./components/RefreshButton";
import Grid from "./components/Grid";
import Loader from "./components/Loader";

function App() {
  const [listings, setListings] = useLocalStorage("properties", []);
  const [loading, setLoading] = useState(false);
  const [error, setError] = useState("");

  const fetchListings = async () => {
    setLoading(true);
    setError("");
    setListings([]);

    try {
      const response = await axios.get("http://localhost:5001/scrape");
      if (response.data.length === 0) {
        throw new Error("No listings found");
      }
      setListings(response.data);
    } catch (err) {
      setError(
        err.response?.data?.error ||
          "Failed to fetch listings. Please try again."
      );
    } finally {
      setLoading(false);
    }
  };

  useEffect(() => {
    if (listings.length === 0) {
      fetchListings();
    }
  }, []);

  return (
    <div className="flex flex-col items-center justify-center min-h-screen bg-gray-100">
      <Header />
      <RefreshButton callback={fetchListings} loading={loading} />
      <main className="flex flex-col items-center justify-center flex-1 w-full px-4 relative">
        {error && <p className="text-red-500">{error}</p>}
        {loading ? <Loader /> : <Grid listings={listings} />}
      </main>
      <Footer />
    </div>
  );
}

export default App;

All components are placed in the components directory (this is what I call a surprise, ah:)). Most of the components are quite simple, and I included them to give the app a more complete appearance.

The Header displays the top bar. The RefreshButton is used to send a new request and get the latest data. In the <main> section, we either show an error message if fetching fails, or we display a Loader component and a Grid component.

The loading part is straightforward. The Grid component is the interesting one. We pass the scraping results to it using a prop called 'listings'. Inside, we use a simple map() function to go through them and display the properties. We use Tailwind to style the grid, ensuring the properties are neatly listed and look good on both desktop and mobile screens.

And in the end with we have the Footer component that renders a simple bar with text. Again, I’ve added it just for completeness.

How to Deploy to render.com

Maybe a little over a year ago, I needed a place to deploy full-stack applications, ideally for free, since they were just for educational purposes.

After some research, I found a platform called Render and managed to deploy an app with both client and server parts, getting it to work online. I left it there until now. Since our scraper requires both parts to function properly, we will deploy it there and have it working online, as you can see here.

To do this, you need to create an account with Render and use their dashboard application. The process is simple, but I'll include a few screenshots below for clarity.

This is the Overview page where you can see all your projects:

Here is the Project page where you can view and manage your projects. In our case, we can see both the server and the client app as separate services.

You can click on each service to open its page, where you can view the deployments and the commits that triggered them. You can find even more details if you explore further.

You should be able to manage the deployment process on your own, as everything is clearly explained. But if you need help, feel free to reach out.

I should mention that I am not affiliated with Render in any way and I am not receiving any benefits for mentioning them here. I just found it to be a useful tool and wanted to share it with you – so I’ve used it here.

Conclusion

A web scraper app can be a powerful tool for gathering data, but there are several areas for improvement and important considerations to keep in mind here.

Firstly, you can enhance the app's performance and efficiency by optimizing the scraping process and ensuring that the data is stored and processed effectively. You can also implement more robust error handling and retry mechanisms to improve the reliability of the scraper.

Also, keep in mind that ethical scraping is crucial, and it's important to always respect the terms of service and privacy policies of the websites you are scraping. This includes not overloading the website with requests and ensuring that the data is used responsibly. Always seek permission from the site if required and consider using APIs provided by the website as a more ethical and reliable alternative.

Lastly, respecting the law is paramount. Ensure that your scraping activities comply with legal regulations and guidelines to avoid any potential legal issues. By focusing on these aspects, you can build a more effective, ethical, and legally compliant web scraper app.