Amazon is the most popular e-commerce website for web scrapers, with billions of product pages being scraped every month.

It is also home to a vast database of product reviews, which can be very useful for market research and competitor monitoring.

You can extract relevant data from the Amazon website and save it in a spreadsheet or JSON format. And you can even automate the process to update the data regularly.

Scraping Amazon product reviews is not always straightforward, especially when a login is required. In this guide, you'll learn how to scrape Amazon product reviews behind a login. You’ll learn the process of logging in, parsing review data, and exporting reviews to CSV.

Important Disclaimer: This tutorial is for educational purposes only. Scraping data from behind logins on websites may violate their terms and conditions (T&Cs). It's crucial to always check the T&Cs of any website before scraping data.

Without further ado, let's get started.

Prerequisites and Project Setup

We’ll use the Node.js Puppeteer library to scrape Amazon reviews. Make sure Node.js is installed on your system. If it is not, go to the official Node.js website and install it.

After Node.js is installed, install Puppeteer. Puppeteer is a Node.js library that provides a high-level, user-friendly API for automating tasks and interacting with dynamic web pages.

Now, let's install and configure Puppeteer.

Open a terminal and create a new folder with any name. (In my case, it is _amazonreviews).

mkdir amazon_reviews

Change your current directory to the folder created above.

cd amazon_reviews

Cool, you're now in the correct directory. Execute the following command to initialize the package.json file:

npm init -y

Finally, install Puppeteer using the following command:

npm install puppeteer

This is what the process looks like:

Now, open the folder in any code editor, and create a new JavaScript file (index.js). Make sure that the hierarchy looks like this:

_Hierarchy showing node_modules, index.js, package-lock.json, and package.json_

All set up successfully. We’re now ready to code the scraper.

Note: Ensure that you have an account on Amazon so you can progress through the rest of this tutorial.

Step 1: Get Access to the Public Page

You're going to scrape the reviews of the product shown below. You’ll extract the author's name, review title, and date.

Here's the product URL: https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/

The product we're using in the example - headphones

First, you’ll log in to Amazon, and then redirect to the product URL to scrape the reviews.

Step 2: Scrape Behind the Login

Amazon's multi-stage login process requires users to enter their username or email, click a Continue button to enter their password, and then finally submit it. Both the username and password fields are typically on different pages.

To enter the email ID, use the selector input[name=email].

HTML of the sign-in field

Now, click on the Continue button using the selector input[id=continue].

HTML of the continue button

Now you should be on the next page. To enter the password, use the selector input[name=password].

HTML of the password field

Finally, click on the Sign In button using the selector input[id=signInSubmit].

HTML of the sign-in button

Here’s the code for the login process:

const selectors = {
  emailid: 'input[name=email]',
  password: 'input[name=password]',
  continue: 'input[id=continue]',
  singin: 'input[id=signInSubmit]',
};


    await page.goto(signinURL);
    await page.waitForSelector(selectors.emailid);
    await page.type(selectors.emailid, "satyam@gmail.com", { delay: 100 });
    await page.click(selectors.continue);
    await page.waitForSelector(selectors.password);
    await page.type(selectors.password, "mypassword", { delay: 100 });
    await page.click(selectors.singin);
    await page.waitForNavigation();

We're following the same steps as discussed above. First, go to the sign-in URL, enter the email ID, and click on the Continue button. Then enter the password, click on the Sign In button, and wait for a moment for the sign-in process to complete.

After the sign-in process is completed, you’ll be redirected to the product page to scrape the reviews.

Product page

Step 3: Parse the Review Data

You've successfully logged in and are now on the product page that you want to scrape. Let's now parse the review data.

On the page, you'll find various reviews. These reviews are contained within a parent div with the ID cm-cr-dp-review-list, which holds all the reviews on the current page. If you want to access more reviews, you'll need to navigate to the next page using the pagination process.

This parent div has multiple child divs, and each child div holds one review. To extract the reviews, you can use the selector #cm-cr-dp-review-list div.review.

const selectors = {
  allReviews: '#cm-cr-dp-review-list div.review',
  authorName: 'div[data-hook="genome-widget"] span.a-profile-name',
  reviewTitle: '[data-hook=review-title]>span:not([class])',
  reviewDate: 'span[data-hook=review-date]',
};

This selector shows that you first go to the element with the ID cm-cr-dp-review-list, then search for all div elements with the data-hook review.

Review data with Author name, Review Title, Description, etc.

The following code snippet shows that you should first go to the product URL, wait for the selector to load, and then scrape all the reviews and store them in the reviewElements variable.

await page.goto(productURL);
await page.waitForSelector(selectors.allReviews);
const reviewElements = await page.$$(selectors.allReviews);

Now, let's extract the author's name, review title, and date.

Targetting Author name, Review Title, and Date

To parse the author name, you can use the selector div[data-hook="genome-widget"] span.a-profile-name. This selector tells us to first search for the div element with the data-hook attribute set to genome-widget, because the names are inside this div element. Then, search for the span element with the class name a-profile-name. This is the element that contains the author's name.

const author = await reviewElement.$eval(selectors.authorName, (element) => element.textContent);

To parse the review title, you can use the CSS selector [data-hook="review-title"] > span:not([class]). This selector tells us to search for the span element that is a direct child of the [data-hook="review-title"] element and that does not have a class attribute.

const title = await reviewElement.$eval(selectors.reviewTitle, (element) => element.textContent);

To parse the date, you can use the CSS selector span[data-hook="review-date"]. This selector tells us to search for the span element that has the data-hook attribute set to review-date. This is the element that contains the review date.

const rawReviewDate = await reviewElement.$eval(selectors.reviewDate, (element) => element.textContent);

Note that you’ll get the entire text, including the location, instead of just the full date. Therefore, you must use a regular expression pattern to extract the date from the text.

After that, combine all of the data into the reviewData and then push it to the final list reviewsData.

const datePattern = /(\w+\s\d{1,2},\s\d{4})/;
      const match = rawReviewDate.match(datePattern);
      const reviewDate = match ? match[0].replace(',', '') : "Date not found";

      const reviewData = {
        author,
        title,
        reviewDate,
      };

      reviewsData.push(reviewData);
    }

The above process will run until it has parsed all of the reviews on the current page. Here’s the code snippet to parse the data:

for (const reviewElement of reviewElements) {
      const author = await reviewElement.$eval(selectors.authorName, (element) => element.textContent);
      const title = await reviewElement.$eval(selectors.reviewTitle, (element) => element.textContent);
      const rawReviewDate = await reviewElement.$eval(selectors.reviewDate, (element) => element.textContent);

      const datePattern = /(\w+\s\d{1,2},\s\d{4})/;
      const match = rawReviewDate.match(datePattern);
      const reviewDate = match ? match[0].replace(',', '') : "Date not found";

      const reviewData = {
        author,
        title,
        reviewDate,
      };

      reviewsData.push(reviewData);
    }

Great! You’ve successfully parsed the relevant data, which is now in JSON format, as shown below:

Scraped the data in JSON format

Step 4: Export Reviews to a CSV

You've parsed the reviews in JSON format, which is a bit human-readable. You can convert this data to CSV format to make it more readable and easier for other purposes.

There are many ways to convert JSON data to CSV, but we'll use a simple and effective approach. Here is a simple code snippet to convert JSON to CSV:

let csvContent = "Author,Title,Date\n
for (const review of reviewsData) {
      const { author, title, reviewDate } = review;
      csvContent += `${author},"${title}",${reviewDate}\n`;
    }

const csvFileName = "amazon_reviews.csv";
await fs.writeFileSync(csvFileName, csvContent, "utf8");

Here’s the output of the CSV file.

Converted JSON data into CSV format

And there you have it!

You can find the full Code uploaded on GitHub here.

Conclusion

In this guide, you learned how to scrape Amazon product reviews behind a login using Puppeteer. You learned how to log in, parse relevant data, and save it to a CSV file.

To practice more, you can extract all the reviews of all the pages using pagination.

Using scripts, we can extract the data we need from a website for various purposes, such as creating databases, doing some analytics, and even more.

Disclaimer: Be careful when doing web scraping. Always make sure you're scraping sites that allow it, and performing this activity within ethical and legal limits.

JavaScript and Node.js offers various libraries that make web scraping easier. For simple data extraction, you can use Axios to fetch an API responses or a website HTML.

But if you're looking to do more advanced tasks including automations, you'll need libraries such as Puppeteer, Cheerio, or Nightmare (don't worry the name is nightmare, but it's not that bad to use 😆).

I'll introduce the basics of web scraping in JavaScript and Node.js using Puppeteer in this article. I structured the writing to show you some basics of fetching information on a website and clicking a button (for example, moving to the next page).

At the end of this introduction, I'll recommend ways to practice and learn more by improving the project we just created.

Prerequisites

Before diving in and scraping our first page together using JavaScript, Node.js, and the HTML DOM, I'd recommend having a basic understanding of these technologies. It'll improve your learning and understanding of the topic.

Let's dive in! 🤿

How to Initialize Your First Puppeteer Scraper

New project...new folder! First, create the first-puppeteer-scraper-example folder on your computer. It'll contain the code of our future scraper.

mkdir first-puppeteer-scraper-example

Now, it's time to initialize your Node.js repository with a package.json file. It's helpful to add information to the repository and NPM packages, such as the Puppeteer library.

npm init -y

After typing this command, you should find this package.json file in your repository tree.

{
  "name": "first-puppeteer-scraper-example",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "puppeteer": "^19.6.2"
  },
  "type": "module",
  "devDependencies": {},
  "description": ""
}

Before proceeding, we must ensure the project is configured to handle ES6 features. To do so, you can add the "types": "module" instruction at the end of the configuration.

{
  "name": "first-puppeteer-scraper-example",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "puppeteer": "^19.6.2"
  },
  "type": "module",
  "description": "",
  "types": "module"
}

The last step of our scraper initialization is to install the Puppeteer library. Here's how:

npm install puppeteer

Wow! We're there – we're ready to scrape our first website together. 🤩

How to Scrape Your First Piece of Data

In this article, we'll use the ToScrape website as our learning platform. This online sandbox provides two projects specifically designed for web scraping, making it a great starting point to learn the basics such as data extraction and page navigation.

For this beginner's introduction, we'll specifically focus on the Quotes to Scrape website.

How to Initialize the Script

In the project repository root, you can create an index.js file. This will be our application entry point.

To keep it simple, our script consists of one function in charge of getting the website's quotes (getQuotes).

In the function's body, we will need to follow different steps:

• Start a Puppeteer session with puppeteer.launch (it'll instantiate a browser variable that we'll use for manipulating the browser)
• Open a new page/tab with browser.newPage (it'll instantiate a page variable that we'll use for manipulating the page)
• Change the URL of our new page to http://quotes.toscrape.com/ with page.goto

Here's the commented version of the initial script:

import puppeteer from "puppeteer";

const getQuotes = async () => {
  // Start a Puppeteer session with:
  // - a visible browser (`headless: false` - easier to debug because you'll see the browser in action)
  // - no default viewport (`defaultViewport: null` - website page will in full width and height)
  const browser = await puppeteer.launch({
    headless: false,
    defaultViewport: null,
  });

  // Open a new page
  const page = await browser.newPage();

  // On this new page:
  // - open the "http://quotes.toscrape.com/" website
  // - wait until the dom content is loaded (HTML is ready)
  await page.goto("http://quotes.toscrape.com/", {
    waitUntil: "domcontentloaded",
  });
};

// Start the scraping
getQuotes();

What do you think of running our scraper and seeing the output? Let's do it with the command below:

node index.js

After doing this, you should have a brand new browser application started with a new page and the website Quotes to Scrape loaded onto it. Magic, isn't it? 🪄

Quotes to Scrape homepage loaded by our initial script

Note: For this first iteration, we're not closing the browser. This means you will need to close the browser to stop the running application.

How to Fetch the First Quote

Whenever you want to scrape a website, you'll have to play with the HTML DOM. What I recommend is to inspect the page and start navigating the different elements to find what you need.

In our case, we'll follow the baby step principle and start fetching the first quote, author, and text.

After browsing the page HTML, we can notice a quote is encapsulated in a <div> element with a class name quote (class="quote"). This is important information because the scraping works with CSS selectors (for example, .quote).

Browser inspector with the first quote <div> selected

An example of how each quote is rendered in the HTML

Now that we have this knowledge, we can return to our getQuotes function and improve our code to select the first quote and extract its data.

We will need to add the following after the page.goto instruction:

• Extract data from our page HTML with page.evaluate (it'll execute the function passed as a parameter in the page context and returns the result)
• Get the quote HTML node with document.querySelector (it'll fetch the first <div> with the classname quote and returns it)
• Get the quote text and author from the previously extracted quote HTML node with quote.querySelector (it'll extract the elements with the classname text and author under <div class="quote"> and returns them)

Here's the updated version with detailed comments:

import puppeteer from "puppeteer";

const getQuotes = async () => {
  // Start a Puppeteer session with:
  // - a visible browser (`headless: false` - easier to debug because you'll see the browser in action)
  // - no default viewport (`defaultViewport: null` - website page will in full width and height)
  const browser = await puppeteer.launch({
    headless: false,
    defaultViewport: null,
  });

  // Open a new page
  const page = await browser.newPage();

  // On this new page:
  // - open the "http://quotes.toscrape.com/" website
  // - wait until the dom content is loaded (HTML is ready)
  await page.goto("http://quotes.toscrape.com/", {
    waitUntil: "domcontentloaded",
  });

  // Get page data
  const quotes = await page.evaluate(() => {
    // Fetch the first element with class "quote"
    const quote = document.querySelector(".quote");

    // Fetch the sub-elements from the previously fetched quote element
    // Get the displayed text and return it (`.innerText`)
    const text = quote.querySelector(".text").innerText;
    const author = quote.querySelector(".author").innerText;

    return { text, author };
  });

  // Display the quotes
  console.log(quotes);

  // Close the browser
  await browser.close();
};

// Start the scraping
getQuotes();

Something interesting to point out is that the function name for selecting an element is the same as in the browser inspect. Here's an example:

After running the document.querySelector instruction in the browser inspector, we have the first quote as an output (like on Puppeteer)

Let's run our script one more time and see what we have as an output:

{
  text: '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',
  author: 'Albert Einstein'
}

We did it! Our first scraped element is here, right in the terminal. Now, let's expand it and fetch all the current page quotes. 🔥

How to Fetch All Current Page Quotes

Now that we know how to fetch one quote, let's trick our code a bit to get all the quotes and extract their data one by one.

Previously we used document.getQuerySelector to select the first matching element (the first quote). To be able to fetch all quotes, we will need the document.querySelectorAll function instead.

We'll need to follow these steps to make it work:

• Replace document.getQuerySelector with document.querySelectorAll (it'll fetch all <div> elements with the classname quote and return them)
• Convert the fetched elements to a list with Array.from(quoteList) (it'll ensure the list of quotes is iterable)
• Move our previous code to get the quote text and author inside the loop and return the result (it'll extract the elements with the classname text and author under <div class="quote"> for each quote)

Here's the code update:

import puppeteer from "puppeteer";

const getQuotes = async () => {
  // Start a Puppeteer session with:
  // - a visible browser (`headless: false` - easier to debug because you'll see the browser in action)
  // - no default viewport (`defaultViewport: null` - website page will be in full width and height)
  const browser = await puppeteer.launch({
    headless: false,
    defaultViewport: null,
  });

  // Open a new page
  const page = await browser.newPage();

  // On this new page:
  // - open the "http://quotes.toscrape.com/" website
  // - wait until the dom content is loaded (HTML is ready)
  await page.goto("http://quotes.toscrape.com/", {
    waitUntil: "domcontentloaded",
  });

  // Get page data
  const quotes = await page.evaluate(() => {
    // Fetch the first element with class "quote"
    // Get the displayed text and returns it
    const quoteList = document.querySelectorAll(".quote");

    // Convert the quoteList to an iterable array
    // For each quote fetch the text and author
    return Array.from(quoteList).map((quote) => {
      // Fetch the sub-elements from the previously fetched quote element
      // Get the displayed text and return it (`.innerText`)
      const text = quote.querySelector(".text").innerText;
      const author = quote.querySelector(".author").innerText;

      return { text, author };
    });
  });

  // Display the quotes
  console.log(quotes);

  // Close the browser
  await browser.close();
};

// Start the scraping
getQuotes();

As an end result, if we run our script one more time, we should have a list of quotes as an output. Each element of this list should have a text and an author property.

[
  {
    text: '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',
    author: 'Albert Einstein'
  },
  {
    text: '“It is our choices, Harry, that show what we truly are, far more than our abilities.”',
    author: 'J.K. Rowling'
  },
  {
    text: '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”',
    author: 'Albert Einstein'
  },
  {
    text: '“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”',
    author: 'Jane Austen'
  },
  {
    text: "“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”",
    author: 'Marilyn Monroe'
  },
  {
    text: '“Try not to become a man of success. Rather become a man of value.”',
    author: 'Albert Einstein'
  },
  {
    text: '“It is better to be hated for what you are than to be loved for what you are not.”',
    author: 'André Gide'
  },
  {
    text: "“I have not failed. I've just found 10,000 ways that won't work.”",
    author: 'Thomas A. Edison'
  },
  {
    text: "“A woman is like a tea bag; you never know how strong it is until it's in hot water.”",
    author: 'Eleanor Roosevelt'
  },
  {
    text: '“A day without sunshine is like, you know, night.”',
    author: 'Steve Martin'
  }
]

Good job! All the quotes from the first page are now scraped by our script. 👏

How to Move to the Next Page

Our script is now able to fetch all the quotes for one page. What would be interesting is clicking on the "Next page" at the page bottom and doing the same on the second page.

"Next" button at the Quotes to Scrape page bottom

Back to our browser inspect, and let's find how we can target this element using CSS selectors.

As we can notice, the next button is placed under an unordered list <ul> with a pager classname (<ul class="pager">). This list has an element <li> with a next classname (<li class="next">). Finally, there is a link anchor <a> that links to the second page (<a href="/page/2/">).

In CSS, if we want to target this specific link there are different ways to do that. We can do:

• .next > a: but, it's risky because if there is an other element with .next as a parent element containing a link, it'll click on it.
• .pager > .next > a: safer, because we make sure the link should be inside the .pager parent element under the .next element. There is a low risk of having this hierarchy more than once.

An example of how the "Next" button is rendered in the HTML

To click this button, at the end of our script after the console.log(quotes);, you can add the following: await page.click(".pager > .next > a");.

Since we're now closing the browser page with await browser.close(); after all instructions are done, you need to comment on this instruction to see the second page opened in the scraper browser.

It's temporary and for testing purposes, but the end of our getQuotes function should look like this:

  // Display the quotes
  console.log(quotes);

  // Click on the "Next page" button
  await page.click(".pager > .next > a");

  // Close the browser
  // await browser.close();

After this, if you run our scraper again, after processing all instructions, your browser should stop on the second page:

Quotes to Scrape second page loaded after clicking the "Next" button

It’s Your Time! Here’s What You Can Do Next:

Congrats on reaching the end of this introduction to scraping with Puppeteer! 👏

Now it's your turn to improve the scraper and make it get more data from the Quotes to Scrape website. Here's a list of potential improvements you can make:

• Navigate between all pages using the "Next" button and fetch the quotes on all the pages.
• Fetch the quote's tags (each quote has a list of tags).
• Scrape the author's about page (by clicking on the author's name on each quote).
• Categorize the quotes by tags or authors (it's not 100% related to the scraping itself, but that can be a good improvement).

Feel free to be creative and do any other things you see fit 🚀

Scraper Code Is Available on GitHub

Check out the latest version of our scraper on GitHub! You're free to save, fork, or utilize it as you see fit.

=> First Puppeteer Scraper (example)

Successful Scraping Start: Thanks for reading the article!

I hope this article gave you a valuable introduction to web scraping using JavaScript and Puppeteer. Writing this was a pleasure, and I hope you found it informative and enjoyable.

Join me on Twitter for more content like this. I regularly share content to help you grow your web development skills and would love to have you join the conversation. Let's learn, grow, and inspire each other along the way!

In this guide, we'll explore the basics of using Puppeteer with Node.js so you can start automating your tests.

Prerequisites

• Basic understanding of Node.js
• Basic understanding of Puppeteer
• A suitable IDE such as VS Code

What You'll Learn

• What is puppeteer?
• What is Node.js?
• How to set up your first test with Puppeteer
• How to run headless Chrome tests on a CI server

What is Puppeteer?

Puppeteer is a Node.js library developed by Google that lets you control headless Chrome through the DevTools Protocol.

It is a tool for automating testing in your application using headless Chrome or Chromebit devices, without requiring any browser extensions like Selenium Webdriver or PhantomJS.

Puppeteer lets you automate the testing of your web applications. With it, you can run tests in the browser and then see the results in real-time on your terminal.

Puppeteer uses the WebDriver protocol to connect with the browser and simulate user interaction with HTML elements or pages.

What is Node.js?

Node.js is an open-source JavaScript runtime built on Chrome's V8 engine that runs on Linux, Mac OS X, and Windows operating systems. It was first released in 2009 by Ryan Dahl who was one of its original contributors (with some help from Douglas Crockford).

Node.js has become immensely popular over the years as an essential part of many software development projects. It has extensive capabilities when it comes down to coding certain tasks like server-side applications or peer-to-peer networking protocols like Websockets.

How to Set Up Node.js and Puppeteer

First make a directory which you will be working with by right clicking on your preferred location and choosing new folder. You can also use the command mkdir dir-name in your terminal.

Then create an app.js file in your folder and add the node.js code as shown below:

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto('https://www.freecodecamp.org/');

    await browser.close();
})();

The code above creates an instance of the browser which lets Puppeteer launch. Let's make sure we understand the code above:

• browser.newPage() creates new page
• page.goto() provides the URL to browser.newPage()
• browser.close() closes the running process

Now open your terminal and cd into the folder. Then run npm init to create a package.json file.

Press enter then type yes if asked 'is this ok'.

Your output will look like this:

package.json

Follow the setup instructions to install the dependencies that we will use in our project.

How to Set Up Your First Test with Puppeteer

To use Puppeteer with Node.js, you'll need to install several packages and set up a few environment variables. This part will walk you through the steps you'll need to follow to use Puppeteer in your tests:

• Download and install Node.js
• Install Puppeteer
• Install Mocha
• Install Chai
• Install Selenium Webdriver

You only need to complete the last step if you want to run tests on an actual browser instead of just testing against web driver scripts.

If this is your case, then go ahead and install the selenium-web driver module from the npm package manager by typing npm i selenium-webdriver --save.

Installing the dependencies will generate node_modules and a package-lock.json file as shown below:

package-lock.json

Screenshots are a great way to capture information in your browser. Well, Puppeteer has got you covered!

To take a screenshot of the webpage you navigated to, add the code snippet below:

  await page.screenshot({path: 'example.png'});

To run the application:

cd puppeter-tut
cd src

Then type the command below in your terminal:

node app.js

You can also create a PDF by adding the following snippet in your code:

    await page.pdf({ path: 'example.pdf' });

The above code snippet will give us the output shown below:

How to Test Your Setup

To test your setup, create a test folder in your code, then add example.test.js.

Your file should contain the following code:

const puppeteer = require('puppeteer')

describe("My first Setup Testing",()=>{
     it("Home landing page",async()=>{
    const browser = await puppeteer.launch({headless:false})
     });
});

Run your test using npm run test. After running your test you will get the following output:

Here's a GitHub link to the tutorial's source-code.

How to Use One Browser Instance with Puppeteer

As a web developer, you can use Puppeteer to run scripts in the headless Chrome browser and access the window object. This is useful when testing apps that need access to web resources like localStorage or cookies.

To use one browser instance with Puppeteer, you just need to pass { headless: false } to the launch method. It's asynchronous so it won't block the main thread and make your application unresponsive.

The best thing about this method is that, once it's launched, it should only be used once. Otherwise you will get an error when trying to access any webpage from Puppeteer again.

Here's an example:

let browser; (async() => { if(!browser) browser = await puppeteer.launch({headless: false});

Conclusion

So, there you have it! Now you know how to get started with Puppeteer and Node.js.

I hope this guide has helped you become more familiar with the tool and its capabilities. Feel free to get in touch with me if you have any questions or suggestions.

‌‌

It often happens that you come across a website and are forced to perform a set of actions to finally get some data. You are then faced with a dilemma: how do you make this data available in a form which can easily be consumed by your application?

Scraping comes to the rescue in such a case. And selecting the right tool for the job is quite important.

API is just a way to look at a Website after all (Source: XKCD Comics)

Puppeteer: Not Just Another Scraping Library

Puppeteer is a Node.js library maintained by the Chrome Devtools Team at Google. It basically runs a Chromium or Chrome (perhaps the more recognizable name) instance in a headless (or configurable) manner and exposes a set of high-level APIs.

From its official documentation, puppeteer is normally leveraged for multiple processes which are not limited to the following:

• Generating screenshots and PDFs
• Crawling an SPA and generating pre-rendered content (i.e. Server Side Rendering)
• Testing Chrome extensions
• Automation testing of Web Interfaces
• Diagnosis of performance issues through techniques like capturing the timeline trace of a website

For our case, we need to be able to access a website and map the data in a form which can be easily consumed by our application.

Sounds simple? The implementation is not that complex, either. Let's start.

Stringing the Code Along

My fondness for Amazon products prompts me to use one of their product listing page as a sample here. We will implement our use case in two steps:

• Extract data from the page and map it in an easily consumable JSON form
• Add a little sprinkle of automation to make our lives a little bit easier

You can find the complete code in this repository.

We will be extracting the data from this link: https://www.amazon.in/s?k=Shirts&ref=nb_sb_noss_2 ( a listing of the top searched shirts as shown in the image) in an API servable form.

Amazon India - Shirts Listing Page

Before we get started using puppeteer extensively in this section, we need to understand the two primary classes provided by it.

• Browser: launches a Chrome instance when we use puppeteer.launch or puppeteer.connect . This works as a simple browser emulation.
• Page: resembles a single tab on a Chrome browser. It provides an exhaustive set of methods you can use with a particular page instance and is invoked when we call browser.newPage. Just like you can create multiple tabs in the browser, you can similarly create multiple page instances at a single time in puppeteer.

Setting Up Puppeteer and Navigating to the Target URL

We start setting up puppeteer by using the npm module provided. After installing puppeteer, we create an instance of the browser and the page class and navigate to the target URL.

const puppeteer = require('puppeteer');

const url = 'https://www.amazon.in/s?k=Shirts&ref=nb_sb_noss_2';

async function fetchProductList(url) {
    const browser = await puppeteer.launch({ 
        headless: true, // false: enables one to view the Chrome instance in action
        defaultViewport: null, // (optional) useful only in non-headless mode
    });
    const page = await browser.newPage();
    await page.goto(url, { waitUntil: 'networkidle2' });
    ...
}

fetchProductList(url);

We use networkidle2 as the value for the waitUntil option while navigating to the URL. This ensures that the page load state is considered final when it has no more than 2 connections running for at least 500ms.

Note: You do not need to have Chrome or an instance of it installed on your system for puppeteer to work. It already ships with a lite version of it bundled with the library.

Page Methods to Extract and Map Data

The DOM has already loaded in the page instance created. We will go ahead and leverage the page.evaluate() method to query the DOM.

Before we start, we need to figure out the exact data-points we need to extract. In the current sample, each of the product objects will look something like this.

{
    brand: 'Brand Name', 
    product: 'Product Name',
    url: 'https://www.amazon.in/url.of.product.com/',
    image: 'https://www.amazon.in/image.jpg',
    price: '₹599',
}

We have laid out the structure we want to achieve. Time to start inspecting the DOM for the identifiers. We check for the selectors that occur throughout the items to be mapped. We will mostly use [document.querySelector](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelector) and [document.querySelectorAll](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelectorAll) for traversing the DOM.

...

async function fetchProductList(url) {
    ...

    await page.waitFor('div[data-cel-widget^="search_result_"]');

    const result = await page.evaluate(() => {
        // counts total number of products
        let totalSearchResults = Array.from(document.querySelectorAll('div[data-cel-widget^="search_result_"]')).length;

        let productsList = [];

        for (let i = 1; i < totalSearchResults - 1; i++) {
            let product = {
                brand: '',
                product: '',
            };
            let onlyProduct = false;
            let emptyProductMeta = false;

            // traverse for brand and product names
            let productNodes = Array.from(document.querySelectorAll(`div[data-cel-widget="search_result_${i}"] .a-size-base-plus.a-color-base`));

            if (productNodes.length === 0) {
                // traverse for brand and product names 
                // (in case previous traversal returned empty elements)
                productNodes = Array.from(document.querySelectorAll(`div[data-cel-widget="search_result_${i}"] .a-size-medium.a-color-base.a-text-normal`));
                productNodes.length > 0 ? onlyProduct = true : emptyProductMeta = true;
            }

            let productsDetails = productNodes.map(el => el.innerText);

            if (!emptyProductMeta) {
                product.brand = onlyProduct ? '' : productsDetails[0];
                product.product = onlyProduct ? productsDetails[0] : productsDetails[1];
            }

            // traverse for product image
            let rawImage = document.querySelector(`div[data-cel-widget="search_result_${i}"] .s-image`);
            product.image =rawImage ? rawImage.src : '';

            // traverse for product url
            let rawUrl = document.querySelector(`div[data-cel-widget="search_result_${i}"] a[target="_blank"].a-link-normal`);
            product.url = rawUrl ? rawUrl.href : '';

            // traverse for product price
            let rawPrice = document.querySelector(`div[data-cel-widget="search_result_${i}"] span.a-offscreen`);
            product.price = rawPrice ? rawPrice.innerText : '';

            if (typeof product.product !== 'undefined') {
                !product.product.trim() ? null : productsList = productsList.concat(product);
            }
        }

        return productsList;
    });

    ...
}

...

// traverse for brand and product names

After investigating the DOM, we see that each listed item is enclosed under an element with the selector div[data-cel-widget^="search_result_"] . This particular selector seeks out all div tags with the attribute data-cel-widget that have a value starting with search_result_.

Similarly, we map out the selectors for the parameters we require as listed. If you want to learn more about DOM traversal, you can check out this informative article by Zell.

• total listed items: div[data-cel-widget^="search_result_"]
• brand: div[data-cel-widget="search_result_${i}"] .a-size-base-plus.a-color-base (i stands for the node number in total listed items)
• product: div[data-cel-widget="search_result_${i}"] .a-size-base-plus.a-color-base or div[data-cel-widget="search_result_${i}"] .a-size-medium.a-color-base.a-text-normal (i stands for the node number in total listed items)
• url: div[data-cel-widget="search_result_${i}"] a[target="_blank"].a-link-normal (i stands for the node number in total listed items)
• image: div[data-cel-widget="search_result_${i}"] .s-image (i stands for the node number in total listed items)
• price: div[data-cel-widget="search_result_${i}"] span.a-offscreen (i stands for the node number in total listed items)

Note: We wait for div[data-cel-widget^="search_result_"] selector named elements to be available on the page by using the page.waitFor method.

Once the page.evaluate method is invoked, we can see the data we require logged.

It works! We have our API data ready to serve what we require

Adding Automation to Ease Flow

So far we are able to navigate to a page, extract the data we need, and transform it into an API-ready form. That sounds all hunky-dory.

However, consider for a moment a case where you have to navigate to one URL from another by performing some actions – and then try to extract the data you need.

Would that make your life a little trickier? Not at all. Puppeteer can easily imitate user behavior. Time to add some automation to our existing use case.

Unlike in the previous example, we will go to the amazon.in homepage and search for 'Shirts'. It will take us to the products listing page and we can extract the data required from the DOM. Easy peasy. Let's look at the code.

...

async function fetchProductList(url, searchTerm) {
    ...
    await page.goto(url, { waitUntil: 'networkidle2' });

    await page.waitFor('input[name="field-keywords"]');
    await page.evaluate(val => document.querySelector('input[name="field-keywords"]').value = val, searchTerm);

    await page.click('div.nav-search-submit.nav-sprite');

    // DOM traversal and data mapping logic
    // returns a productsList array
    ...
}

fetchProductList('https://amazon.in', 'Shirts');

We can see that we wait for the search box to be available and then we add the searchTerm passed using page.evaluate. We then navigate to the products listing page by emulating the 'search button' click action and exposing the DOM.

The complexity of automation varies from use case to use case.

Some Notable Gotchas: A Minor Heads Up

Puppeteer's API is pretty comprehensive but there are a few gotchas I came across while working with it. Remember, not all of these gotchas are directly related to puppeteer but tend to work better along with it.

• Puppeteer creates a Chrome browser instance as already mentioned. However, it is likely that some existing websites might block access if they suspect bot activity. There is this package called [user-agents](https://www.npmjs.com/package/user-agents) which can be used with puppeteer to randomize the user-agent for the browser.

Note: Scraping a website lies somewhere in the grey areas of legal acceptance. I would recommend using it with caution and checking rules where you live.

const puppeteer = require('puppeteer');
const userAgent = require('user-agents');

...

const browser = await puppeteer.launch({ headless: true, defaultViewport: null });
const page = await browser.newPage();
await page.setUserAgent(userAgent.toString());

...

• We came across defaultViewport: null when launching our Chrome instance and I had listed it as optional. This is because it comes in handy only when you are viewing the Chrome instance being launched. It prevents the website's width and height from being affected when it is rendered.
• Puppeteer is not the ultimate solution when it comes to performance. You, as a developer, will have to optimize it to increase its performance efficiency through actions like throttling animations on the site, allowing only essential network calls, etc.
• Remember to always end a puppeteer session by closing the Browser instance by using browser.close. (I happened to miss out on it in the first try) It helps end a running Browser Session.
• Certain common JavaScript operations like console.log() will not work within the scope of the page methods. The reason being that the page context/browser context differs from the node context in which your application is running.

These are some of the gotchas I noticed. If you have more, feel free to reach out to me with them. I would love to learn more.

Done? Let's run the application.

Website to Your API: Bringing it All Together

The application is run in non-headless mode so you can witness what exactly happens. We will automate the navigation to the product listing page from which we obtain the data.

There. You have your own API consumable data setup from the website of your choice. All you need to do now is to wire this up with a server side framework like express and you are good to go.

Conclusion

There is so much you can do with Puppeteer. This is just one particular use case. I would recommend that you spend some time to read the official documentation. I will be doing the same.

Puppeteer is used extensively in some of the largest organizations for automation tasks like testing and server side rendering, among others.

There is no better time to get started with Puppeteer than now.

If you have any questions or comments, you can reach out to me on LinkedIn or Twitter.

In the meantime, keep coding.

Lately I've added continuous integration to my blog using Puppeteer for end to end testing. My main goal was to allow automatic dependency updates using Dependabot. In this guide I'll show you how to create such a pipeline yourself.

As my CI platform, I chose Github Actions, as it is super easy to work with. It also integrates beautifully with any Github repository you already have. The whole thing only took roughly two days of intermittent work, and I think the results are quite awesome.

I do want to give a shout-out to Nick Taylor, who published his article on the subject, and laid the ground work for my efforts here. I encourage you to read his article as well.

My tech stack is quite different though. I chose puppeteer as my end-to-end framework for several reasons. The first is that it is written and maintained by the folks behind the Chrome dev tools, so I'm guaranteed a life-time of support (until Chrome dies out, which is not in the near future), and it is really easy to work with.

Another reason is that at home I'm working on a windows laptop with WSL (on which I'm running zshell with oh-my-zsh). Setting up cypress is quite a bit more difficult (although in our world nothing is impossible). Both reasons led me to choose puppeteer, and so far I'm not regretting it.

End to end testing

End to end (or E2E) tests are different from other types of automated tests. E2E tests simulate a real user, performing actions on the screen. This kind of test should help fill the blank space between "static" tests - such as unit tests, where you usually don't bootstrap the entire application - and component testing, which usually runs against a single component (or a service in a micro-service architecture).

By simulating user interaction you get to test the experience of using your application or service in the same way a regular user would experience it.

The mantra that we try to follow is that it does not matter if your code performs perfectly if the button the user should press is hidden due to some CSS quirk. The end result is that the user will never get to feel the greatness of your code.

Getting started with puppeteer

Puppeteer has a few configuration options that make it really awesome to use for writing and validating tests.

Puppeteer tests can run in a "head-full" state. This means you can open a real browser window, navigate to the site being tested, and perform actions on the given page. This way you - the developers writing the tests - can see exactly what happens in the test, what buttons are being pressed and what the resulting UI looks like.

The opposite of "head-full" would be headless, where puppeteer does not open a browser window, making it ideal for CI pipelines.

Puppeteer is quite easy to work with, but you'll be surprised with the number of actions you can perform using an automated tool.

We'll start with a basic scraper that prints the page title when we go to https://dorshinar.me. In order to run puppeteer tests, we must install it as a dependency:

npm i puppeteer

Now, our basic scraper looks like this:

const puppeteer = require("puppeteer");

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto("https://dorshinar.me");
  console.log(await page.title());

  await browser.close();
})();

What we do here is very simple: we open the browser with puppeteer.launch(), create a new page with browser.newPage() and navigate to this blog with page.goto(), and then we print the title.

There are a bunch of things we can do with the puppeteer API, such as:

Running code in the context of the page:

(async () => {
  await page.evaluate(() => document.querySelector(".awesome-button").click());
})();

Clicking on elements in the screen using a CSS selector:

(async () => {
  await page.click(".awesome-button");
})();

Making use of the $ selector (jQuery style):

(async () => {
  await page.$(".awesome-button");
})();

Taking a screenshot:

(async () => {
  await page.screenshot({ path: "screenshot.png" });
})();

There is a bunch more you can do with the puppeteer API, and I suggest you take a look at it before diving into writing tests. But the examples I've shown should give you a solid foundation to build from.

Integrating puppeteer with Jest

jest is an awesome test runner and assertion library. From their docs:

Jest is a delightful JavaScript Testing Framework with a focus on simplicity.

Jest allows you to run tests, mock imports, and make complex assertions really easily. Jest is also bundled with create-react-app, so I use it often at work.

Writing your first Jest test

Jest tests are super easy to write, and they might be familiar to those who know other testing frameworks (as Jest uses it, test, describe and other familiar conventions).

A basic test could look like:

function subtract(a, b) {
  return a - b;
}

it("subtracts 4 from 6 and returns 2", () => {
  expect(subtract(6, 4)).toBe(2);
});

You can also group multiple tests under one describe, so you can run different describes or use it for convenient reporting:

function divide(a, b) {
  if (b === 0) {
    throw new Error("Can't divide by zero!");
  }
  return a / b;
}

describe("divide", () => {
  it("throws when dividing by zero", () => {
    expect(() => divide(6, 0)).toThrow();
  });
  it("returns 3 when dividing 6 by 3", () => {
    expect(divide(6, 3)).toBe(2);
  });
});

You can, of course, create much more complicated tests using mocks and other type of assertions (or expectations), but for now that's enough.

Running the tests is also very simple:

jest

Jest will look for test files with any of the following popular naming conventions:

• Files with .js suffix in __tests__ folders.
• Files with .test.js suffix.
• Files with .spec.js suffix.

jest-puppeteer

Now, we need to make puppeteer play nicely with jest. This isn't a particularly hard job to do, as there is a great package named jest-puppeteer that comes to our aid.

First, we must install it as a dependency:

npm i jest-puppeteer

And now we must extend our jest configuration. If you don't have one yet, there are a number of ways to do it. I'll go with a config file. Create a file named jest.config.js in the root of your project:

touch jest.config.js

In the file we must tell jest to use jest-puppeteer's preset, so add the following code to the file:

module.exports = {
  preset: "jest-puppeteer"
  // The rest of your file...
};

You may specify a special launch configuration in a jest-puppeteer.config.js file, and jest-puppeteer will pass this configuration to puppeteer.launch(). For example:

module.exports = {
  launch: {
    headless: process.env.CI === "true",
    ignoreDefaultArgs: ["--disable-extensions"],
    args: ["--no-sandbox"],
    executablePath: "chrome.exe"
  }
};

jest-puppeteer will take care of opening a new browser and a new page and store them on the global scope. So in your tests you can simply use the globally available browser and page objects.

Another great feature we can use is the ability of jest-puppeteer to run your server during your tests, and kill it afterwards, with the server key:

module.exports = {
  launch: {},
  server: {
    command: "npm run serve",
    port: 9000,
    launchTimeout: 180000
  }
};

Now jest-puppeteer will run npm run serve, with a timeout of 180 seconds (3 minutes), and listen on port 9000 to see when it will be up. Once the server starts the tests will run.

You can now write a full test suite using jest and puppeteer. The only thing left is creating a CI pipeline, for which we'll use GitHub actions.

You can add a script to your package.json file to execute your tests:

{
  "scripts": {
    "test:e2e": "jest"
  }
}

Github Actions in a gist

Recently, Github released a big new feature called Actions. Basically, actions allow you to create workflows using plain yaml syntax, and run them on dedicated virtual machines.

In your workflow you can do pretty much anything you want, from basic npm ci && npm build && npm run test to more complicated stuff.

I'll show you how to configure a basic workflow running your puppeteer test suite, and prevent merging if your tests don't pass.

The easiest way to start is to click on the Actions tab in your github repo. If you haven't configured any action before, you'll see a list of previously configured workflows, from which you can choose one with some predefined configuration.

For our case, choosing the predefined Node.js action is good enough. The generated yaml looks like this:

name: Node CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [8.x, 10.x, 12.x]

    steps:
      - uses: actions/checkout@v1
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node-version }}
      - name: npm install, build, and test
        run: |
          npm ci
          npm run build --if-present
          npm test
        env:
          CI: true

In the file you can configure the workflow name, jobs to run, and when to run the workflow. You can run your workflow on every push, on new pull requests, or as a recurring event.

Jobs in a workflow run in parallel by default, but can be configured to run in sequence. In the above workflow, there is one job named build.

You can also choose the OS on which your workflow will run (by default you can use Windows Server 2019, Ubuntu 18.04, Ubuntu 16.04 and macOS Catalina 10.15 - at the time of publishing) with the runs-on key.

The strategy key can help us run our tests on a matrix of node versions. In this case we have the latest versions of the latest LTS majors - 8.x, 10.x and 12.x. If you are interested in that you can leave it as is, or simply remove it and use any specific version you want.

The most interesting configuration option is the steps. With it we define what actually goes on in our pipeline.

Each step represents an action you can perform, such as checking out code from the repo, setting up your node version, installing dependencies, running tests, uploading artifacts (to be used later or downloaded) and many more.

You can find a very extensive list of readily available actions in the Actions Marketplace.

The basic configuration will install dependencies, build our project and run our tests. If you need more (for example if you want to serve your application for e2e tests) you may alter it to your liking. Once done, commit your changes and you are good to go.

Forcing checks to pass before merge

The only thing left for us is to make sure no code can be merged before our workflow passes successfully. For that, go to your repo's settings and click on Branches:

We need to set a Branch protection rule so that malicious code (or at least code that doesn't pass our tests) won't be merged. Click on Add rule, and under Branch name pattern put your protected branch (master, dev or whichever one you choose). Make sure Require status checks to pass before merging is checked, and you'll be able to choose which checks must pass:

Click on Save changes below, and you're good to go!

Thank you for reading!
This article was previously published on my blog: dorshinar.me, If you want to read more content, you can check my blog as it would mean a lot to me.

If you want to support me, you can

This post is co-authored by Hricha Kabir, my colleague at Altizon Systems. She was working on this task primarily. I had the chance to learn along the way.

_Photo by [Unsplash](https://unsplash.com/@rvruggiero?utm_source=medium&utm_medium=referral" rel="noopener" target="_blank" title="">Robert V. Ruggiero on <a href="https://unsplash.com?utm_source=medium&utm_medium=referral" rel="noopener" target="blank" title=")

We work on an IoT product focusing on manufacturing industries and build analytics reports. Most of the time, report design and information vary based on the user’s role who is going to consume it.

Example: A Director level person is interested in a consolidated report of a week, whereas a Department Head is interested in the statistics of a single day and Quality Manager is keen about shift data. If we further drill down the hierarchy, then the Production Manager will be the one who wants to look into live production data.

For live data, Production manager can open the Live Dashboard and monitor it, but the user who wants to see a consolidated report won’t prefer to login into Product and check out a report. Instead, they prefer scheduled email communication which will send a report based on scheduled time and repeat it forever until unless a user stopped this scheduled job.

So our task was to write a utility which periodically sends a particular report to a user via email on a daily basis or at a defined scheduled time. For scheduling, we use Sidekiq, but our main challenge was: how to execute/calculate the report on the server side (without opening a browser) and send it over email.

In all of our products, we use React for the front-end and it does the heavy lifting of rendering the report on the browser. To achieve attaching the same report in the email, we would have to design an HTML template for every report. That means we would have had 2 views of each report, one written in HTML and another in ReactJS.

This has following drawbacks:

1. The product code base will have duplicate code which violates the DRY principle for software development.

2. Increases the time for designing a report which impacts product delivery.

We decided to look at other possible solutions instead of writing the same code twice. And we came across following options and explored them one by one until we succeeded in achieving our goal.

1. Print Screen: Print the report using print screen and send it as an attachment.

2. FrontEnd Print Button: In report’s UI, add a Print Button which will convert react component into HTML/PDF page.

3. Ruby Gem: There is a ruby gem which can render a report component on the server side within Rails, generate its HTML and send it as an email body.

4. Server-Side Rendering with Node server: Use serverside rendering using ReactDOMServer and headless browser protocol to render HTML and JS and generate a pdf.

Now we will go through the details of each solution.

Print Screen:

It was the simplest solution we had and it requires user input. A user needs to be present with a system who will do a screen capture as an image and attach it over the mail. But this doesn’t scale. Also what if a user wants a report at a scheduled time or sent periodically?

Example: Every week at 8 pm. It’s not possible for a person to do this. This also suffers from the inability to fully capture a scrollable page, so we had to drop this idea.

FrontEnd Print Button:

After doing some search, we found out there are some browser extensions available which produce a full image of an opened page. Example: Full Page Screen Capture in Chrome. After adding this extension to the browser, we would be able to capture any screen in image(png/jpg) format.

Again this solution requires no development but suffers from some of the previous scaling and scheduling issues. At the same time, we would still need a user logged in to the browser to perform this action, which defeats the purpose of email delivery.

RubyGem:

Soon we realized that our solution cannot require browser interaction. We would need to use server-side rendering. So we started exploring Ruby Gems which support server-side rendering with React. We explored the following Gems

3.1 rails_react_stdio:

It is based on react-stdio which supports server-side rendering irrespective of server-side technology. It acts as a binary which will do the work of rendering react components. For rendering React Component on the server side we need to pass the file path of the react component and props if required. It will return a JSON response which will have the HTML code of the report. Further we can send HTML content over email.

Internally this gem uses popen3 for executing render command. But this means that react-stdio binary needs to be present in the docker container where our rails app is running.

This is not great from a point of view of maintainability and reproducibility. Additionally having large HTML content with charts can be slow to load so we preferred pdf attachment. Yet we gave it a try.

Example: For rendering a report which has the component TestComponent and the file path app/assets/javascripts/components/TestComponent.jsx

First, include the gem in Gemfile:

gem ‘rails_react_stdio’, ‘~> 0.1.0’

Now from the email scheduler call the gem method to render the report and get the HTML from the response. Then send this response to email.

email_body = RailsReactStdio::React.render(‘app/assets/javascripts/components/TestComponent.jsx’, {city: “Pune”})

We tried using the above method but had no success. Also the GitHub repo was not actively maintained, and all of its test cases were failing. So we decided to move forward without this.

3.2 react-rails:

The ReactJS community built this gem. It uses ExecJS for executing render-action of a react component on the server side. We just need to pass one flag ‘prerender’: true.

<%= react_component(‘Dashboard’, {name: ‘Example’}, {prerender: true}) %>

This prerendering process does not have access to the window or document so it does not load runtime JavaScript or CSS. We also use JQuery for a few things so it wouldn’t work as well.

There is an alternative: this gem has another class for server-side rendering, ExecJSRenderer, which helps in availing JavaScript to a component on the server side.

ExecJSRenderer Class has 2 methods: _beforerender and _afterrender, which gives access to the JavaScript required before and after component rendering. But it’d require a lot of changes in the existing code base for supporting server-rendering, in every controller. Apart from this, ExecJS doesn’t provide sandboxing as well as runtime error information. We were still looking for something better.

Server-Side Rendering with Node server:

Most of the ruby gems we explored internally created the node server and rendered react on the server side. So instead of using Ruby, we decided to directly use the node server and achieve this task ourselves.

4.1 ReactDOMServer based solution:

Here we use ReactDOMServer. It is the preferred solution for server-side rendering from the React team. We created a node server which calls the renderToString() method with a react component. It returns the rendered content which we combine with HTML and send over email.

Example:

server.get(‘/’, (req, res, next) => {
  /**
  * renderToString() will take our React app and turn it into a      string
  * to be inserted into our Html template function.
  */
  console.log(‘started’)
  const body = renderToString(<App />);
  const title = ‘Server side Rendering React Components’;
  var result = Html({ body, title })
}

The renderToString() method returns a string response. We pass this response to an HTML template and send the template over mail.

When we tested this out, an email was received as expected except all the images used in the report were broken. The email was not able to resolve the image source relative path.

To get the correct images in the email, we would either need to

• Store images in S3 and use the source URL in the report: So now we would be adding S3 image URLs in the email, and the email server would directly load images from the S3 server. It would require extra cloud space for storing images, and downloading from destination requires another network call from the email inbox.

Or

• Send base64 code of image in the mail: Instead of image URL, we can send the base64 code of an image. Although It increases network payload, many mail servers such as Outlook and Gmail block base64 images.

So we would still need to do something about this.

4.2 Puppeteer:

After exploring the above methods, we discovered Puppeteer Headless Chrome service. Puppeteer is a NodeJS library from the Google Chrome team, used in end to end testing. By default, it uses the Chrome/Chromium browser for the same. Essentially it simulates all the actions a user can perform in the browser. Example: Keyboard Input, Mouse events, Form submission etc.

The result of a puppeteer request can be an HTML page, Screenshot, or PDF. In case of HTML, it renders the full page on the server side along with all images, CSS, and JavaScript. Or a user can render a page on the server side and if required they can generate a screenshot or a pdf.

If we use this library with a Node server, we can schedule a task within Sidekiq which would make a request to this server, render a report, and send it over email. Puppeteer also has a rich set of APIs which support in sending customizing headers in the request which help us in authenticating a background request.

This is exactly what we needed!

So the overall request lifecycle inside puppeteer should look like this:

1. It launches the browser

2. Creates a page on the browser

3. Authenticate with the Report Application server

4. Opens the report URL that the user wants and returns the rendered page content

5. Based on the user’s requirement, stores HTML of the rendered page or takes a Screenshot or generates a PDF.

We can use either the default browser (gets downloaded when we install puppeteer) or we can give a specific browser version. If we give a specific browser version, then we will have to make sure that puppeteer APIs are compatible with the given browser.

All of the above can be achieved using the following steps.

1. Install puppeteer:

It downloads the default Chrome/Chromium browser. So If we want to launch the default browser and install puppeteer:

npm install puppeteer

2. Serve a Request:

Build a server which will receive a request and implement a request lifecycle and return the desired result.

The following is the code snippet we use for generating a pdf of a given page URL.

const puppeteer = require(“puppeteer”);
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.emulateMedia(“screen”);
await page.goto(‘https://www.google.com', {
      timeout: 30 * 1000, 
      waitUntil: “networkidle0”
});
await page.pdf(pdfOptions);
return page;

As we already mentioned in the lifecycle steps above, it first launches the browser. Then it creates and opens a page in the launched browser. Here along with the URL, we have passed timeout and waitUntil params which are given for the following reasons:

• timeout: If we want to restrict a request time then we can pass it to timeout variable
• waitUntil: if networkidle0 is given, then the next request will not be served until or unless the current one is completed.

In the end, rendered content will be passed to the pdf method and it will generate a pdf. We can also provide PDF formatting options like page height, page width, headers, footers, and margins.

Additionally, we have called page.emulateMedia(“screen”); which applies CSS to the page. If we don’t add emaulateMedia, our PDF doesn’t load the CSS.

Apart from what we have used, there are many different configuration options with Puppeteer APIs. Visit the API docs for more information.

If you found this helpful or if you have any suggestions, please feel free to write them it in comments.

That’s all folks.