How to Use Playwright for Web Scraping: A Step-by-Step Guide

Playwright is a powerful browser automation tool perfect for scraping modern, JavaScript-heavy websites. Unlike simple HTTP requests, Playwright can render pages, click buttons, and wait for elements, making it ideal for dynamic content.

In this tutorial, you’ll learn how to set up Playwright, launch a browser, navigate to a site, extract data, and handle common scraping challenges. Let’s dive in.

Article illustration

1. Installation and Setup

Install Playwright via npm (Node.js) or pip (Python). For Node.js:

  • npm install playwright
  • Then npx playwright install to download browser binaries

For Python:

  • pip install playwright
  • playwright install

2. Launching a Browser and Navigating

Create a script that opens a browser instance. Example in Node.js:

const { chromium } = require('playwright');
(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  // ... scraping code
  await browser.close();
})();

Use { headless: false } to see the browser in action.

3. Extracting Data from the Page

Use CSS selectors with page.$eval or page.$$eval:

const titles = await page.$$eval('h2', els => els.map(el => el.textContent));
console.log(titles);

For more complex logic, use page.evaluate() to run JavaScript inside the page context.

4. Handling Dynamic Content and Wait Strategies

Many sites load content after page load. Wait for specific elements:

await page.waitForSelector('.product-list');
const products = await page.$$eval('.product', ...);

Also use page.waitForNavigation() after clicks, and avoid arbitrary timeouts.

Conclusion

Playwright simplifies scraping modern websites by automating browser actions. With its API, you can handle JavaScript rendering, infinite scroll, and logins. Always respect robots.txt and site terms. Start simple, add waits where needed, and scrape responsibly.

sarah antaboga
Author: sarah antaboga

Leave a Reply

Your email address will not be published. Required fields are marked *