How to Use Playwright for Web Scraping: A Step-by-Step Guide
Playwright is a powerful browser automation tool perfect for scraping modern, JavaScript-heavy websites. Unlike simple HTTP requests, Playwright can render pages, click buttons, and wait for elements, making it ideal for dynamic content.
In this tutorial, you’ll learn how to set up Playwright, launch a browser, navigate to a site, extract data, and handle common scraping challenges. Let’s dive in.
1. Installation and Setup
Install Playwright via npm (Node.js) or pip (Python). For Node.js:
npm install playwright- Then
npx playwright installto download browser binaries
For Python:
pip install playwrightplaywright install
2. Launching a Browser and Navigating
Create a script that opens a browser instance. Example in Node.js:
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com');
// ... scraping code
await browser.close();
})();
Use { headless: false } to see the browser in action.
3. Extracting Data from the Page
Use CSS selectors with page.$eval or page.$$eval:
const titles = await page.$$eval('h2', els => els.map(el => el.textContent));
console.log(titles);
For more complex logic, use page.evaluate() to run JavaScript inside the page context.
4. Handling Dynamic Content and Wait Strategies
Many sites load content after page load. Wait for specific elements:
await page.waitForSelector('.product-list');
const products = await page.$$eval('.product', ...);
Also use page.waitForNavigation() after clicks, and avoid arbitrary timeouts.
Conclusion
Playwright simplifies scraping modern websites by automating browser actions. With its API, you can handle JavaScript rendering, infinite scroll, and logins. Always respect robots.txt and site terms. Start simple, add waits where needed, and scrape responsibly.