How to Build a Web Scraper Using BeautifulSoup in Python

BeautifulSoup is a Python library that makes extracting data from HTML documents simple and intuitive. In this tutorial, you’ll learn how to create a basic web scraper that fetches a webpage, parses its HTML, and pulls out the information you need — such as headlines, prices, or product names. We’ll use two core libraries: requests for fetching pages and BeautifulSoup for parsing.

Before you start, ensure Python 3 is installed. You’ll also need to install the requests and beautifulsoup4 packages. Open your terminal and run: pip install requests beautifulsoup4. Now we’re ready to scrape.

1. Fetch the Webpage HTML

Use requests.get() to download the page content. Always check the status code to confirm the request succeeded.

import requests
url = 'https://example.com'
response = requests.get(url)
if response.status_code == 200:
    html = response.text
else:
    print('Failed to retrieve page')

2. Parse the HTML with BeautifulSoup

Pass the HTML string into BeautifulSoup and specify a parser (usually 'html.parser'). This builds a navigable tree of the document.

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'html.parser')

3. Extract Data Using CSS Selectors or Tags

Find elements by tag name, class, or ID. Common methods:

• soup.find('h1') – first <h1>
• soup.find_all('a', class_='link') – all links with that class
• soup.select('div.title') – CSS selector

For example, to scrape all article headlines on a blog:

headlines = soup.find_all('h2', class_='post-title')
for h in headlines:
    print(h.get_text(strip=True))

4. Handle Common Challenges

Respect the site’s robots.txt and set a user-agent header to avoid being blocked. Add a small delay between requests with time.sleep(). If content is loaded by JavaScript, you may need tools like Selenium instead.

Conclusion: With just a few lines of Python, BeautifulSoup turns raw HTML into structured data you can save, analyze, or automate. Start with simple static sites, and always scrape responsibly.