How to Build a Simple Web Scraper with Python: A Step-by-Step Guide

Web scraping lets you automatically extract data from websites. Python makes this easy with powerful libraries. In this tutorial, you’ll build a scraper that fetches headlines from a sample page and saves them to a file.

First, install the two essential libraries: requests (to download web pages) and BeautifulSoup (to parse HTML). Open your terminal and run:

Install Required Libraries

pip install requests beautifulsoup4

Send an HTTP Request and Parse HTML

Use requests.get() to fetch the page. Then pass the response content to BeautifulSoup:

import requests
from bs4 import BeautifulSoup

url = "https://example.com/news"
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

Extract the Data You Need

Inspect the target page to find the HTML tags containing your data. For headlines inside h2 tags with class headline:

headlines = soup.find_all('h2', class_='headline')
for h in headlines:
    print(h.get_text(strip=True))

• find_all() returns a list of matching tags.
• get_text() extracts the visible text.

Save Data to a CSV File

Store every headline in a CSV for later analysis:

import csv

with open('headlines.csv', 'w', newline='', encoding='utf-8') as f:
    writer = csv.writer(f)
    writer.writerow(['Headline'])
    for h in headlines:
        writer.writerow([h.get_text(strip=True)])

Conclusion

With just a few lines of Python, you can scrape structured data from any website. Always check the site’s robots.txt and terms of service before scraping. Experiment with different selectors and data formats to expand your toolset.