How to Build a News Aggregator from Scratch: A Comprehensive Step-by-Step Guide
In an age where information overload is the norm, a news aggregator serves as a powerful tool to curate, filter, and present content from multiple sources in one unified interface. Whether you want to track industry trends, monitor competitors, or simply stay updated on topics you care about, building your own aggregator gives you complete control over which sources are included, how the content is displayed, and how frequently it is refreshed. Many popular platforms such as Google News, Feedly, and Flipboard started as simple aggregators before evolving into sophisticated recommendation engines. The good news is that with modern web development tools and open‑source libraries, you can create a functional news aggregator in a matter of days, even if you are a relatively novice developer. This guide will walk you through the entire process — from planning and data collection to storage, presentation, and deployment — so that by the end you will have a fully working aggregator that can pull in articles from dozens of RSS feeds, APIs, or scraped websites. We will use a stack that is beginner‑friendly but scalable: Node.js for the backend, Express for routing, a lightweight database like MongoDB or SQLite for storage, and a simple front‑end built with HTML, CSS, and JavaScript. However, the concepts discussed here can be adapted to any programming language or framework. So, roll up your sleeves and get ready to build a tool that puts you back in the driver’s seat of your news consumption.
Before we dive into the code, it is essential to understand the architectural components of a news aggregator. At its core, the system must perform three main tasks: fetch content from external sources, parse and normalize that content into a consistent format, and then present it to the user in a readable, searchable manner. Additionally, you will want to handle periodic updates (cron jobs), deduplication of articles, and possibly a simple categorization mechanism. The beauty of building your own aggregator is that you can decide how much complexity to include — for instance, you might start with only RSS feeds and later add web scraping or API integrations. The steps below follow a logical progression: first, you will set up the project and choose your data sources; second, you will write the code to fetch and parse the feeds; third, you will store the parsed articles in a database; fourth, you will build a web interface to display the articles; and fifth, you will schedule automatic updates and add optional features like filtering or bookmarking. Along the way, I will also share best practices for handling errors, avoiding duplicate content, and respecting the terms of service of the sources you aggregate. By the time you finish, you will not only have a functional aggregator but also a deep understanding of how modern media curation works under the hood.
Step 1: Planning Your News Aggregator and Choosing Your Tech Stack
Every successful project begins with a clear plan. For a news aggregator, you need to decide on the scope: will it aggregate only RSS/Atom feeds, or will it also scrape HTML pages and use APIs? For simplicity and reliability, starting with RSS feeds is highly recommended because they provide structured data (title, link, publication date, summary) and are widely supported by blogs, news sites, and even YouTube channels. Once you have the basics working, you can extend to scraping with libraries like Cheerio (Node.js) or Beautiful Soup (Python). The choice of programming language and database should be driven by your familiarity and the expected scale. Node.js is an excellent choice because its non‑blocking I/O handles many concurrent fetch requests efficiently, and the npm ecosystem has mature packages for RSS parsing (rss-parser), HTTP requests (axios), and scheduling (node-cron). For the database, SQLite (using better-sqlite3) is perfect for a single‑user aggregator, while MongoDB (via Mongoose) offers more flexibility if you plan to add user accounts or complex queries later. The front‑end can be as simple as a single HTML page with client‑side JavaScript that fetches articles from your backend API, or you can use a framework like React. For this tutorial, I will assume you are using Node.js with Express for the backend, SQLite for storage, and a plain HTML/CSS/JS front‑end to keep the focus on the core aggregator logic. You will also need a hosting environment — either a VPS, a platform like Heroku or Railway, or even a Raspberry Pi at home. The key is to have a server that can run a Node.js process 24/7 (or at least during the intervals you set for updates).
Beyond the technical stack, take a moment to list the sources you want to aggregate. Choose a mix of high‑volume and low‑volume feeds to test how your system handles different update frequencies. For development, I recommend using the official RSS feeds of major news outlets (e.g., BBC News, The New York Times, TechCrunch) and a few niche blogs. Pay close attention to the feed format — most are RSS 2.0 or Atom, but some may use JSON Feed (a newer format). Your parser should handle all of these. Also, consider the legal and ethical implications: many websites include terms of service that prohibit scraping or automated access without permission. RSS feeds are generally considered fair game because they are designed for syndication, but always check the feed’s license or the site’s robots.txt file. If you plan to display full article content (not just snippets), you may need to obtain explicit permission or use a “fair use” approach. This tutorial assumes you will display only the title, summary, and link back to the original source — which is standard practice for aggregators like Feedly or Inoreader.
Step 2: Setting Up the Project and Installing Dependencies
With your plan in place, it’s time to create the project skeleton. Open a terminal and create a new directory named news-aggregator, then navigate into it. Run npm init -y to generate a package.json file. Now install the core dependencies: express for the web server, rss-parser for parsing RSS and Atom feeds, axios for making HTTP requests, better-sqlite3 for the database, and node-cron for scheduling periodic updates. You can install them all with npm install express rss-parser axios better-sqlite3 node-cron. Optionally, install dotenv for managing environment variables if you plan to store API keys later. Once the installation finishes, create a file named index.js — this will be your main application entry point. Also create a db.js file to handle database initialization and queries, and a feeds.js file where you will define the list of feed URLs and the fetching/parsing logic. For the front‑end, create a public folder with an index.html, a style.css, and a app.js for client‑side JavaScript. Your folder structure should look like this:
news-aggregator/├── index.js(Express server, routes, cron job)├── db.js(SQLite setup and CRUD operations)├── feeds.js(list of sources and fetch logic)├── package.json├── .env(optional)├── public/│ ├── index.html│ ├── style.css│ └── app.js
Now, initialize the database. In db.js, we will create a table called articles with columns for id (integer primary key autoincrement), title (text), link (text unique to prevent duplicates), summary (text), source (text — the feed name), published_at (datetime), and created_at (datetime default current timestamp). The unique constraint on link ensures that even if the same article appears in multiple feeds, or if you accidentally fetch the same feed twice, you won’t store duplicates. We will also create a feeds table to store metadata about each source (name, URL, last fetched time). The code below shows the schema creation:
const Database = require('better-sqlite3');
const db = new Database('aggregator.db');
db.pragma('journal_mode = WAL'); // Better concurrent access
db.exec(`
CREATE TABLE IF NOT EXISTS articles (
id INTEGER PRIMARY KEY AUTOINCREMENT,
title TEXT NOT NULL,
link TEXT UNIQUE NOT NULL,
summary TEXT,
source TEXT,
published_at DATETIME,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS feeds (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
url TEXT UNIQUE NOT NULL,
last_fetched DATETIME
);
`);
module.exports = db;
This is the foundation of your data storage. Notice that we use WAL mode to allow read operations while writes are happening — crucial when your aggregator is fetching new articles in the background while users browse the front‑end.
Step 3: Fetching and Parsing RSS Feeds
Now we will build the core fetch logic in feeds.js. First, define an array of source objects. Each object should have a name (display name for the UI) and a url (the feed URL). I will include a few examples to get you started:
const sources = [
{ name: 'BBC News', url: 'https://feeds.bbci.co.uk/news/rss.xml' },
{ name: 'TechCrunch', url: 'https://techcrunch.com/feed/' },
{ name: 'Hacker News', url: 'https://hnrss.org/frontpage' },
{ name: 'The Verge', url: 'https://www.theverge.com/rss/index.xml' }
];
Next, we need a function fetchAndStoreAll() that iterates over each source, uses axios to download the feed XML, then passes it to rss-parser to convert it into a JavaScript object. The parser handles both RSS and Atom transparently. After parsing, we extract each item (article) and insert it into the database, ignoring duplicates thanks to the INSERT OR IGNORE SQLite syntax. We also update the last_fetched timestamp for the feed. Let’s write that:
const axios = require('axios');
const Parser = require('rss-parser');
const parser = new Parser();
const db = require('./db');
async function fetchAndStoreAll() {
for (const source of sources) {
try {
const response = await axios.get(source.url, { timeout: 10000 });
const feed = await parser.parseString(response.data);
const insert = db.prepare(`
INSERT OR IGNORE INTO articles (title, link, summary, source, published_at)
VALUES (?, ?, ?, ?, ?)
`);
const updateFeed = db.prepare(`
UPDATE feeds SET last_fetched = ? WHERE url = ?
`);
const now = new Date().toISOString();
// Ensure the feed exists in the feeds table
db.prepare('INSERT OR IGNORE INTO feeds (name, url) VALUES (?, ?)').run(source.name, source.url);
for (const item of feed.items) {
const summary = item.contentSnippet || item.content || item.summary || '';
insert.run(item.title, item.link, summary, source.name, item.pubDate || item.isoDate || now);
}
updateFeed.run(now, source.url);
console.log(`Fetched ${feed.items.length} articles from ${source.name}`);
} catch (error) {
console.error(`Error fetching ${source.name}: ${error.message}`);
}
}
}
Notice the INSERT OR IGNORE — if an article with the same link already exists, it will be silently skipped. This is your first line of defense against duplicates. The published_at field uses the feed’s date if available, otherwise falls back to the current timestamp. The summary is extracted from the feed’s content snippet; some feeds include full HTML, but we store only a plain text snippet for simplicity. You can adjust this later.
Now, in your index.js, you can call fetchAndStoreAll() once when the server starts (to populate the database initially) and then schedule it to run every 15 minutes using node-cron:
const cron = require('node-cron');
const { fetchAndStoreAll } = require('./feeds');
// Initial fetch
fetchAndStoreAll().then(() => console.log('Initial fetch complete'));
// Schedule every 15 minutes
cron.schedule('*/15 * * * *', () => {
console.log('Running scheduled fetch...');
fetchAndStoreAll();
});
With this code, your aggregator will continuously refresh its content from the defined sources. The */15 cron expression runs the job every 15 minutes — adjust based on how often you want updates and how many sources you have. Be mindful of rate limits: if you have 50 feeds, fetching them all every 15 minutes may be overwhelming for smaller servers. You can also implement a queue or stagger the fetches between sources.
Step 4: Building the API and Web Interface
Now that your database is filling with articles, you need an API endpoint that the front‑end can call to retrieve them. In index.js, set up Express to serve static files from the public directory and create a /api/articles route that returns JSON. We’ll use pagination to avoid sending thousands of rows at once. The route should accept query parameters for page (default 1) and limit (default 20), and optionally a source filter. Here’s the route code:
const express = require('express');
const app = express();
const db = require('./db');
const path = require('path');
app.use(express.static(path.join(__dirname, 'public')));
app.get('/api/articles', (req, res) => {
const page = parseInt(req.query.page) || 1;
const limit = parseInt(req.query.limit) || 20;
const offset = (page - 1) * limit;
let query = 'SELECT * FROM articles';
let countQuery = 'SELECT COUNT(*) as total FROM articles';
const params = [];
const countParams = [];
if (req.query.source) {
query += ' WHERE source = ?';
countQuery += ' WHERE source = ?';
params.push(req.query.source);
countParams.push(req.query.source);
}
query += ' ORDER BY published_at DESC LIMIT ? OFFSET ?';
params.push(limit, offset);
const articles = db.prepare(query).all(...params);
const { total } = db.prepare(countQuery).get(...countParams);
res.json({ articles, total, page, limit });
});
app.listen(3000, () => console.log('Aggregator running on http://localhost:3000'));
This API returns a JSON object with the articles array, total count, and pagination metadata. Now let’s build the front‑end. In public/index.html, create a minimal structure: a header with the aggregator name, a source filter dropdown (populated from a separate API call to /api/sources), and a container for the articles. In the app.js, use fetch to get the first page of articles, then render them as cards. Each card shows the title (linked to the original article), the source name, and a snippet of the summary. Below is an example of the rendering logic:
async function loadArticles(page = 1, source = '') {
let url = `/api/articles?page=${page}&limit=20`;
if (source) url += `&source=${encodeURIComponent(source)}`;
const res = await fetch(url);
const data = await res.json();
const container = document.getElementById('articles');
container.innerHTML = '';
data.articles.forEach(article => {
const card = document.createElement('div');
card.className = 'card';
card.innerHTML = `
${article.title}
${article.source} · ${new Date(article.published_at).toLocaleDateString()}
${article.summary}
`;
container.appendChild(card);
});
// Update pagination UI...
}
Add a minimal CSS to make the cards look clean — use a white background, slight shadow, and responsive grid. The end result is a functional web page that displays aggregated news content. Don’t forget to add a loading spinner while fetching, and handle errors gracefully.
Step 5: Testing, Deploying, and Adding Enhancements
Before deploying, test your aggregator thoroughly on your local machine. Start the server with node index.js and open http://localhost:3000. You should see articles populating after the initial fetch completes (the fetch might take a few seconds). Check that duplicates are properly ignored, that the pagination works, and that the source filter dropdown behaves correctly. Also, verify the scheduled fetch by checking the console logs — after 15 minutes you should see another fetch cycle. Once you are satisfied, it’s time to deploy. Options include:
- Railway / Heroku: Simple Git‑based deployment. Create a
Procfilewithweb: node index.jsand set the environment. Note that Heroku’s ephemeral filesystem will delete your SQLite database on restart, so for production you should either use a hosted SQLite service (like Turso) or switch to a cloud database (MongoDB Atlas, PostgreSQL). - VPS (DigitalOcean, Linode): More control. Use a process manager like PM2 to keep your Node.js app running. Set up a reverse proxy (nginx) and enable SSL with Let’s Encrypt.
- Docker: Containerize your app along with the SQLite file stored on a persistent volume.
After deployment, consider adding enhancements to make your aggregator more useful. You can implement a search feature using SQLite’s LIKE or Full‑Text Search (FTS5). You could allow users to subscribe to specific sources or categories, store user preferences in a simple JSON file or a separate table. If you want to include non‑RSS sources, you can add web scraping with Cheerio, but be cautious about legality and server load. Another popular feature is to send email digests of the top articles using a scheduled email service like SendGrid. Finally, you might want to integrate a read‑later service (e.g., Pocket API) so users can save articles. The possibilities are vast; this tutorial gives you a solid foundation to build upon.
Best Practices and Tips
1. Respect Rate Limits and Robots.txt
When fetching feeds, especially from smaller blogs, you should add a small delay between requests to avoid overwhelming the server. You can implement a simple throttling mechanism using a queue that waits 500 ms between each fetch. Additionally, always check the robots.txt of the website — some sites disallow automated access to their RSS feeds (though this is rare). If you scrape HTML content (not just RSS), you must respect the Crawl-delay directive and the Disallow rules. A good practice is to set a custom User‑Agent that identifies your aggregator and includes a contact email so site owners can reach you if needed.
2. Handle Feed Errors Gracefully
Feeds can become temporarily unavailable, change URLs, or return malformed XML. Your fetch function should catch exceptions and log them without crashing the entire update cycle. You might also want to track the number of consecutive failures for each feed and disable it after a threshold (e.g., 5 failures in a row) to avoid wasting resources. Store a error_count column in the feeds table and reset it on successful fetch. Notify yourself (via email or dashboard) when a feed has been disabled.
3. Optimize Database Queries with Indexing
As your database grows to tens of thousands of articles, queries will slow down if you don’t have indexes. Add indexes on columns that you frequently query or sort by: published_at, source, and link. In SQLite, you can create indexes like this:
db.exec('CREATE INDEX IF NOT EXISTS idx_articles_published ON articles(published_at DESC);');
db.exec('CREATE INDEX IF NOT EXISTS idx_articles_source ON articles(source);');
Also, consider setting a TTL (time‑to‑live) for old articles. For instance, you can delete articles older than 30 days to keep the database size manageable. Run a cleanup job daily: DELETE FROM articles WHERE published_at < datetime('now', '-30 days').
FAQ: Frequently Asked Questions
Q1: Is it legal to aggregate news from other websites?
Yes, in most cases, aggregating RSS feeds is legal because RSS is designed for syndication. However, if you display full articles (not just headlines and summaries), you may be violating copyright. Fair use varies by jurisdiction, but as a rule, show only the title and a brief excerpt (1-2 sentences) and always link back to the original source. Avoid re‑hosting images or large portions of text. If you scrape content beyond RSS, you absolutely must check the website’s terms of service and robots.txt.
Q2: How do I avoid duplicate articles across multiple feeds?
The INSERT OR IGNORE with a UNIQUE constraint on the link column is the simplest solution. However, some articles may appear with slightly different URLs (e.g., with tracking parameters). To handle that, you can normalize the link by stripping query parameters (except essential ones) before insertion. Another approach is to use a hash of the title+source as a unique key instead of the link, but that can still lead to duplicates if two different sources publish the same article with different titles.
Q3: What if an RSS feed doesn’t provide a publication date?
Some feeds omit the pubDate or isoDate fields. In that case, you should use the current timestamp when the article was first fetched (new Date().toISOString()). This is less accurate but ensures chronological ordering based on when you discovered the article. You can also try to parse the dc:date element if present. For a better experience, consider feeding articles through a date‑inference library or simply accept the fetch timestamp as the “discovered” date.
Q4: How can I add user authentication to my aggregator?
You can implement a simple session‑based authentication using express-session and a local user database. Alternatively, use OAuth providers (Google, GitHub) with Passport.js. The authenticated user can then have personalized source lists, bookmarks, and read‑later features. For a tutorial focused on aggregation, I skipped authentication to keep it simple, but adding it is a natural next step. Ensure you hash passwords with bcrypt and store sensitive data securely.
Q5: My aggregator is slow when fetching many feeds. How can I speed it up?
Use parallel fetching with Promise.all on batches of feeds, but be careful not to exceed the server’s connection limit or the sources’ rate limits. A good strategy is to fetch 5 feeds concurrently, wait for all to finish, then fetch the next 5. You can also use a connection pool for axios. On the database side, batch inserts (using a transaction) dramatically improve performance. Instead of inserting one article at a time, you can collect all articles from one feed and insert them in a single transaction wrapped in db.transaction(), which reduces disk I/O.
Conclusion
Building a news aggregator from scratch is an excellent project that teaches you practical skills in web scraping, data parsing, database design, API development, and front‑end rendering. By following the steps in this guide, you now have a working aggregator that can fetch, store, and display articles from multiple RSS sources, update them automatically, and present them in a clean web interface. Moreover, the architecture is modular enough that you can easily extend it to include web scraping, user accounts, machine‑learning‑based filtering, or even a mobile app front‑end. The real power of building your own aggregator lies in the ability to tailor the curation to your exact needs — whether that means filtering out certain keywords, surfacing breaking news faster, or combining niche sources that no existing service covers. I encourage you to experiment with additional features, contribute to the open‑source community by sharing your code, and never stop learning how to tame the firehose of the internet. Your news aggregator is now live; go ahead and make it your own.
| Library | Language | Ease of Use | Standards Supported | Performance |
|---|---|---|---|---|
rss-parser (npm) |
JavaScript (Node.js) | Very easy, promise‑based | RSS 2.0, Atom, JSON Feed | Good for moderate volumes |
feedparser (npm) |
JavaScript (Node.js) | Streaming, but more complex | RSS 2.0, Atom, RDF | Excellent for high throughput |
feedparser (Python) |
Python | Moderate, extensive flexibility | RSS, Atom, RDF, JSON Feed | Good |
SimplePie (PHP) |
PHP | Easy, well‑documented | RSS, Atom, many extensions | Moderate |
| Table | Column | Type | Notes |
|---|---|---|---|
articles |
id |
INTEGER PRIMARY KEY | Auto‑increment |
title |
TEXT NOT NULL | Article headline | |
link |
TEXT UNIQUE | URL to original article | |
summary |
TEXT | Snippet or first paragraph | |
source |
TEXT | Feed name for display | |
published_at |
DATETIME | Feed‑provided or fetch time | |
feeds |
id |
INTEGER PRIMARY KEY | Auto‑increment |
name |
TEXT | Display name | |
url |
TEXT UNIQUE | RSS/Atom feed URL | |
last_fetched |
DATETIME | Timestamp of last successful fetch |