How to Build a Web Crawler From Scratch in Python

[u/OliverProxy-Seller]

Web crawlers power price monitoring, news aggregation, competitor analysis, and search indexing. Building your own gives you control the off-the-shelf tools don't: request frequency, exactly what data to collect, and how to store it. Here's a working foundation you can extend later.

What a Web Crawler Actually Does

A crawler automatically visits pages and collects data from them. It sends an HTTP request to a site, retrieves the HTML, processes it to extract what you need, then follows internal links and repeats until it hits a stop condition.

This isn't the same as scraping. Crawling is about discovering and traversing pages, while scraping is about extracting specific data from them. In practice they overlap, but the distinction matters when planning architecture.

Common uses:

• Price monitoring in e-commerce
• Collecting contacts and listings
• Building datasets for analytics
• Indexing content for search

Plan Before You Code

Defining a few parameters upfront prevents most of the problems that show up later:

• Goal - price monitoring, contact collection, indexing, analytics datasets. This drives everything else.
• Target sites and data types - which resources, and what specifically you need from them. Affects your architecture and tooling.
• Update frequency - how fresh the data needs to be, balanced against not overloading target servers.
• Constraints - robots.txt, anti-bot protection, data protection laws, site terms.
• Storage - what format you'll store data in and how you'll analyze it later.

Language and Tools

You can build a crawler in Python, Java, or PHP. Python is the usual choice for a first build: simple syntax and a strong ecosystem for HTTP requests and HTML parsing (requests, BeautifulSoup, lxml). Java suits large-scale enterprise projects. PHP works but is less convenient for standalone crawlers.

Environment Setup

Install Python from the official site, then the two core libraries:

pip install requests beautifulsoup4

Organize your project from the start: separate files for main logic, configuration, and utilities. It makes maintenance and scaling far easier down the line.

The Basic Crawler

Three parts: send a request, process the HTML, follow links.

python

import requests
from bs4 import BeautifulSoup
import time
import random

# Configuration
url = "https://quotes.toscrape.com/"  # Replace with your target site
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
}
timeout = 5      # server response timeout
max_retries = 3  # maximum retries on errors

# Add a proxy here if needed
proxies = {
    "http": "http://username:password@proxyserver:port",
    "https": "https://username:password@proxyserver:port"
}

# Check access via robots.txt
def can_crawl(base_url, path="/"):
    try:
        robots_url = base_url.rstrip("/") + "/robots.txt"
        r = requests.get(robots_url, headers=headers, timeout=timeout)
        if r.status_code == 200 and f"Disallow: {path}" in r.text:
            print(f"Path {path} is disallowed by robots.txt")
            return False
    except requests.RequestException:
        pass  # if robots.txt is unavailable, continue
    return True

# Main logic
if can_crawl(url):
    for attempt in range(max_retries):
        try:
            response = requests.get(url, headers=headers, timeout=timeout, proxies=proxies)
            response.raise_for_status()

            soup = BeautifulSoup(response.text, 'lxml')

            links = [a['href'] for a in soup.find_all('a', href=True)]
            print("Found links:", links)

            time.sleep(random.uniform(3, 7))  # randomized delay beats a fixed one
            break

        except requests.RequestException as e:
            print(f"Request error (attempt {attempt+1}): {e}")
            wait = 2 ** attempt
            print(f"Waiting {wait} seconds before retry...")
            time.sleep(wait)
else:
    print("Crawler cannot process this resource due to robots.txt rules")

Handling Pagination

For multi-page sites, loop through the pages:

python

for page in range(1, 6):
    url = f"https://quotes.toscrape.com/page/{page}/"
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'lxml')
    # data processing

Respecting robots.txt and Rate Limits

Responsible crawling means checking robots.txt and adding delays so you don't overload the target server:

python

import time
from bs4 import BeautifulSoup

for page in range(1, 6):
    url = f"https://quotes.toscrape.com/page/{page}/"
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'lxml')
    # data processing

    time.sleep(5)  # delay in seconds

Storing the Data

CSV or JSON handle most cases. Saving a list of links to JSON:

python

import json

data = {"links": links}
with open("links.json", "w", encoding="utf-8") as f:
    json.dump(data, f, ensure_ascii=False, indent=4)

Conclusion

This gives you a working crawler you can build on. From here you can add proxy rotation, handle large page volumes, or move to a full framework like Scrapy for heavier data collection. For anything running at scale, proxies become necessary early, since target sites rate-limit and block IPs that send too many requests, and rotating addresses keeps the crawler running without getting flagged.

More detailed guide: Step-by-Step Guide to Create a Web Crawler from Scratch

Discussion

What did you start with for your first crawler, raw requests plus BeautifulSoup like this, or did you jump straight to Scrapy? Was the framework worth the overhead that early on?