Web scraping with Python: requests, parsing, proxies and retries

Set up the project

Create a virtual environment and install two packages: python -m pip install requests beautifulsoup4. Everything else in the script comes from the standard library.

For crawls of many thousands of pages with scheduling, deduplication and pipelines built in, a framework such as Scrapy saves work. The approach below is the right size for focused jobs and for learning what a framework does for you.

Step 1: fetch a page properly

• Use a `Session`. The requests documentation notes that a session reuses the underlying TCP connection to the same host and persists cookies — faster, and closer to how a browser behaves.
• Always set a timeout. Without one, requests waits indefinitely. A tuple sets connect and read timeouts separately: timeout=(5, 30).
• Identify yourself. Set a User-Agent that names your project and gives a contact. It is courteous and lets an operator reach you instead of blocking you.
• Check the status. raise_for_status() turns 4xx and 5xx responses into exceptions so a 404 page is never parsed as data.

Step 2: parse the HTML

Beautiful Soup turns HTML into a tree you query with CSS selectors: soup.select("article.product") for every product card, select_one("h2") inside each for the name. Use get_text(strip=True) for text and read attributes like a dictionary. Turn relative links into absolute ones with urllib.parse.urljoin.

Validate as you go. If a card has no price, skip it and count it; do not write an empty or guessed value. Before writing selectors at all, compare the page source with what the browser shows: if the data is missing from the source, the page loads it from a JSON endpoint you can usually request directly. Only if that fails do you need a browser — Playwright vs Puppeteer vs Selenium compares the options.

Step 3: follow pagination and stop safely

• Follow the page's own next link — a[rel=next] where the site uses it — rather than incrementing page numbers blindly.
• Cap the number of pages per run so a pagination bug cannot loop forever.
• Keep a set of URLs already visited if the site's pagination can repeat.
• Pause between pages. One to two seconds with a little randomness is a reasonable default for a small job; the site's robots.txt or terms may ask for more.

Step 4: retries and backoff

requests does not retry failed requests by default, but it accepts urllib3's Retry through an HTTPAdapter. With backoff_factor=1, urllib3 sleeps backoff_factor × 2^(retries so far) seconds and starts backing off after the second try, so the first retry is immediate and the following ones wait about 2, 4 and 8 seconds; backoff_jitter adds randomness and backoff_max caps the wait at 120 seconds by default.

status_forcelist decides which status codes are retried — 429 and the transient 5xx codes are the usual set. respect_retry_after_header is on by default and honours Retry-After on 413, 429 and 503. Limit allowed_methods to safe methods such as GET and HEAD. For the reasoning behind each code, see handling 429 rate limits.

Step 5: add a proxy

A proxy URL with credentials goes into a proxies dictionary for both schemes. The requests documentation warns that a proxy set on session.proxies can be overridden by environment variables, so the script passes proxies= on every request. For SOCKS5, install requests[socks] and use socks5h:// if host names should be resolved on the proxy.

Use a rotating username for independent pages and a sticky one when several requests must share an IP and cookies — rotating vs sticky sessions explains the trade-off. Proxies spread a legitimate workload across addresses and let you choose the country; they do not make it acceptable to send more than a site is willing to serve. Setup details for this library are on the Python requests page; pools are on residential proxies.

Step 6: be a polite scraper

• Read robots.txt before crawling a host and skip what it disallows — respecting robots.txt shows how in Python.
• Collect only public pages a logged-out visitor sees. Never work around a login, paywall or CAPTCHA.
• Keep concurrency per host low, and slow down when the site returns 429 or 503.
• Store only the fields you need, and leave personal data out unless you have a lawful basis for it.