How to avoid IP bans and rate limits when scraping

You didn't touch the scraper. You didn't touch the site. Yesterday it pulled two thousand pages clean. Today every third request comes back 429, or worse, every single request comes back 403 no matter which proxy you route through.

The gut reaction is to blame your fingerprint: swap the user agent, bolt on a stealth browser, reach for a fancier TLS impersonation library. Skip that.

Most bans and rate limits are a math problem, not a disguise problem: too many requests, too fast, from too few IPs, with zero patience for a server that's already telling you to slow down.

Fix the math and most of it stops. Here's how, plus a retry loop you can paste into a scraper today.

Rate limit or reputation ban?

Both look the same from your terminal: the script stops getting data. But they're different problems, and treating one like the other wastes hours.

• A 429 Too Many Requests, especially with a Retry-After header, is the server talking to you directly. It's telling you exactly how long to wait. Pacing problem, easy fix.
• A 403 (or a silent redirect to a junk page) on every IP you try, including a brand-new residential one you've never sent a single request from, is reputation. Something about the request, an account, a device fingerprint, a whole ASN, got blocklisted. Slowing down doesn't touch it.
• A response that gets slower and slower, or starts sneaking a CAPTCHA onto pages that never had one, is usually a rate signal wearing a disguise. Same fix as the 429: back off.

The fastest way to tell them apart: send the exact same request from a completely fresh IP with zero history on the target. Succeeds? You were rate-limited on the old one. Still fails? You're not looking at a pacing problem, and no amount of backoff timer fixes that.

Check the actual headers before guessing:

curl -s -D - -o /dev/null https://example.com/products \
  | grep -i -E 'HTTP/|retry-after|x-ratelimit'

See retry-after or an x-ratelimit-remaining counting down toward zero? That's confirmation: this is a quota, and quotas reset.

Concurrency: the knob everyone forgets

Most people tune the delay between requests and never touch how many are running at once. That's backwards. Fire 40 requests at the same time through one session and you can blow through a per-minute limit in under a second, no matter how generous your delay looks averaged over the next sixty.

Cap concurrency per identity, not just globally. A semaphore works fine:

import asyncio
import httpx

CONCURRENCY = 5
sem = asyncio.Semaphore(CONCURRENCY)

async def fetch(client, url):
    async with sem:
        resp = await client.get(url)
        return resp

async def run(urls):
    limits = httpx.Limits(max_connections=CONCURRENCY, max_keepalive_connections=CONCURRENCY)
    async with httpx.AsyncClient(limits=limits, timeout=30) as client:
        return await asyncio.gather(*(fetch(client, u) for u in urls))

Five in flight per proxy is a reasonable starting point for most sites. Push it up slowly while you watch the error rate, not the other way around.

Exponential backoff with jitter

When a request fails, the worst thing you can do is retry it immediately. The second worst thing is retrying after a fixed delay, because if you've got ten workers, they all picked the same delay, and they all slam the server again at the exact same moment.

The fix is two ideas stacked together. First, back off exponentially: double the wait after each failure, so attempt one waits a second, attempt two waits two, attempt three waits four, and so on. Second, add randomness, jitter, so your workers don't retry in lockstep. "Full jitter" is the simplest version that actually works: pick a random delay between zero and the exponential cap, not a fixed wobble around it.

import random

def backoff_delay(attempt):
    ceiling = min(MAX_DELAY, BASE_DELAY * (2 ** attempt))
    return random.uniform(0, ceiling)

Cap the delay so attempt eight doesn't leave you waiting four minutes for one page. Cap the retry count too. Six or seven tries is plenty. Past that you're not being patient, you're being stubborn, and the right move is to log the failure and move on to the next URL.

Honor Retry-After like it's a contract

If the server sends a Retry-After header, use it. Don't estimate a shorter wait because your backoff formula said 8 seconds and the header said 30. The header wins, every time, because it's the server telling you the exact condition under which it'll accept you again.

It shows up in two formats. Sometimes it's a plain integer, seconds to wait. Sometimes it's a full HTTP date. Code that only handles the integer case breaks silently on the other one:

from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def retry_after_seconds(headers):
    value = headers.get("Retry-After")
    if not value:
        return None
    if value.isdigit():
        return int(value)
    try:
        target = parsedate_to_datetime(value)
        return max(0, (target - datetime.now(timezone.utc)).total_seconds())
    except (TypeError, ValueError):
        return None

No Retry-After in sight? Check for X-RateLimit-Reset instead, common on API-style endpoints, usually a Unix timestamp for when your quota refills. Only fall back to your own exponential formula when neither header shows up.

Spread load across a rotating residential pool

Perfect backoff still runs into a ceiling, because a single IP only gets so much trust no matter how politely it behaves. Scrape a few thousand product pages a day from one address and some sites will throttle you even when every request was properly paced.

Residential proxies raise that ceiling by spreading your traffic across many independent identities instead of one. Rotate to a new IP for independent page fetches, and hold a sticky session on one IP when you need continuity, mid-login, mid-checkout, anywhere state has to survive across requests. We cover picking a pool in the residential proxy guide and the rotation mechanics in our Python rotation walkthrough.

One thing a bigger pool won't do: fix bad pacing. Ten thousand IPs each firing at max concurrency with no backoff just means you get banned on ten thousand IPs instead of one, faster than you'd think.

Reuse sessions instead of hammering new ones

Every fresh connection costs the server a TCP handshake and a TLS handshake before it even sees your request. Open a new connection for every page and you look like a thousand strangers showing up one after another instead of one visitor browsing around. That pattern alone gets flagged on plenty of sites, independent of your actual request rate.

Reuse a client instead of building a new one per request:

import httpx

urls = ["https://example.com/products/1", "https://example.com/products/2"]

with httpx.Client(
    limits=httpx.Limits(max_keepalive_connections=10, max_connections=10),
    timeout=30,
) as client:
    for url in urls:
        resp = client.get(url)
        print(resp.status_code)

Same goes for cookies and any session token from a login. Throwing those away and re-authenticating on every request is slower for you and reads as suspicious to the site, since a real user's session just sits there quietly reused for the length of a visit. If you're scraping anything behind a login, this matters even more; see our walkthrough on handling sessions behind a login.

A runnable backoff-and-retry loop

Put it together and here's a fetch wrapper that handles the whole decision tree: succeed, retry with backoff, honor Retry-After, or bail out because you've hit a wall that backoff can't fix.

import random
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

import httpx

MAX_RETRIES = 6
BASE_DELAY = 1.0        # seconds, doubles each retry
MAX_DELAY = 60.0         # never wait longer than this


class BannedError(Exception):
    pass


class RetriesExhausted(Exception):
    pass


def retry_after_seconds(headers):
    value = headers.get("Retry-After")
    if not value:
        return None
    if value.isdigit():
        return int(value)
    try:
        target = parsedate_to_datetime(value)
        return max(0, (target - datetime.now(timezone.utc)).total_seconds())
    except (TypeError, ValueError):
        return None


def backoff_delay(attempt):
    ceiling = min(MAX_DELAY, BASE_DELAY * (2 ** attempt))
    return random.uniform(0, ceiling)


def fetch_with_backoff(client, url, **kwargs):
    for attempt in range(MAX_RETRIES):
        resp = client.get(url, **kwargs)

        if resp.status_code == 200:
            return resp

        if resp.status_code == 429:
            wait = retry_after_seconds(resp.headers)
            delay = wait if wait is not None else backoff_delay(attempt)
            print(f"429 on {url}, waiting {delay:.1f}s (attempt {attempt + 1}/{MAX_RETRIES})")
            time.sleep(delay)
            continue

        if resp.status_code == 403:
            raise BannedError(f"403 on {url}, this isn't a pacing problem")

        resp.raise_for_status()

    raise RetriesExhausted(f"gave up on {url} after {MAX_RETRIES} attempts")


if __name__ == "__main__":
    with httpx.Client(timeout=30) as client:
        try:
            resp = fetch_with_backoff(client, "https://example.com/products")
            print(resp.status_code, len(resp.text))
        except BannedError as e:
            print(f"stop and rotate: {e}")

Notice what it doesn't do: retry a 403 on the same connection. That status means backoff won't help, so the function throws immediately instead of burning six attempts and thirty seconds finding out the slow way. Catch BannedError upstream and that's your signal to rotate to a new IP, not to wait longer.

You got banned anyway. Now what?

It happens. Even careful scrapers eat a ban sometimes, a site changes its threshold overnight, or you underestimated how aggressive it'd be. A few things actually help:

• Stop hitting the blocked identity immediately. Retrying through a banned IP right away can turn a short block into a longer one on some systems. Move on.
• Rotate to a fresh IP, then throttle harder than before. A new proxy with the old aggressive pacing just earns a second ban, usually faster than the first.
• If you were logged in, the ban may follow the account, not the connection. A new IP won't save you if the block is tied to a user or a device fingerprint tied to that account. Slow the whole account down, not just the connection.
• Give it time before you test again. Plenty of blocks are on a timer. Hammering a fresh identity against the same endpoint to "check" if you're still banned is just more of the pattern that got you banned.

FAQ

What's the real difference between a 429 and a 403?

A 429 is the server explicitly rate-limiting you, usually with a Retry-After header telling you exactly when to come back. It's temporary and it's about pace. A 403 that shows up on every IP you try, including fresh ones, is a reputation block. It's not about speed anymore, and waiting longer doesn't fix it.

How long should I wait after hitting a rate limit?

Whatever Retry-After says, if it's present, use that over your own guess. No header? Start an exponential backoff around a second, double it each retry, add jitter, and cap it around a minute so a bad run doesn't stall forever.

Will more proxies fix a reputation ban?

More IPs raise your ceiling, but they don't fix bad behavior. Point the same aggressive, unpaced scraper at a bigger pool and you'll rack up bans across more addresses, just faster. Fix the concurrency and backoff first, then let a larger pool handle the volume that's left.

Is it legal to scrape a site slowly and politely?

Pulling publicly available data at a reasonable pace is broadly fine in a lot of places, but "reasonable" isn't a legal term, and terms of service, robots rules, and local law still apply and vary by where you and the target sit. We break it down in our legality guide. This isn't legal advice, and a high-stakes project deserves an actual lawyer, not a blog post.