Rotating residential proxies in Python

Most "how to rotate proxies in Python" tutorials hand you a list of IPs and a random.choice(). For a static set of datacenter proxies, fine. For residential proxies, that's usually the wrong mental model, and it's how people burn a pool in an afternoon.

Here's the part nobody says up front: with a residential provider, you rarely rotate in code at all. The provider's gateway rotates for you. Your real job is deciding when the IP should change, not shuffling a list.

This guide covers both worlds. Rotating a pool you actually hold, rotating through a backconnect gateway, holding one sticky IP for a login, and what to do when a target bans you mid-run. All in Python, most of it paired with a stealth browser so the IP isn't the only thing you get right.

Who actually does the rotating

Rotating and sticky aren't two products you buy. They're one pool used two ways, and the switch usually lives in your proxy username, not your code.

A residential provider gives you a single gateway endpoint, one host and port, called a backconnect gateway. You send every request to that same address. Behind it, the provider picks a fresh residential IP per request by default and routes your traffic out through it. You never see the individual IPs. You never maintain a list.

So "rotating proxies in Python" is often one line: point your client at the gateway and let it do the work. The interesting decisions are about sessions, which is a section down.

When do you rotate a list yourself? Two cases. You bought a fixed set of datacenter or ISP proxies as raw ip:port lines. Or you're spreading load across a few gateway accounts. Both are real, so let's do that first.

Rotate a pool yourself in Python

If you're holding a list, rotation is a few lines. Round-robin gives every proxy equal turns in a predictable order:

import itertools

PROXIES = [
    "http://user:pass@45.12.0.1:8000",
    "http://user:pass@45.12.0.2:8000",
    "http://user:pass@45.12.0.3:8000",
]

pool = itertools.cycle(PROXIES)      # round-robin, forever
proxy = next(pool)                   # grab the next IP each request

Want unpredictable order instead? Swap in random.choice(PROXIES). For independent page fetches it barely matters. What matters is that you don't hammer one IP while the rest sit idle.

Wiring that into requests looks like this:

import itertools, requests

pool = itertools.cycle(PROXIES)

for url in urls:
    proxy = next(pool)
    r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=30)
    print(url, r.status_code)

That's the whole "rotate proxies" trick people write two hundred lines for. But raw requests against a protected site gets flagged on the TLS handshake, long before the IP even matters. On anything with real defenses, you want a browser.

Rotate through a Camoufox browser

Camoufox is a stealth Firefox build that spoofs your fingerprint at the binary level. It takes a proxy the exact same way Playwright does, a dict with server, username, and password:

from camoufox.sync_api import Camoufox

proxy = {
    "server": "http://gate.example-proxy.com:8080",  # backconnect gateway
    "username": "user",
    "password": "pass",
}

with Camoufox(proxy=proxy, geoip=True, humanize=True) as browser:
    page = browser.new_page()
    page.goto("https://api.ipify.org?format=json")
    print(page.content())   # a different exit IP most runs

Because the server is a rotating gateway, each fresh browser launch tends to exit from a new residential IP. No list, no cycling. The gateway rotates; you just start the browser.

geoip=True is the setting that keeps the rest honest. Camoufox sends a request through the proxy to find its exit IP, then sets the browser's timezone, locale, language, and geolocation to match it. A German exit IP behind a New-York clock is one of the cheapest bans a site can hand out, and this closes that gap for free. There's a full walkthrough of the browser half in the Camoufox guide.

Rotating vs sticky sessions

Rotate on every request and you'll wreck any flow that has state. Log in, and the site watches your "session" jump from Ohio to Portugal between two clicks. Instant flag.

So for logins, carts, and anything multi-step, you hold one IP with a sticky session. On a residential gateway you don't change the server for this. You change the username. Most providers read a session token out of it:

import uuid
from camoufox.sync_api import Camoufox

def proxy_for(session_id):
    return {
        "server": "http://gate.example-proxy.com:8080",
        "username": f"user-session-{session_id}",  # same token = same IP
        "password": "pass",
    }

session = uuid.uuid4().hex[:8]          # hold one IP for this whole flow
with Camoufox(proxy=proxy_for(session), geoip=True) as browser:
    page = browser.new_page()
    page.goto("https://example.com/login")
    # log in, add to cart, all on the same exit IP

Reuse the token to keep the IP. Change the token to jump to a new one. The exact spelling varies by provider (-session-, -sessid-, cc-us-sid-, sometimes a duration like sesstime-1800), so check your provider's docs for the precise format. The idea is identical everywhere.

Rule of thumb: independent pages, rotate per request. Anything with a login, one sticky session per worker. More on picking a pool that does both well in the residential proxy guide.

Back off and retry when you get banned

Even a clean setup eats the occasional block. The move isn't to retry the same IP harder. It's to back off, grab a fresh session, and relaunch, which hands you a new exit IP and a new fingerprint in one shot.

import random, time, uuid
from camoufox.sync_api import Camoufox

BLOCK_TEXT = ("Access denied", "unusual traffic", "verify you are human")

def fresh_proxy():
    session = uuid.uuid4().hex[:8]
    return {
        "server": "http://gate.example-proxy.com:8080",
        "username": f"user-session-{session}",   # new session = new IP
        "password": "pass",
    }

def fetch(url, max_tries=4):
    for attempt in range(max_tries):
        try:
            with Camoufox(proxy=fresh_proxy(), geoip=True, humanize=True) as browser:
                page = browser.new_page()
                resp = page.goto(url, wait_until="domcontentloaded", timeout=45_000)
                body = page.content()
                status_bad = resp is not None and resp.status in (403, 429)
                text_bad = any(m in body for m in BLOCK_TEXT)
                if not (status_bad or text_bad):
                    return body
        except Exception as err:
            print(f"attempt {attempt + 1}: {err}")
        time.sleep(2 ** attempt + random.random())   # backoff + jitter
    return None

Three things earn their keep here. The 2 ** attempt backoff spaces retries out instead of stampeding a target that's already annoyed with you. The jitter keeps a fleet of workers from retrying in lockstep. And tearing the browser down between attempts means the next try looks like a genuinely different visitor, not the same bot knocking twice.

For a light job, four tries with backoff clears most transient blocks. For a hard 403 on every IP, stop. That's the site telling you the IP layer isn't your problem, and no amount of rotation fixes a fingerprint or a behavior tell.

The leak that deanonymizes you

Here's the one that quietly ruins runs. If your proxy goes unreachable mid-scrape, Firefox, and so Camoufox, falls back to a direct connection. Your real IP hits the target wearing a perfect fake fingerprint, and you never see an error. The scrape "works." You just deanonymized yourself.

Guard against it by checking the exit IP before you trust the session:

MY_REAL_IP = "203.0.113.7"   # your egress, fetched once with no proxy

with Camoufox(proxy=proxy, geoip=True) as browser:
    page = browser.new_page()
    page.goto("https://api.ipify.org")
    exit_ip = page.inner_text("body").strip()
    assert exit_ip != MY_REAL_IP, "proxy fell through, aborting"

The other quiet killer is the geo mismatch from earlier, which is why geoip=True isn't optional on serious targets. And rotation has its own failure mode: over-rotate inside a logged-in session and you look more like a bot than if you'd never rotated at all. The IP is one layer of several. The full bot-detection checklist covers the rest.

Put it together

Let the gateway rotate per request for independent pages. Pin a sticky session for anything with a login. Match the browser to the IP with geoip, verify the exit before you trust it, and back off to a fresh session when a target pushes back. Do that and rotation stops being the thing that breaks your scraper and goes back to being plumbing you don't think about.