How to bypass Cloudflare bot detection when scraping

Cloudflare sits in front of a big chunk of the web, and when it decides your scraper is a bot you get one of two things: a hard 403, or an interstitial that says it's checking your browser and loops forever. Neither one tells you why. That's the maddening part. You're left guessing which of a dozen checks you failed, changing your User-Agent, and wondering why nothing helps.

So let's take the mystery out of it. Below is what Cloudflare inspects, how to tell which wall you hit, and the fixes that still work in 2026, in the order I'd try them. The frame throughout is pulling public data at a reasonable pace, not hammering someone's login.

What the block is actually telling you

Cloudflare doesn't serve one block. It serves a few, and they mean different things.

• A plain 403 that says "Sorry, you have been blocked" is usually a firewall or WAF rule (often shown as error 1020). A rule looked at your request and said no. It was never a maybe.
• A 403 or 503 with a spinner and scripts loading from /cdn-cgi/challenge-platform/ is a challenge, not a ban. Check the response headers: a cf-mitigated: challenge header means Cloudflare swapped your content for a challenge page. The body comes back as text/html even when you asked for JSON.
• Pass a challenge and you get a cf_clearance cookie. That cookie is your proof of passage. Send it back on the next request and you skip the challenge until it expires. Change your IP or your TLS fingerprint and the cookie stops being valid, so you're challenged again.

Which one you're getting tells you where to aim. A firewall 403 is almost always an IP or fingerprint problem. A challenge that never clears is a "can you run our JavaScript like a real browser" problem. Two different fixes.

How Cloudflare decides you're a bot

Cloudflare scores every request across a stack of independent signals and rolls them into a trust score. No single check blocks you. The score does. Roughly from the network layer up:

• IP reputation. Datacenter ranges are known and scored badly. This is the first thing checked and the most common reason a clean-looking browser still gets blocked.
• TLS fingerprint. Before a single HTTP header arrives, your TLS ClientHello has a shape to it, hashed as JA3 or, these days, JA4. Cloudflare computes it from the handshake it terminates, so you can't fake it with a header. A default Python or curl handshake looks nothing like a browser's.
• HTTP/2 fingerprint. The SETTINGS frame, pseudo-header order, and stream priorities differ between real browsers and HTTP libraries.
• Headers and their order. Real Chrome sends a specific set of headers in a specific order, with client hints that match. A thin, out-of-order header set is a cheap tell.
• Active checks. Once JavaScript runs, Cloudflare reads canvas and WebGL output, navigator.webdriver, Chrome DevTools Protocol artifacts, and a pile of other environment properties.
• Behavior. Mouse paths, timing, scrolling, how fast you jump from page to page.

The signal that catches most scrapers isn't any one of these. It's incoherence. Your User-Agent claims Chrome 120, your TLS fingerprint says OpenSSL, your header set is three lines long. Any single mismatch reads as a high-confidence bot. This is why swapping your User-Agent string does nothing on its own: it changes the label, not the handshake underneath it.

One 2026 note: JA3 is legacy. Chrome randomizes its ClientHello extension order, which broke plain JA3, so Cloudflare and other vendors score on JA4 and normalized variants now. If an old blog told you to match a JA3 hash, that advice has rotted.

Step 1: read the response before you change anything

Before you touch your code, look at what you're getting. Half the time people rebuild the whole stack when a two-line check would've pointed at the IP.

curl -s -D - -o /dev/null https://example.com/ \
  | grep -i -E 'HTTP/|server|cf-mitigated|cf-ray'

A server: cloudflare line confirms who you're up against. A cf-mitigated: challenge header means you hit a challenge, not a ban, so the fix is a real browser, not a new IP. No cf-mitigated but a flat 403 points at a firewall rule keyed on your IP or fingerprint. Diagnose first, then act.

Step 2: fix the IP

Start here, because it's the highest-impact layer and the one people skip. If you're scraping from a cloud box, the target sees a datacenter IP, and on a Cloudflare-protected site that alone can end the request before your browser fingerprint ever matters.

For anything protected you want residential proxies, real IPs from home internet connections, so your traffic reads as an ordinary person. Rotate them for independent pages, and keep a sticky session when you're logged in so the IP doesn't change mid-flow. We go deeper on picking a pool in the residential proxy guide. A clean home IP at low volume might get through without one. At any real scale, it won't.

Step 3: fix the TLS and browser fingerprint

Now the handshake. You have two roads, and which one you take depends on whether the site runs a JavaScript challenge.

If it's only TLS fingerprinting, skip the browser. A library like curl_cffi impersonates a real browser's TLS and HTTP/2 handshake without launching anything heavy, so your JA4 lines up with the User-Agent you're claiming:

from curl_cffi import requests

r = requests.get(
    "https://example.com/api/products",
    impersonate="chrome",
    proxies={"https": "http://user:pass@gate.example-proxy.com:8080"},
)
print(r.status_code, r.text[:200])

That clears the passive fingerprint checks that stop most Python scrapers, and it's fast. But it runs no JavaScript, so it can't solve a "checking your browser" interstitial or a Turnstile widget. For those, you need a real browser.

For sites that throw a JavaScript challenge, drive a real browser that spoofs at the source. This is where Camoufox earns its place. It's a patched Firefox that hides automation tells at the binary level instead of injecting evasion scripts a detector can catch, and Cloudflare treats Firefox TLS differently from the Chromium tools it watches hardest. Two settings matter most here:

from camoufox.sync_api import Camoufox

with Camoufox(
    headless="virtual",   # virtual display on a Linux server, trusted more than pure headless
    geoip=True,           # match timezone, locale, and language to the proxy's exit IP
    humanize=True,
    proxy={
        "server": "http://gate.example-proxy.com:8080",
        "username": "user",
        "password": "pass",
    },
) as browser:
    page = browser.new_page()
    page.goto("https://example.com/")
    page.wait_for_load_state("networkidle")
    print(page.title())

geoip=True is the one people forget. It sets the browser's timezone, locale, and geolocation to match the proxy's exit IP, so a residential IP in Frankfurt doesn't show up next to a browser on New York time. Cloudflare cross-checks exactly that. On a Linux server, headless="virtual" runs the browser on a virtual display, which some detectors trust more than pure headless mode.

Step 4: getting past Turnstile and managed challenges

Cloudflare's managed challenge is adaptive. It reads your IP reputation, headers, and browser environment, then decides how hard to make you work. Get the earlier layers right and it often resolves silently, no puzzle shown, and hands you the cf_clearance cookie in the background. A genuine browser like Camoufox clears these invisible challenges far more often than a headless-looking Chromium does, which is most of why the browser layer matters here.

Turnstile is the widget version of the same logic. In its managed mode it only shows an interaction when the signals look automated, so a clean IP and a real browser usually get waved through without a click. When it does put a visible checkbox or puzzle in the way, you can't fingerprint past it, and solving by hand doesn't scale.

Step 5: slow down and act human

A perfect browser on a clean IP still gets flagged if it behaves like a machine. Instant page-to-page jumps, zero mouse movement, requests spaced exactly 500ms apart. Real people are messier than that.

• Randomize delays between actions. A fixed sleep is its own fingerprint.
• Let the page settle and reuse the cf_clearance cookie you earned instead of re-triggering a challenge on every request.
• Give each identity its own browser process. Several identities in one Firefox process can be correlated regardless of proxy rotation. One identity, one process.
• Don't run in a mathematically perfect rhythm. Jitter everything.

What stopped working

A few methods that used to work are dead ends now, and chasing them wastes days.

• User-Agent swapping. Changing the string doesn't change the TLS handshake behind it. If the handshake says Python, the fanciest Chrome UA string in the world still gets you blocked.
• cloudscraper and old header-spoof scripts. They were built for a version of Cloudflare's challenge that no longer ships. On current defenses they mostly loop.
• Datacenter proxy rotation. Rotating through more flagged IPs is still flagged IPs. Reputation is the check, and cloud ranges fail it.

No single trick beats a modern Cloudflare setup, and anyone selling you one is selling last year's bypass. What works is stacking the layers: a clean residential IP, a matching handshake and browser, geo that lines up, human-shaped timing, and a solver fallback for the rest. Miss one and the score tips against you.

FAQ

Is it legal to scrape a site behind Cloudflare?

Scraping publicly available data is broadly legal in a lot of places, and Cloudflare being in front of a site doesn't change that either way. But the site's terms of service, its robots rules, rate limits, and privacy law still apply, and they vary by where you and the site sit. This isn't legal advice. If a project is high-stakes, talk to a lawyer before you point a scraper at someone's servers.

Why do I still get blocked with residential proxies?

Because the IP is only one layer. A clean residential IP paired with a raw Python handshake is still incoherent, and Cloudflare scores the mismatch. Fix the TLS fingerprint and the browser too, and make sure your browser's timezone matches the proxy's exit location.

Can I bypass Cloudflare without running a browser?

Sometimes. If the target only does passive TLS fingerprinting with no JavaScript challenge, a TLS-impersonating client like curl_cffi plus a residential proxy is enough, and far faster than a browser. The moment you see a "checking your browser" page or a Turnstile widget, you need something that runs JavaScript.

What is the cf_clearance cookie?

It's the token Cloudflare hands you after you pass a challenge. Send it back on later requests and you skip the challenge until it expires. It's tied to your IP and fingerprint, so reusing it from a different IP or a different TLS stack won't work.