How to avoid bot detection when scraping

People treat bot detection like a single wall to climb. It isn't. Vendors like Cloudflare, DataDome, and Akamai build a risk score from a stack of independent signals, and any one of them can sink you on its own. Nail the browser and forget the IP, you're blocked. Nail both and hammer the site fifty times a second, blocked.

So think in layers. Here they are, roughly bottom to top.

1. IP reputation

The first thing a site sees. Datacenter IP ranges are known and scored badly, so on any protected target you want residential or mobile IPs that look like ordinary home users. This is the single highest-impact layer, and it's where most scrapers lose before the browser even loads.

We go deep on choosing a pool in the residential proxy guide. The short version: real residential IPs, rotate for independent pages, sticky sessions for logins.

2. TLS and HTTP fingerprint

Before a single line of HTML loads, your client's TLS handshake has a shape to it, often called a JA3 fingerprint, and plain Python HTTP libraries have handshakes that look nothing like a real browser's. HTTP/2 frame ordering gives you away the same way. A real browser engine sends real browser handshakes, which is one more reason to drive an actual browser instead of raw requests against a serious target.

3. Browser fingerprint

This is the big one people obsess over, and it matters, just not in isolation. Sites read navigator.webdriver, canvas and WebGL rendering, installed fonts, screen size, audio stack, and dozens of other properties, then check whether they're internally consistent and whether they smell automated.

Injected-JavaScript stealth tools patch these from the outside and can be caught doing it. A browser that spoofs at the binary level, like Camoufox, is much harder to unmask because the values read back as native. If you only fix one layer beyond the IP, fix this one, and fix it at the source.

4. Headers and their order

Real browsers send a specific set of headers in a specific order, with matching User-Agent, Accept-Language, and client hints. A mismatched or out-of-order header set is a cheap, reliable tell. If you drive a real browser, you get this for free. If you're hand-rolling requests, you have to get it exactly right, which is fiddly and easy to botch.

5. Timing and behavior

Even a perfect browser on a perfect IP gets caught if it acts like a machine. Instant page-to-page jumps, zero mouse movement, clicks landing dead-center on every button, requests spaced exactly 800ms apart. Real people are messier.

• Add randomized delays between actions, not a fixed sleep.
• Move the cursor along curved paths before clicking. Camoufox's humanize option does this, though it won't randomize your timing for you.
• Scroll before you interact. People read before they click.
• Don't run in a mathematically perfect rhythm. Jitter everything.

6. CAPTCHAs and active challenges

Get everything else right and a site may still throw a challenge, especially on login or checkout. A real browser slips past invisible challenges more often than a headless-looking one does, but visible reCAPTCHA and hCaptcha are a wall you can't fingerprint your way through. Solving them by hand doesn't scale, so for volume you hand the challenge to a solving service that returns a token your script submits.

Put it together

No single trick beats a modern anti-bot system. What works is stacking clean IPs, a real browser that spoofs at the source, correct headers, and human-shaped behavior, then adding a CAPTCHA fallback for the times all of that still isn't enough. Miss a layer and the score tips against you.