How to scrape Google search results without getting blocked

Type the same handful of searches into Google fifteen times in two minutes from one IP, and you'll meet a page that says "Our systems have detected unusual traffic from your computer network," with a reCAPTCHA checkbox sitting between you and your results. Most sites give a scraper a few hundred requests before they blink. Google gives you a few dozen.

That's the whole problem in one line: Google fingerprints harder and reacts faster than almost anything else you'll scrape. Rank trackers, price watchers, anyone building a dataset out of search results, they all hit this same wall eventually. Below is the setup that actually holds up in 2026, layer by layer, plus something most guides skip: sometimes the right move is not scraping it yourself at all.

Why Google walls you off so fast

Google doesn't publish its bot-detection rulebook, and anyone who says they've fully reverse-engineered it is selling something. But people who run this for a living agree on the rough shape of it: how many requests are coming from your IP and how fast, whether your cookies and session look continuous from one request to the next, and a browser-environment check that likely feeds something similar to an invisible reCAPTCHA v3 score. Trip enough of those at once and you're redirected to a captcha wall before a single result renders. It's a different flavor of aggressive than Cloudflare or DataDome, too: those vendors protect thousands of sites and score everyone the same way, while Google is scoring you against nothing but its own search traffic, at a volume nobody else on the internet sees.

Check what you're actually getting back before you touch any code:

curl -s "https://www.google.com/search?q=web+scraping" \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
  | grep -o "unusual traffic"

If that prints anything, you got walled on request one, and no amount of clever parsing further down the pipeline fixes an IP-and-fingerprint problem. Fix the cause, not the symptom.

The honest option: use an API instead

Before you build any of this, know that Google sells a version of it, sort of. The Custom Search JSON API gives you 100 free queries a day, then bills you past that, and by default it searches an engine you configure rather than the open web. Turn on "search the entire web" and it gets closer, but the ranking and layout still won't match what a logged-out person sees in a browser.

Closer to the real thing: third-party SERP APIs like SerpApi, Serper, and Bright Data's SERP API scrape Google's actual results pages behind their own proxy and browser infrastructure, then hand you clean JSON. Pricing typically runs somewhere between $1 and $5 per thousand searches depending on how deep you need to go, organic only versus every panel and ad slot, and it's worth comparing against what a proxy pool alone costs before you assume building it yourself is cheaper.

So why scrape it yourself? Control, mostly. An API gives you what its parser decided to extract, and if you need something outside that shape, a specific render, a params combination nobody built a field for, you're back to doing it by hand. It's also the better deal at real scale if you already run proxy infrastructure for other jobs, since adding Google costs you almost nothing extra. Neither of those apply to you? Go buy the API and skip the rest of this page.

Layer 1: get off datacenter IPs

Start here. It's the highest-impact change and the one people skip because it costs money instead of time. Scrape from a cloud box and Google sees a datacenter IP right away, a known range with a bad reputation, and that alone can end the request before your browser fingerprint is worth anything.

You want residential proxies: real IPs handed out by ISPs to actual home connections, so your traffic reads as a person on their laptop instead of a rack in a data center. Rotate across independent searches, keep a session sticky while you're paging through one query's results, and don't run hundreds of queries through the same exit. The residential proxy guide covers picking a pool, and the rotation guide covers wiring it into Python without leaking your real IP on a failed request.

Layer 2: drive a real browser

A plain HTTP client with a browser User-Agent slapped on it doesn't hold up here. Google's checks lean on things a bare request can't fake: a JavaScript environment that behaves like a browser, canvas and font output, timing between page events. Playwright-stealth patches some of that from the outside and is fine for lower-stakes jobs, but for anything you actually depend on, Camoufox is the safer bet: a patched Firefox that fixes automation tells at the binary level instead of injecting scripts a detector can catch.

from urllib.parse import quote_plus
from camoufox.sync_api import Camoufox

def search_google(query, proxy):
    with Camoufox(headless="virtual", geoip=True, humanize=True, proxy=proxy) as browser:
        page = browser.new_page()
        page.goto(f"https://www.google.com/search?q={quote_plus(query)}&num=10&hl=en&gl=us")
        page.wait_for_selector("#search")

        return page.eval_on_selector_all(
            "#search a:has(h3)",
            "els => els.map(el => ({title: el.querySelector('h3').innerText, url: el.href}))",
        )

proxy = {"server": "http://gate.example-proxy.com:8080", "username": "user", "password": "pass"}
print(search_google("best residential proxies 2026", proxy))

geoip=True matters more here than on most targets: it lines up the browser's timezone and locale with the proxy's exit location, and a proxy exit in São Paulo paired with a browser clock still set to Berlin time is exactly the kind of mismatch that reads as suspicious. One more thing worth knowing: Google's result container class names are auto-generated and rotate every so often. Anchor your selectors on structure instead, like "the link wrapping an h3 inside the results container," and your parser survives the next redesign instead of silently returning nothing.

Layer 3: pace it like a person

This might be the layer that matters most for Google specifically. Measuring request patterns at web scale is the company's whole business, so query velocity from a single IP is something it almost certainly weighs harder than most sites bother to.

Nobody outside Google will give you an exact number, and it moves. As a working rule: a few dozen searches an hour from one residential IP is comfortable, a few hundred is pushing it, and a perfectly even gap between requests gets noticed no matter the volume, since real people don't search on a metronome.

import random
import time
from collections import defaultdict

MAX_PER_PROXY_PER_HOUR = 40
usage = defaultdict(int)

def throttled_search(query, proxy_pool):
    proxy = min(proxy_pool, key=lambda p: usage[p["server"]])
    if usage[proxy["server"]] >= MAX_PER_PROXY_PER_HOUR:
        raise RuntimeError("proxy pool exhausted for this hour, add more exits")
    usage[proxy["server"]] += 1
    time.sleep(random.uniform(12, 35))
    return search_google(query, proxy)

Spread real volume across more IPs rather than pushing one harder. A pool that looks lightly used everywhere beats one identity working overtime.

Layer 4: solving the wall when it fires anyway

Even a clean setup eats a captcha sometimes. Google's version is a standard reCAPTCHA checkbox sitting right on the "unusual traffic" page, and it can escalate to an image grid if the signals still look bad after you click it. You can't fingerprint your way past a challenge that's already on screen, and solving it by hand doesn't scale past a handful of runs a day.

For volume, hand it to a solving service. Most take the same shape: submit the sitekey and page URL, poll for a token, then hand that token back to the page. We cover the provider-neutral version of this pattern in the CAPTCHA services guide; here's how it looks against Google's block page specifically, picking up right where the browser step above hit the wall:

import requests, time

API = "https://api.captcha-provider.example/v1"
KEY = "YOUR_API_KEY"

sitekey = page.get_attribute("div.g-recaptcha", "data-sitekey")

# 1. Submit the task: what to solve, and where.
job = requests.post(f"{API}/createTask", json={
    "clientKey": KEY,
    "task": {
        "type": "RecaptchaV2TaskProxied",
        "websiteURL": page.url,
        "websiteKey": sitekey,
        "proxyType": "http",
        "proxyAddress": "gate.example-proxy.com",
        "proxyPort": 8080,
        "proxyLogin": "user",
        "proxyPassword": "pass",
    },
}).json()
task_id = job["taskId"]

# 2. Poll until a worker returns the token.
while True:
    time.sleep(5)
    res = requests.post(f"{API}/getTaskResult", json={
        "clientKey": KEY, "taskId": task_id,
    }).json()
    if res["status"] == "ready":
        token = res["solution"]["gRecaptchaResponse"]
        break

# 3. Hand the token back to the page Google rendered, then submit.
page.evaluate(
    """(token) => {
        document.getElementById('g-recaptcha-response').innerHTML = token;
        document.querySelector('form').submit();
    }""",
    token,
)

One honest note: solving the captcha clears that one wall. It doesn't fix why you hit it, and if the same IP earns another one ten minutes later, that's your pacing or your IP layer talking, not a reason to buy more solves.

Parsing results without your scraper breaking every month

A results page is busier than it looks: organic listings, ads, "People also ask," a featured snippet, sometimes a knowledge panel or a local pack, all interleaved. Decide what you actually need before you write a selector for all of it. Most jobs only want the organic list and maybe the snippet text, and grabbing everything just gives you more surface area to break.

A few query params save real work: num controls results per page, start pages through them, and hl / gl set language and country so you're not scraping the wrong market by accident. Pagination isn't always a clean increment either: some layouts load the next page with an AJAX call instead of a fresh URL, so check what your browser's network tab actually does before assuming a start bump will work. Archive the raw HTML alongside your parsed output, too. When Google reshuffles its markup, and it will, you fix the parser and replay it against what you already saved instead of re-scraping the whole job from zero.

ToS, robots.txt, and playing it straight

Google's Terms of Service bar automated querying of Search without permission, and its robots.txt disallows crawling the results pages for bots that respect it. That doesn't automatically make scraping illegal wherever you sit, public data with no login involved is a defensible position in a lot of places, but it does mean you're outside the rules Google actually set. Sustained volume from one place is what gets IP ranges, and sometimes accounts, blocked. That's doubly true if you or whoever you're scraping for sits in the EU: GDPR adds a data-protection layer on top of Google's own terms the moment personal data shows up in a result, which happens more than you'd expect. Read the full legal breakdown before you point anything at real scale.

Keep it to public, logged-out SERP data, keep the pace sane, and don't go anywhere near a login you don't have permission to touch. Past that, how much risk fits your project is your call, not ours.

FAQ

Is it legal to scrape Google search results?

Scraping public, logged-out search results is broadly legal in a lot of places, the same reasoning that covers most public-web scraping. What's not on your side is Google's own Terms of Service, which bars automated querying without permission, so you can be breaking a contract even where you're not breaking a law. Keep it to public SERP data, keep the volume sane, and read the full breakdown in our legal guide before you point anything at scale.

Why do I still get blocked with a good residential proxy?

Because the IP is one layer out of four. A clean residential IP running a bare HTTP client with default headers is still an obvious mismatch, and forty queries a minute through one IP burns it no matter how clean it started. Fix the browser and the pacing too.

Should I just pay for a SERP API instead?

For most people, yes. SerpApi, Serper, and similar services have already solved the proxy, browser, and captcha problem, and they charge per search instead of per headache. Build it yourself when you need control an API doesn't give you, or when you're already running proxy infrastructure for other jobs and the extra cost of adding Google is close to nothing.

How many searches can one IP do before it gets flagged?

Nobody outside Google knows the exact number, and it moves. As a working rule, a few dozen searches an hour from one residential IP is comfortable, a few hundred is pushing it, and a fixed interval between requests gets noticed regardless of volume. Spread real volume across more IPs instead of squeezing one harder.