How to scrape JavaScript-rendered websites (2026)

So the page renders fine in Chrome and your scraper gets a blank. This catches everyone who learned scraping on static HTML, and the fix isn't a bigger hammer. It's realizing that a modern site hands your code and your browser two different things.

Below: why that happens, a 30-second test to prove it, and three ways to get the data. They go fastest to slowest. Try them in that order, because the first one that works saves you hours.

Why you get empty HTML

Your browser makes two trips, not one. First it downloads a thin HTML skeleton and a big bundle of JavaScript. Then it runs that JavaScript, which calls the site's own API, gets back the data, and builds the page you actually see. The products, the prices, the reviews, all of it gets injected into the DOM after the fact.

requests only makes the first trip. It grabs the skeleton and stops. No JavaScript runs, so the data never loads. That blank you're staring at is the page before it filled itself in.

This is client-side rendering, and it's how most front-ends ship now. React, Vue, Svelte, Angular, they all send an app shell first and hydrate it in the browser. The tell in the raw HTML is a near-empty mount point like <div id="root"></div> or <div id="app"></div> sitting above a wall of <script> tags. That div is where the whole page is about to appear. In the HTML you fetched, it's still empty.

Confirm it in 30 seconds

Before you change a line, prove that rendering is the problem and not, say, a header you forgot. Two fast checks.

One. Grep the raw HTML for text you can see on the page:

curl -s https://example.com/products | grep -i "some product name"

If it's not in there, the server didn't send it. JavaScript did.

Two. In the browser, right-click and hit View Page Source, not Inspect. View Source shows the exact HTML the server returned, which is what your scraper sees. Inspect shows the live DOM after JavaScript ran, which is what you see. Data in Inspect but missing from View Source means it's rendered client-side. Confirmed.

Now the part people skip: a JavaScript site does not automatically mean you need a browser. The data's often one quiet request away.

Route 1: find the JSON API (fastest)

The fastest scraper is the one that never opens a browser. Your React app didn't invent those products. It fetched them, as JSON, from somewhere. Hit that somewhere directly and you skip the rendering entirely, and you get clean structured data instead of scraping text out of HTML.

Two places that JSON hides.

Baked into the page. Server-rendered frameworks often dump their whole data payload into the HTML so the client can hydrate without a second round trip. Next.js ships it inside a <script id="__NEXT_DATA__"> tag. Nuxt sets a window.__NUXT__. Plenty of apps drop a window.__INITIAL_STATE__. It's all sitting in the HTML that requests already downloaded. Pull the script tag and parse it.

import httpx, json
from bs4 import BeautifulSoup

html = httpx.get("https://example.com/products").text
soup = BeautifulSoup(html, "html.parser")

blob = soup.find("script", id="__NEXT_DATA__").string
data = json.loads(blob)
products = data["props"]["pageProps"]["products"]

for p in products:
    print(p["name"], p["price"])

No browser. No waiting. Full structured data from a single GET.

A separate API call. When the data isn't in the HTML, the browser fetched it after load. Go find that request. Open DevTools (F12), click Network, filter to Fetch/XHR, then reload the page. You'll watch the background requests fire. The one whose Response is your data as JSON is the endpoint you want.

Click it, confirm the Response tab, then right-click and Copy as cURL. That hands you the full URL, headers, and params the browser used. Translate it to Python:

import httpx

r = httpx.get(
    "https://example.com/api/v2/products",
    params={"page": 1, "per_page": 48},
    headers={
        "Accept": "application/json",
        "Referer": "https://example.com/products",
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
    },
)
for item in r.json()["results"]:
    print(item["title"], item["price"])

Watch for a few things. The endpoint may want a Referer, an X-Requested-With: XMLHttpRequest header, or an auth token the page set earlier. Pagination is usually a query param you can walk. And once in a while the token is signed and short-lived, which pushes you toward Route 2 or 3.

When it works, this route wins on everything. It's 10 to 50 times faster than a browser, it returns data you don't have to scrape, and it doesn't shatter the moment the site reshuffles its CSS classes. Always look for it first.

Route 2: run a headless browser

Sometimes there's no clean API to hit. The data comes from six chained calls, or the endpoint is signed in a way you can't replay, or honestly it's just faster to let the page render than to reverse-engineer the network tab. Fine. Run the page in a real browser engine, let the JavaScript execute, wait for the content, and read it out of the DOM.

Playwright is the right tool here. Selenium still works, but Playwright's waiting is smarter and the API hurts less.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/products")

    page.wait_for_selector(".product-card")   # wait for real content, not a fixed timer

    for card in page.query_selector_all(".product-card"):
        name = card.query_selector("h3").inner_text()
        price = card.query_selector(".price").inner_text()
        print(name, price)

    browser.close()

The mistake I see constantly: time.sleep(5) after goto. Don't. A fixed sleep is either too short, so you scrape a half-rendered page and get garbage, or too long, so you burn seconds on every single run. Wait for a specific selector that only exists once the data rendered. wait_for_selector returns the millisecond that element shows up and throws if it never does, so you get speed and a real error instead of a silent blank.

Infinite scroll or a "load more" button? Scroll, wait, repeat, or watch the Network tab for the XHR each scroll fires, which drops you straight back into Route 1. If the site starts sniffing for automation, how you configure the browser matters a lot, and we get into that in the Playwright stealth guide.

Route 3: when the API fights back

Here's the case nobody warns you about. You found the JSON endpoint, replayed it byte for byte, and got a 403. Or the headless browser loaded and the page served a challenge instead of listings. You're not fighting JavaScript anymore. You're fighting an anti-bot system: Cloudflare, DataDome, Akamai, HUMAN. They fingerprint the client and score it before they'll hand over a thing.

Two things change.

First, plain headless=True Chromium is easy to spot. It leaks navigator.webdriver, Chrome DevTools Protocol artifacts, and a headless user-agent, and detectors read those in milliseconds. You need a browser that hides the automation at the source. Camoufox is a patched Firefox that spoofs those signals at the binary level instead of injecting evasion scripts a detector can catch. It reads as a real browser because the values come back native.

Second, your datacenter IP is already flagged. A scraper running from a cloud box wears its origin on its sleeve, and protected sites score datacenter ranges badly before the browser fingerprint even matters. You want residential IPs that read as ordinary home connections. We go deep on picking a pool in the residential proxy guide.

Put both together, an anti-detect browser over a residential proxy, and match the geo so the two don't contradict each other:

from camoufox.sync_api import Camoufox

with Camoufox(
    headless="virtual",   # virtual display on a Linux box, trusted more than pure headless
    geoip=True,           # match timezone and locale to the proxy's exit IP
    humanize=True,
    proxy={
        "server": "http://gate.example-proxy.com:8080",
        "username": "user",
        "password": "pass",
    },
) as browser:
    page = browser.new_page()
    page.goto("https://example.com/listings")
    page.wait_for_selector("[data-testid='listing']")
    for row in page.query_selector_all("[data-testid='listing']"):
        print(row.inner_text())

One trick that pays for itself: drive the anti-detect browser once to clear the wall and pick up a valid session cookie, then lift those cookies and replay the JSON API with plain httpx. You get Route 1 speed on every request that follows, sitting behind a Route 3 defense. If the wall is specifically Cloudflare, the Cloudflare bypass guide walks the fingerprint fight in detail.

Which route, when

Quick map. Top is best:

• Data sits in View Source or a __NEXT_DATA__ blob → parse the HTML, no browser. Seconds per page.
• Data comes from a clean XHR endpoint with no auth wall → hit the API directly. Fast and stable.
• No usable API, or too many chained calls → headless browser, wait on a selector.
• 403s, challenges, or a fingerprint wall → anti-detect browser plus a residential IP, or clear it once and reuse the cookies.

The opinion, since you came here for one: try Route 1 every single time before you reach for a browser. Half the "I need Selenium" scrapes I get handed were a single JSON GET the whole time. People spin up a browser out of reflex and pay 20x the runtime and 100x the flakiness for it. The Network tab is right there. Open it first.

FAQ

Why does BeautifulSoup return an empty page?

Because BeautifulSoup only parses the HTML you handed it, and requests handed it the pre-JavaScript skeleton. The data loads a beat later, when the browser runs the site's JavaScript and calls its API. BeautifulSoup never sees that step. Find the API (Route 1) or render the page in a real browser (Route 2).

Do I always need Selenium or Playwright for a JavaScript site?

No, and defaulting to them wastes time. Most of the time the data's sitting in a JSON endpoint or embedded in the page as a __NEXT_DATA__-style blob, and a plain HTTP request gets it faster and cleaner. A browser is the fallback for when there's no replayable API, not the first move.

How do I find the hidden API a site calls?

Open DevTools, go to the Network tab, filter to Fetch/XHR, and reload the page. Watch which background request returns your data as JSON. Right-click it, Copy as cURL, and replay that in Python with the same headers. If a required header or token is missing you'll get a 401 or 403, so copy them all.

Is scraping JavaScript-rendered sites legal?

Pulling publicly available data is broadly legal in a lot of places, and a site using client-side rendering doesn't change that. But terms of service, robots rules, rate limits, and privacy law still apply and vary by jurisdiction. Keep the pace reasonable and stick to public data. This isn't legal advice; for a high-stakes project, talk to a lawyer. More in is web scraping legal.