How to scrape a website behind a login

The data's right there once you sign in. Your orders, your messages, the analytics the site shows you but won't hand you as a file. Getting a script to see the same page is the trick, because point plain requests at a URL behind a login and you get the login screen back, not your data.

The fix is to make your script carry the same proof of login your browser does. Two ways to do that, and most serious jobs use both. But before any code, the part that matters more than the code.

Are you actually allowed to?

This guide is about pulling data you already have a right to see. Your own account. A client's account, with the client's written say-so. A free-signup account on a site whose terms let you take out what's yours. That's the whole scope, and it isn't a throwaway line.

What this guide is not about: logging into someone else's account. Buying, selling, sharing, or guessing credentials. Credential stuffing, which is firing leaked username-and-password pairs at a login form to see what opens. Account takeover. Scraping private data that belongs to other people. None of those are gray areas you can finesse with a better proxy. In a lot of places they're crimes under computer-misuse laws, and no scraping trick makes them not crimes. Simple test: if you'd be fine telling the site's owner exactly what you're doing, you're on solid ground. If the plan only works because nobody knows it's you, stop.

Terms of service carry more weight here than for public-page scraping. When you logged in, you clicked "I agree" to something, and that something usually has words about automated access. Read them. Breaking a ToS is a contract problem, not a criminal one, but it can still get your account closed and your data locked away.

None of this is legal advice. I'm a scraper, not your lawyer. Laws differ by country and state, they change, and the facts of your case decide everything. If the stakes are real, money, a business built on the data, anything you can't afford to get wrong, pay a lawyer for an hour of their time. We laid out the wider picture in is web scraping legal, but that's a map, not a permission slip.

Two ways in, and when to use each

Both approaches start the same way: a real login happens once, with a real browser, run by a real person or a well-behaved script. The difference is what you do after. Reuse the session and you replay its cookie on plain HTTP requests, fast and cheap, no browser after login. Drive the browser and you keep scraping inside the same authed browser, slower, but able to survive JavaScript pages, rotating tokens, and challenges the fast path can't.

Rule of thumb: try the reuse path first. If the data comes back as clean JSON from an API the site's own frontend already calls, you're done in an afternoon. Drop to a real browser when that returns garbage, gets logged out fast, or the login won't complete without running scripts.

Approach 1: grab the session and replay it

When you log in, the site sets something in your browser that says "this request is you." Usually a cookie. Sometimes a token in a header. Find that thing, send it yourself, and the server can't tell your script apart from the tab you logged in with.

Finding it: open DevTools, go to the Application tab, and look under Cookies for the site's domain. One or two cookies do the real work, with names like session, sid, _session, connect.sid, or a __Secure- prefix. Ignore the analytics junk. Not sure which one matters? The Network tab settles it: open the page with your data, find the request that returns it, and read the Cookie header it sent.

Single-page apps often skip the session cookie. They carry a bearer token, a JWT, in an Authorization header and stash it in localStorage. If the data request shows Authorization: Bearer eyJhbGci..., that string is your key, and you send it as a header instead of a cookie:

# Single-page apps often carry a JWT instead of a session cookie.
from curl_cffi import requests

headers = {
    "Authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
    "Accept": "application/json",
}
r = requests.get("https://example.com/api/me", headers=headers, impersonate="chrome")

CSRF tokens trip people up. If your reads work but writes come back 403, the site wants an anti-forgery token, usually a header like X-CSRF-Token whose value has to match a cookie the site set. Copy both, send both.

Here's the fast path end to end. It loads a saved cookie jar, the one the browser login further down writes out, and hits an authenticated JSON endpoint through a sticky proxy:

import json
from curl_cffi import requests

# state.json is the storage state saved by the browser login below.
# It holds the cookies the site set once you were authenticated.
with open("state.json") as f:
    state = json.load(f)

jar = {c["name"]: c["value"] for c in state["cookies"]}

session = requests.Session(impersonate="chrome")
r = session.get(
    "https://example.com/api/account/orders",
    cookies=jar,
    headers={"Accept": "application/json"},
    proxies={"https": "http://USER-session-abc123:PASS@gate.example-proxy.com:7777"},
)

if r.status_code in (401, 403):
    raise SystemExit("session died: re-run the browser login to refresh state.json")

for order in r.json()["orders"]:
    print(order["id"], order["total"])

Why try this first: it's fast, and there's no browser to babysit. Why it breaks: cookies expire, sometimes in weeks, sometimes in twenty minutes, and a JWT often dies in fifteen. When requests start coming back 401 or bouncing to the login page, the session's dead, and retrying won't fix it. You log in again and mint a fresh one, which is the next approach, and the reason the two fit together.

Approach 2: drive a real browser and keep the session

When the fast path won't cooperate, stop fighting it and drive a real browser. A real browser runs the site's JavaScript, so rotating tokens, JS-rendered tables, and script-guarded logins all just work, because you're doing exactly what a person does.

Use a browser that doesn't announce itself as automation. Plain Playwright Chromium sets navigator.webdriver and trips detectors on sight. Camoufox is a patched Firefox that hides those tells at the binary level, which counts double on a login page, where sites watch hardest. The full picture of what they score is in how to avoid bot detection.

Log in once and save the session to disk. Playwright calls this the storage state: a JSON file with your cookies and localStorage. Do this run with a visible window so you can clear MFA by hand the first time:

from camoufox.sync_api import Camoufox

PROXY = {
    "server": "http://gate.example-proxy.com:7777",
    "username": "USER-session-abc123",   # sticky session id lives in the username
    "password": "PASS",
}

# Visible window on the first run so you can clear MFA by hand.
with Camoufox(headless=False, geoip=True, humanize=True, proxy=PROXY) as browser:
    page = browser.new_page()
    page.goto("https://example.com/login")

    page.fill("input[name=email]", "you@example.com")
    page.fill("input[name=password]", "your-own-password")
    page.click("button[type=submit]")

    # Do MFA now if the site asks, then wait until you're actually inside.
    page.wait_for_url("**/dashboard", timeout=90000)

    # Cookies + localStorage for the logged-in session, written to disk.
    page.context.storage_state(path="state.json")
    print("saved state.json")

Every run after that skips the login. Load the saved state into a fresh context and go straight to the data:

from camoufox.sync_api import Camoufox

# Same proxy and session settings as the login run above.
with Camoufox(headless="virtual", geoip=True, proxy=PROXY) as browser:
    context = browser.new_context(storage_state="state.json")
    page = context.new_page()

    page.goto("https://example.com/account/orders")
    page.wait_for_load_state("networkidle")

    for row in page.query_selector_all(".order-row"):
        print(row.inner_text())

That state.json is the same file approach 1 reads, which is the sturdy pattern for a big job: browser for the door, to clear MFA and any challenge once, then plain HTTP for the hauling with the harvested cookies.

Keep the same IP, or get logged out

Here's the mistake that kills authenticated scrapes: rotating the IP mid-session. For public-page scraping you rotate constantly, a fresh IP per request, to spread load and dodge rate limits. Do that while logged in and you're telling the site your account just teleported from Ohio to Vietnam between two clicks. Sites read that as a stolen-session signal. Best case they log you out. Worst case they flag the account.

So pin one IP for the whole session. Providers call this a sticky session: the same residential exit IP stays yours for minutes or hours instead of rotating every request. You set it with a session id in the proxy username, the USER-session-abc123 bit above. Same id, same IP.

You still want residential IPs, not datacenter ones, because a logged-in account coming from a known cloud range is its own red flag. Residential plus sticky: a real home IP, held steady for the whole session. We cover picking a pool in the residential proxy guide, and the rotate-versus-stick mechanics in rotating residential proxies in Python.

MFA and CAPTCHA live at the login

Notice where the friction sits. It's the login. Once you're in, the site trusts your session and mostly leaves you alone. So spend your effort, and your money, at the door, and only there.

MFA. If your account uses two-factor, the first login needs the code. Run that one with a visible browser and type it in. After you save the session, later runs reuse it and skip MFA, until the session expires. If it's TOTP, the authenticator-app kind, generate the code in your script from the shared secret and drop the manual step. Just don't scatter that secret across machines.

CAPTCHA. Some logins throw a CAPTCHA when the signals look automated. A real browser on a clean residential IP clears the invisible ones most of the time without a click. When a visible puzzle does show up, you can't fingerprint past it, and solving it by hand every run defeats the point. This is where a solving service earns its fee: it takes the challenge, returns a token your script submits, and the login goes through. You pay per solve, and since you only solve at login, not on every data request, the bill stays tiny. We compared the ones worth paying for in the CAPTCHA solver roundup. If the login sits behind Cloudflare, that's its own puzzle, covered in bypassing Cloudflare bot detection.

Rules that keep a login scrape alive

The stuff that separates a scrape that runs for months from one that dies Tuesday:

• Reuse the session, don't re-login every run. Every login is risk: a CAPTCHA, an MFA prompt, one more chance to look like an attack. Log in as rarely as you can, save the session, and re-auth only when it actually expires.
• One identity per browser process. Don't juggle three accounts in one browser behind three proxies and hope they stay separate. Shared state leaks between them and ties the accounts together no matter how you route the traffic. One account, one browser, one IP.
• Match the browser's geo to the IP. A residential IP in Berlin behind a browser set to New York time is a mismatch sites check for. Camoufox's geoip=True lines up timezone, locale, and language with the proxy's exit. Leave it on.
• Back off when something breaks. A 401, a 403, a sudden bounce to the login page: that's the site talking. Hammering it harder gets the account flagged. Stop, log in fresh, and if that fails too, wait. Build the retry as detect, re-login, retry once, then give up and alert yourself. Not an infinite loop.
• Go at a human pace. You're logged in as a person, and a person doesn't pull four hundred pages a minute. Randomize the gaps, keep the rate sane, and you'll outlast every scraper that sprinted.

FAQ

Is it legal to scrape behind a login?

It depends on whose data it is and what you agreed to. Scraping your own account, or one you're authorized to access, for data you're allowed to see, is a different thing from breaking into someone else's. The first is usually fine. The second, credential stuffing, account takeover, or pulling other people's private data, runs into computer-misuse law and isn't fine anywhere. And the site's terms probably say something about automated access, which you agreed to by logging in. This isn't legal advice. For anything high-stakes, ask a lawyer.

How do I keep a scraping session logged in?

Save the session and reuse it instead of logging in every run. With a browser, that's Playwright's storage state, a JSON file of your cookies and localStorage that you load into each run. With plain HTTP, it's the session cookie you replay on every request. Either way, pin one sticky residential IP so the account doesn't look like it's hopping countries, and log in again only when the session expires.

Why do I get logged out when scraping?

Three usual reasons. One, you rotated the IP mid-session, so the site saw your account jump locations and killed it. Two, the cookie or token expired, which some sites do fast, and you kept using the dead one. Three, your requests looked automated, thin headers, a non-browser TLS handshake, a robotic pace, so the site invalidated the session on purpose. Fixes, in order: sticky IP, re-login when you get a 401 instead of retrying, and send a browser-shaped request so you don't get singled out.

Do I need proxies for authenticated scraping?

For a light job on your own account, often no. Log in, pull your data at a calm pace from your own IP, and you'll usually be fine. You want proxies when the volume climbs, when you're on a datacenter IP a logged-in account shouldn't come from, or when the site is strict about where a session connects from. When you do, reach for sticky residential sessions, not rotating datacenter IPs, so the account stays put on one believable IP.