Scrapy AutoThrottle: Set Respectful Crawl Rates and Concurrency
AutoThrottle adjusts a Scrapy crawl's delay using response latency. It helps control request pacing, but the site's published limits and your authorization still define the job's boundaries.
Start with the limits you have permission to use
Before choosing Scrapy settings, check the service's allowed access, rate limits and data-use terms. A target concurrency value is not permission to send that many requests. If a provider specifies a fixed request budget or offers an API, design around that contract rather than increasing traffic until the site objects.
How AutoThrottle chooses a delay
Scrapy calculates a target delay from observed response latency divided by AUTOTHROTTLE_TARGET_CONCURRENCY, then adjusts subsequent delays toward that target. Non-200 response latency cannot decrease the delay. The AutoThrottle documentation describes the algorithm and the start and maximum delay settings.
Target concurrency is an average the extension aims toward, not a strict cap. CONCURRENT_REQUESTS_PER_DOMAIN remains a separate limit, and AutoThrottle respects DOWNLOAD_DELAY as a lower bound. A service that requires a particular time window may need a separate rate limiter because an average and a concurrency cap don't enforce every possible quota.
Retries also spend the budget
Scrapy's RetryMiddleware handles configured transient failures. Review the status codes, retry count and failure behavior for your installation. Each retry is another request; include it in the provider's allowance. A refusal, authentication failure or challenge is a reason to stop and investigate, not an invitation to retry around the restriction.
Use a small, authorized test dataset first. Inspect request counts, delays and error responses before increasing scope, and set an explicit stopping condition for persistent errors. Scrapy's practices guide discusses polite crawling settings. Different jobs and versions may have different defaults, so check your deployed configuration rather than assuming a sample setting applies.
Keep the outcome visible
Record which items were fetched, skipped, retried or failed. That distinction helps a downstream job avoid interpreting an incomplete crawl as an empty catalog. Keep sensitive response data and credentials out of logs, and ask the service operator for a supported export when a required refresh can't fit the allowed budget.
Example: conservative starting settings
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
DOWNLOAD_DELAY = 2.0
RANDOMIZE_DOWNLOAD_DELAY = False
CONCURRENT_REQUESTS_PER_DOMAIN = 1
RETRY_TIMES = 2These illustrative settings target one concurrent request, set a two-second base delay, and permit at most two retries under the configured retry rules. They are not a universal safe rate or a strict hourly quota. Test against a local fixture, including non-200 responses, and lower the rate or add a quota limiter if the owner requires it.
If you need help planning an authorized automation workflow, ask for a scoped assessment. Share the documented limits and a redacted example, without credentials or private data.