Robots.txt Explained: What Crawl Rules Do and Do Not Authorize
A robots.txt file gives participating crawlers instructions about paths on a host. Read it before planning a crawl, then check the site's access terms and your permission separately.
Find the file for the exact origin
The Robots Exclusion Protocol places the file at the top-level /robots.txt path. Scheme, host and port matter when determining which service the file applies to. A file on one subdomain doesn't automatically describe another. The protocol specification describes retrieval, parsing and error handling; check the crawler's own implementation for its behavior.
Read the group and path together
A group starts with one or more User-agent lines and contains path rules such as Allow and Disallow. Match the crawler's product token case-insensitively. Combine the rules from every matching group; use the generic * group only when no specific group matches. Matching isn't a quick substring search over the whole file: groups, case handling and path precedence all matter.
When applicable rules conflict, RFC 9309 says to use the most specific matching path, measured by octets. If matching allow and disallow rules are equivalent, the RFC says the allow rule should be used. Percent-encoding and wildcard behavior also have specific rules, so use a compliant parser and test representative URLs rather than writing a one-line parser from memory.
Don't treat a crawl rule as permission
Robots instructions aren't an authentication system. A disallowed URL can still be reachable, and a participating crawler can encounter a URL without being permitted to fetch its contents. Google's documentation also cautions against using robots.txt to hide private pages. Protect private material with actual access controls.
An allowed path isn't a license to collect personal data, ignore terms, exceed a service's rate limits or use an account you aren't authorized to access. Conversely, a particular crawler's treatment of a missing or unavailable file doesn't settle the permission question. Review the site's published terms and seek permission when needed; this guide isn't legal advice.
Keep the decision with the job
Record the origin, crawler name, relevant rule and time checked with the job's other inputs. Recheck when the origin or job changes. If an authorized crawl is disallowed or the rules are unclear, ask the site operator about a supported feed or permitted access. Keep crawl limits and error handling in the plan too; robots.txt doesn't replace those controls.
Example: combine matching groups
User-agent: ExampleBot
Disallow: /private/
User-agent: ExampleBot
Allow: /private/public/For ExampleBot, combine the two groups. /private/report matches the disallow rule; /private/public/help matches both and the longer allow rule applies. This is a parsing example, not permission to access either path. Check the actual crawler implementation before relying on its behavior.
If you need help planning an authorized automation workflow, ask for a scoped assessment. Share the documented limits and a redacted example, without credentials or private data.