Robots Exclusion Protocol
Extracted elements (26)
Crawlers MAY interpret other records not part of the robots.txt protocol (e.g., Sitemaps) and MAY be lenient with misspellings, but such parsing MUST NOT interfere with defined records. This allows extensibility while preserving core protocol integrity.
The longest-match rule (most octets) is used for selecting between conflicting allow/disallow rules, rather than order of appearance, to provide deterministic and more intuitive behavior for overlapping path patterns.
The Robots Exclusion Protocol is explicitly not a form of access authorization. Listing paths in robots.txt exposes them publicly, making them discoverable. Valid content security (e.g., HTTP Authentication) must be used separately to control actual access.
Crawlers MAY use standard cache control as defined in RFC 9111 for the fetched robots.txt file, enabling interoperability with HTTP caching infrastructure.
Implementors MAY bridge encoding mismatches if they detect that the robots.txt file is not UTF-8 encoded, providing tolerance for legacy or misconfigured servers.
Crawlers MUST support the special characters '#' (line comment), '$' (end of match pattern), and '*' (zero or more instances of any character).
Crawlers MUST try to parse each line of the robots.txt file and MUST use the parseable rules.
Crawlers MUST use case-insensitive matching to find the group that matches the product token. If more than one group matches, the matching groups' rules MUST be combined into one group.
Crawlers SHOULD follow at least five consecutive redirects, even across authorities. If a robots.txt file is reached within five redirects, it MUST be fetched, parsed, and its rules followed in the context of the initial authority.
Crawlers SHOULD impose a parsing limit to protect their systems; the parsing limit MUST be at least 500 kibibytes (KiB).
Crawlers SHOULD NOT use a cached robots.txt version for more than 24 hours, unless the file is unreachable.
If a server status code indicates robots.txt is unavailable (e.g., HTTP 400-499), the crawler MAY access any resources on the server.
If an 'allow' rule and a 'disallow' rule are equivalent, the 'allow' rule SHOULD be used. If no match is found among rules, the URI is allowed. The /robots.txt URI is implicitly allowed.
If no matching group exists, crawlers MUST obey the group with a user-agent line with the '*' value, if present. If no group matches and no '*' group exists, no rules apply.
If robots.txt is unreachable due to server or network errors (e.g., HTTP 500-599), the crawler MUST assume complete disallow. If undefined for a reasonably long period (e.g., 30 days), crawlers MAY assume unavailability or continue using a cached copy.
If the crawler successfully downloads the robots.txt file, the crawler MUST follow the parseable rules.
Octets in the URI and robots.txt paths outside the ASCII range and those in the reserved range defined by RFC 3986 MUST be percent-encoded prior to comparison. Percent-encoded ASCII octets in the URI MUST be unencoded prior to comparison unless reserved or outside the unreserved character range.
Parsing of other records (e.g., Sitemaps) MUST NOT interfere with parsing of explicitly defined records. For example, a 'Sitemaps' record MUST NOT terminate a group.
The product token MUST contain only uppercase and lowercase letters ('a-z' and 'A-Z'), underscores ('_'), and hyphens ('-'). It SHOULD be a substring of the identification string the crawler sends to the service.
The robots.txt file MUST be accessible at '/robots.txt' (all lowercase) in the top-level path of the service, MUST be UTF-8 encoded, and MUST have Internet Media Type 'text/plain'.
To evaluate if access to a URI is allowed, a crawler MUST match paths in 'allow' and 'disallow' rules against the URI starting with the first octet of the path; the most specific match (most octets) MUST be used.
A 'Group' consists of one or more user-agent lines followed by one or more rules, terminated by another user-agent line or end of file. The last group may have no rules, implicitly allowing everything.
A 'Rule' is a key-value pair line using 'allow' or 'disallow' directives with a path pattern, defining how a crawler may access URIs.
Parsers should reject out-of-bound characters as invalid to limit attack vectors, treat robots.txt content as untrusted, and enforce the 500 KiB parsing limit to protect against out-of-memory attacks.
The Robots Exclusion Protocol is not a substitute for valid content security measures. Paths listed in robots.txt are publicly exposed and discoverable; service owners should use application-layer security (e.g., HTTP Authentication per RFC 9110) to control actual access.
The robots.txt file uses an ABNF-defined format: groups start with 'user-agent' lines (product-token), followed by 'allow'/'disallow' rule lines with path patterns. Lines are terminated by optional whitespace and optional comment preceded by '#', then NL (%x0D / %x0A / %x0D.0A).