draft-illyes-aipref-cbcp

Crawler best practices

groupnone stateactive pages7 rev04 expires2026-10-11 canonicalhttps://datatracker.ietf.org/doc/draft-illyes-aipref-cbcp/

This document describes best practices for web crawlers. Discussion Venues This note is to be removed before publishing as an RFC. Source for this draft and an issue tracker can be found at https://github.com/garyillyes/cbcp.