Best Practices for Responsible Web Data Collection

The IETF develops standards and protocols to make the internet work better, adhering to principles of openness and decentralization. Industry best practices and protocols for automated web data collection have long existed, but have not been documented at the IETF. For decades, researchers, universities, journalists, public interest groups, and commercial entities have used automated tools to access and collect public web data (sometimes referred to as data scraping, web crawling, or text and data mining) for a wide range of uses [I-D.farzdusa-aipref-enduser]. Examples of these uses include extraction of pricing information for market intelligence or to create a consumer price index, comparative real estate analysis to support underwriting of loans and mortgages, webpage archiving to preserve human knowledge, preserving government websites to hold political powers accountable, journalist research and reporting, and university and scientific research. Recently, innovations in artificial intelligence have significantly increased the automated collection of public web data, creating tensions between the use of AI to equalize and increase access to knowledge and the disruption of existing Internet models, including non-profit repositories that face increased demands for access and businesses that profit from free web access to human viewers. This document lists a set of technical best practices that are prevalent across industries for the automated collection of public web data. It provides protocols for how automated tools access and collect publicly available web data, including volume control, transparency, documentation, and access, that can be implemented by any automated data collector. It applies principles of net neutrality to the collection of data, providing uniform guidance regardless of the identity of the data collector or website operator, the location of the collection, or the applicable legal jurisdiction.