Robots Exclusion Protocol
RFC 9309, “Robots Exclusion Protocol”, is a Proposed Standard document published in September 2022 by M. Koster, G. Illyes, H. Zeller, L. Sassman. The canonical text is published by the RFC Editor.
Abstract
This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching.
What “Proposed Standard” means
An entry-level standards-track specification: stable, peer-reviewed and a solid basis for implementation, though it may still evolve before becoming an Internet Standard.
The canonical text of RFC 9309 is hosted at rfc-editor.org. Available in HTML,TXT,PDF,XML.
- RFC 9308 Applicability of the QUIC Transport Protocol
- RFC 9307 Report from the IAB Workshop on Analyzing IETF Data 2021
- RFC 9311 Running an IETF Hackathon
- RFC 9306 Vendor-Specific LISP Canonical Address Format
- RFC 9312 Manageability of the QUIC Transport Protocol
- RFC 9305 Locator/ID Separation Protocol Generic Protocol Extension
- RFC 9313 Pros and Cons of IPv6 Transition Technologies for IPv4-as-a-Service
- RFC 9304 Locator/ID Separation Protocol : Shared Extension Message and IANA Registry for Packet Type Allocations