Ontify
⌘
S
⌘
S
robots.txt - Ontify
Share
robots.txt
robots.txt Protocol
Standard used to advise web crawlers and scrapers not to index a web page or site.
Wikipedia | Robots.txt
Statements about
robots.txt
Instance Of
Computer Network Protocol
File Format
Technical Standard
Official Website
https://www.robotstxt.org/
Stack Exchange Tag
https://stackoverflow.com/tags/robot ...
https://webmasters.stackexchange.com ...
Described at URL
https://tools.ietf.org/html/draft-gi ...
https://tools.ietf.org/html/draft-ko ...
https://wiki.archiveteam.org/index.p ...
Described by Source
RFC 9309: Robots Exclusion Protocol
Image
Wikidata