Robots.txt: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

14 June 2023

  • curprev 16:4216:42, 14 June 2023‎ WHC-admin talk contribs‎ 2,676 bytes +2,676‎ Created page with "=Topic Overview= The '''robots.txt''' file is part of the ''Robots Exclusion Protocol'' (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. The REP also includes directives like meta robots and x-robots-tag, but the robots.txt file is the most well-known part of the protocol. The primary function of the robots.txt file is to manage crawler traffic to the site, prevent certain parts of the..." Tag: Visual edit
Retrieved from "https://wiki.wolfheadconsulting.com/index.php/Robots.txt"