Robots.txt

A standard text file in a website's root directory that instructs crawlers which pages are accessible or forbidden, serving as t