Learn more about robots.txt – Often pages that are set on “disallowed in robots.txt” anyway appear in Google’s search results. In the following robots.txt Video Matt Cutts from Google explains why this happens.Follow Matt Cutts >@mattcuttson Twitter. 1. “disallowed in robots.txt
A complaint heard often is: “I have blocked Google from crawling a page and robots.txt and Google clearly violated that robot.txt by crawling that page because it’s showing up in Google search results anyway.” From GoogleWebmasterHelp – Matt Cutts explains why a page that is disallowed in robots.txt may still appear in Google’s search results.
