Main Section

robots.txt

Blogs

disallowed in robots.txt

Learn more about robots.txt – Often pages that are set on “disallowed in robots.txt” anyway appear in Google’s search results. In the following robots.txt Video Matt Cutts from Google explains why this happens.
Follow Matt Cutts >@mattcuttson Twitter.

1. “disallowed in robots.txt

Watch this video on YouTube.

robots.txt – uncrawled URLs in search results

A complaint heard often is: “I have blocked Google from crawling a page and robots.txt and Google clearly violated that robot.txt by crawling that page because it’s showing up in Google search results anyway.”

From GoogleWebmasterHelp – Matt Cutts explains why a page that is disallowed in robots.txt may still appear in Google’s search results.

Watch this video on YouTube.