Main Section



robots.txt – uncrawled URLs in search results

A complaint heard often is: “I have blocked Google from crawling a page and robots.txt and Google clearly violated that robot.txt by crawling that page because it’s showing up in Google search results anyway.”

From GoogleWebmasterHelp – Matt Cutts explains why a page that is disallowed in robots.txt may still appear in Google’s search results.

Watch this video on YouTube.

Autor: Ortwin Oberhauser
Veröffentlicht in: SEO, robots.txt

Leave a Comment

Your email address will not be published.