Published: 6. October 2009

robots.txt – uncrawled URLs in search results

Ortwin Oberhauser Avatar
By:
Ortwin Oberhauser

A complaint heard often is: “I have blocked Google from crawling a page and robots.txt and Google clearly violated that robot.txt by crawling that page because it’s showing up in Google search results anyway.”

From GoogleWebmasterHelp – Matt Cutts explains why a page that is disallowed in robots.txt may still appear in Google’s search results.

Author: Ortwin Oberhauser
INITIATOR OF SEOLOGY & WORLD’S FIRST SEOLOGE

BSc Applied Computer Science
SEM / SEO & Conversion Optimization Geek

Founder of Oberhauser.com
Co Founder of bobdo.com International Film & Digital Solution Agency
film production . web design . app & web development data driven performance search & social media marketing ai and search engine content & visibility optimization conversion optimizing . datacenter for web data analytics
Bahnhofstraße 10 . 6900 Bregenz . Austria

bobdo.com

Let’s discuss it

Leave a Reply

Your email address will not be published. Required fields are marked *

Privacy settings

Decide which cookies you want to allow.

You can change these settings at any time. However, this can result in some functions no longer being available. For information on deleting the cookies, please consult your browser’s help function.

Learn more about the cookies we use.

With the slider, you can enable or disable different types of cookies:

This website will

  • Remember which cookies group you accepted

This website won't

  • Remember your login details
  • Essential: Remember your cookie permission setting
  • Essential: Allow session cookies
  • Essential: Gather information you input into a contact forms newsletter and other forms across all pages
  • Essential: Keep track of what you input in a shopping cart
  • Essential: Authenticate that you are logged into your user account
  • Essential: Remember language version you selected
  • Functionality: Remember social media settings
  • Functionality: Remember selected region and country
  • Analytics: Keep track of your visited pages and interaction taken
  • Analytics: Keep track about your location and region based on your IP number
  • Analytics: Keep track of the time spent on each page
  • Analytics: Increase the data quality of the statistics functions
  • Advertising: Tailor information and advertising to your interests based on e.g. the content you have visited before. (Currently we do not use targeting or targeting cookies.
  • Advertising: Gather personally identifiable information such as name and location

This website will

  • Essential: Remember your cookie permission setting
  • Essential: Allow session cookies
  • Essential: Gather information you input into a contact forms newsletter and other forms across all pages
  • Essential: Keep track of what you input in a shopping cart
  • Essential: Authenticate that you are logged into your user account
  • Essential: Remember language version you selected

This website won't

  • Remember your login details
  • Functionality: Remember social media settings
  • Functionality: Remember selected region and country
  • Analytics: Keep track of your visited pages and interaction taken
  • Analytics: Keep track about your location and region based on your IP number
  • Analytics: Keep track of the time spent on each page
  • Analytics: Increase the data quality of the statistics functions
  • Advertising: Tailor information and advertising to your interests based on e.g. the content you have visited before. (Currently we do not use targeting or targeting cookies)
  • Advertising: Gather personally identifiable information such as name and location

This website will

  • Essential: Remember your cookie permission setting
  • Essential: Allow session cookies
  • Essential: Gather information you input into a contact forms newsletter and other forms across all pages
  • Essential: Keep track of what you input in a shopping cart
  • Essential: Authenticate that you are logged into your user account
  • Essential: Remember language version you selected
  • Functionality: Remember social media settings
  • Functionality: Remember selected region and country

This website won't

  • Analytics: Keep track of your visited pages and interaction taken
  • Analytics: Keep track about your location and region based on your IP number
  • Analytics: Keep track of the time spent on each page
  • Analytics: Increase the data quality of the statistics functions
  • Advertising: Tailor information and advertising to your interests based on e.g. the content you have visited before. (Currently we do not use targeting or targeting cookies)
  • Advertising: Gather personally identifiable information such as name and location

This website will

  • Essential: Remember your cookie permission setting
  • Essential: Allow session cookies
  • Essential: Gather information you input into a contact forms newsletter and other forms across all pages
  • Essential: Keep track of what you input in a shopping cart
  • Essential: Authenticate that you are logged into your user account
  • Essential: Remember language version you selected
  • Functionality: Remember social media settings
  • Functionality: Remember selected region and country
  • Analytics: Keep track of your visited pages and interaction taken
  • Analytics: Keep track about your location and region based on your IP number
  • Analytics: Keep track of the time spent on each page
  • Analytics: Increase the data quality of the statistics functions

This website won't

  • Advertising: Tailor information and advertising to your interests based on e.g. the content you have visited before. (Currently we do not use targeting or targeting cookies)
  • Advertising: Gather personally identifiable information such as name and location

This website will

  • Functionality: Remember social media settings
  • Functionality: Remember selected region and country
  • Analytics: Keep track of your visited pages and interaction taken
  • Analytics: Keep track about your location and region based on your IP number
  • Analytics: Keep track of the time spent on each page
  • Analytics: Increase the data quality of the statistics functions
  • Advertising: Tailor information and advertising to your interests based on e.g. the content you have visited before. (Currently we do not use targeting or targeting cookies)
  • Advertising: Gather personally identifiable information such as name and location

This website won't

  • Remember your login details
Save & Close