Lake Lariba
Understanding And Using SEO Robots.txt Files!
We all get a bit excited when the search engines visits our web site frequently and indexes our content. However, there are certain things that we don’t want the search engines to spider because of private information that we don’t want the world to see.
This is where we use SEO Robots.txt files. Another scenario would be that we may have more than one version of a page on our site. We can tell the search engines in our robot.txt file which page to crawl and which ones to ignore. We definitely don’t want both of these pages to be crawled and end up with the search engine nailing you for spam because of the duplicate content in the two similar versions of one page.
Understanding And Using SEO Robots.txt Files!
We all get a bit excited when the search engines visits our web site frequently and indexes our content. However, there are certain things that we don’t want the search engines to spider because of private information that we don’t want the world to see.
This is where we use SEO Robots.txt files. Another scenario would be that we may have more than one version of a page on our site. We can tell the search engines in our robot.txt file which page to crawl and which ones to ignore. We definitely don’t want both of these pages to be crawled and end up with the search engine nailing you for spam because of the duplicate content in the two similar versions of one page.