Comments on: Crawl-Delay in Robots.txt http://www.evilgeniuschronicles.org/wordpress/2006/11/13/crawl-delay-in-robotstxt/?utm_source=rss&utm_medium=rss&utm_campaign=crawl-delay-in-robotstxt A journal of geekery, music and joy Fri, 03 Feb 2012 23:04:56 -0500 hourly 1 http://wordpress.org/?v=3.3.1 By: vlkn http://www.evilgeniuschronicles.org/wordpress/2006/11/13/crawl-delay-in-robotstxt/#comment-58509 vlkn Mon, 28 Jul 2008 19:56:01 +0000 http://www.evilgeniuschronicles.org/wordpress/2006/11/13/crawl-delay-in-robotstxt/#comment-58509 Setting a crawl-delay might limit the coverage and freshness of your content representation in search results. Set to maximum 10. Setting a crawl-delay might limit the coverage and freshness of your content representation in search results.

Set to maximum 10.

]]>