# robots.txt zu http://www.checker-pig.com User-agent: * Allow: / Disallow: /js/ Disallow: /kontakt/ Disallow: /pdf/ Disallow: /fahrrad-2007/ User-agent: WebReaper Disallow: / User-agent: Slurp Crawl-delay: 180 User-agent: Googlebot Disallow: /js/ Disallow: /kontakt/ Disallow: /pdf/ Disallow: /haendler/ Disallow: /fahrrad-2007/