User-agent: TurnitinBot Disallow: / #Will disallow TurnitinBot to all urls on site User-agent: * Disallow: /cache_maps Crawl-delay: 60 User-agent: Slurp Crawl-delay: 30 Disallow: /cgi-bin/ User-agent: MSIECrawler Disallow: /cgi-bin/ Crawl-delay: 30