# www.robotstxt.org/ # https://developers.google.com/webmasters/control-crawl-index/ User-agent: * Disallow: /search User-agent: * Crawl-Delay: 15