User-agent: * Allow: /staff/$ Disallow: /staff/* Disallow: /staff_category/ Disallow: /staff_category/* # High traffic crawlers User-agent: AhrefsBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: Cloudfind Disallow: / User-agent: dotbot Disallow: / User-agent: MegaIndex.ru Disallow: / User-Agent: MJ12bot Disallow: / User-agent: seek Disallow: / User-agent: SemrushBot Disallow: / User-Agent: trendkite-akashic-crawler Disallow: /