Filtration by User-Agents in the Anti-Crawler Option


We've extended the feature for working with user-agents for the Anti-Spam plugin.  This feature is available for the WordPress plugin only. To enable user-agent filtration please open your Anti-Spam plugin settings:

WordPress Dashboard -> Settings -> Antispam by Cleantalk 

And enable SpamFireWall and Anti-Crawler option:


SpamFireWall option


Then open advanced settings (WordPress Dashboard -> Settings -> Anti-Spam by CleanTalk -> Advanced settings) and enable Block bots by User Agents:


Filter UA option


You can see user-agent in your SpamFireWall logs when Anti-crawler blocks the request. With "To Personal black & white lists" button you can add a user-agent to your personal lists.

Please note that you can add user-agents existing in our list only. Google bot and Bing bot whitelisted by default, you can't add them to your personal lists.


SpamFireWall logs


You can manage your user-agents personal lists here. Press "Add new filter" and set filter type as User-Agent:


User-agent filter


Then select agents from the drop-down list and Blacklist them:


User-agents list


Here is the list of all available User-Agents:

  • AhrefsBot
  • Akamai crawler
  • Alexabot
  • Applebot
  • archive.org_bot
  • AspiegelBot - HuaweiWebCatBot
  • (Barkrowler)
  • Baidu
  • Better Uptime Bot
  • Bing
  • CleanTalk Uptime bot
  • CloudFlare crawler
  • DuckDuckGo
  • FacebookBot
  • Google
  • GTmetrix
  • HuaweiWebCatBot
  • Majestic MJ12bot
  • Petalbot
  • bot
  • Pinterest bot
  • Printful WooCommerce Integration
  • Rambler bot
  • Rogerbot -
  • Screaming Frog SEO Spider
  • Semrush
  • SEOkicks-Robot
  • serpstatbot
  • Seznam
  • ShipStation
  • ShortPixel Image Optimizer
  • Site24x7
  • Sitechecker
  • sogou spider
  • Wikipedia crawler
  • wordoftravel
  • WP Engine Smart Plugin Manager
  • Yandex

These User-Agent are system-defined and can't be added to personal lists:

  • Baidu
  • Bing
  • CleanTalk
  • Google
  • Yandex



Perhaps it would also be interesting