Filtration by User-Agents in the Anti-Crawler Option

 

We've extended the feature for working with user-agents for the Anti-Spam plugin.  This feature is available for the WordPress plugin only. To enable user-agent filtration please open your Anti-Spam plugin settings:

WordPress Dashboard -> Settings -> Antispam by Cleantalk 

And enable SpamFireWall and Anti-Crawler option:

 

SpamFireWall option

 

Then open advanced settings (WordPress Dashboard -> Settings -> Anti-Spam by CleanTalk -> Advanced settings) and enable Block bots by User Agents:

 

Filter UA option

 

You can see user-agent in your SpamFireWall logs when Anti-crawler blocks the request. With "To Personal black & white lists" button you can add a user-agent to your personal lists.

Please note that you can add user-agents existing in our list only. Google bot and Bing bot whitelisted by default, you can't add them to your personal lists.

 

SpamFireWall logs

 

You can manage your user-agents personal lists here. Press "Add new filter" and set filter type as User-Agent:

 

User-agent filter

 

Then select agents from the drop-down list and Blacklist them:

 

User-agents list

 

Here is the list of all available User-Agents:

  • AhrefsBot
  • Akamai crawler
  • Alexabot
  • Applebot
  • archive.org_bot
  • AspiegelBot - HuaweiWebCatBot
  • Babbar.tech (Barkrowler)
  • Baidu
  • Better Uptime Bot
  • Bing
  • CleanTalk Uptime bot
  • CloudFlare crawler
  • DuckDuckGo
  • FacebookBot
  • Google
  • GTmetrix
  • HuaweiWebCatBot
  • Mail.ru
  • Majestic MJ12bot
  • Meetedgar.com
  • Petalbot
  • Pingdom.com bot
  • Pinterest bot
  • Printful WooCommerce Integration
  • Rambler bot
  • Rogerbot - MOZ.com
  • Screaming Frog SEO Spider
  • Semrush
  • SEOkicks-Robot
  • serpstatbot
  • Seznam
  • ShipStation
  • ShortPixel Image Optimizer
  • Site24x7
  • Sitechecker
  • sogou spider
  • Wikipedia crawler
  • wordoftravel
  • WP Engine Smart Plugin Manager
  • Yandex

These User-Agent are system-defined and can't be added to personal lists:

  • Baidu
  • Bing
  • CleanTalk
  • Google
  • Yandex

 

 


Perhaps it would also be interesting