Filtration by User-Agents in the Anti-Crawler Option

 

We've extended the feature for working with user-agents for the Anti-Spam plugin.  This feature is available for the WordPress plugin only. To enable user-agent filtration please open your Anti-Spam plugin settings:

WordPress Dashboard -> Settings -> Anti-Spam by CleanTalk 

And enable SpamFireWall and Anti-Crawler option:

 

SpamFireWall option

 

Then open advanced settings (WordPress Dashboard -> Settings -> Anti-Spam by CleanTalk -> Advanced settings) and enable "Block bots by User-Agents":

 

Filter UA option

 

You can see User-Agent in your SpamFireWall logs when Anti-Crawler blocks the request. With "To Personal Black & White Lists" button you can add a User-Agent to your Personal Lists.

Please note that you can add user-agents existing in our list only. Google bot and Bing bot whitelisted by default, you can't add them to your personal lists.

 

SpamFireWall logs

 

You can manage your User-Agent Personal Lists here. Press "Add new filter" and set filter type as User-Agent:

 

User-agent filter

 

Then select agents from the drop-down list and Blacklist them:

 

User-agents list

 

Here is the list of all available User-Agents:

  • admantx.com
  • AhrefsBot
  • Akamai crawler
  • Alexabot
  • Applebot
  • archive.org_bot
  • AspiegelBot - HuaweiWebCatBot
  • Babbar.tech (Barkrowler)
  • Baidu
  • Better Uptime Bot
  • Bing
  • BrightLocalBot
  • CleanTalk Uptime bot
  • CloudFlare crawler
  • DuckDuckGo
  • FacebookBot
  • FeedBurner
  • Feedly Fetcher
  • FreshpingBot
  • Google
  • Grapeshot
  • GTmetrix
  • GumGum-Bot
  • HuaweiWebCatBot
  • Hypefactors
  • Lighthouse
  • Mail.ru
  • MailChimp
  • Majestic MJ12bot
  • Meetedgar.com
  • MirrorWeb
  • Petalbot
  • Pingdom.com bot
  • Pinterest bot
  • Printful WooCommerce Integration
  • proximic
  • Qwant Web crawler
  • Rambler bot
  • RankMathApi
  • RapidLoad (rapidload.io)
  • Reddit
  • Rogerbot - MOZ.com
  • Screaming Frog SEO Spider
  • Semrush
  • SEOkicks-Robot
  • serpstatbot
  • Seznam
  • ShipStation
  • ShortPixel Image Optimizer
  • Site24x7
  • Sitechecker
  • sogou spider
  • StatusCake
  • Sucuri Uptime Monitor
  • UptimeRobot.com
  • Wikipedia crawler
  • WooCommerce API
  • wordoftravel: Find Travel Blogs and Explore Destinations
  • WP Engine Smart Plugin Manager
  • WP Rocket
  • WPCompress
  • Yandex

These User-Agents are system defined and can't be added to your Personal Lists:

  • Baidu
  • Bing
  • CleanTalk
  • Google
  • Yandex

 

 

If you haven't found the answer to your question, please, contact our support team:

https://cleantalk.org/my/support/open

 

 

Was this information helpful?

No

Yes




Perhaps it would also be interesting