CleanTalk Anti-Flood and Anti-Crawler Options
SpamFireWall Anti-Flood and Anti-Crawler options are intended for blocking unwanted bots that can search vulnerabilities on website, attempts to hack a site, collect personal data, parsing prices or content and images, generate 404 error pages, or aggressive website scanning bots. Aggressive website scanning bots can load website servers a lot therefore the speed of websites can go down as well. As a result, Google Ranking System will lower the website position.
Learn more about what problems bad bots can cause on your site on our Blog.
By default, all bots are added to the whitelist. You can change it and add the desired bots to the blacklist.
If you need to allow access for any bots that are not in the exceptions, you can use your own white lists allowing any IP to visit your website.
More information about white lists is here: https://cleantalk.org/help/sfw-blacklist-usage.
How to manage your Private list by User-Agents: https://cleantalk.org/help/filter-ua
How the Options Work
CleanTalk Anti-Crawler — this option is meant to block all types of bots visiting website pages. The initial visit of your website by any IP triggers the checking for spam, if the checking fails then the visitor will be shown the CleanTalk blocking screen for 3 seconds. The screen has the spam checking mechanism so the visitor will be transferred to the website if everything goes OK when the time runs out.
Read more about how to use CleanTalk anti-Crawler for filtering by user-agents here.
CleanTalk Anti-Flood — this option is meant to block aggressive bots. You set the maximum number of website pages your visitors can click on within 1 minute. If any IP exceeds the set number it will get the CleanTalk blocking screen for 30 seconds. It's impossible for the IP to open any website pages while the 30 second timer takes place. When the timer ends the IP will be able to continue visiting your pages and the Anti-Flood option starts counting the number of page visits again.
For example, by default it's 20 visits per 1 minute. That means that any visitor who opens 10 website pages within 1 minute will be blocked for 30 seconds and they will not be able to see your website during that time.
You can set your number of how many website pages an IP can visit before getting blocked. The option is in the CleanTalk plugin settings: Advanced Settings —> Anti-Flood Page Views Limit.
Statistics of blocks for the options Anti-Flood и Anti-Crawler is available in the SpamFireWall Log: https://cleantalk.org/my/show_sfw