CleanTalk Anti-Flood and Anti-Crawler Options

 

Anti-Flood and Anti-Crawler options are intended for blocking unwanted bots, content parsing, shop goods prices parsing or aggressive website scanning bots.

Aggressive website scanning bots can load website servers a lot therefore speed of websites can go down as well. As a result Google Ranking System will lower the website position.

Web crawling bots such as Google, Bing, MSN, Yandex are excluded and will not be blocked. However, you can use your own white lists allowing any IP to visit your website. More information about white lists is here: https://cleantalk.org/help/sfw-blacklist-usage

 

How the Options Work

 

CleanTalk Anti-Flood — this option is meant to block aggressive bots. You set the maximum number of website pages your visitors can click on within 1 minute. If any IP exceeds the set number it will get the CleanTalk blocking screen for 30 seconds. It's impossible for the IP to open any website pages while the 30 second timer takes place. When the timer ends the IP will be able to continue visiting your pages and the Anti-Flood option starts counting the number of page visits again.

For example, by default it's 10 visits per 1 minute. That means that any visitor who opens 10 website pages within 1 minute will be blocked for 30 seconds and they will not be able to see your website during that time.

You can set your number of how many website pages an IP can visit before getting blocked. The option is in the CleanTalk plugin settings: Advanced Settings —> Anti-Flood Page Views Limit.

anti-flood option settings value number


CleanTalk Anti-Crawler — this option is meant to block all types of bots visiting website pages. The initial visit of your website by any IP triggers the checking for spam, if the checking fails then the visitor will be shown the CleanTalk blocking screen for 30 seconds. The screen has the spam checking mechanism so the visitor will be transferred to the website if everything goes OK when the time runs out.

Statistics of blocks for the options Anti-Flood и Anti-Crawler is available in the SpamFireWall Log: https://cleantalk.org/my/show_sfw