Nowadays more and more web crawlers visit a website, some are good for you, such as google search engine, they respect robots.txt protocol. But some of them have bad behavior that has negatively impact to your site. How to block the bad bots These crawlers are called bot. A bot is a software program that operates on the Internet and performs repetitive tasks. While some bot traffic is from good bots, bad bots can have a huge negative impact on a website or application. So we want to block those bad bots from visiting our site. Normally, we can set rule in robots.txt file, but the bad bots do not respect these rules. So setting up robots.txt file is only for good bots. To be sure the bad bots are blocked, we have to use .htaccess file. Here are the steps: 1. find out the bot keyword from the user-agent from the log. 2. add the following script to the top of .htaccess file: a) set rewrite condition: It uses regular expression to match multiple user agents in one line. RewriteEngine On RewriteCond %{HTTP_USER_AGENT} "dataforseobot|Yandex|AhrefsBot|BLEXBot|SemrushBot" [NC] RewriteRule "^.*$" - [F,L] Or b) using SetEnvIf directives: SetEnvIfNoCase User-Agent "AhrefsBot" badbots SetEnvIfNoCase User-Agent "BLEXBot" badbots SetEnvIfNoCase User-Agent "SemrushBot" badbots SetEnvIfNoCase User-Agent "YandexBot" badbots SetEnvIfNoCase User-Agent "dataforseobot" badbots <Limit GET POST HEAD> Order Allow,Deny Deny from env=badbots </Limit> Both scripts are equiverant, just pick the one you like. By using the script above the page will return 403 (forbidden) to the bad bots. Test the effects Here's an example that uses chrome browser…