banned AhrefsBot - been attacking us similarly like baidubot

User avatar
andy1
Posts: 1295
Joined: February 22nd, 2010, 9:42 am

banned AhrefsBot - been attacking us similarly like baidubot

Unread post by andy1 »

.. in the past.

added this text at the end of .htaccess file because just banning the IP range didn't seem to do the trick.

Code: Select all

BrowserMatchNoCase AhrefsBot/3.1 bad_bot
Order Deny,Allow
Deny from env=bad_bot
and so far so good! no fooling around with .htaccess!:)

as per: http://www.thesitewizard.com/apache/blo ... cess.shtml


User avatar
Blue Frost
Posts: 1442
Joined: July 22nd, 2010, 11:48 am

Re: banned AhrefsBot - been attacking us similarly like baid

Unread post by Blue Frost »

I tried posting here last night, but you likely was updating .

I have seen that code when looking for a way to ban those, but never followed up with it.
Good thing since you banned them I have not seen them, so since being linked to you they are likely gone.
I never had them till you put a link up on the site.

You can also change that script, and add other sneaks I saw.


User avatar
Blue Frost
Posts: 1442
Joined: July 22nd, 2010, 11:48 am

Re: banned AhrefsBot - been attacking us similarly like baid

Unread post by Blue Frost »

Guest IP: 180.76.5.196 » Whois
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Still getting that one :(


User avatar
andy1
Posts: 1295
Joined: February 22nd, 2010, 9:42 am

Re: banned AhrefsBot - been attacking us similarly like baid

Unread post by andy1 »

ha! yes will add that one now - i had it banned, the whole range of IPs through MCP of the other forum but i want to add them to .htaccess of the main domain so they can't come close to neither one of the forums!

thanks for that blue!:) i'll go add them now, pls tell me if you still see that bot around.

k, this is the addition to my .htaccess now.

Code: Select all

BrowserMatchNoCase AhrefsBot/3.1 bad_bot
BrowserMatchNoCase Baiduspider/2.0 bad_bot
Order Deny,Allow
Deny from env=bad_bot


User avatar
Blue Frost
Posts: 1442
Joined: July 22nd, 2010, 11:48 am

Re: banned AhrefsBot - been attacking us similarly like baid

Unread post by Blue Frost »

:) cool Ill keep an eye, it's a shame they are suck hogs .
One day I had like 12 baidu at once, and a few others I forget what the names was.


User avatar
Blue Frost
Posts: 1442
Joined: July 22nd, 2010, 11:48 am

Re: banned AhrefsBot - been attacking us similarly like baid

Unread post by Blue Frost »

I still have this one, but far less than I usually do

Guest IP: 180.76.5.166 » Whois
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Maybe i need to add the script now, might just know where I am now.


User avatar
andy1
Posts: 1295
Joined: February 22nd, 2010, 9:42 am

Re: banned AhrefsBot - been attacking us similarly like baid

Unread post by andy1 »

damn. maybe. i'm not seeing it here or the other forum tho.. yup i'm afraid you'll have to add the script now blue, sorry lol:(


User avatar
Blue Frost
Posts: 1442
Joined: July 22nd, 2010, 11:48 am

Re: banned AhrefsBot - been attacking us similarly like baid

Unread post by Blue Frost »

Test :001_unsure:


User avatar
Blue Frost
Posts: 1442
Joined: July 22nd, 2010, 11:48 am

Re: banned AhrefsBot - been attacking us similarly like baid

Unread post by Blue Frost »

As I tried to post before I think I need to add the script to my forum, I guess it just gets put at the very end of the .ht access .
Been seeing more different ones lately, google even has one now I noticed.


User avatar
andy1
Posts: 1295
Joined: February 22nd, 2010, 9:42 am

Re: banned AhrefsBot - been attacking us similarly like baid

Unread post by andy1 »

hi! yes, do it blue:) and let me know if it worked. it completely removed them from here as far as i can see.


User avatar
Blue Frost
Posts: 1442
Joined: July 22nd, 2010, 11:48 am

Re: banned AhrefsBot - been attacking us similarly like baid

Unread post by Blue Frost »

Ill try it later this evening, I have no use for them or any other.
Resource hogs is all they are.


User avatar
sidv220
Posts: 83
Joined: October 16th, 2012, 6:23 am
Location: Burlington, MA
Contact:

Re: banned AhrefsBot - been attacking us similarly like baid

Unread post by sidv220 »

AHrefs Bots operate from various leased clouds, particularly choopa.net and amazon Cloud services, in addition to their servers in Ukraine. They are notorious of doing IP hopping if you ban them at htaccess level. The ahrefsbot somehow will get into your site through amazon cloud service, choopa.net or one of the unresolvable IP address. It is the most unethical bot/content scrapper, I had seen. Your best bet is; install ZBBlock from http://www.spambotsecurity.com.
I have ZBBlock, which blocks all the attempts from ahrefsbots. I see almost over 100 IP hopping attempts on daily basis, in my ZBBlock log files, by ahrefsbot. It is the most notorious bandwitdth sucking content scrapper. It has no business at any sites. You do not benefits from it in any way. Ban it permanently. Below is a snip from my log files to give an impression how they do IP hopping.

#: 553 @: Tue, 16 Oct 2012 06:09:12 -0500 Running: 0.4.10a1

Host: 213.186.127.28.utel.net.ua

IP: 213.186.127.28

Score: 4

Violation count: 1

Why blocked: Bandwidth eating research bot (UA-136). Possibly hostile scraper/harvester (SPD-104).RBN. Bandwidth eating Ukranian bot. Stay in Ukrainian ghetto. You are not welcome elsewhere

Query:

Referer:

User Agent: Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/)

Reconstructed URL: http:// http://www.shaadiconnections.com /blog/



#: 554 @: Tue, 16 Oct 2012 06:17:10 -0500 Running: 0.4.10a1

Host: 173.199.115.59.choopa.net

IP: 173.199.115.59

Score: 3

Violation count: 1 INSTA-BANNED

Why blocked: Cloud Services. Not an ISP. Allows IP hopping. INSTA-BAN (CLD-210). Bandwidth eating research bot (UA-136). Possibly hostile scraper/harvester (SPD-104).You have been instantly banned due to extremely hazardous behavior!

Query:

Referer:

User Agent: Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)

Reconstructed URL: http:// http://www.shaadiconnections.com /members/



#: 555 @: Tue, 16 Oct 2012 06:22:44 -0500 Running: 0.4.10a1

Host: 173.199.115.155.choopa.net

IP: 173.199.115.155

Score: 3

Violation count: 1 INSTA-BANNED

Why blocked: Cloud Services. Not an ISP. Allows IP hopping. INSTA-BAN (CLD-210). Bandwidth eating research bot (UA-136). Possibly hostile scraper/harvester (SPD-104).You have been instantly banned due to extremely hazardous behavior!

Query:

Referer:

User Agent: Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)

Reconstructed URL: http:// http://www.shaadiconnections.com /affiliate-area/



#: 556 @: Tue, 16 Oct 2012 06:24:41 -0500 Running: 0.4.10a1

Host: 173.199.120.131.choopa.net

IP: 173.199.120.131

Score: 3

Violation count: 1 INSTA-BANNED

Why blocked: Cloud Services. Not an ISP. Allows IP hopping. INSTA-BAN (CLD-210). Bandwidth eating research bot (UA-136). Possibly hostile scraper/harvester (SPD-104).You have been instantly banned due to extremely hazardous behavior!

Query:

Referer:

User Agent: Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)

Reconstructed URL: http:// http://www.shaadiconnections.com /shaadi-register-free/


User avatar
andy1
Posts: 1295
Joined: February 22nd, 2010, 9:42 am

Re: banned AhrefsBot - been attacking us similarly like baid

Unread post by andy1 »

with all due respect, what can your program do that rewriting .htaccess alone won't do?


User avatar
sidv220
Posts: 83
Joined: October 16th, 2012, 6:23 am
Location: Burlington, MA
Contact:

Re: banned AhrefsBot - been attacking us similarly like baid

Unread post by sidv220 »

It is not my program, by the way. It from Zaphad, available under GPL from http://www.spambotsecurity.com. Here is what it can do from his web site. In my experience, you do not need to edit htaccess each time you have a spam or hacking issue. It takes care of it... so anyway, here is from spambotsecurity web site

What ZB Block is Excellent at:

Saves money by reducing hacker bandwith usage! (by 2,500% on this site's index page alone!)
Strengthing your site against defacement.
Preventing PHP script exploitation.
Ending Remote File Include (RFI) exploits.
Protecting against directory traversal attacks.
Stopping MySQL database injection and tampering.
Removing access from known bad addresses and domain names.
Blocking access from top level domains, like .cn (China) and .kp (North Korea).

What ZB Block is Good at:

Avoiding website scraping/content theft.
Deterring bad user agents.
Halting referrer spam.
Impeding some Cross Site Scripting (XSS) attacks.

ZB Block is also fast, not only does ZB Block check for over 100 million bad IPs/Hostnames and many thousands of bots, but standard execution times are around 1/10th of a second on an aged PIII 930, which is unnoticable to the web surfer.

Why ZB Block is BETTER than .htaccess methods...

Under certiain tasks, it is FASTER than htaccess due to only polling the server for data once per execution. An example of this is domain blocking.
It will run on webservers that do not support the full gamut of .htaccess commands (And there are quite a few).
It allows for intelligent detection of problem clients without previous knowledge of their address.
It can sniff query strings to find attack sequences from all IPs, while allowing legitimate requests to go through.
Through proper signature use, it can automatically remove some blocks that have met a condition. (such as registration of domain)
It can ban whole whole ranges of IPs written in classic decimal quadot notation. You can put your own custom ones in the signatures like 193.189.126.5 through 193.189.127.252 . (.htaccess gets a big FAIL! on dealing with IPs as it uses tricky to maintain CIDR ranges that only work in a most signifigant bit (MSB) method, sometimes requiring multiple entries for oddball ranges. 'Did I really include all the IPs? Did I accidentally go to far?')
Some hosts don't like custom 403s, so they don't allow you to use your own .htaccess driven 403. ZB Block doesn't care if the .htaccess is emplaced.
It logs banned accesses for later review in plain, easy to read english, with a description as to why said session was blocked.
It's simple and easy to use, and requires no authorization beyond the ability to upload files to your php equipped web-server.
Most importantly, it slows down evil robot machines to a crawl (sometimes) and helps alleviate (we hope) your fellow hosts/webmasters from some of the unwanted traffic!


User avatar
andy1
Posts: 1295
Joined: February 22nd, 2010, 9:42 am

Re: banned AhrefsBot - been attacking us similarly like baid

Unread post by andy1 »

thanks sid that's detailed and very informative.

however i don't really have any more robots bugging me since i did the code edits so i'll wait and see if i need to do anything on top of that. thanks anyway for the info.


Post Reply