Banning Baidu bot

User avatar

Topic author
andy1
Posts: 1295
Joined: February 22nd, 2010, 9:42 am

Banning Baidu bot

Unread post by andy1 » November 28th, 2011, 12:15 pm

i did that on the other forum and it decreased tremendously the load on the server - we still have blackouts because our host overextended themselves (and that's the opinion i got at the phpBB support forum too - apparently it's quite common as they tend to oversell their cheap hosting and then they start underperforming), but not as much.

this bot would come from a crazy number of IPs at a time over 20 or so, so i just banned its whole range of IPs for now.

User avatar

Blue Frost
Posts: 1442
Joined: July 22nd, 2010, 11:48 am

Re: Banning Baidu bot

Unread post by Blue Frost » November 28th, 2011, 4:24 pm

Interesting, I banned all bots but google because the some thing. The host even though you have unlimited bandwidth will give you a timeout and blame you.
Shared servers are bad at that and really advertise misleading info.

User avatar

GuardianFlash
Posts: 83
Joined: July 16th, 2010, 3:14 pm

Re: Banning Baidu bot

Unread post by GuardianFlash » November 28th, 2011, 4:39 pm

what are bots? Are they really people browsing your site?

User avatar

Blue Frost
Posts: 1442
Joined: July 22nd, 2010, 11:48 am

Re: Banning Baidu bot

Unread post by Blue Frost » November 28th, 2011, 4:46 pm

They are automated crawlers getting info to advertise your site.
Google bot is one and that's how you are put in line in there search engine

User avatar

Topic author
andy1
Posts: 1295
Joined: February 22nd, 2010, 9:42 am

Re: Banning Baidu bot

Unread post by andy1 » November 29th, 2011, 6:26 pm

Shared servers are bad at that and really advertise misleading info.
absolutely. a ton of that stuff on phpbb support forum.

the question to ask your host before getting an account with then is how many user connections will they allow on their shared servers (apparently the bandwidth does not translate to that - you can have unlimited bandwidth yet no more than 100 users allowed at a time.

User avatar

Blue Frost
Posts: 1442
Joined: July 22nd, 2010, 11:48 am

Re: Banning Baidu bot

Unread post by Blue Frost » November 29th, 2011, 6:42 pm

Oh one person can have 100 connections also, most don't but some have. One person don't = one connection most the time, also pages can stay open and not close in your database, that can severely ad up fast.

User avatar

Topic author
andy1
Posts: 1295
Joined: February 22nd, 2010, 9:42 am

Re: Banning Baidu bot

Unread post by andy1 » November 29th, 2011, 7:17 pm

right! which is why it's important to limit the session length to some reasonable value (the default is 3600s).

btw this is what i got from that host i'm talking to so far. doesn't look too bad (i think cheap hosts typically limit max_connections to 100 if i'm not mistaken)
I just spoke to one of the techs and they tell me we usually set max_connections
to 500 and max_user_connections to 50-100. These values can be raised upon
request as long as it doesn't cause any issues for other clients on the server.
and what if it does cause any issues lol:(.. they just need a better, more precise description of their services..

User avatar

Blue Frost
Posts: 1442
Joined: July 22nd, 2010, 11:48 am

Re: Banning Baidu bot

Unread post by Blue Frost » November 29th, 2011, 8:12 pm

I don't even know what to limit to, what a reasonable value is I have no idea.
Mine is limited to 100 even though it tells you unlimited, on the hour it resets and the board lags.

User avatar

Topic author
andy1
Posts: 1295
Joined: February 22nd, 2010, 9:42 am

Re: Banning Baidu bot

Unread post by andy1 » November 30th, 2011, 12:43 am

blue why don't you ask them specifically how many connections do they allow and can you hope for an increase in the number of connections they allow or maybe a move to a less busy server?

User avatar

Blue Frost
Posts: 1442
Joined: July 22nd, 2010, 11:48 am

Re: Banning Baidu bot

Unread post by Blue Frost » November 30th, 2011, 8:54 am

They allow 100 is all I was told, and it was to protect the other clients.
I think it's automatically done shutting us off

User avatar

Topic author
andy1
Posts: 1295
Joined: February 22nd, 2010, 9:42 am

Re: Banning Baidu bot

Unread post by andy1 » December 3rd, 2011, 3:40 pm

yeah, i think you're right. it's probably the same here. still banning this bot did make a huge difference we basically haven't been down since.

User avatar

Blue Frost
Posts: 1442
Joined: July 22nd, 2010, 11:48 am

Re: Banning Baidu bot

Unread post by Blue Frost » December 3rd, 2011, 6:38 pm

My site has been better since I booted them off, but I understand the need for them that you have.
It's just a shame they are so intrusive and over staying their welcome taking up needed bandwidth.

Oh I changed my session length to 3000, I'm not sure if that helped much but I have not sen so much lag time on the hour :) maybe it helped.

User avatar

Topic author
andy1
Posts: 1295
Joined: February 22nd, 2010, 9:42 am

Re: Banning Baidu bot

Unread post by andy1 » December 7th, 2011, 12:56 am

on monday (yesterday) i did get a short note from somebody at "invancouver" to say that they'll forward my email to their sales department but haven't heard anything since.. *sigh*..

do you remember blue that guy who owns chatanplay forum where is he hosted? we talked about that some time ago, he said he was paying something like 12? or 20 dollars per month for a designated option or semi-designated option.. i wish i remembered.. problem with twitter is if you tweeted something only a few month ago, you basically can't retrieve it anymore (assuming you tweeted some in the mean time, which i did:). ugh.

User avatar

Blue Frost
Posts: 1442
Joined: July 22nd, 2010, 11:48 am

Re: Banning Baidu bot

Unread post by Blue Frost » December 7th, 2011, 8:52 am

I remember the site and conversation, but not the statistics.
I wish I knew of some to check on for you, it's not any fun to be down like you have been lately.

User avatar

Topic author
andy1
Posts: 1295
Joined: February 22nd, 2010, 9:42 am

Re: Banning Baidu bot

Unread post by andy1 » December 9th, 2011, 10:13 am

can't wait to move, if for no other reason than to stop supporting an unethical business that is our current host. :angry:

Post Reply