Yahoo crawlers - killing our site

[.QUACK.Major.Pain]

Your phpBB Version: 2.0.
phpBB Type: Integramod 141
MODs: No
Your knowledge: Beginner
Board URL: http://www.aaquacks.com/forum

PHP Version:
MySQL Version:


What was done before the problem appeared?



What was done to try to solve the problem?




De.scription and Message

Starting to find yahoo is getting more intense lately.
Seems to be as many as 6 crawlers from yahoo on at a time.
Also getting in the range of 40-50 pending yahoo bots a day over the past week.
Curious why such an agressive move by yahoo all of a sudden?

[ZacFields]

It's just how yahoo crawls. Yahoo will send a bunch of bots to the same site, whereas google tends just to do one or two bots to each site and cover a lot more content.

They just have conflicting views on how to spider. One thing I will tell you is that I blocked all search engine spiders for a couple weeks once, and my site ran way faster during that time. But it just wasn't worth it knowing that I was taking my site out of the public eye.

Zac

[.QUACK.Major.Pain]

Yea - only had a couple google crawlers since we started.

Google:

|66.249.65.110|66.249.65.80|66.249.66.138|66.249.66.236|66.249.67.26|66.249.73.11|66.249.66.44|66.249.66.11

Yahoo:

|74.6.72.37|74.6.73.35|74.6.71.154|74.6.71.92|74.6.68.144|74.6.72.73|74.6.70.233|74.6.72.123|74.6.71.70|72.30.177.185|74.6.68.211|74.6.73.165|74.6.68.146|74.6.74.62|74.6.73.77|74.6.73.61|74.6.68.117|74.6.70.6|74.6.70.246|74.6.70.177|74.6.69.216|74.6.69.118|74.6.71.59|74.6.69.23|74.6.69.172|74.6.73.248|74.6.69.21|74.6.71.186|74.6.71.57|74.6.69.204|72.30.177.165|74.6.73.92|74.6.69.169|74.6.70.48|74.6.73.251|74.6.71.36|74.6.72.99|72.30.177.245|74.6.72.79|74.6.73.123|74.6.73.90|74.6.70.165|74.6.69.14|74.6.67.145|74.6.73.6|74.6.68.18|74.6.70.225|74.6.68.31|74.6.68.79|74.6.69.105|74.6.72.247|74.6.66.38|74.6.72.180|74.6.65.233|74.6.72.184|74.6.68.41|74.6.73.184|74.6.74.222|74.6.68.223|74.6.67.230|74.6.66.47|74.6.69.112|74.6.70.169|74.6.67.111|74.6.69.226|74.6.74.230|74.6.68.215|74.6.71.119|74.6.68.74|74.6.69.163|74.6.67.210|74.6.65.254|74.6.68.160|74.6.68.17|74.6.70.79|74.6.72.236|72.30.177.124|74.6.71.50|74.6.70.215|74.6.69.100|72.30.177.237|74.6.72.182|74.6.67.72|72.30.177.216|72.30.177.77|74.6.71.179|74.6.75.40|74.6.69.227|74.6.69.20|74.6.67.119|74.6.73.245|74.6.73.156|74.6.67.121|74.6.74.40|74.6.73.124|74.6.66.56|74.6.67.215|74.6.73.173|74.6.68.236|74.6.75.29|74.6.74.74|74.6.69.134|74.6.72.120|74.6.71.159|74.6.72.222|74.6.72.217|74.6.70.180|72.30.177.160|74.6.68.106|74.6.73.70|74.6.72.55|74.6.69.121|74.6.66.50|74.6.68.164|74.6.67.156|74.6.74.189|74.6.70.8|74.6.71.239|74.6.73.42|74.6.67.165|74.6.70.178|74.6.68.48|74.6.67.152|74.6.72.170|74.6.72.52|74.6.73.44|74.6.73.34|74.6.68.135|74.6.68.221|74.6.72.157|74.6.69.221|72.30.177.188|74.6.68.109|74.6.74.224|74.6.72.220|74.6.73.8|74.6.75.30|74.6.72.70|74.6.70.171|74.6.67.134|74.6.74.79|74.6.68.123|74.6.69.229|74.6.65.242|74.6.73.180|74.6.73.50|74.6.71.224|74.6.74.119|74.6.69.198|74.6.74.212|74.6.66.44|74.6.69.157|74.6.69.107|74.6.73.247|74.6.65.222|74.6.70.59|74.6.73.40|74.6.74.82|74.6.70.220|74.6.67.159|74.6.71.83|74.6.71.234|74.6.71.243|74.6.69.22|74.6.70.158|74.6.65.241|74.6.74.124|74.6.68.235|74.6.72.82|72.30.177.171|74.6.73.186|74.6.74.86|74.6.68.224|74.6.68.226|74.6.71.80|74.6.71.33|74.6.74.125|74.6.69.40|74.6.73.93|74.6.68.218|72.30.177.121|74.6.68.232|74.6.69.113|74.6.74.8|74.6.73.171|74.6.69.41|74.6.72.56|74.6.70.162|74.6.67.235|74.6.72.51|74.6.73.157|74.6.73.229|74.6.68.217|74.6.68.138|74.6.70.243|74.6.71.72|74.6.72.235|72.30.177.187|74.6.68.209|74.6.70.52|74.6.72.57|74.6.69.155|74.6.65.202|74.6.68.165|74.6.69.167|72.30.177.175|74.6.73.254|74.6.70.179|74.6.75.24|74.6.70.28|74.6.72.250|74.6.73.227|74.6.68.136|74.6.74.214|74.6.72.223|72.30.177.233|74.6.72.117|74.6.70.43|74.6.71.88|72.30.177.123|74.6.71.160|74.6.72.159|74.6.74.95|74.6.69.13|74.6.74.90|74.6.70.160|74.6.68.23|74.6.67.155|74.6.71.125|72.30.177.156|74.6.70.189|74.6.70.125|74.6.70.161|74.6.67.123|74.6.68.206|74.6.65.232|72.30.177.151|74.6.74.43|74.6.69.106|74.6.67.171|74.6.68.213|74.6.70.46|74.6.74.47|74.6.70.93|74.6.73.52|74.6.68.107|74.6.70.123|74.6.70.229|74.6.74.168|74.6.66.60|74.6.68.145|74.6.72.189|74.6.69.37|74.6.69.45|72.30.177.240|74.6.66.51|74.6.68.42|74.6.65.245|74.6.70.117|74.6.73.160|74.6.69.24|74.6.74.36|74.6.68.152|74.6.67.109|74.6.70.42|74.6.72.173|74.6.70.190|74.6.72.40|74.6.68.118|74.6.73.219|74.6.67.209|74.6.72.160|74.6.70.30|74.6.68.141|74.6.68.124|74.6.74.186|74.6.75.49|74.6.70.94|74.6.69.30|74.6.65.248|74.6.74.209|74.6.72.80|74.6.74.165|74.6.67.100|74.6.67.150|74.6.65.247|74.6.67.138|74.6.70.157|74.6.69.31|74.6.72.162|74.6.69.11|74.6.71.240|74.6.69.32|74.6.73.241|74.6.74.231|74.6.67.148|74.6.68.20|74.6.67.115|74.6.73.181|74.6.67.112|74.6.72.221|74.6.72.171|74.6.71.220|74.6.73.78|74.6.70.99|74.6.69.80|74.6.69.162|74.6.72.122|74.6.73.167|74.6.70.118|74.6.65.204|74.6.68.44|74.6.68.73|74.6.70.153|74.6.71.246|74.6.68.15|74.6.66.32|74.6.68.110|74.6.69.16|74.6.72.244|72.30.177.218|74.6.70.174|72.30.177.186|74.6.67.216|72.30.177.190|72.30.177.183|74.6.70.76|74.6.72.61|74.6.73.91|74.6.71.156|74.6.70.55|74.6.71.52|72.30.177.117|74.6.72.59|74.6.70.92|74.6.71.117|74.6.73.237|74.6.72.178|74.6.72.164|74.6.71.157|74.6.74.213|74.6.73.56|74.6.67.172|74.6.72.226|74.6.71.151|74.6.72.126|74.6.74.157|74.6.72.30|74.6.67.141|74.6.68.100|74.6.73.235|74.6.66.8|74.6.69.158|74.6.71.120|74.6.71.73|74.6.67.80|74.6.73.225|74.6.74.167|74.6.71.219|74.6.71.248|74.6.74.72|74.6.74.57|74.6.74.126|74.6.72.93|74.6.74.99|74.6.73.41|74.6.67.162|74.6.67.124|74.6.73.58|74.6.71.152|74.6.68.24|72.30.177.119|74.6.71.71|74.6.68.214|74.6.73.215|74.6.73.7|74.6.71.163|74.6.68.228|74.6.75.23|74.6.67.168|74.6.72.175|72.30.177.70|74.6.68.13|74.6.71.124|74.6.74.76|74.6.70.234|74.6.71.165|74.6.69.122|74.6.74.60|74.6.71.230|74.6.69.124|74.6.71.34|74.6.68.225|74.6.71.94|74.6.67.223|74.6.72.152|74.6.65.244|74.6.71.222|74.6.68.140|74.6.66.57|74.6.68.43|74.6.74.117|74.6.71.170|74.6.68.169|74.6.73.236|74.6.69.70|74.6.67.151|74.6.67.231|74.6.67.200|74.6.66.31|74.6.69.42|74.6.73.239|74.6.72.218|74.6.67.202|74.6.68.229|74.6.71.172|74.6.74.227|72.30.177.90|74.6.74.56|74.6.67.208|72.30.177.231|74.6.68.122|74.6.67.142|74.6.70.240|74.6.68.45|74.6.71.85|74.6.72.27|74.6.66.27|74.6.71.35|74.6.71.247|74.6.73.94|74.6.74.180|74.6.68.71|74.6.67.136|74.6.74.228|74.6.70.173|74.6.68.32|74.6.66.28|74.6.73.51|74.6.72.36|74.6.70.98|74.6.72.42|74.6.71.79|74.6.73.31|74.6.73.97|74.6.67.77|74.6.73.59|72.30.177.173|74.6.68.143|74.6.74.172|74.6.70.38|74.6.70.74|74.6.70.245|74.6.74.163|74.6.68.167|74.6.72.216|74.6.72.88|72.30.177.92|74.6.71.233|74.6.72.252|72.30.177.88|74.6.65.223|74.6.65.240|74.6.68.78|72.30.177.174|74.6.71.39|74.6.74.226|74.6.71.54|74.6.72.166|74.6.72.227|74.6.73.238|74.6.68.29|74.6.73.38|74.6.69.82|74.6.66.49|74.6.72.249|74.6.70.163|74.6.74.7|74.6.69.199|74.6.70.156|74.6.72.186|74.6.74.53|74.6.72.174|74.6.67.74|74.6.68.125|74.6.69.159|74.6.72.54|74.6.69.76|74.6.65.229|74.6.71.158|74.6.72.177|74.6.67.217|74.6.72.8|74.6.67.70|74.6.74.187|74.6.65.253|74.6.72.125|74.6.70.223|74.6.71.45|74.6.66.52|74.6.68.120|72.30.177.166|74.6.70.164|74.6.71.187|74.6.73.74|74.6.70.47|74.6.71.218|74.6.71.62|74.6.74.41|74.6.70.242|74.6.74.217|74.6.68.227|74.6.74.156|74.6.73.176|74.6.71.229|74.6.72.188|74.6.70.186|74.6.66.54|74.6.71.93|72.30.177.232|74.6.71.221|74.6.68.102|74.6.70.181|74.6.69.125|74.6.67.101|74.6.70.182|74.6.68.168|72.30.177.251|74.6.66.59|74.6.71.55|72.30.177.164|74.6.67.153|74.6.72.34|74.6.65.246|74.6.74.169|72.30.177.172|72.30.177.177|74.6.69.15|74.6.72.83|74.6.71.216|74.6.72.240|74.6.65.221|74.6.70.33|74.6.65.225|74.6.71.251|74.6.69.34|74.6.68.82|74.6.67.204|74.6.71.241|74.6.74.177|74.6.72.87|74.6.72.233|74.6.68.142|72.30.177.221|74.6.68.19|74.6.67.234|74.6.67.207|74.6.71.7|74.6.68.114|74.6.73.57|74.6.73.175|74.6.68.163|72.30.177.253|74.6.74.211|74.6.69.152|74.6.65.234|74.6.68.47|74.6.71.37|72.30.177.227|74.6.65.250|74.6.72.183|74.6.75.19|72.30.177.238|74.6.73.95|74.6.71.47|74.6.72.7|74.6.67.105|74.6.71.226|74.6.71.42|74.6.74.181|74.6.70.254|74.6.74.34|74.6.73.178|74.6.73.32|72.30.177.241|74.6.67.212|74.6.71.162|74.6.68.101|74.6.73.226|74.6.65.235|74.6.71.183|74.6.74.55|74.6.68.37|74.6.69.26|74.6.71.181|74.6.74.42|74.6.67.120|74.6.74.61|74.6.67.79|74.6.72.89|74.6.74.174|74.6.68.149|72.30.177.181|74.6.69.72|74.6.74.236|74.6.70.221|72.30.177.91|74.6.70.72|74.6.72.33|74.6.72.229|74.6.69.77|74.6.68.22|74.6.65.227|74.6.67.227|74.6.73.230|74.6.72.46|74.6.73.99|72.30.177.152|74.6.69.38|74.6.72.228|74.6.74.199|74.6.66.48|74.6.74.70|74.6.72.81|74.6.67.75|74.6.66.7|74.6.70.244|74.6.74.28|74.6.73.218|74.6.70.27|74.6.67.164|74.6.70.119|74.6.72.246|74.6.68.111|74.6.68.21|74.6.74.33|74.6.67.106|74.6.71.84|72.30.177.249|74.6.69.102|74.6.68.77|74.6.67.163|74.6.73.246|74.6.70.73|74.6.72.155|74.6.75.25|74.6.69.171|74.6.69.33|74.6.68.115|74.6.70.78|72.30.177.220|74.6.72.77|74.6.74.225|74.6.73.220|74.6.67.81|74.6.72.187|74.6.70.227|74.6.70.241|74.6.74.123|74.6.71.217|74.6.74.159|74.6.69.36|74.6.74.58|72.30.177.126|74.6.73.223|74.6.72.84|74.6.68.113|74.6.71.177|74.6.71.28|74.6.73.250|74.6.73.45|74.6.71.58|74.6.72.165|74.6.73.98|74.6.67.149|74.6.67.125|74.6.66.58|74.6.74.30|74.6.70.44|74.6.67.169|74.6.69.10|74.6.74.35|72.30.177.230|74.6.72.230|74.6.71.161|74.6.66.40|74.6.67.135|74.6.71.118|74.6.72.121|74.6.74.179|74.6.74.98|74.6.71.169|74.6.66.33|74.6.74.205|74.6.70.166|72.30.177.228|74.6.70.95|74.6.68.222|74.6.73.174|74.6.73.232|74.6.72.190|72.30.177.236|74.6.71.180|74.6.69.39|74.6.70.249|74.6.71.188|74.6.75.10|74.6.67.236|74.6.71.153|74.6.73.253|74.6.68.40|72.30.177.167|74.6.70.183|74.6.74.46|74.6.73.96|74.6.70.39|74.6.72.38|74.6.68.166|74.6.70.232|74.6.70.49|72.30.177.229|74.6.67.103|74.6.70.251|74.6.74.37|74.6.69.79|74.6.70.96|74.6.74.232|74.6.67.161|74.6.71.82|74.6.73.163|74.6.71.96|74.6.68.25|74.6.69.43|74.6.71.60|74.6.73.155|74.6.69.27|74.6.71.90|74.6.72.169

And I have another 40+ pending yahoo bots that came over night. At 11:30PM, I had cleared all the pending bots. At 7AM, there were 40+ yahoo pending. Just a little agressive.

Does there come a time when you have all the bots added and there are no more pending bots? LOL
Wishful thinking.

[tekguru]

So anyway of blocking Yahoo bots as I only rely on Google coming in.

[dan0042]

need some help have 247 tahoo bots on my forums + cont send pm...saying this
You cannot make another post so soon after your last; please try again in a short while.


Some one help me ..Been like this for 2 days now

[CaNNon]

You can use admin > crackertracker > IP & UserAgent Blocker.

Notice " *ia_archiver* " this is the Web crawler used by the Alexa search engine
and they block it by default. Just do yahoo the same way if you wish to block it.

[.QUACK.Major.Pain]

They hit my site really bad for about 3 weeks and then almost stopped for about 6 weeks.
Now it started again and all new IP's. Getting about 50-60 pending yahoo bots a day.
I must be up to about 3000 yahoo ip's added already.

[dan0042]

tba with u .im 141.. dont look like goin help u ate me...

[CaNNon]

You can use admin > crackertracker > IP & UserAgent Blocker.

Notice " *ia_archiver* " this is the Web crawler used by the Alexa search engine
and they block it by default.

Just do yahoo the same way if you wish to block it.

Do that dan0042, just block them for now! when some one has a better idea you can allow them again.