Page 1 of 1
Yahoo crawlers - killing our site
Posted:
Wed May 09, 2007 8:10 pmAuthor: .QUACK.Major.Pain
Your phpBB Version: 2.0.
phpBB Type: Integramod 141
MODs: No
Your knowledge: Beginner
Board URL: http://www.aaquacks.com/forum PHP Version: MySQL Version: What was done before the problem appeared? What was done to try to solve the problem? De.scription and MessageStarting to find yahoo is getting more intense lately.
Seems to be as many as 6 crawlers from yahoo on at a time.
Also getting in the range of 40-50 pending yahoo bots a day over the past week.
Curious why such an agressive move by yahoo all of a sudden?
Posted:
Wed May 09, 2007 8:49 pmAuthor: ZacFields
It's just how yahoo crawls. Yahoo will send a bunch of bots to the same site, whereas google tends just to do one or two bots to each site and cover a lot more content.
They just have conflicting views on how to spider. One thing I will tell you is that I blocked all search engine spiders for a couple weeks once, and my site ran way faster during that time. But it just wasn't worth it knowing that I was taking my site out of the public eye.
Zac
Re: Yahoo crawlers - killing our site
Posted:
Thu May 10, 2007 4:39 amAuthor: .QUACK.Major.Pain
Yea - only had a couple google crawlers since we started.
Google:
|66.249.65.110|66.249.65.80|66.249.66.138|66.249.66.236|66.249.67.26|66.249.73.11|66.249.66.44|66.249.66.11
Yahoo:
|74.6.72.37|74.6.73.35|74.6.71.154|74.6.71.92|74.6.68.144|74.6.72.73|74.6.70.233|74.6.72.123|74.6.71.70|72.30.177.185|74.6.68.211|74.6.73.165|74.6.68.146|74.6.74.62|74.6.73.77|74.6.73.61|74.6.68.117|74.6.70.6|74.6.70.246|74.6.70.177|74.6.69.216|74.6.69.118|74.6.71.59|74.6.69.23|74.6.69.172|74.6.73.248|74.6.69.21|74.6.71.186|74.6.71.57|74.6.69.204|72.30.177.165|74.6.73.92|74.6.69.169|74.6.70.48|74.6.73.251|74.6.71.36|74.6.72.99|72.30.177.245|74.6.72.79|74.6.73.123|74.6.73.90|74.6.70.165|74.6.69.14|74.6.67.145|74.6.73.6|74.6.68.18|74.6.70.225|74.6.68.31|74.6.68.79|74.6.69.105|74.6.72.247|74.6.66.38|74.6.72.180|74.6.65.233|74.6.72.184|74.6.68.41|74.6.73.184|74.6.74.222|74.6.68.223|74.6.67.230|74.6.66.47|74.6.69.112|74.6.70.169|74.6.67.111|74.6.69.226|74.6.74.230|74.6.68.215|74.6.71.119|74.6.68.74|74.6.69.163|74.6.67.210|74.6.65.254|74.6.68.160|74.6.68.17|74.6.70.79|74.6.72.236|72.30.177.124|74.6.71.50|74.6.70.215|74.6.69.100|72.30.177.237|74.6.72.182|74.6.67.72|72.30.177.216|72.30.177.77|74.6.71.179|74.6.75.40|74.6.69.227|74.6.69.20|74.6.67.119|74.6.73.245|74.6.73.156|74.6.67.121|74.6.74.40|74.6.73.124|74.6.66.56|74.6.67.215|74.6.73.173|74.6.68.236|74.6.75.29|74.6.74.74|74.6.69.134|74.6.72.120|74.6.71.159|74.6.72.222|74.6.72.217|74.6.70.180|72.30.177.160|74.6.68.106|74.6.73.70|74.6.72.55|74.6.69.121|74.6.66.50|74.6.68.164|74.6.67.156|74.6.74.189|74.6.70.8|74.6.71.239|74.6.73.42|74.6.67.165|74.6.70.178|74.6.68.48|74.6.67.152|74.6.72.170|74.6.72.52|74.6.73.44|74.6.73.34|74.6.68.135|74.6.68.221|74.6.72.157|74.6.69.221|72.30.177.188|74.6.68.109|74.6.74.224|74.6.72.220|74.6.73.8|74.6.75.30|74.6.72.70|74.6.70.171|74.6.67.134|74.6.74.79|74.6.68.123|74.6.69.229|74.6.65.242|74.6.73.180|74.6.73.50|74.6.71.224|74.6.74.119|74.6.69.198|74.6.74.212|74.6.66.44|74.6.69.157|74.6.69.107|74.6.73.247|74.6.65.222|74.6.70.59|74.6.73.40|74.6.74.82|74.6.70.220|74.6.67.159|74.6.71.83|74.6.71.234|74.6.71.243|74.6.69.22|74.6.70.158|74.6.65.241|74.6.74.124|74.6.68.235|74.6.72.82|72.30.177.171|74.6.73.186|74.6.74.86|74.6.68.224|74.6.68.226|74.6.71.80|74.6.71.33|74.6.74.125|74.6.69.40|74.6.73.93|74.6.68.218|72.30.177.121|74.6.68.232|74.6.69.113|74.6.74.8|74.6.73.171|74.6.69.41|74.6.72.56|74.6.70.162|74.6.67.235|74.6.72.51|74.6.73.157|74.6.73.229|74.6.68.217|74.6.68.138|74.6.70.243|74.6.71.72|74.6.72.235|72.30.177.187|74.6.68.209|74.6.70.52|74.6.72.57|74.6.69.155|74.6.65.202|74.6.68.165|74.6.69.167|72.30.177.175|74.6.73.254|74.6.70.179|74.6.75.24|74.6.70.28|74.6.72.250|74.6.73.227|74.6.68.136|74.6.74.214|74.6.72.223|72.30.177.233|74.6.72.117|74.6.70.43|74.6.71.88|72.30.177.123|74.6.71.160|74.6.72.159|74.6.74.95|74.6.69.13|74.6.74.90|74.6.70.160|74.6.68.23|74.6.67.155|74.6.71.125|72.30.177.156|74.6.70.189|74.6.70.125|74.6.70.161|74.6.67.123|74.6.68.206|74.6.65.232|72.30.177.151|74.6.74.43|74.6.69.106|74.6.67.171|74.6.68.213|74.6.70.46|74.6.74.47|74.6.70.93|74.6.73.52|74.6.68.107|74.6.70.123|74.6.70.229|74.6.74.168|74.6.66.60|74.6.68.145|74.6.72.189|74.6.69.37|74.6.69.45|72.30.177.240|74.6.66.51|74.6.68.42|74.6.65.245|74.6.70.117|74.6.73.160|74.6.69.24|74.6.74.36|74.6.68.152|74.6.67.109|74.6.70.42|74.6.72.173|74.6.70.190|74.6.72.40|74.6.68.118|74.6.73.219|74.6.67.209|74.6.72.160|74.6.70.30|74.6.68.141|74.6.68.124|74.6.74.186|74.6.75.49|74.6.70.94|74.6.69.30|74.6.65.248|74.6.74.209|74.6.72.80|74.6.74.165|74.6.67.100|74.6.67.150|74.6.65.247|74.6.67.138|74.6.70.157|74.6.69.31|74.6.72.162|74.6.69.11|74.6.71.240|74.6.69.32|74.6.73.241|74.6.74.231|74.6.67.148|74.6.68.20|74.6.67.115|74.6.73.181|74.6.67.112|74.6.72.221|74.6.72.171|74.6.71.220|74.6.73.78|74.6.70.99|74.6.69.80|74.6.69.162|74.6.72.122|74.6.73.167|74.6.70.118|74.6.65.204|74.6.68.44|74.6.68.73|74.6.70.153|74.6.71.246|74.6.68.15|74.6.66.32|74.6.68.110|74.6.69.16|74.6.72.244|72.30.177.218|74.6.70.174|72.30.177.186|74.6.67.216|72.30.177.190|72.30.177.183|74.6.70.76|74.6.72.61|74.6.73.91|74.6.71.156|74.6.70.55|74.6.71.52|72.30.177.117|74.6.72.59|74.6.70.92|74.6.71.117|74.6.73.237|74.6.72.178|74.6.72.164|74.6.71.157|74.6.74.213|74.6.73.56|74.6.67.172|74.6.72.226|74.6.71.151|74.6.72.126|74.6.74.157|74.6.72.30|74.6.67.141|74.6.68.100|74.6.73.235|74.6.66.8|74.6.69.158|74.6.71.120|74.6.71.73|74.6.67.80|74.6.73.225|74.6.74.167|74.6.71.219|74.6.71.248|74.6.74.72|74.6.74.57|74.6.74.126|74.6.72.93|74.6.74.99|74.6.73.41|74.6.67.162|74.6.67.124|74.6.73.58|74.6.71.152|74.6.68.24|72.30.177.119|74.6.71.71|74.6.68.214|74.6.73.215|74.6.73.7|74.6.71.163|74.6.68.228|74.6.75.23|74.6.67.168|74.6.72.175|72.30.177.70|74.6.68.13|74.6.71.124|74.6.74.76|74.6.70.234|74.6.71.165|74.6.69.122|74.6.74.60|74.6.71.230|74.6.69.124|74.6.71.34|74.6.68.225|74.6.71.94|74.6.67.223|74.6.72.152|74.6.65.244|74.6.71.222|74.6.68.140|74.6.66.57|74.6.68.43|74.6.74.117|74.6.71.170|74.6.68.169|74.6.73.236|74.6.69.70|74.6.67.151|74.6.67.231|74.6.67.200|74.6.66.31|74.6.69.42|74.6.73.239|74.6.72.218|74.6.67.202|74.6.68.229|74.6.71.172|74.6.74.227|72.30.177.90|74.6.74.56|74.6.67.208|72.30.177.231|74.6.68.122|74.6.67.142|74.6.70.240|74.6.68.45|74.6.71.85|74.6.72.27|74.6.66.27|74.6.71.35|74.6.71.247|74.6.73.94|74.6.74.180|74.6.68.71|74.6.67.136|74.6.74.228|74.6.70.173|74.6.68.32|74.6.66.28|74.6.73.51|74.6.72.36|74.6.70.98|74.6.72.42|74.6.71.79|74.6.73.31|74.6.73.97|74.6.67.77|74.6.73.59|72.30.177.173|74.6.68.143|74.6.74.172|74.6.70.38|74.6.70.74|74.6.70.245|74.6.74.163|74.6.68.167|74.6.72.216|74.6.72.88|72.30.177.92|74.6.71.233|74.6.72.252|72.30.177.88|74.6.65.223|74.6.65.240|74.6.68.78|72.30.177.174|74.6.71.39|74.6.74.226|74.6.71.54|74.6.72.166|74.6.72.227|74.6.73.238|74.6.68.29|74.6.73.38|74.6.69.82|74.6.66.49|74.6.72.249|74.6.70.163|74.6.74.7|74.6.69.199|74.6.70.156|74.6.72.186|74.6.74.53|74.6.72.174|74.6.67.74|74.6.68.125|74.6.69.159|74.6.72.54|74.6.69.76|74.6.65.229|74.6.71.158|74.6.72.177|74.6.67.217|74.6.72.8|74.6.67.70|74.6.74.187|74.6.65.253|74.6.72.125|74.6.70.223|74.6.71.45|74.6.66.52|74.6.68.120|72.30.177.166|74.6.70.164|74.6.71.187|74.6.73.74|74.6.70.47|74.6.71.218|74.6.71.62|74.6.74.41|74.6.70.242|74.6.74.217|74.6.68.227|74.6.74.156|74.6.73.176|74.6.71.229|74.6.72.188|74.6.70.186|74.6.66.54|74.6.71.93|72.30.177.232|74.6.71.221|74.6.68.102|74.6.70.181|74.6.69.125|74.6.67.101|74.6.70.182|74.6.68.168|72.30.177.251|74.6.66.59|74.6.71.55|72.30.177.164|74.6.67.153|74.6.72.34|74.6.65.246|74.6.74.169|72.30.177.172|72.30.177.177|74.6.69.15|74.6.72.83|74.6.71.216|74.6.72.240|74.6.65.221|74.6.70.33|74.6.65.225|74.6.71.251|74.6.69.34|74.6.68.82|74.6.67.204|74.6.71.241|74.6.74.177|74.6.72.87|74.6.72.233|74.6.68.142|72.30.177.221|74.6.68.19|74.6.67.234|74.6.67.207|74.6.71.7|74.6.68.114|74.6.73.57|74.6.73.175|74.6.68.163|72.30.177.253|74.6.74.211|74.6.69.152|74.6.65.234|74.6.68.47|74.6.71.37|72.30.177.227|74.6.65.250|74.6.72.183|74.6.75.19|72.30.177.238|74.6.73.95|74.6.71.47|74.6.72.7|74.6.67.105|74.6.71.226|74.6.71.42|74.6.74.181|74.6.70.254|74.6.74.34|74.6.73.178|74.6.73.32|72.30.177.241|74.6.67.212|74.6.71.162|74.6.68.101|74.6.73.226|74.6.65.235|74.6.71.183|74.6.74.55|74.6.68.37|74.6.69.26|74.6.71.181|74.6.74.42|74.6.67.120|74.6.74.61|74.6.67.79|74.6.72.89|74.6.74.174|74.6.68.149|72.30.177.181|74.6.69.72|74.6.74.236|74.6.70.221|72.30.177.91|74.6.70.72|74.6.72.33|74.6.72.229|74.6.69.77|74.6.68.22|74.6.65.227|74.6.67.227|74.6.73.230|74.6.72.46|74.6.73.99|72.30.177.152|74.6.69.38|74.6.72.228|74.6.74.199|74.6.66.48|74.6.74.70|74.6.72.81|74.6.67.75|74.6.66.7|74.6.70.244|74.6.74.28|74.6.73.218|74.6.70.27|74.6.67.164|74.6.70.119|74.6.72.246|74.6.68.111|74.6.68.21|74.6.74.33|74.6.67.106|74.6.71.84|72.30.177.249|74.6.69.102|74.6.68.77|74.6.67.163|74.6.73.246|74.6.70.73|74.6.72.155|74.6.75.25|74.6.69.171|74.6.69.33|74.6.68.115|74.6.70.78|72.30.177.220|74.6.72.77|74.6.74.225|74.6.73.220|74.6.67.81|74.6.72.187|74.6.70.227|74.6.70.241|74.6.74.123|74.6.71.217|74.6.74.159|74.6.69.36|74.6.74.58|72.30.177.126|74.6.73.223|74.6.72.84|74.6.68.113|74.6.71.177|74.6.71.28|74.6.73.250|74.6.73.45|74.6.71.58|74.6.72.165|74.6.73.98|74.6.67.149|74.6.67.125|74.6.66.58|74.6.74.30|74.6.70.44|74.6.67.169|74.6.69.10|74.6.74.35|72.30.177.230|74.6.72.230|74.6.71.161|74.6.66.40|74.6.67.135|74.6.71.118|74.6.72.121|74.6.74.179|74.6.74.98|74.6.71.169|74.6.66.33|74.6.74.205|74.6.70.166|72.30.177.228|74.6.70.95|74.6.68.222|74.6.73.174|74.6.73.232|74.6.72.190|72.30.177.236|74.6.71.180|74.6.69.39|74.6.70.249|74.6.71.188|74.6.75.10|74.6.67.236|74.6.71.153|74.6.73.253|74.6.68.40|72.30.177.167|74.6.70.183|74.6.74.46|74.6.73.96|74.6.70.39|74.6.72.38|74.6.68.166|74.6.70.232|74.6.70.49|72.30.177.229|74.6.67.103|74.6.70.251|74.6.74.37|74.6.69.79|74.6.70.96|74.6.74.232|74.6.67.161|74.6.71.82|74.6.73.163|74.6.71.96|74.6.68.25|74.6.69.43|74.6.71.60|74.6.73.155|74.6.69.27|74.6.71.90|74.6.72.169
And I have another 40+ pending yahoo bots that came over night. At 11:30PM, I had cleared all the pending bots. At 7AM, there were 40+ yahoo pending. Just a little agressive.
Does there come a time when you have all the bots added and there are no more pending bots? LOL
Wishful thinking.
Posted:
Thu May 10, 2007 9:14 amAuthor: tekguru
So anyway of blocking Yahoo bots as I only rely on Google coming in.
Posted:
Mon Jun 25, 2007 8:25 amAuthor: dan0042
need some help have 247 tahoo bots on my forums + cont send pm...saying this
You cannot make another post so soon after your last; please try again in a short while.
Some one help me ..Been like this for 2 days now
Re: Yahoo crawlers - killing our site
Posted:
Mon Jun 25, 2007 11:42 amAuthor: CaNNon
You can use admin > crackertracker > IP & UserAgent Blocker.
Notice " *ia_archiver* " this is the Web crawler used by the Alexa search engine
and they block it by default. Just do yahoo the same way if you wish to block it.
Re: Yahoo crawlers - killing our site
Posted:
Mon Jun 25, 2007 1:20 pmAuthor: .QUACK.Major.Pain
They hit my site really bad for about 3 weeks and then almost stopped for about 6 weeks.
Now it started again and all new IP's. Getting about 50-60 pending yahoo bots a day.
I must be up to about 3000 yahoo ip's added already.
Posted:
Tue Jun 26, 2007 6:55 amAuthor: dan0042
tba with u .im 141.. dont look like goin help u ate me...
Re: Yahoo crawlers - killing our site
Posted:
Tue Jun 26, 2007 3:57 pmAuthor: CaNNon
"CaNNon";p="26830" wrote:You can use admin > crackertracker > IP & UserAgent Blocker.
Notice " *ia_archiver* " this is the Web crawler used by the Alexa search engine
and they block it by default.
Just do yahoo the same way if you wish to block it.
Do that dan0042, just block them for now! when some one has a better idea you can allow them again.