Bots Management - Could it be that this Mod blocks bots?

[jolin]

Could it be that the Bots Management Mod is blocking bots from crawling my site?

What does it mean when a bot is in the "Pending Bots" list? Does it mean this mod is blocking this Bot until I add it to the list or it just means that it does not track this bot or a particular new IP found for this bot?


Can I disable this Mod alltogether?

[Frost]

That mod is for showing you what all bots are crawling your site. As for the pending bots, that is just a feature that allows you to tell the mod which are bots and which are your members or visitors.

The mod does not prevent any bots from crawling/indexing your site at all. The only thing that is for is to show you the statistics in the upper section of how many times they have crawled.

The mod uses the ip/range and user agent to determine what bot is what. The bots (and regular visitors) shown in the pending area are just the ones that will not be shown in the upper panel until you add/delete them.

Suggestions: Make sure that when you see the ip/user agent of the "bot" that you know the difference. All spiders/bots will usually have something in the user agent that tells you if its a bot.

------------------------------------------------------------------------------------

Examples-

Bot = Mozilla/5.0 (compatible; Googlebot/2.1; +http]http://help.yahoo.com/help/us/ysearch/slurp[/url])


User/Visitor = Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv]Note// You should simply delete the ones that you find are just normal users. If you are unsure copy and paste the user agent into google.com and most of the time you will find out. You can also look up the ip if you are unsure. Try http://www.arin.net/whois/

Hope this helps, Frost[/b]

[jolin]

Thanx, I thought that was the case but were concerned as I'm not getting indexed as I should although I have done everything to make the board crawlable.

[Frost]

You're welcome <img>

And you'll find that bots have a much harder time indexing phpbb/forum sites because of the overwhelming content. Still, a little indexing is better than no indexing. lol

Frost