Bots & Crawlers???
Posted: Mon Oct 02, 2006 6:53 pmActually I have two questionsà¢Ãƒ ¢Ã¢â‚¬Å¡Ã‚ ¬Ãƒâ€šÃ‚ ¦.
First:
Has anyone either moded the bots mod, or found a better way of inserting the bots.
When I started my site it was easy to keep up, but then after a while I was overwhelmed with Google, Inktomi, and Yahoo Bots, (Yahoo the lesser of the three).
I tried wildcards, but still have too many inserts to ad any more manually.
I have hundreds, and hundreds of bots to add.
Can a select all, add all be adapted???
Even if it would be a couple hour process like rebuilding the search, it would definetly be worth it.
My second question is:
I use to have a script from scriptwiz on my sites that indexed all my public forums and made them into fake .html docs with topic urls.
I m guessing the reason that I can t get it to work correctly anymore is because of the added security measures since the hacking episodes.
Does anyone know a good script for this, or something I can change to get the scriptwiz version to work without creating a security hole???
First:
Has anyone either moded the bots mod, or found a better way of inserting the bots.
When I started my site it was easy to keep up, but then after a while I was overwhelmed with Google, Inktomi, and Yahoo Bots, (Yahoo the lesser of the three).
I tried wildcards, but still have too many inserts to ad any more manually.
I have hundreds, and hundreds of bots to add.
Can a select all, add all be adapted???
Even if it would be a couple hour process like rebuilding the search, it would definetly be worth it.
My second question is:
I use to have a script from scriptwiz on my sites that indexed all my public forums and made them into fake .html docs with topic urls.
I m guessing the reason that I can t get it to work correctly anymore is because of the added security measures since the hacking episodes.
Does anyone know a good script for this, or something I can change to get the scriptwiz version to work without creating a security hole???