Helo..KH Team,
A good SE is always important.
Now a day there are lots of SE bots/crawler/ripper are working around..
and there are lots of bot that are very hurmfull for sites..
they stole bandwidth,increase server load,consume cpu,dont follow rules,spam,attack ..and many things..
And every server has a limit- load limit/cpu using limit..
and Hence we dont have unlimited resources so we have to keep them miles away from our site..
Personally i always try to keep them far away from my site...
Firstly I have tried using robots.txt ,once i found my robots.txt got large & lager...
but Google/Google Adsense wont verify robots.txt larger than 5000 char(it burns).And as we are adsense user we had to make shorter our robots.txt
then i go for .htaccess ..and yes..slowly it also staterd big & bigger
matching stirng/pattern of spider's name also need resources..also a cause of server load + cpu resource.
And we dont have lots lots of resources..
Finally Now I need knownhost's help...to block bad-bots..
Personally for our vps(domain: golpo.net)
I can ensure all of the bots/spiders name which we want to block..
using httpd.conf or any other way..
Looking For knownhost help...
I am ready to provide bots name..needed to be blocked...
Waiting for reply.
Thnx.
A good SE is always important.
Now a day there are lots of SE bots/crawler/ripper are working around..
and there are lots of bot that are very hurmfull for sites..
they stole bandwidth,increase server load,consume cpu,dont follow rules,spam,attack ..and many things..
And every server has a limit- load limit/cpu using limit..
and Hence we dont have unlimited resources so we have to keep them miles away from our site..
Personally i always try to keep them far away from my site...
Firstly I have tried using robots.txt ,once i found my robots.txt got large & lager...
but Google/Google Adsense wont verify robots.txt larger than 5000 char(it burns).And as we are adsense user we had to make shorter our robots.txt
then i go for .htaccess ..and yes..slowly it also staterd big & bigger
matching stirng/pattern of spider's name also need resources..also a cause of server load + cpu resource.
And we dont have lots lots of resources..
Finally Now I need knownhost's help...to block bad-bots..
Personally for our vps(domain: golpo.net)
I can ensure all of the bots/spiders name which we want to block..
using httpd.conf or any other way..
Looking For knownhost help...
I am ready to provide bots name..needed to be blocked...
Waiting for reply.
Thnx.