Blocking Some Bad-Bots.

Discussion in 'Linux VPS/Dedicated - cPanel' started by Nneel, Dec 17, 2006.

  1. Nneel

    Nneel New Member

    Helo..KH Team,

    A good SE is always important.
    Now a day there are lots of SE bots/crawler/ripper are working around..
    and there are lots of bot that are very hurmfull for sites..
    they stole bandwidth,increase server load,consume cpu,dont follow rules,spam,attack ..and many things..


    And every server has a limit- load limit/cpu using limit..

    and Hence we dont have unlimited resources so we have to keep them miles away from our site..
    Personally i always try to keep them far away from my site...
    Firstly I have tried using robots.txt ,once i found my robots.txt got large & lager...
    but Google/Google Adsense wont verify robots.txt larger than 5000 char(it burns).And as we are adsense user we had to make shorter our robots.txt

    then i go for .htaccess ..and yes..slowly it also staterd big & bigger :(
    matching stirng/pattern of spider's name also need resources..also a cause of server load + cpu resource.
    And we dont have lots lots of resources..

    Finally Now I need knownhost's help...to block bad-bots..
    Personally for our vps(domain: golpo.net)

    I can ensure all of the bots/spiders name which we want to block..
    using httpd.conf or any other way..

    Looking For knownhost help...

    I am ready to provide bots name..needed to be blocked...

    Waiting for reply.

    Thnx.
     

Share This Page