GoogleBot

Discussion in 'Linux VPS/Dedicated - cPanel' started by csuihko, Feb 24, 2009.

  1. csuihko

    csuihko New Member

    Is it normal get hit hundred of times a day - is way cut down hits?
     
  2. geetee

    geetee New Member

    You can try using a robots.txt file in the home dir of the site. This can be used to either disallow bots from certain parts of the site (use Disallow) or to slow them down when they crawl (use Crawl-delay and/or Request-rate).

    You can also set specific times of day to be crawled with Visit-time.

    Not all bots respect robots.txt while others only obey certain comands in it - but it's worth a shot if you're being hit by one of the bigger search engine bots, they are usually quite well behaved.

    Info:
    http://www.webresourcesdepot.com/how-to-use-robotstxt-file/

    Handy robots.txt file generator:
    http://www.mcanerin.com/en/search-engine/robots-txt.asp
     

Share This Page