Using robots.txt - Great for SEO

Discussion in 'SEO, Traffic and Revenue' started by Kaiser, Jun 10, 2011.

  1. Kaiser

    Kaiser Regular Member

    Joined:
    Nov 15, 2010
    Messages:
    6,744
    Likes Received:
    1,132
    Do you use a robots.txt file on your website?

    Basically what it does is it disallows bots from visiting certain places that you put in the robots.txt file, if you use Google's Webmaster tools you can see the places where you get crawl errors, and you can put those places in your robots.txt to prevent Google from crawling it again. Its great for better SEO results.

    Here is AdminBB's robots.txt http://admin-talk.com/robots.txt
     
  2. SpacewardAsh

    SpacewardAsh Lurking From Space

    Joined:
    Jan 2, 2011
    Messages:
    211
    Likes Received:
    683
    Location:
    Falmouth, Cornwall, UK
    First Name:
    Ashley
    Yes, I use robots.txt or I did, I must have mis-placed it when the new era of PH arrived so I'll have to correct that later :oops:

    EDIT: Fixed http://plutohosting.net/robots.txt :cool: please tell me what it's like as you know what my SEO knowledge is like :(
     
  3. Kaiser

    Kaiser Regular Member

    Joined:
    Nov 15, 2010
    Messages:
    6,744
    Likes Received:
    1,132
    Well yours robots.txt is basically empty, just has the sitemap and that's about it. Do you use Google Webmaster tools? If you do you should check for crawl errors and puts those places in your robots.txt file. If you dont use Webmaster Tools I suggest you do :thumbsup:
     
  4. SpacewardAsh

    SpacewardAsh Lurking From Space

    Joined:
    Jan 2, 2011
    Messages:
    211
    Likes Received:
    683
    Location:
    Falmouth, Cornwall, UK
    First Name:
    Ashley
    I do but sometimes I don't have enough time to fully look into things :(
     
    Kaiser likes this.
  5. Kaiser

    Kaiser Regular Member

    Joined:
    Nov 15, 2010
    Messages:
    6,744
    Likes Received:
    1,132
    Well if you dont get too many crawl errors than its fine. But if you have alot I suggest you look into it as it might cause a problem with SEO.
     
  6. SpacewardAsh

    SpacewardAsh Lurking From Space

    Joined:
    Jan 2, 2011
    Messages:
    211
    Likes Received:
    683
    Location:
    Falmouth, Cornwall, UK
    First Name:
    Ashley
    Only Issue I seem to have is 404 errors, most of which should be reasonably easy to fix :D
     
    Kaiser likes this.
  7. Kaiser

    Kaiser Regular Member

    Joined:
    Nov 15, 2010
    Messages:
    6,744
    Likes Received:
    1,132
    Yea thats not a big problem. And you might also want to dissalow your private staff forum in the robots.txt so that google wont visit it. Simply put something like this:
    Disallow: /staffboarddirectory/
     
  8. SpacewardAsh

    SpacewardAsh Lurking From Space

    Joined:
    Jan 2, 2011
    Messages:
    211
    Likes Received:
    683
    Location:
    Falmouth, Cornwall, UK
    First Name:
    Ashley
    I've got nothing to disallow as far as I know, as private items are hosted elsewhere at the moment
     

Share This Page