Robots.txt

Discussion in 'SEO, Traffic and Revenue' started by Nick, Jun 4, 2009.

  1. Nick

    Nick Regular Member

    Joined:
    Jul 27, 2008
    Messages:
    7,444
    Likes Received:
    219
    What areas of your site (if any) do you 'ask' the search engine spiders to not crawl, via a robots.txt file?
     
    Nick, Jun 4, 2009
    #1
  2. theboss

    theboss Newcomer

    Joined:
    Jun 14, 2009
    Messages:
    10
    Likes Received:
    0
    nope i want it to find all :P
     
    theboss, Jun 14, 2009
    #2
  3. Peggy

    Peggy Regular Member

    Joined:
    May 18, 2009
    Messages:
    1,641
    Likes Received:
    58
    no you don't, really. Most members don't want their profile info listed in the search engines ;)

    You want the spidies to concentrate on content, not your registration page, or usercp page, etc.

    This is what my robots.txt file looks like -

     
    Peggy, Jun 14, 2009
    #3
    2 people like this.
  4. Demo

    Demo Regular Member

    Joined:
    Jun 9, 2009
    Messages:
    172
    Likes Received:
    6
    Location:
    Europe
    First Name:
    Demo
    Mostly vbulletin files, which can create double content or aren't content-related like ajax.php
     
    Demo, Jun 14, 2009
    #4
  5. Bundy

    Bundy Admin Talk Staff

    Joined:
    Apr 19, 2005
    Messages:
    842
    Likes Received:
    36
    Location:
    Boston
    Yeah you want a lot blocked off. Not that many bot really pay attention to the text anymore :/

    You also never want to mention your admincp or modcp areas or files in your robots.txt. It's a big no no

     
    Bundy, Jun 14, 2009
    #5
  6. Michael Biddle

    Michael Biddle vBSEO Mentor

    Joined:
    Jun 6, 2009
    Messages:
    11
    Likes Received:
    6
    First Name:
    Michael
    Just a FYI, do not copy Kevin's above. It looks like he is using vBSEO without an extension (.html). If you use his, all of your content would be blocked.
     
    Michael Biddle, Jun 14, 2009
    #6
    2 people like this.
  7. Bundy

    Bundy Admin Talk Staff

    Joined:
    Apr 19, 2005
    Messages:
    842
    Likes Received:
    36
    Location:
    Boston

    Yes! Thank you. hahaha Every single one of your files would be blocked ;)

    I use no extensions at all. I actually use briansol's setup.

    I probably should have mentioned that.
     
    Bundy, Jun 14, 2009
    #7
  8. doodles

    doodles Adept

    Joined:
    May 28, 2009
    Messages:
    114
    Likes Received:
    2
    First Name:
    Lisa
    Mine

     
    doodles, Jun 14, 2009
    #8
  9. Nick

    Nick Regular Member

    Joined:
    Jul 27, 2008
    Messages:
    7,444
    Likes Received:
    219
    We don't use extensions here (unless I forgot to change some of them...), so I'm assuming it's safe to use your robots.txt?
     
    Nick, Jun 14, 2009
    #9
  10. Bundy

    Bundy Admin Talk Staff

    Joined:
    Apr 19, 2005
    Messages:
    842
    Likes Received:
    36
    Location:
    Boston
    Yeah, it looks like your setup is pretty much the same as mine. Except you use 'Keywords' in your url's and I use all numbers for the most part. But that wont matter at all ;)
     
    Bundy, Jun 14, 2009
    #10
    2 people like this.
  11. Michael Biddle

    Michael Biddle vBSEO Mentor

    Joined:
    Jun 6, 2009
    Messages:
    11
    Likes Received:
    6
    First Name:
    Michael
    Looking at your URLs you can use his as well.
     
    Michael Biddle, Jun 14, 2009
    #11
    2 people like this.

Share This Page