What's in your robots.txt?

Discussion in 'SEO, Traffic and Revenue' started by Nick, Jun 29, 2009.

  1. Nick

    Nick Regular Member

    Joined:
    Jul 27, 2008
    Messages:
    7,444
    Likes Received:
    219
    Post what forum script you use (vBulletin, IP.Board, MyBB, phpBB, etc.) and what your robots.txt file consists of (to block search engines from crawling unnecessary pages).
     
  2. Michael

    Michael Regular Member

    Joined:
    Jan 18, 2004
    Messages:
    166
    Likes Received:
    35
    vBulletin of course :P

    Code:
    User-agent: *
    Disallow: /ajax.php
    Disallow: /attachment.php
    Disallow: /calendar.php
    Disallow: /cron.php
    Disallow: /editpost.php
    Disallow: /global.php
    Disallow: /image.php
    Disallow: /inlinemod.php
    Disallow: /joinrequests.php
    Disallow: /login.php
    Disallow: /memberlist.php
    Disallow: /misc.php
    Disallow: /moderator.php
    Disallow: /newattachment.php
    Disallow: /newreply.php
    Disallow: /newthread.php
    Disallow: /online.php
    Disallow: /poll.php
    Disallow: /postings.php
    Disallow: /printthread.php
    Disallow: /private.php
    Disallow: /profile.php
    Disallow: /register.php
    Disallow: /report.php
    Disallow: /reputation.php
    Disallow: /search.php
    Disallow: /sendmessage.php
    Disallow: /showgroups.php
    Disallow: /subscription.php
    Disallow: /threadrate.php
    Disallow: /usercp.php
    Disallow: /usernote.php
    Disallow: /login.php?do=lostpw
    Disallow: /view.php?pg=advertise
    Disallow: /home.php
    Disallow: /arcade/
    Disallow: /arcade.php
    Disallow: /UTF/
    Disallow: /sexes/
    Disallow: /ads/
    Disallow: /cgi-bin/
    Disallow: /images/
    Disallow: /register/
    
    That is ours :D
     
  3. Bundy

    Bundy Admin Talk Staff

    Joined:
    Apr 19, 2005
    Messages:
    842
    Likes Received:
    36
    Location:
    Boston
    Did I already do this here...or am I going crazy?
     
  4. Nick

    Nick Regular Member

    Joined:
    Jul 27, 2008
    Messages:
    7,444
    Likes Received:
    219
    I think it was in another thread for something else. :lol:
     
  5. FullMetalBabe

    FullMetalBabe Zealot

    Joined:
    May 30, 2009
    Messages:
    2,912
    Likes Received:
    339
    For MyBB

     
  6. Dunhamzzz

    Dunhamzzz Newcomer

    Joined:
    Jun 14, 2009
    Messages:
    13
    Likes Received:
    0
    First Name:
    Matthew
    You guys should have a sitemap entry in there too.
     
  7. tryfuhl

    tryfuhl Champion

    Joined:
    Jun 27, 2009
    Messages:
    391
    Likes Received:
    14
    First Name:
    Shawn
    to disallow?
     
  8. Nick

    Nick Regular Member

    Joined:
    Jul 27, 2008
    Messages:
    7,444
    Likes Received:
    219
    No, to point out where exactly your sitemap is located. Look up "sitemap AutoDiscovery". ;)

    Add Sitemap To robots.txt For Autodiscovery
     
  9. Soliloquy

    Soliloquy Regular Member

    Joined:
    Jun 3, 2009
    Messages:
    2,402
    Likes Received:
    66
    Location:
    New York City
    Can you add more than one sitemap through the robots.txt? (And have it be recognized, that is.)
     
  10. Dan

    Dan Future Proof

    Joined:
    Jul 16, 2009
    Messages:
    368
    Likes Received:
    84
    Location:
    Staffordshire
    First Name:
    Dan

Share This Page