650 visitor Bots

Discussion in 'Security and Legal' started by Rasty, Feb 28, 2014.

  1. Rasty

    Rasty Regular Member

    Joined:
    Jan 31, 2014
    Messages:
    23
    Likes Received:
    5
    Is 650 visitor bots preventable? What can or should be done as it's slowing the site down.
     
  2. AWS

    AWS Administrator

    Joined:
    Feb 1, 2010
    Messages:
    1,612
    Likes Received:
    695
    Location:
    Joliet, IL U.S.A.
    First Name:
    Bob
    What is it? I never heard of it before.
     
  3. we_are_borg

    we_are_borg Regular Member

    Joined:
    May 8, 2013
    Messages:
    305
    Likes Received:
    168
    Location:
    Netherlands
    First Name:
    Jeroen
    Some search engines dont care how many bots they send to your site that is causing the slowdown of your site. If you know who is sending what you can block them just do a google search and you find many ways to block.
     
    Rasty likes this.
  4. Rasty

    Rasty Regular Member

    Joined:
    Jan 31, 2014
    Messages:
    23
    Likes Received:
    5
    Majestic 12 was the cause.
     
  5. GTB

    GTB Regular Member

    Joined:
    Jun 30, 2009
    Messages:
    1,792
    Likes Received:
    270
    That's a lot for Majestic bot, it's not one you'd usually link with coming in very large numbers
     
  6. Rasty

    Rasty Regular Member

    Joined:
    Jan 31, 2014
    Messages:
    23
    Likes Received:
    5
    I looked at the last 1000 visitors and it was everywhere. I didn't see much activity from ips that I see the regular users using. I don't understand if Majestic is legit or not. I see conflicting information when I looked it up.
     
  7. GTB

    GTB Regular Member

    Joined:
    Jun 30, 2009
    Messages:
    1,792
    Likes Received:
    270
    Majestic is a bot or crawler. Not sure which of the two but have had it listed on my other forums before in detected bots that show up.

    http://www.majestic12.co.uk/projects/dsearch/mj12bot.php

    Actually it shows as MJ12bot, I have it listed on mine. And they say you can can block it in robots.txt using.

    User-agent: MJ12bot
    Disallow: /

    You should add a delay in your robots.txt file anyway really to try and limit all best you can, even if not all follow it.

    User-agent: *
    Crawl-Delay: 10

    That will limit all bots that follow it to not hit your forum in huge numbers all at once, such as Yahoo Slurp!
     
    Last edited: Mar 1, 2014
    Rasty likes this.

Share This Page