• Last couple of days with my binkp mailer..

    From Janis Kracht@1:261/38 to All on Thu Aug 19 22:46:50 2010
    Hi There,

    My system has been absolutely tied up by my (not)friend Google-bot as they tried to scan my Internet fidonet nodes (geez :) ). What a pia.. I took care of it as soon as I noticed it though.. Luckily, bbbs has a feature that lets you keep all connections or some, whichever you like, that try to tie up your tcpip nodes.. I have 7 each of binkp, www, telnet, and raw tcpip.. so the idiots at google-bot were trying anyway they could.. I laughed when I saw them try the port on 24555 (my bbbs binkp port).. still a problem because the attempt is made, but once I got their ip address in with the / at the end of the ip address to specify any port they used, that took care of it.

    For those of you that use BBBS, and run into this, this is what I put in inet.bbb:

    [bbbsd]

    !ftp 66.249.65/$
    !www 66.249.65/$
    !tcpip 66.249.65/$
    !binkp 66.249.65/$

    Seemed like it should work fine, and it did <smile>

    Take care,
    Janis

    --- BBBS/LiI v4.01 Flag
    * Origin: Prism bbs (1:261/38)
  • From Janis Kracht@1:261/38 to All on Fri Aug 20 14:29:54 2010
    Well, the following didn't work as I thought it was working.. <grin>

    Later, Google was able to connect.. more coffee was obviously needed:

    For those of you that use BBBS, and run into this, this is what I put in inet.bbb:

    [bbbsd]

    !ftp 66.249.65/$
    !www 66.249.65/$
    !tcpip 66.249.65/$
    !binkp 66.249.65/$


    But THIS works (so far so good <g>):

    [bbbsd]
    !FTP 66.249.65.83/10
    !HTTP 66.249.65.83/10
    !raw 66.249.65.83/10
    !TCPIP 66.249.65.83/10

    Don't know where my brain was last night <g>

    Take care,
    Janis

    --- BBBS/LiI v4.01 Flag
    * Origin: Prism bbs (1:261/38)
  • From Sean Dennis@1:18/200 to Janis Kracht on Fri Aug 20 22:24:43 2010
    Hello, Janis.

    Friday August 20 2010 at 14:29, you wrote to All:

    !FTP 66.249.65.83/10
    !HTTP 66.249.65.83/10
    !raw 66.249.65.83/10
    !TCPIP 66.249.65.83/10

    Good idea; I'll just block the entire block from coming in. My web server is professionally hosted so there's no need for Google to go poking around.

    Later,
    Sean

    ... You're only young once. How long that once lasts is the question.
    --- GoldED/2 3.0.1
    * Origin: Nocturnal State BBS - (423) 926-7999 - bbs.kd5col.info (1:18/200)
  • From Janis Kracht@1:261/38 to Sean Dennis on Sat Aug 21 11:58:24 2010
    Hi Sean,

    !FTP 66.249.65.83/10
    !HTTP 66.249.65.83/10
    !raw 66.249.65.83/10
    !TCPIP 66.249.65.83/10

    Good idea; I'll just block the entire block from coming in. My web server is professionally hosted so there's no need for Google to go poking around.

    Exactly :) Mr. Google-bot was KILLING my already crummy DSL system here - They
    constantly scan my main apache server on port 80, so it's not like they can't get updates. When they started scanning my BBBS ports (ftp 6071, http 8080, binkp 24555, telnet 23) I'd had enough.. people couldn't even connect to pickup
    their mail via the internet since I've only got 7 nodes.. and they didn't take up just "one" node...

    Now thankfully, this is what I'm seeing in the log for just one port, but BBBS is blocking all of BBBS's ports when Google-bot tries:

    100821 11:22 Got FTP from 66.249.65.137:38215, blocked by inet.bbb!
    100821 11:23 Got FTP from 66.249.65.137:43860, blocked by inet.bbb!
    100821 11:23 Got FTP from 66.249.65.137:42786, blocked by inet.bbb!
    100821 11:23 Got FTP from 66.249.65.137:43327, blocked by inet.bbb!
    100821 11:23 Got FTP from 66.249.65.137:39070, blocked by inet.bbb!
    100821 11:23 Got FTP from 66.249.65.137:45389, blocked by inet.bbb!
    100821 11:23 Got FTP from 66.249.65.137:45336, blocked by inet.bbb!
    100821 11:23 Got FTP from 66.249.65.137:46020, blocked by inet.bbb!
    100821 11:23 Got FTP from 66.249.65.137:46428, blocked by inet.bbb!
    100821 11:23 Got FTP from 66.249.65.137:45457, blocked by inet.bbb!
    100821 11:23 Got FTP from 66.249.65.137:43847, blocked by inet.bbb!
    100821 11:23 Got FTP from 66.249.65.137:44818, blocked by inet.bbb!
    100821 11:23 Got FTP from 66.249.65.137:46356, blocked by inet.bbb!
    [yay! <bg> The block then allowed this:]
    100821 11:28 Got raw from 173.52.216.2:3038, spawning node 5.
    100821 11:28 Closing raw at node 5, 0kB in, 15kB out.
    100821 11:28 Got raw from 69.55.228.237:59366, spawning node 6.
    100821 11:28 Closing raw at node 6, 0kB in, 7kB out.
    100821 11:29 Got raw from 216.66.97.101:4954, spawning node 7.

    The above shows them hitting the ftp port on BBBS.. but the dogs were hitting all of the BBBS ports, and constantly as shown above for bbbs's ftp.. I even emailed them first.. no response of course..Robots don't talk much and I guess neither does Googles' admin.

    Who needs it.. Once I saw them hitting all the ports of my bbs, I said, enough!


    I should just kill them period even on port 80 for apache, and I still may do so <grin>. I have free-find indexing my site automatically, and they do it in such a way that it doesn't interfere with anything.. so it's not like google-bot couldn't do it that way if they wanted to!

    Take care,
    Janis

    --- BBBS/LiI v4.01 Flag
    * Origin: Prism bbs (1:261/38)
  • From Sean Dennis@1:18/200 to Janis Kracht on Sun Aug 22 09:20:03 2010
    Hello, Janis.

    Saturday August 21 2010 at 11:58, you wrote to me:

    I should just kill them period even on port 80 for apache, and I still
    may do so <grin>. I have free-find indexing my site automatically,
    and they do it in such a way that it doesn't interfere with anything..
    so it's not like google-bot couldn't do it that way if they wanted to!

    I had to block that Chinese search engine, Baidu, from my site-their bot would hit it nearly every hour...and it was killing my bandwidth.

    Later,
    Sean

    ... It's not whether you win or lose -- it's how you look when you play the game.

    --- GoldED/2 3.0.1
    * Origin: Nocturnal State BBS - (423) 926-7999 - bbs.kd5col.info (1:18/200)
  • From Janis Kracht@1:261/38 to Sean Dennis on Sun Aug 22 11:56:10 2010
    Hi Sean,

    I should just kill them period even on port 80 for apache, and I still
    may do so <grin>. I have free-find indexing my site automatically,
    and they do it in such a way that it doesn't interfere with anything..
    so it's not like google-bot couldn't do it that way if they wanted to!

    I had to block that Chinese search engine, Baidu, from my site-their bot would
    hit it nearly every hour...and it was killing my bandwidth.

    I'm not familiar with that one.. I should do a grep for them to see if they've hit over here as well <grin>

    Hey, last night I had the greatest chat with a user on my system.. he's from Zone 6 (he's in Tokyo) - and he wonders what happened to Z6.. sad :( Well, he is looking to get it going again.. Super I think :)

    Take care,
    Janis

    --- BBBS/LiI v4.01 Flag
    * Origin: Prism bbs (1:261/38)
  • From Jame Clay@1:120/546 to Janis Kracht on Sun Aug 22 08:41:46 2010
    Janis,
    so the idiots at google-bot were trying anyway they could..
    I laughed when I saw them try the port on 24555 (my bbbs binkp port)..

    Are you sure that it was actually googlebot? It only follows http links, which won't lead to that binkp port... (How'd the IP show up in a reverse dns?)

    Also; do you have a robots.txt? I use it to disallow "/bbbs"...




    Jame

    --- BBBS/LiI v4.01 Flag
    * Origin: Rocasa BBS (1:120/546)
  • From Jame Clay@1:120/546 to Sean Dennis on Sun Aug 22 08:46:58 2010
    Sean,

    Hello, Janis.
    Friday August 20 2010 at 14:29, you wrote to All:
    !FTP 66.249.65.83/10
    !HTTP 66.249.65.83/10
    !raw 66.249.65.83/10
    !TCPIP 66.249.65.83/10
    Good idea; I'll just block the entire block from coming in.

    Do keep in mind that the IPs have a tendancy to change...


    My web server is professionally hosted so there's no need
    for Google to go poking around.

    I don't see what the first thing as to do with the second...?




    Jame
    ]

    --- BBBS/LiI v4.01 Flag
    * Origin: Rocasa BBS (1:120/546)
  • From Janis Kracht@1:261/38 to Jame Clay on Mon Aug 23 09:14:26 2010
    Hi Jame,

    Are you sure that it was actually googlebot? It only follows http links,

    I guess since the web version of the bbs is http, it went there on my port 8080.. and from there, there are links on that page for ftp, etc. and yes I'm sure it was google bot. MSN-Bot is another one that likes to run all over my site - I've seen them in my bbbs inet.log as well <grin>.

    which won't lead to that binkp port... (How'd the IP show up in a reverse dns?)

    I'm sure it was Google-bot :)

    janis:~$ host 66.249.71.53
    53.71.249.66.in-addr.arpa domain name pointer crawl-66-249-71-53.googlebot.com.

    That is for sure the google spider :)

    Also; do you have a robots.txt? I use it to disallow "/bbbs"...

    I haven't put that in there, but I will.. Should work, though there's no need right now. Good idea for the bbs, though, thanks :)

    Google called me on the phone quite a while back and their representative asked
    me to allow Google-bot to index my main site that runs apache, with robots.txt - I don't mind that since it doesn't involve the bbs nodes.. this is how I allow them to scan my main site on port 80:

    User-agent: *
    Disallow: /
    User-agent: Googlebot
    Allow: /

    Take care,
    Janis

    --- BBBS/LiI v4.01 Flag
    * Origin: Prism bbs (1:261/38)
  • From Sean Dennis@1:18/200 to Janis Kracht on Mon Aug 23 09:54:14 2010
    Hello, Janis.

    Sunday August 22 2010 at 11:56, you wrote to me:

    I'm not familiar with that one.. I should do a grep for them to see if they've hit over here as well <grin>

    Yeah, it's called BaiduSearch or BaiduBot; I've seen both of them in my logs before.

    Hey, last night I had the greatest chat with a user on my system..
    he's from Zone 6 (he's in Tokyo) - and he wonders what happened to
    Z6.. sad :( Well, he is looking to get it going again.. Super I think
    :)

    That's great! It'd be nice to see Z6 up and running again.

    Later,
    Sean

    ... A good scare is worth more to a man than good advice. - E.E. Howe
    --- GoldED/W32 3.0.1
    * Origin: Nocturnal State BBS - (423) 926-7999 - bbs.kd5col.info (1:18/200)
  • From Sean Dennis@1:18/200 to Jame Clay on Mon Aug 23 09:55:03 2010
    Hello, Jame.

    Sunday August 22 2010 at 08:46, you wrote to me:

    Do keep in mind that the IPs have a tendancy to change...

    Yep and I can ban them also by domain name or my system will look it up if need
    be. I've got three firewalls on my LAN, so if one doesn't catch it, one of the
    others will.

    My web server is professionally hosted so there's no need
    for Google to go poking around.

    I don't see what the first thing as to do with the second...?

    Everything. I don't run a web server here so Google has no need to go probing the other ports on my LAN. If port 80 is closed, their bot should move on.

    Later,
    Sean

    ... For success: try aspiration, inspiration and perspiration.
    --- GoldED/W32 3.0.1
    * Origin: Nocturnal State BBS - (423) 926-7999 - bbs.kd5col.info (1:18/200)
  • From Janis Kracht@1:261/38 to Sean Dennis on Mon Aug 23 12:05:30 2010
    Hi Sean,

    I'm not familiar with that one.. I should do a grep for them to see if
    they've hit over here as well <grin>

    Yeah, it's called BaiduSearch or BaiduBot; I've seen both of them in my logs before.

    Wow Baidubot is surely hitting on BBBS' webserver over here :( Oh well.. off to inet.bbb to get rid of them haha I mean, I really don't mind if they hit the main web server here.. but bbbs is limited to only those 7 nodes.

    Hey, last night I had the greatest chat with a user on my system..
    he's from Zone 6 (he's in Tokyo) - and he wonders what happened to
    Z6.. sad :( Well, he is looking to get it going again.. Super I think
    :)

    That's great! It'd be nice to see Z6 up and running again.

    Yeah really :) He's got some friends who'll be coming in, and if we count the z6 folks that we've got listed in Z3 right now, it should make a good basis for
    reinstating that zone.

    Take care,
    Janis

    --- BBBS/LiI v4.01 Flag
    * Origin: Prism bbs (1:261/38)
  • From mark lewis@1:3634/12 to Janis Kracht on Wed Sep 29 19:57:57 2010

    I'm not familiar with that one.. I should do a grep for them to see if
    they've hit over here as well <grin>

    Yeah, it's called BaiduSearch or BaiduBot; I've seen both of them in
    my logs before.

    Wow Baidubot is surely hitting on BBBS' webserver over here :( Oh
    well.. off to inet.bbb to get rid of them haha

    you could just deny them access to those links in BBBS that lead to places like
    the messages, files and games... they are robots.txt friendly...

    I mean, I really don't mind if they hit the main web server here..
    but bbbs is limited to only those 7 nodes.

    i assume by that you mean that it is similar to apache in that there's seven HTTP handlers that are allowed to run and you don't allow any more than that??

    Hey, last night I had the greatest chat with a user on my system..
    he's from Zone 6 (he's in Tokyo) - and he wonders what happened to
    Z6.. sad :( Well, he is looking to get it going again.. Super I think
    :)

    That's great! It'd be nice to see Z6 up and running again.

    Yeah really :) He's got some friends who'll be coming in, and if
    we count the z6 folks that we've got listed in Z3 right now, it
    should make a good basis for reinstating that zone.

    that might be a good thing... especially considering the reasons why Z6 went away last time it was operational...

    )\/(ark


    * Origin: (1:3634/12)
  • From mark lewis@1:3634/12 to Janis Kracht on Wed Sep 29 20:02:40 2010

    Wow Baidubot is surely hitting on BBBS' webserver over here :( Oh
    well.. off to inet.bbb to get rid of them haha

    you could just deny them access to those links in BBBS that lead to
    places like the messages, files and games... they are robots.txt friendly...

    just to follow up and clarify this... an example is my gallery site... i may not want the spiders trapsing thru the exhibits and looking at the actual photos available... only the microthumbs and thumbs, sure so i just set up a disallow for the exhibits area and let them wander over the others... then, if i see them in the exhibits areas, then i know that they're not following robots.txt and i can then contact their admins with my complaint or just block them at the perimeter and deny them access to everything... depending on my mood, of course ;)

    )\/(ark


    * Origin: (1:3634/12)
  • From Janis Kracht@1:261/38 to mark lewis on Thu Sep 30 09:46:10 2010
    Hi Mark,

    Yeah, it's called BaiduSearch or BaiduBot; I've seen both of them in
    my logs before.

    Wow Baidubot is surely hitting on BBBS' webserver over here :( Oh
    well.. off to inet.bbb to get rid of them haha

    you could just deny them access to those links in BBBS that lead to places lik
    the messages, files and games... they are robots.txt friendly...

    Well, that's not exactly the problem.. the content is so not 'private' in other
    words.. but when they tie up my bbbs web/telnet/binkp nodes, 'real fido people'
    can't connect.. that is a drag <g>

    I mean, I really don't mind if they hit the main web server here..
    but bbbs is limited to only those 7 nodes.

    i assume by that you mean that it is similar to apache in that there's seven
    HTTP handlers that are allowed to run and you don't allow any more than that??

    Yes, that's right, but it's not that I don't allow any more than 7, it's because the way bbbs works is you register the number of nodes you want for the
    bbbs daemons.. so I have 7 nodes registered... that's 1 phone-modem node, 6 http instances, 6 telnet nodes, 6 binkp nodes, etc. I also run BinkD stand-alone mailer on the standard binkp port to pick up more binkp connections
    since I know the bbbs binkp nodes get a bit busy with the number of downlinks here <g>. BBBS's binkp daemon runs on port 24555.

    I guess what really bugged me was that these spiders were hitting all the ports
    here.. not just 80 :( I mean, what could a spider get out of attemping repeated connections to my binkp port on 24555?? <bg>

    Great thing this week: when I contacted Kim Heino (he's the author of bbbs) about my dead motherboard and all that, etc., (he uses ftp to connect so we needed to set that up on the main ftp server), he sent me a beta 64 bit version
    of BBBS.. Really cool :) It's running really well.

    That's great! It'd be nice to see Z6 up and running again.

    Yeah really :) He's got some friends who'll be coming in, and if
    we count the z6 folks that we've got listed in Z3 right now, it
    should make a good basis for reinstating that zone.

    that might be a good thing... especially considering the reasons why Z6 went away last time it was operational...

    Understand.. these kinds of things take time, but we're hoping.

    Take care,
    Janis

    --- BBBS/Li6 v4.10 Dada-1
    * Origin: Prism bbs (1:261/38)
  • From Janis Kracht@1:261/38 to mark lewis on Thu Sep 30 09:53:12 2010
    Hi Mark,

    just to follow up and clarify this... an example is my gallery site... i may not want the spiders trapsing thru the exhibits and looking at the actual photos available... only the microthumbs and thumbs, sure so i just set up a

    Sure, because they are all on the web, and you want some control there.. I understand that :)

    disallow for the exhibits area and let them wander over the others... then, if
    i see them in the exhibits areas, then i know that they're not following
    robots.txt and i can then contact their admins with my complaint or just block
    them at the perimeter and deny them access to everything... depending on my mood, of course ;)

    <laugh>.. now you sound like me :) :) But seriously, that week that they were hitting BBBS's ports was kind of insane to me <g>

    Take care,
    Janis

    --- BBBS/Li6 v4.10 Dada-1
    * Origin: Prism bbs (1:261/38)
  • From mark lewis@1:3634/12 to Janis Kracht on Thu Sep 30 18:18:15 2010

    Wow Baidubot is surely hitting on BBBS' webserver over here :( Oh
    well.. off to inet.bbb to get rid of them haha

    you could just deny them access to those links in BBBS that lead to
    places like the messages, files and games... they are robots.txt friendly...

    Well, that's not exactly the problem.. the content is so not
    'private' in other words.. but when they tie up my bbbs
    web/telnet/binkp nodes, 'real fido people' can't connect.. that is
    a drag <g>

    yeah, i can see that with a server that has limited handlers allowed... but the
    idea was to limit what they have access to so they're in and out as fast as possible ;)

    I mean, I really don't mind if they hit the main web server here..
    but bbbs is limited to only those 7 nodes.

    i assume by that you mean that it is similar to apache in that
    there's seven HTTP handlers that are allowed to run and you don't
    allow any more than that??

    Yes, that's right, but it's not that I don't allow any more than 7,
    it's because the way bbbs works is you register the number of nodes
    you want for the bbbs daemons.. so I have 7 nodes registered...
    that's 1 phone-modem node, 6 http instances, 6 telnet nodes, 6
    binkp nodes, etc.

    yup... pretty much the same idea... close enough for the analogy ;)

    I also run BinkD stand-alone mailer on the standard binkp port to
    pick up more binkp connections since I know the bbbs binkp nodes
    get a bit busy with the number of downlinks here <g>. BBBS's
    binkp daemon runs on port 24555.

    i'd really hate it if the bots started hitting the telnet and binkd stuff... i think that google does the http and ftp stuff now... i know i've seen references to it in my ftp logs somewhere...

    I guess what really bugged me was that these spiders were hitting
    all the ports here.. not just 80 :( I mean, what could a spider
    get out of attemping repeated connections to my binkp port on
    24555?? <bg>

    trying to connect to a web server that it thinks is running there... if it is doing that, lodge a complaint and/or block it at the perimeter and don't let it
    in at all ;) depending on the methods, it could be blocked at the perimeter for
    all ports except for 80 :P

    Great thing this week: when I contacted Kim Heino (he's the
    author of bbbs) about my dead motherboard and all that, etc., (he
    uses ftp to connect so we needed to set that up on the main ftp
    server), he sent me a beta 64 bit version of BBBS.. Really cool :)
    It's running really well.

    i saw reference to you running 64bit earlier... that's a GoodThing<tm> ;)

    That's great! It'd be nice to see Z6 up and running again.

    Yeah really :) He's got some friends who'll be coming in, and if
    we count the z6 folks that we've got listed in Z3 right now, it
    should make a good basis for reinstating that zone.

    that might be a good thing... especially considering the reasons why
    Z6 went away last time it was operational...

    Understand.. these kinds of things take time, but we're hoping.

    word up! :P

    )\/(ark


    * Origin: (1:3634/12)
  • From Janis Kracht@1:261/38 to mark lewis on Wed Oct 13 16:10:58 2010
    Hi Mark,

    'private' in other words.. but when they tie up my bbbs
    web/telnet/binkp nodes, 'real fido people' can't connect.. that is
    a drag <g>

    yeah, i can see that with a server that has limited handlers allowed... but th
    idea was to limit what they have access to so they're in and out as fast as possible ;)

    I think bbbs makes it more than quick.. they connect, they get immediately rejected <g>

    I mean, I really don't mind if they hit the main web server here..
    but bbbs is limited to only those 7 nodes.

    i assume by that you mean that it is similar to apache in that
    there's seven HTTP handlers that are allowed to run and you don't
    allow any more than that??

    Yes, that's right, but it's not that I don't allow any more than 7,
    it's because the way bbbs works is you register the number of nodes
    you want for the bbbs daemons.. so I have 7 nodes registered...
    that's 1 phone-modem node, 6 http instances, 6 telnet nodes, 6
    binkp nodes, etc.

    yup... pretty much the same idea... close enough for the analogy ;)

    Yep, for sure :)

    pick up more binkp connections since I know the bbbs binkp nodes
    get a bit busy with the number of downlinks here <g>. BBBS's
    binkp daemon runs on port 24555.

    i'd really hate it if the bots started hitting the telnet and binkd stuff... i
    think that google does the http and ftp stuff now... i know i've seen references to it in my ftp logs somewhere...

    for the main ftp server, again it's no problem since I don't allow anonymous ftp on my vsftp server..but on bbb's ftp server anonymous is allowed.. not working well for people from what I understand but.. have to look into that :)


    I guess what really bugged me was that these spiders were hitting
    all the ports here.. not just 80 :( I mean, what could a spider
    get out of attemping repeated connections to my binkp port on
    24555?? <bg>

    trying to connect to a web server that it thinks is running there... if it is
    doing that, lodge a complaint and/or block it at the perimeter and don't let i >in at all ;) depending on the methods, it could be blocked at the perimeter fo
    all ports except for 80 :P

    ah one of these days.. no time for it right now.. just trying to catch up <bg>

    Great thing this week: when I contacted Kim Heino (he's the
    author of bbbs) about my dead motherboard and all that, etc., (he
    uses ftp to connect so we needed to set that up on the main ftp
    server), he sent me a beta 64 bit version of BBBS.. Really cool :)
    It's running really well.

    i saw reference to you running 64bit earlier... that's a GoodThing<tm> ;)

    It's a really nice system.. when the conexant internmodem started locking up the system, I decided to back up to the release 32bit version of bbbs.. in case
    it was causing problems as well.. for now I'll just let it sit as is <g>.. one of these days I'll reinstall the 64bit version of bbbs..

    That's great! It'd be nice to see Z6 up and running again.

    Yeah really :) He's got some friends who'll be coming in, and if
    we count the z6 folks that we've got listed in Z3 right now, it
    should make a good basis for reinstating that zone.

    that might be a good thing... especially considering the reasons why
    Z6 went away last time it was operational...

    Understand.. these kinds of things take time, but we're hoping.

    word up! :P

    I'll say :)

    Take care,
    Janis

    --- BBBS/LiI v4.01 Flag
    * Origin: Prism bbs (1:261/38)
  • From Benny Pedersen@2:237/53 to Janis Kracht on Sat Oct 16 15:05:28 2010
    Hello Janis!

    21 Aug 2010 11:58, Janis Kracht wrote to Sean Dennis:

    with anything.. so it's not like google-bot couldn't do it that way
    if they wanted to!

    time to get a mirror :)

    but:

    ----- .htaccess begins -----
    RewriteEngine on

    # Allow only GET and POST verbs
    RewriteCond %{REQUEST_METHOD} !^(GET|POST)$ [NC,OR]

    # Ban Typical Vulnerability Scanners and others
    # Kick out Script Kiddies
    RewriteCond %{HTTP_USER_AGENT} ^(java|curl|wget).* [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*(libwww-perl|curl|wget|python|nikto|wkito|pikto|scan|acunetix).* [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*(winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner). * [NC,OR]

    # Ban Search Engines, Crawlers to your administrative panel
    # No reasons to access from bots
    # Ultimately Better than the useless robots.txt
    # Did google respect robots.txt?
    # Try google: intitle:phpMyAdmin intext:"Welcome to phpMyAdmin *.*.*" intext:"Log in" -wiki -forum -forums -questions intext:"Cookies must be enabled"
    RewriteCond %{HTTP_USER_AGENT} ^.*(AdsBot-Google|ia_archiver|Scooter|Ask.Jeeves|Baiduspider|Exabot|FAST.Enter prise.Crawler|FAST-WebCrawler|www\.neomo\.de|Gigabot|Mediapartners-Google|Goog le.Desktop|Feedfetcher-Google|Googlebot|heise-IT-Markt-Crawler|heritrix|ibm.co m\cs/crawler|ICCrawler|ichiro|MJ12bot|MetagerBot|msnbot-NewsBlogs|msnbot|msnbo t-media|NG-Search|lucene.apache.org|NutchCVS|OmniExplorer_Bot|online.link.vali dator|psbot0|Seekbot|Sensis.Web.Crawler|SEO.search.Crawler|Seoma.\[SEO.Crawler \]|SEOsearch|Snappy|www.urltrends.com|www.tkl.iis.u-tokyo.ac.jp/~crawler|Synoo Bot|crawleradmin.t-info@telekom.de|TurnitinBot|voyager|W3.SiteSearch.Crawler|W 3C-checklink|W3C_Validator|www.WISEnutbot.com|yacybot|Yahoo-MMCrawler|Yahoo\!. DE.Slurp|Yahoo\!.Slurp|YahooSeeker).* [NC]
    RewriteRule .* - [F]

    ----- .htaccess ends -----

    if bbbs http server cant do this make it as proxy in apache !


    Regards Benny

    ... there can only be one way of life, and it works :)

    --- Msged/LNX 6.2.0 (Linux/2.6.34-gentoo-r6 (i686))
    * Origin: www.region23.dk/ www.fido.dk/ <128000K ram after bootup (2:237/53)
  • From Janis Kracht@1:261/38 to Benny Pedersen on Wed Oct 20 14:27:34 2010
    Hi Benny,

    with anything.. so it's not like google-bot couldn't do it that way
    if they wanted to!

    time to get a mirror :)

    Hehe.. Yep, very good idea as we discussed. The more mirrors the better.. That
    way my system won't get so boggled down :)

    but:

    ----- .htaccess begins -----
    RewriteEngine on

    # Allow only GET and POST verbs
    RewriteCond %{REQUEST_METHOD} !^(GET|POST)$ [NC,OR]

    # Ban Typical Vulnerability Scanners and others
    # Kick out Script Kiddies
    RewriteCond %{HTTP_USER_AGENT} ^(java|curl|wget).* [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*(libwww-perl|curl|wget|python|nikto|wkito|pi
    to|scan|acunetix).* [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*(winhttp|HTTrack|clshttp|archiver|loader|ema
    l|harvest|extract|grab|miner). * [NC,OR]

    # Ban Search Engines, Crawlers to your administrative panel
    # No reasons to access from bots
    # Ultimately Better than the useless robots.txt
    # Did google respect robots.txt?
    # Try google: intitle:phpMyAdmin intext:"Welcome to phpMyAdmin *.*.*" intext:"Log in" -wiki -forum -forums -questions intext:"Cookies must be enabled"
    RewriteCond %{HTTP_USER_AGENT} ^.*(AdsBot-Google|ia_archiver|Scooter|Ask.Jeeve >|Baiduspider|Exabot|FAST.Enter prise.Crawler|FAST-WebCrawler|www\.neomo\.de|Gi >abot|Mediapartners-Google|Goog le.Desktop|Feedfetcher-Google|Googlebot|heise-I >-Markt-Crawler|heritrix|ibm.co m\cs/crawler|ICCrawler|ichiro|MJ12bot|MetagerBo >|msnbot-NewsBlogs|msnbot|msnbo t-media|NG-Search|lucene.apache.org|NutchCVS|Om >iExplorer_Bot|online.link.vali dator|psbot0|Seekbot|Sensis.Web.Crawler|SEO.sea >ch.Crawler|Seoma.\[SEO.Crawler \]|SEOsearch|Snappy|www.urltrends.com|www.tkl.i >s.u-tokyo.ac.jp/~crawler|Synoo Bot|crawleradmin.t-info@telekom.de|TurnitinBot| >oyager|W3.SiteSearch.Crawler|W 3C-checklink|W3C_Validator|www.WISEnutbot.com|y
    cybot|Yahoo-MMCrawler|Yahoo\!. DE.Slurp|Yahoo\!.Slurp|YahooSeeker).* [NC] RewriteRule .* - [F]

    ----- .htaccess ends -----

    if bbbs http server cant do this make it as proxy in apache !

    Sure, it can .. :) I have always used mod-rewrite to ban filehogs from places like those where I find companies that are selling the files <g>.

    I like the way you used it though :) Thanks for a good idea.

    Take care,
    Janis

    --- BBBS/LiI v4.01 Flag
    * Origin: Prism bbs (1:261/38)