• Message index statistics

    From g00r00@1:129/215 to Dan Richter on Sat Jun 24 10:47:28 2017
    Hey Dan,

    Ever since you mentioned it was taking you a while to calculate those statistics on your system with 1.2 million messages or whatever it was, I've been working on retooling how it gets those stats.

    Check this out from the A35 whatsnew, I am excited to see these gains!

    + Reduced the time it takes for calculation of area index reader statistics
    by an enormous amount, at the expensive of some accuracy. In A33 my test
    took 6 seconds to complete a list of 300 areas with around 250,000
    messages. In A34 I reduced it to 2.6 seconds. In A35 its now taking
    around 50 milliseconds (1/20th of a second)!

    You should see an epic change in the next alpha in the time it takes you to get to where you need to go, assuming this system proves accurate enough.

    On a side note, stuff like this is the a benefit of ditching Squish and only having JAM. It allows me to focus on coding things optimally just for one specific format instead of coding to make things work on many different
    systems at the cost of performance.

    --- Mystic BBS v1.12 A35 (Windows/32)
    * Origin: Sector 7 [Mystic BBS WHQ] (1:129/215)
  • From Paul Hayton@3:770/100 to g00r00 on Sun Jun 25 13:56:43 2017
    On 06/24/17, g00r00 pondered and said...

    I've been working on retooling how it gets those stats.

    Check this out from the A35 whatsnew, I am excited to see these gains!

    + Reduced the time it takes for calculation of area index reader statistics by an enormous amount, at the expensive of some accuracy. In A33 my test took 6 seconds to complete a list of 300 areas with around

    Very impressive and a welcome speed gain. I use this reader all the time and reducing the number of Usenet groups at Agency was largely down to time it
    was taking me to wait for it to load. Guess I'll add them back in again now :)

    --- Mystic BBS v1.12 A33 (Windows/32)
    * Origin: Agency BBS | telnet://agency.bbs.geek.nz (3:770/100)
  • From Dan Richter@1:317/3 to g00r00 on Sat Jun 24 23:11:01 2017
    On 06/24/17, g00r00 said the following...

    You should see an epic change in the next alpha in the time it takes you to get to where you need to go, assuming this system proves accurate enough.

    It is a lot better. It was taking between 70-100 seconds to load under A33. With A34, that time is down to 30-40 seconds. Great work!

    On a side note, stuff like this is the a benefit of ditching Squish and only having JAM. It allows me to focus on coding things optimally just for one specific format instead of coding to make things work on many different systems at the cost of performance.

    I can imagine that would help a lot. Just out of curiosity, did Mystic ever support the Hudson message bases in the past? I'd think that would have been
    a pain, trying to juggle three different formats.


    ---

    Dan Richter
    aka Black Panther
    Sysop - Castle Rock BBS (RCS)
    telnet://castlerockbbs.com
    The sparrows are flying again...

    --- Mystic BBS v1.12 A34 (Windows/64)
    * Origin: Castle Rock BBS - castlerockbbs.com (1:317/3)
  • From g00r00@1:129/215 to Paul Hayton on Sun Jun 25 10:09:55 2017
    Very impressive and a welcome speed gain. I use this reader all the time and reducing the number of Usenet groups at Agency was largely down to time it was taking me to wait for it to load. Guess I'll add them back
    in again now :)

    I'll send it over for a testing if you'd like to give it a go.

    --- Mystic BBS v1.12 A35 (Windows/32)
    * Origin: Sector 7 [Mystic BBS WHQ] (1:129/215)
  • From g00r00@1:129/215 to Dan Richter on Sun Jun 25 10:39:06 2017
    It is a lot better. It was taking between 70-100 seconds to load under A33. With A34, that time is down to 30-40 seconds. Great work!

    And that should hopefully be reduced again to 1-2 seconds now in A35.

    I can imagine that would help a lot. Just out of curiosity, did Mystic ever support the Hudson message bases in the past? I'd think that would have been a pain, trying to juggle three different formats.

    No, it did not. The reason it did not is (if my memory serves me) was because Hudson had way too many technical limitations by the time Mystic came into play.

    The earliest versions of Mystic (pre 0.01 circa 1995) had its own message base format that did on-the-fly LZH compression, which was a neat idea at the
    time. Back then hard drive space actually mattered so LZH compressed bases could save you quite a bit!

    I ditched that for JAM and Squish, and then eventually ditched Squish.

    --- Mystic BBS v1.12 A35 (Windows/32)
    * Origin: Sector 7 [Mystic BBS WHQ] (1:129/215)
  • From Dan Richter@1:317/3 to g00r00 on Sun Jun 25 12:36:43 2017
    On 06/25/17, g00r00 said the following...

    It is a lot better. It was taking between 70-100 seconds to load unde A33. With A34, that time is down to 30-40 seconds. Great work!

    And that should hopefully be reduced again to 1-2 seconds now in A35.

    If A35 loads in 1-2 seconds, I will buy you a drink... Or 6... :)

    No, it did not. The reason it did not is (if my memory serves me) was because Hudson had way too many technical limitations by the time Mystic came into play.

    I was just wondering, as it seemed as though a lot of the programs tried to support everything at the time.

    Hudson really did have limitations. I only used it for a couple message bases back in the 90s. Jam was what I used primarily.

    I never even tried Squish. I don't remember why... I hate getting old... :)


    ---

    Dan Richter
    aka Black Panther
    Sysop - Castle Rock BBS (RCS)
    telnet://castlerockbbs.com
    The sparrows are flying again...

    --- Mystic BBS v1.12 A34 (Windows/64)
    * Origin: Castle Rock BBS - castlerockbbs.com (1:317/3)
  • From mark lewis@1:3634/12.73 to Dan Richter on Sun Jun 25 14:48:46 2017
    On 2017 Jun 24 23:11:00, you wrote to g00r00:

    On a side note, stuff like this is the a benefit of ditching Squish
    and only having JAM. It allows me to focus on coding things
    optimally just for one specific format instead of coding to make
    things work on many different systems at the cost of performance.

    I can imagine that would help a lot. Just out of curiosity, did Mystic ever support the Hudson message bases in the past? I'd think that
    would have been a pain, trying to juggle three different formats.

    the original library viewed each message base type as an object... each message
    base object did its thing as it needed to... instead of using each message base
    object individually, you use a generic message base object... this generic message base object is simply derived from the message base object for the current message base format... the generic message base object simply hands everything to the current message base object and lets it have a go... if you change message bases and the new one is of a different format, you simply dispose of the generic message base object and initialize it again using the message base object for the new message base... now everything is done in the new message base's format... change bases again? dispose of the generic object and init it again with the new message base's object... you just always use the
    generic message base object and the others are just black boxes... you feed data in for storage and you request data out for use...

    no one ever needs to mess around in the deep guts of the library (other than to
    fix or track down possible bugs or oversights)... all the calls in the generic message object are the same... trust me, i know because i use the same library and i don't have any problems writting to any of the formats supported... it is
    great and is one of the biggest features of object programming!

    i have, however, done my digging in the guts and yes, fixing some things was easily done... other things not so much... one example being the dates as stored in the various message base formats... one doesn't store seconds at all,
    another stores only even seconds, the rest store all available seconds... using
    this black box object stuff, you just send exactly the same date string and the
    actual message base object will handle the seconds according to their format...

    )\/(ark

    Always Mount a Scratch Monkey
    Do you manage your own servers? If you are not running an IDS/IPS yer doin' it wrong...
    ... Food is an important part of a balanced diet.
    ---
    * Origin: (1:3634/12.73)
  • From Tony Langdon@3:633/410 to g00r00 on Mon Jun 26 09:48:00 2017
    g00r00 wrote to Dan Richter <=-

    No, it did not. The reason it did not is (if my memory serves me) was because Hudson had way too many technical limitations by the time
    Mystic came into play.

    I can certainly understand that. I started out with RA 1.1 and the Hudson messagebase. I remember there was a size limit of 16 MB, which if exceeded would lead to severe corruption, causing all sorts of random crossposts. There was also a limit on the number of messages, IIRC. Hudson was a real cow to manage - miss a pack, or get your maintenance settings wrong and it was so easy to trash the messagebase on a busy board, because of these limitations. In the early 90s, this format was already past its useby date.

    When JAM came along with RA 2.0, it was a breath of fresh air. I migrated most of my message areas to JAM and available disk space became the bigger concern (finally!).

    The earliest versions of Mystic (pre 0.01 circa 1995) had its own
    message base format that did on-the-fly LZH compression, which was a
    neat idea at the time. Back then hard drive space actually mattered so LZH compressed bases could save you quite a bit!

    Yes, that would have been handy in the early days!


    ... Real knowledge is to know the extent of ones ignorance.
    --- MultiMail/Win32 v0.49
    * Origin: Freeway BBS - freeway.apana.org.au (3:633/410)
  • From Paul Hayton@3:770/100 to g00r00 on Mon Jun 26 13:26:38 2017
    On 06/25/17, g00r00 pondered and said...

    I'll send it over for a testing if you'd like to give it a go.

    Sounds good, I can certainly flood the BBS with lots of Usenet groups to push it.

    --- Mystic BBS v1.12 A33 (Windows/32)
    * Origin: Agency BBS | telnet://agency.bbs.geek.nz (3:770/100)