I have one echo here where I really let things pile up, to something over 4000
messages, until I decided to do something about it a while ago. Wrote out stuff I wanted to keep, deleted bunches of messages, and then thought I'd have a go at packing it.

No go.

First attempt with me trying to use sqpack got me an error message when it tried to open the ".sqd" file. The second attempt was using sqfix, which gave
me the same error message.

I finally got past it by creating another echo in my squish.cfg, using TimED to "move" stuff over to that, and then deleting the original file and renaming
the new ones to what the old ones were.

The .sqd file in question was a bit over 8M in size. Is this a problem for Squish running under dos?

---
* Origin: TANSTAAFL BBS 717-838-8539 (1:270/615)

Hello Roy!

28 Jul 04 20:07, Roy J. Tellason wrote to all\:

The .sqd file in question was a bit over 8M in size. Is this a
problem for Squish running under dos?

Nope...I have many that are larger without issue. SQFIX and GOLDED DOS versions start misbehaving when areas hit 5400 msgs or so here, though, so I cap mine at 5000. If they go over and break, then SQFIX32 might work...but not
always. Even with 5K caps, here's my top 5:

ALLFIX_F SQD 28,155,429 07-29-04 3:44a
WEATHER SQD 27,625,285 07-29-04 3:43a
U-F1O4Q SQD 20,507,829 07-29-04 4:09a
U-RV6BX SQD 15,777,708 07-29-04 4:09a
U-RVG8O SQD 15,584,469 07-29-04 3:39a

If you don't already have SQPACK in a regular maintenance routine (daily here),
it is probably a good idea to run it both before and after your manual cleanup activity, to insure the integrity of the area. There were probably some pointers that were confused before you started (in the .sqi), which got really confused by the time you were done.

.\\ike

--- GoldED 2.50+
* Origin: -=( The TechnoDrome )=- Austin,TX 512-327-8598 33.6k (1:382/61)

Mike Tripp wrote in a message to Roy J. Tellason:

Hello Roy!

28 Jul 04 20:07, Roy J. Tellason wrote to all\:

The .sqd file in question was a bit over 8M in size. Is this a
problem for Squish running under dos?

Nope...I have many that are larger without issue. SQFIX and GOLDED
DOS versions start misbehaving when areas hit 5400 msgs or so here, though, so I cap mine at 5000. If they go over and break, then
SQFIX32 might work...but not always. Even with 5K caps, here's my
top 5:

ALLFIX_F SQD 28,155,429 07-29-04 3:44a
WEATHER SQD 27,625,285 07-29-04 3:43a
U-F1O4Q SQD 20,507,829 07-29-04 4:09a
U-RV6BX SQD 15,777,708 07-29-04 4:09a
U-RVG8O SQD 15,584,469 07-29-04 3:39a

If you don't already have SQPACK in a regular maintenance routine
(daily here), it is probably a good idea to run it both before and
after your manual cleanup activity, to insure the integrity of the
area. There were probably some pointers that were confused before
you started (in the .sqi), which got really confused by the time
you were done.

I do have it running daily, but it doesn't do anything in those cases where I haven't set the message area to any particular limits, like the one in question:

- 672 - Old=1131352; New=1131352

This is after I'd trimmed out a bunch of stuff...

5400 messages? There were something over 8000 of them in there, I forget how many now. And yes, this is the dos version.

---
* Origin: TANSTAAFL BBS 717-838-8539 (1:270/615)

Hello Roy!

31 Jul 04 12:13, Roy J. Tellason wrote to Mike Tripp:

- 672 - Old=1131352; New=1131352

This is after I'd trimmed out a bunch of stuff...

The file should've gotten smaller the first run after you deleted messages. Even without parameters, SQPACK should purge the "slack space" occupied by the deleted messages. Even if you trim to a specific quantity of messages, the 5K you have today may be fewer bytes than the 5K you had yesterday. If that's not
the case, then you'll see the same bytes reported for Old/New as above.

5400 messages?

That seems to be where GoldEd/DOS starts complaining...GoldEd/2 is still fine beyond there...and your editor may have no trouble. I forget the magic number where SQFIX starts suggesting that you use SQFIX32 instead...but I use the lowest common denominator, which for me happens to be the editor's limit.

There were something over 8000 of them in there, I
forget how many now.

672 according to the line above...and now you have a 1mb file instead of the 8mb you originally asked about.

And yes, this is the dos version.

And you will have different luck between the 16- and 32-bit DOS versions of SQFIX, but my experience is that is the message quantity (.SQI index entries) and not the filesize of the .SQD that makes or breaks you. It's not that rare that there is some index cleanup even on areas with low caps and small .SQDs, and both SQFIX and SQPACK have to manage these while doing their thing.

.\\ike

--- GoldED 2.50+
* Origin: -=( The TechnoDrome )=- Austin,TX 512-327-8598 33.6k (1:382/61)

Mike Tripp wrote in a message to Roy J. Tellason:

Hello Roy!

31 Jul 04 12:13, Roy J. Tellason wrote to Mike Tripp:

- 672 - Old=1131352; New=1131352

This is after I'd trimmed out a bunch of stuff...

The file should've gotten smaller the first run after you deleted messages.

One would think so. It had gotten pretty large, something over 4000 msgs, and over 8M in size. I wrote out what I wanted to keep, by month, deleted the rest, and tried to run sqpack manually -- and that's when I got the error.
A shot at sqinfo -b gave nothing, and sqfix gave me the same error.

Even without parameters, SQPACK should purge the "slack space"
occupied by the deleted messages. Even if you trim to a specific quantity of messages, the 5K you have today may be fewer bytes
than the 5K you had yesterday. If that's not the case, then
you'll see the same bytes reported for Old/New as above.

I'm seeing the same numbers in both cases there because I have no settings in the area. Most of them here are set to some specific time limit (commonly 60 days at this point), one or two of the real busy ones are also set to a maximum number of messages, but I try to avoid that because it slows down tossing. That way I don't have to worry about feeding any parameters to sqpack
in my batch file other than a wildcard pointing to the directory, and can change the settings on the fly here in TimED, as well as with sqset.

5400 messages?

That seems to be where GoldEd/DOS starts complaining...GoldEd/2 is
still fine beyond there...and your editor may have no trouble.

It didn't seem to. I used to use GoldEd but stopped, I can't recall why just now. I know the last time I tried it out I didn't care for it, or was too used to this, or something. <shrug>

I forget the magic number where SQFIX starts suggesting that you
use SQFIX32 instead...but I use the lowest common denominator,
which for me happens to be the editor's limit.

There were something over 8000 of them in there, I
forget how many now.

672 according to the line above...and now you have a 1mb file
instead of the 8mb you originally asked about.

Right. But to get to that point I had to *move* the remaining messages I wanted to keep from one area to another. I created a new area, same name as the other one but with a "2" at the end of the area name and no links listed in
squish.cfg, fired up TimED again, and it found it. I moved everything over there (and didn't get that "moved" string implanted in the messages thankfully), exited the editor, and in my file manager deleted the old area and renamed the new one, then fixed my squish.cfg.

In other words the software didn't work for me the way I expected it to, and I
found a workaround. Which is pretty typical. But also why I posted on this in
the first place, to find out if there's some known point when things get messed up. I know that this is definitely the case with largish text files under dos!

And yes, this is the dos version.

And you will have different luck between the 16- and 32-bit DOS
versions of SQFIX, but my experience is that is the message
quantity (.SQI index entries) and not the filesize of the .SQD that
makes or breaks you. It's not that rare that there is some index
cleanup even on areas with low caps and small .SQDs, and both SQFIX
and SQPACK have to manage these while doing their thing.

That would be the .sqi file? Here's what I'm looking at now for the area in question:

.sqb 20008
.sqd 1208034
.sqi 8628
.sql 4

All with today's date, and a timestamp within the past hour when I was reading
it last. What's that sqi file used for anyhow?

---
* Origin: TANSTAAFL BBS 717-838-8539 (1:270/615)

Hello Roy!

01 Aug 04 04:07, Roy J. Tellason wrote to Mike Tripp:

I'm seeing the same numbers in both cases there because I have no
settings in the area.

Lack of settings alone won't keep the numbers static. If you manually kill one
message and SQPACK again, Old should be larger than New. If you manually edit one existing message to remove a few lines from the message, Old should be larger than New.

Right. But to get to that point I had to *move* the remaining
messages I wanted to keep from one area to another.

Gotcha...and obviously the stats you're posting are for the new version of the area and not the original hosed version. It is the hosed version that should've been shrinking as messages were moved out of it. The new version was
only growing to swallow the new messages being appended, so there's no reason for them to be occupying more than the actual number of bytes required to do so.

All with today's date, and a timestamp within the past hour when I
was reading it last. What's that sqi file used for anyhow?

Those that've been into the source up to their elbows can probably find plenty of holes with my analogy, but if you understand the basics of how DOS manages files and space allocated on a FAT hard drive: the .SQI is the FAT table for the files (messages) within Squish's hard drive (.SQD).

The .SQI basically has the pointers to where the header for each message starts
and how many blocks it occupies. If you delete a message, Squish just updates the .SQI file that references the deleted data to remove the pointer to its header and updates it's usage map to show that space as available for writing new data again. The size of the .SQD is not necessarily representative of the actual number of bytes required by the currently retained messages. Squish will expand the .SQD if more space is needed to accomodate a new retention peak, but it will not shrink it just because fewer are required today than yesterday.

Just as bad things can happen to a FAT if operations fail or are interrupted by
a system lockup or power failure, the same is true for Squish and .SQI/.SQD manipulations. SQFIX does for the Squishbase what CHKDSK and/or Disk Doctor does for the integrity of the FAT. It looks to make sure that a message header
actually starts where each .SQI entry says it does and looks for space that the
.SQI says should be in use by a retained message but actually isn't (akin to lost clusters and cross-linked files) and attempts to either fix or remove .SQI
entries that don't seem to match reality found in the .SQD.

SQPACK is analagous to DEFRAG. If you have specified parameters, it first marks the messages that fail those parameters as deleted and then "compresses" the .SQD data to the beginning of the .SQD file and shrinks the .SQD file down to the actual size required to store the actual bytes associated with those messages that are still retained, rather than the size being some former worst-case maximum that was required to store messages that met the SQSET parameters at some point in the past since you last SQPACKed.

So if you use no purge parameters, never manually delete/edit a message in an area, and never have any problem that causes the .SQI to lose synch with the actual contents of the .SQD, it will appear that SQPACK never does anything...just as running DEFRAG wouldn't accomplish anything for your hard drive if you only appended new files one at a time and never deleted or modified an existing file since your last DEFRAG. If, however, you make any of
those changes or SQFIX frees a pointer to an existing message because of a data
integrity issue, it is possible/probable that there will be some delta between the old/new bytes when you SQPACK next, even if you haven't assigned specific purge criteria to the area.

.\\ike

--- GoldED 2.50+
* Origin: -=( The TechnoDrome )=- Austin,TX 512-327-8598 33.6k (1:382/61)

Mike Tripp wrote in a message to Roy J. Tellason:

Hello Roy!

01 Aug 04 04:07, Roy J. Tellason wrote to Mike Tripp:

I'm seeing the same numbers in both cases there because I have no
settings in the area.

Lack of settings alone won't keep the numbers static. If you
manually kill one message and SQPACK again, Old should be larger
than New. If you manually edit one existing message to remove a few
lines from the message, Old should be larger than New.

Sure.

Right. But to get to that point I had to *move* the remaining
messages I wanted to keep from one area to another.

Gotcha...and obviously the stats you're posting are for the new
version of the area and not the original hosed version.

Right. That one's gone. Old habit, getting rid of stuff like that, from when disk space was a scarcer resource.

It is the hosed version that should've been shrinking as messages
were moved out of it.

No, the hosed version wouldn't have been shrinking as sqpack wouldn't touch it... Too bad my "pack.log" gets rewritten every day, or I'd pull a line out
of that one to show it.

The new version was only growing to swallow the new messages being appended, so there's no reason for them to be occupying more than
the actual number of bytes required to do so.

Yup. And the way things looked it wasn't ever gonna get any smaller. I'm looking at something like 38M currently on that drive, not so much that I need
to worry about an 8M file eating too much space but still...

All with today's date, and a timestamp within the past hour when I
was reading it last. What's that sqi file used for anyhow?

Those that've been into the source up to their elbows can probably
find plenty of holes with my analogy, but if you understand the
basics of how DOS manages files and space allocated on a FAT hard
drive: the .SQI is the FAT table for the files (messages) within
Squish's hard drive (.SQD).

Ok.

The .SQI basically has the pointers to where the header for each
message starts and how many blocks it occupies. If you delete a
message, Squish just updates the .SQI file that references the
deleted data to remove the pointer to its header and updates it's
usage map to show that space as available for writing new data
again. The size of the .SQD is not necessarily representative of
the actual number of bytes required by the currently retained
messages. Squish will expand the .SQD if more space is needed to accomodate a new retention peak, but it will not shrink it just
because fewer are required today than yesterday.

Makes sense. This also explains some why TimED's undelete function seems to grab *all* deleted messages, I guess that was just simpler to code, or something.

Just as bad things can happen to a FAT if operations fail or are interrupted by a system lockup or power failure, the same is true
for Squish and .SQI/.SQD manipulations. SQFIX does for the
Squishbase what CHKDSK and/or Disk Doctor does for the integrity of
the FAT. It looks to make sure that a message header actually
starts where each .SQI entry says it does and looks for space that
the .SQI says should be in use by a retained message but actually
isn't (akin to lost clusters and cross-linked files) and attempts
to either fix or remove .SQI entries that don't seem to match
reality found in the .SQD.

SQPACK is analagous to DEFRAG. If you have specified parameters,
it first marks the messages that fail those parameters as deleted
and then "compresses" the .SQD data to the beginning of the .SQD
file and shrinks the .SQD file down to the actual size required to
store the actual bytes associated with those messages that are
still retained, rather than the size being some former worst-case
maximum that was required to store messages that met the SQSET
parameters at some point in the past since you last SQPACKed.

It also seems to write the results out to a new file, so for example if a single message area is larger than the available space on disk, it will *never* get packed down. I have one such area at present, the .sqd file is something around 66 megs. This was from an internet mailing list I was carrying for a while, back when I had that capability on the system, and I figured that I'd just delete stuff as I read 'em. I somehow haven't found the time to mess with that one lately, and if I don't eventually I may end up just
deleting it.

So if you use no purge parameters, never manually delete/edit a
message in an area, and never have any problem that causes the .SQI
to lose synch with the actual contents of the .SQD, it will appear
that SQPACK never does anything...just as running DEFRAG wouldn't accomplish anything for your hard drive if you only appended new
files one at a time and never deleted or modified an existing file
since your last DEFRAG. If, however, you make any of those changes
or SQFIX frees a pointer to an existing message because of a data integrity issue, it is possible/probable that there will be some
delta between the old/new bytes when you SQPACK next, even if you
haven't assigned specific purge criteria to the area.

There are only a few areas like that here, most are set for specified time frames, typically 60 days.

And there do seem to be times when random errors creep in, I don't know if it's memory glitches, power surges, or what, though for the most part things
are pretty quiet around here in the small hours when maintenance is done. Every so often I'll scan my pack.log file for "err" and find some, like that one mentioned above, but mostly it won't find it, and I'll figure things are okay at that point as there's plenty of areas and they all get packed down each
night.

---
* Origin: TANSTAAFL BBS 717-838-8539 (1:270/615)

Hello Roy!

02 Aug 04 04:07, Roy J. Tellason wrote to Mike Tripp:

The new version was only growing to swallow the new messages
being appended, so there's no reason for them to be occupying
more than the actual number of bytes required to do so.

Yup. And the way things looked it wasn't ever gonna get any smaller.
I'm looking at something like 38M currently on that drive, not so
much that I need to worry about an 8M file eating too much space but still...

I was referring to your new replacement files here, not the original hosed version. If you don't assign it any purge paramaters either, then there's no reason for SQPACK to ever shrink the file, and it should head straight back for
8mb or however far it grows until it a data integrity issue stops one utility or another from being able to interact with it correctly any more.

If you want to keep areas uncapped, it might help to do a monthly SQREIDX or SQFIX event to catch little issues before they become big/fatal issues.

Makes sense. This also explains some why TimED's undelete
function seems to grab *all* deleted messages, I guess that was
just simpler to code, or something.

Yep...and probably has the same pitfoils as UNDELETE in DOS. "Deleted" data is
overwritten by new data arriving...so some deleted msgs are unrecoverable. TimEd is probably just grabbing all of the puzzle pieces it can find to start with, figuring out which fit together, and then throwing away the few leftovers
that don't seem to fit anywhere when done.

There are only a few areas like that here, most are set for
specified
time frames, typically 60 days.

SQUISH and SQPACK work together to maintain the quantity limit. SQPACK alone does the limit by days. SQPACK should =always= show some shrinkage on each daily run for areas with a days limit, unless the area is newer than the limit or the area has days without mail.

.\\ike

--- GoldED 2.50+
* Origin: -=( The TechnoDrome )=- Austin,TX 512-327-8598 33.6k (1:382/61)

Mike Tripp wrote in a message to Roy J. Tellason:

Hello Roy!

02 Aug 04 04:07, Roy J. Tellason wrote to Mike Tripp:

The new version was only growing to swallow the new messages
being appended, so there's no reason for them to be occupying
more than the actual number of bytes required to do so.

Yup. And the way things looked it wasn't ever gonna get any smaller.
I'm looking at something like 38M currently on that drive, not so
much that I need to worry about an 8M file eating too much space but still...

I was referring to your new replacement files here, not the
original hosed version. If you don't assign it any purge
paramaters either, then there's no reason for SQPACK to ever shrink
the file, and it should head straight back for 8mb or however far
it grows until it a data integrity issue stops one utility or
another from being able to interact with it correctly any more.

A data integrity issue? Hmm.

If you want to keep areas uncapped, it might help to do a monthly
SQREIDX or SQFIX event to catch little issues before they become
big/fatal issues.

I never thought about sqreidx, missed that one completely. Sure enough, there it is, right in my squish directory where it should be!

Actually what I plan on doing is archiving the stuff I want to keep about once a month (keeping it in month-sized chunks) and then deleting those from the active message base. What I put off for *way* too long with this particular area, since the archives I did end up creating go all the way back to last November.

Makes sense. This also explains some why TimED's undelete
function seems to grab *all* deleted messages, I guess that was
just simpler to code, or something.

Yep...and probably has the same pitfoils as UNDELETE in DOS.
"Deleted" data is overwritten by new data arriving...so some
deleted msgs are unrecoverable. TimEd is probably just grabbing all
of the puzzle pieces it can find to start with, figuring out which
fit together, and then throwing away the few leftovers that don't
seem to fit anywhere when done.

Or something. I don't use it that often to be able to guess.

There are only a few areas like that here, most are set for
specified time frames, typically 60 days.

SQUISH and SQPACK work together to maintain the quantity limit.

Yeah, which is why I use it seldom, since it slows down message tossing.

SQPACK alone does the limit by days. SQPACK should =always= show
some shrinkage on each daily run for areas with a days limit,
unless the area is newer than the limit or the area has days
without mail.

Sounds right to me!

I just wonder what the problem was. Oh well. I won't likely be letting any area get that big again so I probably won't run into the problem again...
---
* Origin: TANSTAAFL BBS 717-838-8539 (1:270/615)