Can someone point me to a reference somewhere on the structure
of a *.pkt file? I can find bits and pieces strewn about the
Internet, but I haven't found any definitive source yet.

uh... what's wrong with the FTSC documents? /they/ are the definitive documents
on FTN stuffs...

While I'm on the topic - I'm looking to ditch REX for my
outbound gating and want to parse the pkt files myself for
insertion into my web forums database. I can easily get the
To, From and Subject field out because they're null
terminated, but I can't see any way to get just the message
text by itself out without taking all the msgid's, times, and
everything else that falls between the end of the subject and
the beginning of the actual message. Anyone got any bright
ideas there?

those control lines are in there for a reason... some of those control lines are what i was speaking of in other conversations with you on this topic...

)\/(ark
* Origin: (1:3634/12)

======>>> mark lewis, 1:3634/12 wrote:

Originally to: Jon Watson

Can someone point me to a reference somewhere on the structure
of a *.pkt file? I can find bits and pieces strewn about the
Internet, but I haven't found any definitive source yet.

uh... what's wrong with the FTSC documents? /they/ are the definitive documents

on FTN stuffs...


<<<====== end quote


Uh..what's wrong with them is that I haven't found them, wasn't I clear on that? Can you point me to the FTSC documents? That would be helpful.

======>>> mark lewis, 1:3634/12 wrote:


those control lines are in there for a reason... some of those control lines are
what i was speaking of in other conversations with you on this topic...


<<<====== end quote


So you don't have any bright ideas for me?

I'm not going to strip them out of anything leaving my system, I just don't see
the point in storing them in my web forums database. I know you don't fully understand what I'm doing so just take my word for it that anything leaving my system will be a properly formatted packet.

Jon


-FOTW: read your
Fidonet On The Web!
http://www.theheatsinkbbs.ca :=-
--- Internet Rex 2.29
* Origin: The gateway at The HeatSink BBS (1:134/703)

======>>>> mark lewis, 1:3634/12 wrote:

Originally to: Jon Watson

Can someone point me to a reference somewhere on the structure
of a *.pkt file? I can find bits and pieces strewn about the
Internet, but I haven't found any definitive source yet.

uh... what's wrong with the FTSC documents? /they/ are the
definitive documents
on FTN stuffs...

<<<====== end quote

argh! that quoting stuff just ain't working, man... i can't tell what you wrote
and what i wrote in your replies... the above make it look like someone other than you wrote the first paragraph and that you wrote my part... :((

that stuff /really/ needs to have >'s put on each line to conform with what the
rest of the network is doing... :(

Uh..what's wrong with them is that I haven't found them,
wasn't I clear on that? Can you point me to the FTSC
documents? That would be helpful.

lordy lordy, where to start...

1. http://www.ftsc.org
2. your *C
3. most any well stocked fidonet site like mine

OB-)

======>>>> mark lewis, 1:3634/12 wrote:

those control lines are in there for a reason... some
of those control lines are what i was speaking of in
other conversations with you on this topic...

<<<====== end quote

So you don't have any bright ideas for me?

I'm not going to strip them out of anything leaving my system,

how can you if your system doesn't generate them?

I just don't see the point in storing them in my web forums
database.

i don't see why not... what's any different than storing them in a JAM, SQUish,
MSG, Hudson, EZYcomm, or other message base?

I know you don't fully understand what I'm doing so
just take my word for it that anything leaving my system will
be a properly formatted packet.

i do fully understand, dude... but something is getting lost in the translation
from me to you...

its not what you send out but what you do with what you receive... how can you link

MSGID: 1:3634/12@fidonet 900a0e67
REPLY: 1:134/703@fidonet 40d66fcb

back to

MSGID: 1:134/703@fidonet 40d66fcb
REPLY: 1:2800/18 314D5F0E

and that back to

MSGID: 1:2800/18 314D5F0E
REPLY: 1:2800/18 314CB451

if you don't store them somewhere??

additionally, how can your system display the message properly if it doesn't recognise and use the CHRS and CODEPAGE control lines? and there's also the TZUTC control line that carries the UTC offset so that the true local time of the message's creation can be known... should i also mention the REPLYTO and REPLYADDR control lines that tell who to address and where to send private responses to public messages... and FWDFROM and FWDADDR and such for forwarded messages so that a private response can be written back to the original poster in addition to or instead of the person that did the forwarding?

i dunno... maybe i just know too much about this stuff and how it ties together
to see just bitbucketing valid information that is needed at times... i dunno... sorry 'bout that...

like i said before... this is one of the reasons why i'm not going to try to fit FTN stuff into a message base format that simply can't handle it...

actually, though... it can in the same way that the Hudson format does/did... they are stored with the message bodies and not displayed by the code... then the system can make use of them for creating replies and such and you could even have a setting that would display them if the user turned on a/the switch to display them... hummm... guess i've been working with JAM bases too long... i should have seen that option before... with that in mind, i might just see about doing something with phpBB ;)

)\/(ark
* Origin: (1:3634/12)

Quoting Jon Watson from a message to <--

Can someone point me to a reference somewhere on the structure of a
*.pkt file? I can find bits and pieces strewn about the Internet, but
I haven't found any definitive source yet.

From the very excellent SWAG Resources Archive:
{

do you know the format the .PKT files have?

From FSC-0039. There may have been a few changes since this doc so you may need to play with it a little.

Type-2 Packet Format (proposed, but currently in use) -----------------------------------------------------
Field Ofs Siz Type Description Expected value(s)
------- --- --- ---- -------------------------- -----------------
OrgNode 0x0 2 Word Origination node address 0-65535
DstNode 2 2 Word Destination node address 1-65535
Year 4 2 Int Year packet generated 19??-2???
Month 6 2 Int Month " " 0-11 (0=Jan)
Day 8 2 Int Day " " 1-31
Hour A 2 Int Hour " " 0-23
Min C 2 Int Minute " " 0-59
Sec E 2 Int Second " " 0-59
Baud 10 2 Int Baud Rate (not in use) ????
PktVer 12 2 Int Packet Version Always 2
OrgNet 14 2 Word Origination net address 1-65535
DstNet 16 2 Word Destination net address 1-65535
PrdCodL 18 1 Byte FTSC Product Code (lo) 1-255
* PVMajor 19 1 Byte FTSC Product Rev (major) 1-255
Password 1A 8 Char Packet password A-Z,0-9
* QOrgZone 22 2 Int Orig Zone (ZMailQ,QMail) 1-65535
* QDstZone 24 2 Int Dest Zone (ZMailQ,QMail) 1-65535
Filler 26 4 Byte Spare Change ?
* PrdCodH 2A 1 Byte FTSC Product Code (hi) 1-255
* PVMinor 2B 1 Byte FTSC Product Rev (minor) 1-255
* CapWord 2C 2 Word Capability Word BitField
* OrigZone 2E 2 Int Origination Zone 1-65535
* DestZone 30 2 Int Destination Zone 1-65535
* OrigPoint 32 2 Int Origination Point 1-65535
* DestPoint 34 2 Int Destination Point 1-65535
* ProdData 36 4 Long Product-specific data Whatever
PktTerm 3A 2 Word Packet terminator 0000

* - extensions to FTS-0001

Ofs, Siz are in hex, other values are decimal.


Zone/Point Aware Mail Processors (probably a partial list) ----------------------------------------------------------
Prod
Code Name - Uses QOrg/QDstZone Orig/DestZone Orig/DestPoint
---- ----------- ------------- ------------- --------------
0x0C FrontDoor Reads/Updates Yes Yes
0x1A DBridge ????? Yes Yes
0x23 XRS Reads/Updates Yes Yes
0x29 QMail Yes ????? Not point-aware
0x35 ZMailQ Yes ????? Not point-aware
0x3F TosScan Reads/Updates Yes Yes

}

...hope this helps. :)

Later,
Sean
--- Telegard/2 v3.09.g2-sp4/mL
* Origin: Outpost BBS - outpostbbs.us (1:18/200)

======>>> mark lewis, 1:3634/12 wrote:


i do fully understand, dude... but something is getting lost in the translation

from me to you...


<<<====== end quote


No, you don't and that's partially my fault. Let's try a paradigm shift here....I'm not trying to toss fidonet messages into a MySQL database. I'm just
replicating my JAM message bases with a MySQL database. My MySQL database never
sees the light of day. Everything that comes or goes from my node is processed either by my BBS or by IREX. Is it perfect? No, but it's pretty f***cking fantastic considering I've been working on it for about 10 days and other people have been working on it for years and have nothing to show for it.

I agree with the 'lost in translation' thing. I find most of your posts pretty irritating, but I don't think that's your fault - I just don't think I can relate to you. Whatever the reason, I think this subject has lived its usefulness so if we could just let this thread die, that would be marvy.

As for the quoting - yeah, there's a few things on my to do list. Many more pressing than quoting, but it's there.

Jon




-FOTW: read your
Fidonet On The Web!
http://www.theheatsinkbbs.ca :=-
--- Internet Rex 2.29
* Origin: The gateway at The HeatSink BBS (1:134/703)

======>>> Sean Dennis, 1:18/200 wrote:

Originally to: Jon Watson

From FSC-0039.á There may have been a few changes since this doc so you may need to play with it a little.

...snip...


<<<====== end quote


Wicked, thanks Sean. I've stumbled (with a little help from Mark) onto the FTS and FSC docs but I'm having trouble figuring out the message structure.

I understand that everything before the actualy message body is null terminated, and I can parse that out without a problem. What I think is odd, and therefore assume I'm misunderstanding, is that the message body appears to start with the AREA: keyword. What I've seen is that different messages have varying fields between the AREA: keyword and the beginning of the actual text of the message. This makes it pretty hard to separate the part of the message body I want to see, and the part I don't want to see.

I *think* I've figured out (need a few more packets to confirm) that the fields
that comprise the part I don't want to see are separated by a ^M^A token, whereas the message body starts with a single ^M.

Is this the way it's supposed to be? Seems kinda laisse faire (no, I can't spell that).

Thanks!

Jon

-FOTW: read your
Fidonet On The Web!
http://www.theheatsinkbbs.ca :=-
--- Internet Rex 2.29
* Origin: The gateway at The HeatSink BBS (1:134/703)

Quoting Jon Watson from a message to sean dennis <--

Wicked, thanks Sean. I've stumbled (with a little help from Mark)
onto the FTS and FSC docs but I'm having trouble figuring out the
message structure.

No problem. I have the entire SWAG archive here-it's about 10MB compressed, but it's a godsend when you're programming. I don't know if you know Pascal or
not (seems like you do), but it's great to have. Even comes with its own reader program.

Is this the way it's supposed to be? Seems kinda laisse faire (no, I
can't spell that).

Your spelling looks good to me. About the parsing of the AREA: keyword-to be honest, I know very, very little about it. I've never yet figured out how to work with the .PKT format, but I may soon so I can write my own message stuff.

I'd assume that there's people in here that could help more than I could (Mark Lewis comes to mind, so does Deuce [Stephen Hurd]).

Good luck with it and if you need any more stuff, please feel free to ask. :)

Later,
Sean
--- Telegard/2 v3.09.g2-sp4/mL
* Origin: Outpost BBS - outpostbbs.us (1:18/200)

Wicked, thanks Sean. I've stumbled (with a little help from
Mark) onto the FTS and FSC docs but I'm having trouble
figuring out the message structure.

let's see what we can do about that ;)

I understand that everything before the actualy message body
is null terminated, and I can parse that out without a
problem. What I think is odd, and therefore assume I'm
misunderstanding, is that the message body appears to start
with the AREA: keyword.

i'm assuming that you've gotten past the PKT header? if not, it is 48 bytes long... then you have the first message header... packed message headers are 34
bytes long and then variable lengths for the To, From and Subj lines...

so, to read the packed message header, you pump out the first 34 bytes and then
read three times for the null terminated strings... after that, everything till
the terminating null is all message body... everything...

What I've seen is that different messages have varying fields
between the AREA: keyword and the beginning of the actual text
of the message. This makes it pretty hard to separate the part
of the message body I want to see, and the part I don't want
to see.

and you can't go by the assumtion that all control lines are up at the top of the message body right after the header block... why? because some control lines are place within the message body...

the thing to do is to get past the header and then start reading everything up to the terminating null character... stuff starting with the CTRL-A (0x01) character and terminating with the CR or CRLF pair is the control lines that you are wanting to throw away so do so while reading the message body for the content...

when you get done skipping over the 0x01...CR|CRLF stuff, you should only have the message body remaining... you'll also want to be filtering (in some cases) the ALT-141 "" character as QWK and some bbs' use that as a line wrap indicator... and then you have the seenby and path lines that you'll also probably be throwing away...

it is a good idea to read those control lines and adapt the message body to the
proper character set by looking for and acting upon the CHRS and/or CODEPAGE control lines to convert from them to the 8859-1 or whatever it is that is the default in your message stuff...

I *think* I've figured out (need a few more packets to
confirm) that the fields that comprise the part I don't want
to see are separated by a ^M^A token, whereas the message body
starts with a single ^M.

can't rely on that... some stuff is CR (^M) and other stuff is CRLF (^M^J)... the ^A stands on its own as the start of the control line indicator...

Is this the way it's supposed to be? Seems kinda laisse faire
(no, I can't spell that).

neither can it... hope the above helps... enjoy!

)\/(ark
* Origin: (1:3634/12)

======>>> Sean Dennis, 1:18/200 wrote:

Originally to: Jon Watson

Your spelling looks good to me.á About the parsing of the AREA: keyword-to beá honest, I know very, very little about it.á I've never yet figured out how toá work with the .PKT format, but I may soon so I can write my own message stuff.


<<<====== end quote


It's a mystery...I have about 50 packets to look at now, and there doesn't seem
to be any consistent delimeter where the headers end and the message text truly
begins. On some packets there is a double carriage return between the last header and the first line of message text, in other packets there's only a single carriage return.

Most of the packets (perhaps all, not sure yet) have a ^M^A sequence between all the headers and then EITHER just a ^M before the text or a ^M^M before the text. So....I *think* that a single ^M followed by anything other than a ^A *may* work. But since I can't get preg_replace or ereg_replace to work on the whole string, I have to read it and evaluate it char by char. Slooooow business.

We'll see how it goes....

Jon


-FOTW: read your
Fidonet On The Web!
http://www.theheatsinkbbs.ca :=-
--- Internet Rex 2.29
* Origin: The gateway at The HeatSink BBS (1:134/703)

======>>> mark lewis, 1:3634/12 wrote:

Originally to: Jon Watson

... snip ...

i'm assuming that you've gotten past the PKT header? if not, it is 48 bytes long... then you have the first message header... packed message headers are 34

bytes long and then variable lengths for the To, From and Subj lines...

... snip ...


<<<====== end quote


Uncanny. My uplink went down for a few days so in the deafening silence I went to work on my code. And I came to the same conclusions, and coded exactly what you just posted. Well, OK..I didn't know that control lines could come in the middle of the message body and I'll have to look out for that, but other than that...pretty close.

I've got the code and can now rip directly from packet to MySQL database. I don't have it implemented yet though, I want to make sure I have some free time
in case the whole thing blows up in my face. Probably Friday when I do it.

Thanks!

Jon
-FOTW: read your
Fidonet On The Web!
http://www.theheatsinkbbs.ca :=-
--- Internet Rex 2.29
* Origin: The gateway at The HeatSink BBS (1:134/703)