Yes, Dave ..

As I wrote in my follow-up to your identical request
on Usenet, the way to
go about this is to perform the raw extraction on the server. If you view this request as a database table and you wish to extract selected row(s) from said table, the way to go about that is to have the server run some code that extracts the row(s) you want and download them.

This means that ftp itself does not need any real modification -- just smarter servers. I offered the U.K. mirror of Hobbes
as an example, but it
uses Web pages to offer the zip file's contents for selection. While http is somewhat slower than ftp, the difference over a
cable modem is sod all.

You are totally correct, Dave, And the reason I didn't go forward with what was offered there, is that it seems it's more civilized to kick things around here in Fido, so I'll try to do that.

The problem, as I see it is in that last sentence you offered:

While http is somewhat slower than ftp, the difference
over a cable modem is sod all.

Under the conventional way of handling FTP for information and data recovery, we can recover a PAGE or PAGES of data. When we see it displayed on the download end, it is what is in the PAGE that is what we see, Since the content
in the page is accurate, the underpinning file that is a part of it is transparent to the user, but is *NOT* the same. It does not have the same date
and time stamp as the source.

HTTP has no underling method, which I know about, to match the file date and time stamp between the systems, It does match the contents of the file. If you
think about the question I posed, you'll realize that what I was shooting at was replication of FILES, based on and including their exact time and date stamp. The core of the question spoke toward knowing whether or not to get or not get any particular file, or record, as a result of its file stamp time. We
absolutely have to maintain file stamp date and time,and, yes, in some case, even the EA's, just to precisely match the information package between the boxes!

If you look at a couple of threadlettes on this that have been in the Usegroups
too, you'll see that there simply isn't any provision that has ever been set forth for HTTP that will provide this. If you use HTTP to create a maintenace database of files, names, dates and times, for example,to chart the on-going fix-everything for any OS/2 box, or complicated system; you'll fail. That's because there is no way to display the entire collection so that you are *SURE*
that the file you have is the right one.

A prime example of that is what has happened to the various files that make or break the Adaptec SCSI driver operation for OS/2.

There are super-critical applications now coming to focus which have legal requirements concerning this in medicine for example. In short, the FDA here in the USA is very specific. What was stored, in total, has to be what is transmitted, stored, and recovered, in exactly the same fashion as it was archived. 100% lossless is the rule. That's everything, including file dates and times, just so that there is absolutely no question that what was archived,
is exactly the precise same thing that the next person gets, sees,and uses in making their decision on what to do in any given case.

Yes, the other side of this is there too. Once you get it, and you, as qualified party, decide to do something with the data, at the requesting site, it may change form, get bound up in some other file or data form. That's just fine. The issue is that under no circumstances can the sommunications forum, in a quaint sort of way, be allowed to alter the data in any way between two different institutional parties.

The confusion, I think, as to the applicability of all this arrises on not realizing what the rules really are for the transmission between parties!

I'll attempt to explain.

A given institution, internal to itself, may do what it wants to in respect to how it faces eventual liability for errors. Different institutions will have, in every respect, complete information identity 100 lossless between them. It's
been so ordained; case closed.

I request a specific couple of chunks of data from a different institution that
is using even my code, yet operated by different folks. What is requested is going to be just one file of, say hundreds of thousands that are part of an 'Empire Central' archive. It's the file stamp which is the key toward applicatbility!

It may, indeed be shown as necessary in that it has a later date than the one locally. That's how we know it is needed. And that extends down even into the
code locally everywhere too! But that's my choice as the design feller. Not everyone thinks like that at all!

Forget, for the moment, check-sum authenticity and all that. At the moment,we can't even handle any of that, in practice, with HTTP, as I see it. And,as others have illustrated, in answer to my question, .ZIP's have the index of all
the freight cars in the train are in the Brakeman's hip pocket in the crummy! Which is why the railroads all got rid of the Cabooses and park the Brakeman in
the second diesel where the ride is mighty lonely, mighty lonely, if you know a
bit about whatever happened to Jimmy Rogers and so on,dey's a whole lot more lonely brakemen now riding the rails then there ever used to be!

You know, Dave, if only a few requesting sites were involved, and the value of the information could really command a price for archival and retrieval,that would be one thing. However what I contemplate requires that hundreds of thousands of boxes be able to make trivial requests, not often, but when needed
for tiny snippets.

In this information railroad, I think we're all going to be more like that than
many suspect in the near future, but then, that's just my opinion.

In reality, for mission-critical applications, traditional client-server, as I see it, is really a bad setup. As I posted elsewhere, each box of hundreds of thousands of boxes, is an embedded system, as 'tis, even without a keyboard, incidentally! It has to stand on its own. Even if it loses contact with Empire Central, it has to go right on working at the level of intelligence it has in it based on last connectivity. Slowly, over time, it picks up the patterns and nuances of what it needs to do to fight the daily battle from tiny
common snippet files that it needs for update.

But if someone walks in and empties a 45 ACP into whatever server and it can't reach the commander, it goes right on working until it discovers, "Hey I can talk to them again!" Status reports are exchanged, new orders are cut. We are
all happy unless something went terminal during that comm loss that there was no way to have prevented.

That's what lawyers are for.. chuckle.

All this, of course, is what, "On demand", really means, isn't it? And yes, Virgina, for most cases of Santa Claus, maybe a minute max is good enough, in many cases, for yanking whatever little snip you need out of a 100 terabyte tape in an IBM RSC-6000 AIX-400 or bigger. Grotesque? Not hardly. One IBM storage guy here looking at all this, noted, "Well, they have more than 1700 Linux users on a single IBM Mainframe now, Mike, and it isn't even IBM! But we
are a monopoly in the mainframe world, you know."

I said, quietly, "Yes, I know."

He said, "I suppose you do want On Demand?"

I said, quietly, "Of course!"

Further affiant sayeth not. From you and others far better informed than I will ever be, I am trying to learn, trying to learn.

Thanks for whatever time you've spent and can spend on things like this!


Mike @ 1:117/3001


--- Maximus/2 3.01
* Origin: Ziplog Public Port (1:117/3001)

Hi Mike,

Replying to a message of Mike Luther to David Noon:

This means that ftp itself does not need any real modification --
just smarter servers. I offered the U.K. mirror of Hobbes as an
example, but it uses Web pages to offer the zip file's contents for
selection. While http is somewhat slower than ftp, the difference
over a cable modem is sod all.

You are totally correct, Dave, And the reason I didn't go forward
with what was offered there, is that it seems it's more civilized to
kick things around here in Fido, so I'll try to do that.

I agree: Fidonet is far more civilized than Usenet.

The problem, as I see it is in that last sentence you offered:

While http is somewhat slower than ftp, the difference
over a cable modem is sod all.

Under the conventional way of handling FTP for information and data recovery, we can recover a PAGE or PAGES of data.

I don't understand this. ftp shifts entire files around. Did you mean http?

When we see it
displayed on the download end, it is what is in the PAGE that is what
we see, Since the content in the page is accurate, the underpinning
file that is a part of it is transparent to the user, but is *NOT*
the same. It does not have the same date and time stamp as the
source.

That depends on the ftp client embedded in your Web browser.

HTTP has no underling method, which I know about, to match the file
date and time stamp between the systems,

http is used for the interactive part: selecting the zip file to have its directory listed, and to make selections from the list. The transfer should then be done by ftp, which should also preserve the file's timestamp.

file, or record, as a result of its file stamp time. We absolutely
have to maintain file stamp date and time,and, yes, in some case,
even the EA's, just to precisely match the information package
between the boxes!

Now EA's *do* present a problem.

[snip]

What you are asking for is a form of journalling and recovery using ftp, or some moral equivalent. Moreover, you need this processing to be so water-tight that it will keep the FDA [and other government departments around the world] happy as to its accuracy. This is a job for which ftp is not really suited.

Data recovery from journals is a task that is best handled by a program that makes the recovery as automatic as possible. This program could use TCP/IP sockets for its "network tape silo" instead of conventional tape [or other backup] media. But its real smarts have to be in the way the files to be recovered are identified. This should require as little human interaction as possible. The upshot is that some script wrapping an ftp client is like using chewing gum and string where high tensile steel is required.

The reason for this is that the DASD farm of the system to be recovered should not be trusted. Any client system that has been compromised is really untrustworthy. This includes the accuracy of file timestamps. The definitive source of such information should be the journals on the recovery server. This means that the recovery server drives the entire operation.

Data propagation can be handled using the same tools. The journalling activity can be used to propagate files automatically as they change. The recovery utility can be used to rebuild one system on another machine by making the recovery parametric: change the parameter that identifies the "lost" system, rather than run with the current system's id.

For none of this is ftp a really suitable tool.

And that's one reason why Lotus Notes costs an arm and a leg!

Regards

Dave
<Team PL/I>

--- FleetStreet 1.25.1
* Origin: My other computer is an IBM S/390 (2:257/609.5)

Boy, David, when I make errors, do I ever!

Hi Mike,

Replying to a message of Mike Luther to David Noon:

I don't understand this. ftp shifts entire files
around. Did you mean http?

Yes, of course. Wrong term. Stupidity has it's own reward...

Now EA's *do* present a problem.

Yep.

What you are asking for is a form of journalling and
recovery using ftp, or some moral equivalent.
Moreover, you need this processing to be so water-
tight that it will keep the FDA [and other government
departments around the world] happy as to its
accuracy. This is a job for which ftp is not really
suited.

I'm drifting toward the same conclusion. We all become children in our old age, and I keep going back to childhood and hearing old man Wilson in the Missouri Pacific railroad switch tower at about seven years of age for me. He was teaching me how to switch through all the trains with the big steel levers on Sunday's as a kid when I'd discovered the tower! As we'd haul back the levers to signal and switch the Sunbeam and the freights, he'd be listening to gospel on the cathedral radio up there and he'd be echoing with the singer, "Through out the Lifeline, through out the Lifeline, someone is drifting away ........"

;)

Funny how accuracy in progamming logic and the results needed are so similar to
even the mechanical equivalence of the same sort of things....

Data recovery from journals is a task that is best handled by a program that makes the recovery as automatic as possible. This
program could use TCP/IP sockets for its "network tape
silo" instead of conventional tape [or other backup]
media. But its real smarts have to be in the way the
files to be recovered are identified. This should
require as little human interaction as possible. The
upshot is that some script wrapping an ftp client is
like using chewing gum and string where high tensile
steel is required.

"May the circle, be unbroken, be unbroken, in the sky .."

The reason for this is that the DASD farm of the system to be recovered should not be trusted. Any client system that has been
compromised is really untrustworthy. This includes the
accuracy of file timestamps. The definitive source of
such information should be the journals on the
recovery server. This means that the recovery server
drives the entire operation.

But, per the FDA, *ONLY* on request! No Evangelism is allowed!

Data propagation can be handled using the same tools.
The journalling activity can be used to propagate
files automatically as they change. The recovery
utility can be used to rebuild one system on another
machine by making the recovery parametric: change the
parameter that identifies the "lost" system, rather
than run with the current system's id.

FDA don't 'llow no faith-based systems! "Momma don' 'llow no guitar pickers in here!" To which Microsoft (And others.. ) have refrained,"I don't care what Momma don't 'llow, we'll pick that guitar anyhow, since Momma don't 'llow no guitar pickers in here."

For none of this is ftp a really suitable tool.

Of which, "Ridin on the City of New Orleans .. ", is now playing in my mind as I read this.

And that's one reason why Lotus Notes costs an arm and a leg!

A long, long time ago, I bought my ticket at the station on the Rock Island Lines to ride it from North Zulch to Dallas, Texas, one weekend. As a kid,who didn't really know much about such things, I ambled back to the observation car
and opened the door to walk in and have a look. There were,indeed, a few folks
playing cards at the tables, penny a point and, for real, paper sacks on the floor. But the Porter came up to me and said, "Can I see your ticket son? It costs extra to ride in here."

Having none, I had to leave.

On the way home, an old man got on in Dallas with my ex-wife and I, but he was carrying a double barrel 12 gauge shotgun and a fiddle case plus a little bag.
Nobody said a word, nor did the conductor. We wound up in the same car, and as
the train pulled out of Dallas, this bearded old man took out that fiddle and started playing .. no mind to anyone in the coach.

All the old fiddle tunes, you know, Turkey in The Straw ..

You know, he was DARNED good. But more important, that was no fiddle, it was a
violin and a VERY good one. My mother was a concert level pianist, I can tell the difference. A number of us wandered up to listen. When he got tired, I mentioned, "That's a garned good violin!" He said, "Would you like to hold it,
son?" I said, "Yes!" He handed it to me.

I looked, held it just right to the light and looked inside. Strad ... ;)

I thanked him and carefully handed it back to him. He wiped it off and put it back in the case, then turned away and put his hand on the twelve gauge.

Not to long after that, the Conductor came through punching tickets. I quipped, "But the shotgun?" Conductor said, "Oh he's from Jewette. That shotgun's for protection. We all know him and don't care."

Indeed. Yes, he got off the train in Jewette.

Red Glover who used to work for us and had reached about sixth or so in the fiddle contests in Texas was in the shop that next Monday. I asked Red,"Say, you know .. ", and told him the story. Red smiled, said, "Well you've just met
Pat Foley, the fiddle champion of Texas, Mike." That's Red Foley's cousin, Mike."

Indeed.

And that's one reason why Lotus Notes costs an arm and a leg!

Regards

Dave
<Team PL/I>

I think I understand what you've just said.

But one more thing ..

However, as I also recall, when Lotus Notes was rolled out to the USA it was rolled out at the Houston Area League of PC Users, (HAL-PC). I went up and asked the architect after the show, if it should be used for exactly what we are talking about? In that automatic synchronization across a project was what
was being hawked, I thought even then that it was a solution, perhaps.

What I was told was, "No, it is not!" Lotus Notes is designed to be used for collaborative work to develop a project, with this synchronized data need solution. But then it is designed to put that project away and never look at it again over and over as well. It will not support what you want to do, Sir.
Don't try it for a generalized solution for millions of bitty pieces to a puzzle and expect it to work. It can't handle that."

The average X-Ray, to our research, for example, only is requested to be recovered out of archive, after six months, about 10% of the time. But any X-Ray taken of, for example a breast exam for a woman, must be available for life! As another example, an X-Ray of an undescended testicle taken of a small
boy, must be available until that child reaches at least the legal age of adulthood, whatever that means.

Conversely, the X-ray's of a certain star which might prove murder by overdose of a sedative in an unusual location, are wholely improper if they are pushed into every medical record file in the tabloids, even years later,if they even exist! Momma don't 'llow no push technology round here...

Now, when I got the thrill of hearing Pat Foley, the song, "The Devil Went Down
to Georgia!", hadn't ever been written yet!

But Dave .. the Devil *IS* in the details. If I'm absolutely entitled to know and have proved without a doubt that I am what I am, and querry this database for one of literally billions of transaction piecoids, to coin a term, is Lotus
Notes really what is needed. Not only is it necessary to know that you do have
a Scarlet Letter on your forehead, but how you got it,every blessed detail .. every transactio, every transaction!

;)

So is it really Lotus, or is it DB2, or not even written ... yet ...


Forgive the literature . It's a fatal flaw in Mikey.

Sleep well; OS/2's still awake! ;)

Mike @ 1:117/3001






--- Maximus/2 3.01
* Origin: Ziplog Public Port (1:117/3001)

HTTP has no underling method, which I know about, to match the file
date and time stamp between the systems, [...]

As I recall, HTTP has a Last-Modified: header. One simply needs an HTTP client
that recognises this header.

If you use HTTP to create a maintenace database of files, names, dates and times, for example,to chart the on-going fix-everything for any OS/2 box,

or

complicated system; you'll fail. That's because there is no way to

display the

entire collection so that you are *SURE* that the file you have is the

right one.

Why not ? If the problem is maintaining the last modification datestamps on the files at the client end, then that's easily solved.

It's also worth noting that HTTP has a mechanism for retrieving files only if they have been modified later than a given date. Caching Web browsers, such as
Netscape, use this mechanism all of the time.

» JdeBP «

--- FleetStreet 1.22 NR
* Origin: JdeBP's point, using Squish <yuk!> (2:257/609.3)