In conjunction with my ProxHash R&D project,
in late April I started work on a new dataflow automatic-updater
utilty which is somewhat like the Unix 'make' utility.

Background: Unix's 'make' uses a list of dependencies between various
disk files to automatically update a final output file such as an
executable. Typical use is to make sure each source file is compiled
to yield a more recent compiled file, and to make sure each
executable file is at least as recent as all compiled modules it's
built from. The Java's 'ant' utility goes further to support
updating of JAR files containing Class files in a similar manner.

But each of those utilities works *only* in using dependencies
between various disk files, and each effectively runs like a shell
script, starting a new utility (compiler, loader, JAR updater) for
each step in the dataflow that needs to be re-done.

New: My new utility 2008-4-MayLoad.lisp checks dependencies between
both disk files and in-memory data values (in a Lisp environment),
relating in-memory data values with the precursor data needed to
compute them and/or disk files used to back them up to avoid
time-consuming re-calculation. It does all its work from within
that single Lisp environment, not needing to start up any other
executables.

The first draft of my description of the algorithms is here:
<http://www.rawbw.com/~rem/NewPub/MayLoadSpec.txt>
I'm soliciting feedback in two areas:
- Any obvious flaws in the algorithms, especially where I claim
that some fact is "provably true". Can you produce any
counterexample? Based on the described algorithms, can you find
any case where any value is unnecessarily re-calculated or
re-loaded or re-saved?
- Any suggested changes in wording to make the description easier
to understand without changing the meaning.

Also I'm curious whether anybody else has already written a utility
like this, and if not I'm curious whether anybody likes my idea and
might have a practical use for it and would like to beta-test it
later when the code is more stable.

On May 7, 12:07 am, make1.5.CalRob...@SpamGourmet.Com (Robert Maas,
tables.
I'm going to take a look at it. In the meantime, do you know this
paper:

http://cs-www.cs.yale.edu/homes/dvm/papers/lisp05.pdf


I'm very sure I will have use for this, please include me in that
testing phase.

Leslie

make1.5.CalRobert@SpamGourmet.Com (Robert Maas, http://tinyurl.com/uh3t) writes:

Well, it goes further only if you mean >=, but not >.
make is also able to update ar files containing .o files in a similar manner.
You just have to say so, using the syntax: libmine.a(fun.o)

-*- mode: compilation; default-directory: "/tmp/" -*-
Compilation started at Wed May 7 09:34:41

cd /tmp ; cat Makefile ; make clean all
#-------------------------
all:libmine.a
libmine.a:libmine.a(fun.o)
libmine.a(fun.o):fun.c
clean:
rm -f libmine.a
#-------------------------
rm -f libmine.a
cc -c -o fun.o fun.c
ar rv libmine.a fun.o
ar: creating libmine.a
a - fun.o
rm fun.o

Compilation finished at Wed May 7 09:34:41


ant uses checksums, because timestamps don't have enough resolution
anymore. That's even more true when you work in RAM instead of
launching external disk based processes.

I can hear Kenny crying "Cells!" here...


--
__Pascal Bourguignon__

On May 7, 12:07 am, make1.5.CalRob...@SpamGourmet.Com (Robert Maas,

What is "precursor data"? Can you give an example?

I have no way to read PDF files here.

Google Groups doesn't show your full e-mail address, so I needed to
TELNET to a regular NNTP server to pull up your article. I found:
leslie.polzer@gmx.net
I suppose I could have guessed that from your family name,
but it's 4 hours past my bed time and I'm not fully alert.
I'll make a memo of your address to contact you by direct e-mail, OK?

For my current R&D project, there is just one manually-created
input file, and everything else chains from that source.

If A -> B -> C,
then A is a precursor to B, and B is a precursor to C.
If A -> D ---\
then C and D are both precursors to E.

The single input file (at this time) is just a list of descriptions
of Transferrable/Soft skills, a private file which is essentially:
<http://www.rawbw.com/~rem/NewPub/ProxHash/labsatz.txt>
except without the labels at the left margin.
This list of sentences is loaded into memory, and used to generate
two items, the list of labels you see at the left in that file
above, and the parse into a list of words normalized to lower case.
Each list-of-words is converted to histograms of bigrams trigrams
and tetragrams. These are summed to yield the whole-corpus
histograms. The single-record historgrams are divided by the
whole-corpus histograms to yield the ratio histograms. These are
merged into a single histogram per record:
<http://www.rawbw.com/~rem/NewPub/ProxHash/trans-skills-freqrats.txt>
These are normalized to yield:
<http://www.rawbw.com/~rem/NewPub/ProxHash/trans-skills-normfreqrats.txt>
Those are converted to ProxHash vectors:
<http://www.rawbw.com/~rem/NewPub/ProxHash/trans-skills-proxhashvecs.txt>
Most of those lists of intermediate values are attached to
properties on the labels, making them randomly accessible in later
algorithms. The first use of the ProxHash vectors, randomly
accessed via the properties on symbols, is to generate the optimal
graph in 2-d (using just two of the ProxHash components):
<http://www.rawbw.com/~rem/NewPub/ProxHash/trans-skills-links2d.txt>
That graph is then upgraded to successively higher dimensional metrics:
<http://www.rawbw.com/~rem/NewPub/ProxHash/trans-skills-links3d.txt>
<http://www.rawbw.com/~rem/NewPub/ProxHash/trans-skills-links4d.txt>
<http://www.rawbw.com/~rem/NewPub/ProxHash/trans-skills-links10d.txt>
<http://www.rawbw.com/~rem/NewPub/ProxHash/trans-skills-links64d.txt>
That's as far as I have the dataflow utilized so-far.

The thread on ProxHash success has more details about the
methodology, but the above should give you a good enough idea of
the dataflow in general, which is all you need to understand to get
an idea what I'm using my new MayLoad utility for. The use of
normfreqrats as input to compute proxhashvecs is the most
time-consuming task in all the above, so it was important to be
able to save at least those items to disk to avoid needing to
re-compute from scratch every time the modem loses carrier and I
have to re-start my Lisp environment.

moi wrote:
That's definitely not ASCII, the quotation marks are outside of the
range of the ASCII character set.


John
--
Perl isn't a toolbox, but a small machine shop where you
can special-order certain sorts of tools at low cost and
in short order. -- Larry Wall

usenet1.3.CalRobert@SpamGourmet.Com (Robert Maas, http://tinyurl.com/uh3t) writes:
#+sbcl sb-ext:save-lisp-and-die
#+clisp ext:saveinitmem
#+cmu ext:save-lisp
etc.

--
__Pascal Bourguignon__

Robert Maas, http://tinyurl.com/uh3t wrote:

If you are using a Unix command prompt, try "screen". If you loose the
connection, the session is not closed, but you can re-attach it on the next
login with screen -r (maybe depends on nohup settings for your user
account, but works great for root accounts :-)

--
Frank Buss, fb@frank-buss.de
http://www.frank-buss.de, http://www.it4-systems.de

usenet1.3.CalRobert@SpamGourmet.Com (Robert Maas, http://tinyurl.com/uh3t) writes:

You're too pessimistic!

Go to google advanced search, and give the criteria that will select this file:

http://www.google.com/search?hl=en&l...df&btnG=Search

Then "click" on the "View as HTML", or the "Google recommands
visiting our text version of this document." link.

The proof is in w3m:
------------------------------------------------------------------------
Location: http://www.google.com/search?hl=en&l...df&btnG=Search
Web Images Maps News Shopping Gmail more â–¼ Video Groups Books Scholar Finance Blogs
YouTube Calendar Photos Documents Reader
even more »
Sign in

Google [allinurl: dvm papers lisp05 site:cs.yale.edu filetypedf ] Search Advanced Search
Preferences

Web Results 1 - 1 of 1 from cs.yale.edu for allinurl: dvm papers lisp05 filetypedf. (0.27 seconds)
Tip: Save time by hitting the return key instead of clicking on "search"
[PDF] A Framework for Maintaining the Coherence of a Running Lisp Drew ...
File Format: PDF/Adobe Acrobat - View as HTML
Your browser may not have a PDF reader available. Google recommends visiting our text version of this document.
A Framework for Maintaining the Coherence. of a Running Lisp. Drew McDermott. Yale Computer Science Department. PO
Box 208285. New Haven, CT 06520-8285 ...
www.cs.yale.edu/homes/dvm/papers/lisp05.pdf - Similar pages

[allinurl: dvm papers lisp05 site:cs.yale.edu filetypedf ] Search

Search within results | Language Tools | Search Tips | Dissatisfied? Helpus improve | Try Google Experimental

����������������┠�����������������⠔����������������� ����������������┠�����������������⠔����������������� ��������������
©2008 Google - Google Home - Advertising Programs - Business Solutions - About Google
------------------------------------------------------------------------


You must be more optimistic to be able to find quick and easy
solutions to your problems...


--
__Pascal Bourguignon__

John W. Krahn wrote:
It strikes me that the phrase "an ASCII file" has been used as a shorthand
way of saying "a file which is a stream of simple characters". But now that
Unicode is overtaking ASCII in popularity, maybe we need some better
terminology.

People have said "plain text file" for a while, but is that a good term
for something that's in Unicode? Is UTF-8 plain enough to call it "plain"?

I suppose the term "plain text" is ambiguous, so it's sort of a judgment
call whether you co-opt it for UTF-8.

- Logan

På Fri, 09 May 2008 05:12:39 +0200, skrev Logan Shaw
<lshaw-usenet@austin.rr.com>:

It isn't ambiguous to me. It is a file containing only text. The format is
unspecified.
Most editors manage auto detect so they just don't care. If this is a
problem how about writing such a auto-detect library? (Gave me an idea
anyhow.)

LispWorks editor uses a emacs header to determine coding.
Looks like:

;;-*-mode: lisp; coding: utf-16;-*-

In an email header it is spesified like:

Content-Type: text/plain; charset=ISO-8859-1; format=flowed

... and in a web page like this:

<META http-equiv="Content-Type" content="text/html; charset=UTF-8">

Note that content is text/plain for all plain text and seperate from
charset.

--------------
John Thingstad

På Fri, 09 May 2008 05:12:39 +0200, skrev Logan Shaw
<lshaw-usenet@austin.rr.com>:

It isn't ambiguous to me. It is a file containing only text. The format is
unspecified.
Most editors manage auto detect so they just don't care. If this is a
problem how about writing such a auto-detect library? (Gave me an idea
anyhow.)

LispWorks editor uses a emacs header to determine coding.
Looks like:

;;-*-mode: lisp; coding: utf-16;-*-

In an email header it is spesified like:

Content-Type: text/plain; charset=ISO-8859-1; format=flowed

... and in a web page like this:

<META http-equiv="Content-Type" content="text/html; charset=UTF-8">

Note that content is text/plain for all plain text and seperate from
charset.


--
--------------
John Thingstad

Hey, thanks. The way you did it is much better than the way Google
does it. Your way, it's actually legible! How did you do it?

Although that's vaguely similar to what I do, there's a major
difference: I do all my editing on my Macintosh, then copy&paste
across dialup modem to Unix, the only place where CMUCL is
available for me to use, where I do all line-by-line testing, all
full-function testing, and all R&D testing. When I need to restart
Lisp, because I lost my dialup connection or it's another day and
I'm logging in again, I upload (to Unix) all the files that have
changed locally since last uploaded, then start Lisp and run the
initialization sequence to get all active source files loaded into
Lisp. Then I can start re-building whatever data I have in the Lisp
environment. In the past I needed to recompute everything from
original input, so I ran a script for doing that. Now with my
automatic dataflow software, I just call a few functions to request
bringing up-to-date the input for whatever I'm in the middle of
developing, which automatically loads expensively-computed data
from disk instead of recomputing it. Thus I have no need to
automatically keep track of write-date on sourcefiles, and it
wouldn't do any good because there's no way to automatically upload
files from Macintosh to Unix. I only need to watch timestamps on
the data, not the sourcefiles. The system described in this PDF
(now ASCII text) document deals with both sourcecode and data.

A "chunk" per the paper is similar to a "control point" as I used
the term. Perhaps "chunk" is the data itself and "control point" is
the logical record that tells about the "chunk" and determines how
the data will be automatically brought up to date if it isn't
already. Thus from a nitpicky technical point, chunks and control
points are different aspects of the same process, but they match
1-1 so we can talk about either just the same without going astray.

Most of the time a piece of data either is or is not generated or
loaded already, and the timestamp is overkill. The main place where
timestamps would be useful is if I change the definition of how
some data in the chain is computed, such as if I change the
ProxHash algorithm to use a different random number generator. I
would simply delete the backup copy of that one data value from
both Lisp memory and disk, thereby forcing the dataflow system to
re-compute it and re-save it. At that point, since the timestamp is
the date saved, all data dependent on it would show as obsolete and
needing re-computing if I ever ask for them. The timestampes would
save me the burden of trying to manually invalidate each later data
value, and possibly overlook one of them resulting in inconsistent
values.

Yes, that's the basic problem expressed nicely.

Yes. The key is that it's acylcic, else it's impossible to
terminate recursion. This kind of dataflow is very different from
the feedback loops to converge on fixed points of functions during
interval arithmetic calculations.

Conjunctions are easy to understand: One resultant chunk depends
on two supporting chunks. (One control point has two inputs.)

Disjunctions aren't so obvious. Is this like when there might be a
backup copy of the data on disk, whereby the value can either be
re-computed or loaded from backup file depending on which is more
recent, but if output is as recent as latest of supporting and
saved chunk then neither re-compute nor load is needed?

Agreed.

I don't write my code that way. Loading a file doesn't initialize a
table. Instead, loading a file merely makes available the functions
needed to initialize the table, and the functions which determine
under what circumstances the table would need to be initialized. If
the table has already been initialized, the reloaded software won't
have any reason to require it to be initialized again.

This is why I used to have a script that computed all the values
that were needed for my current R&D work, and why *now* I have
instead the automatic dataflow to compute or reload those same
values in a more optimal way.

Note that several years ago I had a weaker form of automatic
dataflow. It used timestamps to load or recompute data as needed,
but if I wanted to save to disk I needed to call that function
manually. It also used two globals per control point, one of which
was the actual data value, and one of which was the timestamp and
other info, both on the value cell of the global symbol. The new
automatic dataflow is essentially a refactoring of that old code to
have only a single symbol per control point, using properties
rather than value cell to store timestamp and data value (and
eventually other info about the control point).

On Thu, 08 May 2008 22:54:19 -0700, Robert Maas, http://tinyurl.com/uh3t
wrote:

Well, basically just cut(select all from within the PDF-viewer) && paste
(via an editor, since my newsreader does not seem to like big chunks).

AvK

Robert Maas, http://tinyurl.com/uh3t wrote:
Assuming you are doing everything at the terminal, which is a fair
assumption if you're using dialup, try out the "screen" program
under Unix. It's a terminal emulator within a terminal. It
supports multiple sessions, which is to say that it can emulate
more than one terminal at a time and let you switch between them.

The reason it's interesting in your situation is that it has
essentially a client-server architecture. When your modem hangs
up, the client will die, but the server (which maintains the state
of the emulated terminals -- and thus all the programs running
within them) remains, and you can reattach to it with "screen -r".

Obviously, you cannot use it to keep your programs alive forever
because there are other limits, but the unreliability of modems
is, I would guess, among the more annoying limits.

Sure there is. Drop to the command line on the Mac and use "rsync".
This assumes you have TCP/IP connectivity between the Mac and the
Unix machine, but that can be accomplished over dialup. Use
"rsync -a localdir remotehost:remotedir", for example.

- Logan

Ah, sort of like what I do at some public computer center which has
Note Pad available (only 2 such places that I know of in all of
Santa Clara County), when I'm looking at a MS-Word document that
I've previously created at the public library
(where MS-IE was the *only* program available until a few months
ago, and even now Note Pad isn't available, but MS-Word has been
available for the past few months, so that's the only way I can
accumulate notes to myself then e-mail at end of session)
with lots of photos and descriptions of where I found them and what
they look like. Except even though I can e-mail that MS-Word
document to myself from the library, I can't view it from home. So
now at the one-of-two public computer lab I want to extract just
the text so that I can view those descriptions from home where I
can't see images or MS-Word documents at all: I load the document
into MS-Word, select all, start a Note Pad, paste into it, which
copies only the text parts and converts the type to TXT, then
finally select-all and paste into MS-IE Yahoo! Mail. It does no
good to copy from MS-Word and paste into MS-IE Yahoo! Mail
directly, because then the images are included and the whole e-mail
is MIME format MS-Word or somesuch which defeats the objective.

Anyway, thanks again for the service, so I don't have to spend
hours commuting to a public computer lab by public transit or
bicycle just to look at one file in the midst of discussion. Too
bad it requires manual copy&paste, so it's not feasible to set it
up as a Web service. (What Google does is crap by comparison, as I
mentionned before: Appx. 1.5 lines of text then a blank line then
1.5 lines of text then a blank line etc. all through a document,
making it totally painful to try to comprehend it.)

On Sat, 10 May 2008 10:05:47 -0700, Robert Maas, http://tinyurl.com/uh3t
wrote:

You're welcome.

My guess is that a wordprocessor-like program has an option "save as",
(sometimes you have to deselect some none-features, agree with losing the
formatting, etc. I don't know if it is scriptable. I can give it a try,
but I am afraid that most of the work will be in avoiding to be used as
an open mail-relay.


Google does what it can. It *needs* the plain ascii just to tokenise.
Reconstructing the formatting is much harder. PDF sucks. Good old
PostScript did an excellent job, and so did ps2asc et.al.

"if it works, it's obsolete" (Marhall McLuhan)
http://www.marshallmcluhan.com/poster.html

AvK

I don't know what you mean by "terminal". My Macintosh, as any
Macintosh since they were first invented, has multiple windows on
screen, overlapping so that I can select a mostly-hidden window by
clicking on some visible corner of it. One window is the VT100
terminal-emulator, others are Finder or McSink. All my editing is
in various McSink windows, not the one VT100 terminal-emulator
window. So it's not correct to say that everything I do is at the
(VT100) terminal, since most typing is done at some other window.

I don't know what you mean by "dialup". When most people nowadays
talk about "dialup", they mean PPP or SLIP or DSL, not VT100. I
don't have anything available except VT100 here. VT100 emulator on
my Mac goes directly through my modem then through voice-grade
phone connection to modem at ISP which then goes through some
TELNET-like digital link to the actual shell machine.

Several people have suggested that over the past several years, but
it has been too much of a nuisance to even consider it, until last
night. Normally the modem loses carrier just a few times per day,
some days not at all for ten hours I'm dialed in, so it's only a
minor inconvenience to have to re-dial and re-establish whatever I
was doing at the time. But last night things got **HORRIBLE**, with
modem hanging up on the average every five minutes, all evening and
through the night until I went to bed. I finally decided times were
desperate enough to try screen. (I didn't see your article until
after I got up this morning, Saturday, so your article was a
coincidence!) But every time I tried to 'man screen', the modem
would hang up before I could get to the important commands I
needed. Finally the modem stayed connected enough to see page 5
where said
If you're impatient and want to get started without doing
a lot more reading, you should remember this one command:
"C-a ?". Typing these two characters will display a list
of the available screen commands and their bindings.
before it disconnected again. So the next time I dialed in I said
'screen' and then tried that list-commands command, but it was
inscrutable, so I had to go back to the manual to see more, except
it disconnected again, so I was back to trying over and over to
display enough of the manual to see how to attach to my
disconnected screen. It looked like -d -r was closest to what I
wanted, even though -D -R was the author's favorite. But I read
further and decided to try just -r by itself, saving -d -r for the
case where -r didn't work because the old screen had failed to
become detached when the modem lost carrier. It had taken a half
hour of re-dialings over-and-over just to find out about C-a ? and
start screen the first time, and it took another half hour
of re-dialings over-and-over before I could finally see that info
about -r etc. and decide what to try, and -r worked fine.
So for the rest of the night I was re-dialing over and over, about
every 2-5 minutes, it was **horribly bad**, and each time I'd
compose one line of code on my Mac while I was disconnected and
then re-dial then screeen -r then copy&paste the one line of code
into screen-CMUCL before the modem would disconnect again.

So now I'm using screen to protect myself from lost of context when
modem loses carrier. Already this morning, modem lost carrier once
while I was composing the previous followup (to moi, the nice
person who converted PDF text to plain ASCII text and posted it),
but so-far it hasn't dropped carrier again while posting that and
composing this new followup to you. So the modems seem to be in
halfway decent shape now. But since I've made the investment to
learn how to use screen, I might continue using it.

One thing about it I don't like: Last night, after I finally got my
new test routine written and debugged and running, where it was
spewing out lots of text (with SLEEPs every so often to avoid
overflowing modem buffers), I was letting it run a few minutes,
then ctrl-C to interrupt it to a breakpoint, then scrolling the
VT100 emulator back to copy&paste the stuff that had gone
off-screen during that batch, then starting it again then ctrl-C to
copy&paste. If the modem lost carrier while it was stopped in a
breakpoint, I'd just finish my batch of copy&paste before
re-dialing. But one time the modem lost carrier while my program
was spewing out, before I could interrupt it, and by the time I
came back it had spewed out much more than a screenful, which is
*not* saved by screen, so I lost all that output and had to stop
the whole process at that point. I spent a half hour browsing the
manual, interrupted by lost modem about every 2-5 minutes, and
never did find any way to scroll back within screen to see anything
that had scrolled off-screen while the modem was disconnected. I
saw some command for starting some mode that saves stuff that
scrolled off-screen, but I couldn't find any way to scroll back
even after that mode had been entered.

I'm not sure that makes any sense, unless by "terminal" you mean a
Unix termcap sort of thing. I guess that's what you mean.

For my purpose, that isn't necessary. I'm content to ctrl-Z and
switch jobs just as I did before starting to use screen.

*now* you tell me "screen -r"!! Last night it would have taken me
three hours of re-dialing to find your article, and I might never
have found it at all because Google Groups would need to be
re-started from scratch each time the modem lost carrier, and I
wouldn't be able to remember where I left off, that's even *if* I
could somehow magically read your mind that you had posted a
suggestion to use screen and you had included "screen -r" within
it. It took only an hour to find the info via repeatedly re-dialing
and re-starting 'man screen', which turned out to be faster and
less frustrating. But thanks for trying. If I had seen your article
just *before* the modem went totally bad, and if it had been fresh in
my mind when the modem did go bad, it would have saved me a half
hour.

Hmmm, still dialed in, modem hasn't disconnected again the whole
time I've been composing this followup, much better than last night!!

Yeah, like the admin here doesn't condone tying up resources that
are not being used for long period of time. It's explicitly
forbidden for PPP users to run a program that automatically
transmits something through the connection at regular intervals to
keep it from timing out the connection and needing (automatic)
re-dial. I think if I were to keep a screen active when I go to bed
at night, or when I go away from home, that would be frouned on
too. So last night, when modem crapped out just as I was getting
ready to go to bed, before I could shut down screen, I deliberately
re-dialed just to properly shut down screen to avoid having it sit
there while I was sleeping.

*** At this point in my composing followup, modem crapped out again ***
Re-dialing at 10:50 PDT ... ISP's dialup phone line is busy, trying
again ... not busy, establishing 19200 bps modem, logging in,
reattaching screen, not 10:52 PDT.

**what** command line???
Macintosh System 7.5.5 doesn't have a command line!!
None of the applications that I've ever used on a Mac have command line either!

No I don't. Didn't you read the part about using VT100 dialup into
Unix shell?? Please read this FYI:
<http://www.rawbw.com/~rem/NewPub/mySituation.html>

The VT100 emulator understands **only** VT100. AFAIK there's no way
to run rsync over VT100 terminal emulator. Only Kermit and Zterm
work over it. And Kermit only half works, because the Kermit on
FreeBSD Unix has a bug: It fails to convert between DOS and Unix
newlines. So before I download any file via Kermit, I need to go
into Emacs to change all newlines to (*) CR-LF combinations, and
after I upload any file from Mac I need to create a one-line
Unix-format file then append the DOS-format uploaded file after it
then go into Emacs which recognizes to change all but the first
newline from CR-LF back to just LF. Copy&paste directly across
modems is much more convenient, so I use Kermit only when I need to
upload or download a really big file that would be too much trouble
to chop into 30k pieces or copy&paste. But in any case, I'm pretty
sure Kermit can't do rsync. And zterm doesn't even exist on FreeBSD
Unix. (man and whereis both turn up empty.)

(*) MODEM LOST CARRIER AGAIN AT 10:57, REDIALING, BACK ON AT 10:59

If I had a machine that could do it I would, but I don't.