Hi. I'm Jeff Shrager. I mainly hack Lisp, but have dabbled in pretty much everything. I’ve been both a classic and modern AI hacker, but these days I’m hacking atmospheric chemistry. I was also an Adjunct Professor in the Symbolic Systems program at Stanford for over 20 years. DM me @ JShrager at gMail.
Yeah. That's actually one of the things that makes this story so interesting: It took place exactly at the dawn of the inter(arap)net, and physically close to BBN (which was the arpanet implementor), so there was lots of sharing between MIT and BBN, in particular, McCarthy put a lisp on BBN's PDP-1, and Bernie Cosell's Lisp ELIZA, which he did from the algorithm in Wiezenbaum's CACM paper, was the one that became public, so everyone thought that ELIZA had originally been implemented in Lisp, which was wrong!
Not sure what you consider "production", but I used *Lisp extensively on the CM2 at Xerox PARC in the late 1980s. In fact, I published several papers based on this research.
We've just added two new runnable ELIZA adaptations to the ElizeGen collection. One is Steve Cherry's Apple II P-Lisp ELIZA, c. ~1980, put online by Ron Garret. (This one's been around for a while.) The other (brand new ... well, brand old :-) is one built in HyperCard by Brendan Bellina, and runnable through infintemac.org. I highly recommend trying Brendan's version as it speaks its respones and the natural stupidity of ELIZA, combined with :yoda mode and the hysterical voices absolutely kills!
BTW, I didn’t check every detail, but a card reader reading “up to 200 cards/min” sounded way wrong, as I recall it. And it is way wrong! The IBM 3505 could read up to 1200 cards/min.
You (and everyone else) is missing the point of my post. (I admit to having thrown it poorly.) Forget the QC part; It confidently described an algorithm to square a number, which literally any beginning CS student could do, that didn’t even come close.
I will admit to using it all the time for simple programming tasks, and it occasionally does them correctly. Often it comes close enough that I can fix them. (Interestingly in most of these cases I can’t talk it into fixing itself. It kinda gets into wrong-approach ruts), and sometimes it’s horribly wrong (like here).
If it was trained on CS textbooks, they weren't very good ones. I asked it (GPT4) to write a quantum computer algorithm to square a number. It very confidently told me that to simplify the problem it would use two bits. Okay, fine. But then the algorithm it (again confidently) implemented did a left shift (which it reminded me was multiplying by 2, so it definitely intended this!) and then add the number to itself. It then wrote that in terms of QC gates. Tada! It took me a half beat to realize that rather than this being some new version of squaring a number that I somehow wasn't aware of, it's completely wrong. It only works on 00! Confronted, of course it did the usual "So sorry... I guess I don't know how to do this." dance. I don't get why anyone thinks that this thing is worth anything at all, except for cheating on creative writing tests.
I’ve thought many times - like literally every time I use Tf - that I should hack a Lisp interface to it. But, of course, PotAU, I never get a round twit.
The thing that is a verb isn’t “knowledge”, it’s “explanation”. Giving an appropriate explanation (or, more generally, answering a question appropriately) is a problem to be solved. Problem solving (whether one is expecting explicit goal seeking or fluently generating from an LLM that encodes “knowledge”, or some other method) is always an interACTIVE process.
I just finished hand converting ~1k lines of js to cl. I’ve been using lisp for multiple decades, and every time I have to use another language it’s so painful, either per syntax or speed, that if I want to extend the code, it’s often worth the time to convert it. (In this case, I wanted to wrap the core algorithm in an mcmc optimizer, but the js was entangled in its ui, so it had to be untangled anyway so I got both the de-tangled algorithm, and it running over 200x faster by the conversion!) I’ve done this sort of thing before - quite a lot, actually - and macros are very very useful in this setting because you can make lisp do whatever random thing your source language syntax does quite naturally. The other thing that drives me f’ing nuts about almost every other language than lisp is that there’s always some aspect of the (non-lisp) language that fails to have syntactic scoping. Python is, of course, the worst case of this (at least among common languages) - white space is just an insane way to scope because you literally cannot see certain bugs. The one that kept making me crazy in js is the waterfall if-elses they use as case statements. It’s essentially impossible to figure out the scoping of complex nested ifs, except by tracking back every subclause. (Or putting it in a js-smart editor and using the auto-indent, and hoping that the author put line-turns in relevant places, which is what I ended up doing.) This is yet another often unrecognized advantage of the somewhat verbose but highly uniform lisp syntax (or lack of syntax!); Yeah, okay, you do end up with lots of irritating strings of parens at the end of some long code blocks, but you can absolutely always tell in any case by the same uniform operation the scope of every code block and sub block, and better still, reorg them as needed by just grabbing everything in the relevant parens, and blasting it where you want it, and 9 times out of 10 that’s all you need to do for most local refactoring.))))) :-)
I've been a hard core lisper for almost 50 years, and have coded in pretty much every other language in that time, either professionally or for fun. But I always return to lisp, because Lisp isn't a programming language. There are no parens. There is no syntax. There is no spoon. Lisp isn't a programming language, it's hard to describe what it is with a simple term, "meta-language" is the closest I can come, but that doesn't get to it deeply enough. You don't write a program for some problem in lisp, you think beyond the problem to it's meta-problem-space and design a language in which you naturally express prose in that (meta)problem space. Then, given that you now have a natural language (and I mean that in the true sense of "natural language", but not for a human problem space, but for the problems space of the problem you are working on) for that problem space, the problem you were after to begin with is a natural expression in that new language. And you're done.
Ahhhh! A breath of fresh air in this world of ugly Algol-wedged hyper-utilitarianism. (I was going to say “hyper-functionalism”, but the unintended pun confused the sense.) [Although, I must admit that js is actually pretty descent, and can almost be used like Lisp if you look at it hard enough and cross-eyed.]
Although that's structurally approximately correct it understates the influence that SA had, and continues to have (IMHO, see below). Mathematically, Spreading Activation is essentially contextually driven probability integration, sort an unruly version of Bayesian Networks -- iterative matrix multiplies of a weighted connectivity matrix given a set of initial "context" weights. My hypothesis is what embeddings are doing for the GPTs can be understood as essentially spreading activation over distributed representations connected to each token. In classical (GOFAI) SA, per Collins and Loftus, you spread to/from the tokens, whereas in modern SA (by my hypothesis) you are spreading through a high dimensionality network represented by the embedding vectors, instead of via the tokens.
Okay, so I asked ChatGPT4 to "Please program a short pattern matcher in Utero." At first it complained: "I'm sorry, but as of my knowledge cut-off in September 2021, there's no programming language called "Utero". Utero is the Latin term for womb, it's not known as a coding or programming language." But I insisted: "Make up what a language that might be called "Utero" would look like if you were to program factorial in it." Here's what it said:
Designing a hypothetical programming language named "Utero" is a creative exercise, and the language's structure and syntax could take any form. For the sake of this example, let's envision "Utero" as a language inspired by human biology, in which functions are organs and code blocks are cells.
Factorial is the product of an integer and all the integers below it, and here's how it might look in our imagined Utero:
Organ factorial(Num n) cells:
if (n equals 0) then:
give 1;
else:
give n times factorial(n minus 1);
endif;
endcells;
Sorry. The couple of typos don’t help. (I tried to edit them but it was too late.)
Yes, per @fhars, “fmi” is pretty common text-speak for “for more info” (which google will tell you if you search for it).
“Morally equivalent” is a term used in the classical hacker community to mean something like “essentially the same as, although not technically Computationally equivalent”. (And, as I noted below, it is also a standard, if a bit obscure, English phrase: https://simple.m.wikipedia.org/wiki/Moral_equivalence)
I've worked with the various LLMS quite a lot at this point (for someone who doesn't do it professionally ... well, I guess I sort of do ... but not for any of the companies who make them.) Anyway, it's amazing how close these are to what our old (i.e., GOFAI) Spreading Activation models form the 1970s would do. You can wikipedia it FMI, but in practice the SA models would be loaded up with a context, and then the SA would keep it one track for a while, but as you got farther from the priming context it was very very hard to get it not to eventually spin off into some quasi-relevant confabulation. I think that the embeddings is just a modern way of creating a non-discrete semantic net, and the GPT is just implementing the moral equivalent of spreading activation.