Doug McIlroy: McCarthy Presents Lisp
paulgraham.com
paulgraham.com
The "high level language" which shipped on every microcomputer platform of the 1970s and 1980s was Basic. I have often wondered why. Compared to Lisp, it is inelegant and inexpressive. Perhaps people who hacked larger computers of the era did not interact with microcomputer hobbyists. Perhaps it was too difficult to get Lisp fast enough to be usable on the old hardware, although if it was possible on big iron in the late 1950s, it should have also been possible on microprocessors in the 1970s.
An entire generation of programmers grew up using GOTO <line number> to call subroutines and execute loops. That same generation could have grown up using lambdas. How sad.
Lisp would also probably have overloaded the hardware of early microcomputers.
And then of course there is the fact that Basic seemed accessible, and Lisp seemed frighteningly advanced. Even Lisp hackers seem to have thought so in the 1970s.
It was not that bad. In the end, it led to Windows.
Oh wait...
Now, seriously, we used GOSUB for subroutines.
"As a Lisp programmer continued to link simpler functions into more complex ones, he or she would eventually reach a point where the whole program was a function-which, of course, would also be just another list.. So to execute that program, the programmer would simply give a command for the list to evaluate itself in the context of all the definitions that had gone before, And in a truly spectacular exercise in self-reference, it would do precisely that. In effect, such a list provided the purest possible embodiment of John von Neumann's original conception of a stored program: it was both data and executable code, at one and the same time
In Mathematics, the technical name for this sort of thing is recursive function theory, which was why McCarthy called his first public description of Lisp 'Recursive Functions of Symbolic Expressions and Their Computation By Machine' Today ranked as one of the most influential documents in the history of computer languages, that paper established that a language could have a rigorous mathematical foundation. And it signified that John McCarthy had finally come up with a framework that was precise enough, rigorous enough, and compelling enough to satisfy even him"
His paper can be found here: http://www-formal.stanford.edu/jmc/recursive.pdf
I am just amazed how often his name appears on this book (the first time he is mentioned, Waldrop talks about how strange people said he is). But in my book he will always be a legend and a visionary.
However, I don't think that is the main story here. It sure is a big part, but not main for sure. I mean, there were LISP systems in usage at the big boys houses where money/hardware was no object. Later on, Moore law brought us supercomputers of the past to our desktops and we got C++, Java and Python; Even Ada was formed and deployed.
So, what's the deal? My primary focus in life has been computer graphics, and my first language was C. I was born in 1980 and my early serious computer endeavor is inherently tied to the Amiga platform (SAS C and alike), so I didn't have exposure to other language paradigms at all until later.
Later on, through a marvelous turns of event I was exposed to Scheme and at first I didn't "get it" - but there was this epiphany moment or two where I could see almost everything I needed in computer language for my needs, yet it was so simple.
Not to bore you more with details - I always wanted to get into game development. Career of TV and Commercials direction happened in the mean time, and now I am back into making my game development ideal a reality. My weapon of choice is D/Tango, since it really best suites my mindset. However, after seeing a PPT/PDF from naughty dog earlier this year about how they have utilized an embedded PLT Scheme, I came to wonder once again about maybe using Scheme as a scripting language after all.
tl;dr; Can anyone, preferably a pro Lisp or Scheme developer give a rationale why it isn't used more often? Everything I've read makes absolutely no sense why it isn't. Oh, and about the python argument that gets thrown around - I like the python as the next guy, but let's not fool ourselves.
</rant>
* The AI winter (with Lisp, at the time, being seen as "the AI language")
* The heavy-cost and closed nature of the toolchain, and lack of standardization between toolchains, when industry was searching for a language to rally behind several decades ago
* The continued lack of standardization on all the "included batteries" when the open source movement was looking for languages to adopt for scripting a decade ago
* The deep semantic rabbit-hole a Lisp codebase can turn into when any macro-like functionality is used, meaning that a programmer has to basically learn every project as its own language (that is, DSL) before they can get to work—further meaning that projects grow a sense of individual stewardship/artistic direction, and become harder to work on by hiring committees and one-off contractors
* The continued outside impression of Lisps as slow, pure-functional and academic (even though most today are none of those)
* The unwillingness for most Lisps to embrace the Unix philosophy (meaning that it's easier to write a web browser in e-lisp than to loosely couple to one already written in some other language. Clojure is basically the only Lisp that escapes this, cleanly interoperating with the APIs of other languages... on the JVM, which itself throws Unix design principles out the window.)
I'm sure there are more well-founded answers, but all of those came to mind on my road to accepting Lisp.
I've been wondering how Lisp would look if it were redesigned from the ground up to include Unix's philosophy, and I'm beginning to think that Erlang would be a good candidate. (With a healthy dose of Prolog as well.)
Several points have been touched on already, I will mention one: DSL. I agree that the perception of it as a barrier to entry in a project may exist. I think it is as incorrect as tying Lisp to AI, albeit for a very different reason. Indeed, a new member of a team has to learn DSL if one is used in the project. However, new members always have to learn the conventions and patterns used in the project they are joining. DSL makes some of them explicit and more understandable.
I'll go read up on AI winter again, shame I wasn't around to witness that era. On a related note, I just got an email a few days ago for "Third Conference on Artificial General Intelligence, AGI-10 (in Lugano, Switzerland, March 5-8 2010)".
You will need to bite bullets, take responsibilities and own projects and budgets if you want to use lisp at "work". I am doing my current project in Common Lisp, all the way down. It wasn't always like that; I started as project manager for a tiny but growing PHP codebase. The application domain is both too broad, and also error prone; a large web application platform. There are both budget and time constraints and corporate buyers are on our ass daily.
Between conference calls and discussions with the outsourcing company that was doing our project, I decided to prototype it on the side in CL and see how far I can go. Honestly, I prototyped it just to see if I understood the application domain. The programmers where coming to me on an hourly basis asking questions and it came down to the level where I had to spoon-feed them fully specified PHP classes, well documented API designs and database models. These guys were just typing out my sweat and getting paid for it.
I bit the bullet, took full responsibility, looked my boss in the eye and and said it was now MY project.
Right on. What Common Lisp Operating System did you find suitable for what you are doing?
http://common-lisp.net/project/movitz/
Chew it.
[Edit: this, of course, is a wise-ass reply to something I considered a wise-ass question. I develop on Win32 and deploy on Linux]
What did you mean by Turtles all the way down if you actually aren't using a Lisp OS?
What is "Chew it"?
"Chew it", that's my most well-behaved curse word. It's the second person imperative version of "darn-it!" and "blimey!"
Sorry for the confusion:)
Usually when people think Lisp is dead, they mean (b), but you've been here long enough to have noticed the volume of stories about Clojure on HN, so the probability of (a) and (b) seem about equal.
However as for b) - Of course I have seen a trend on HN about Lisp, so what's better place to look for answers than here? It's a safe haven, and that's it. Even if all people on HN wrote nothing but code in one of the Lisp dialects, would that be statistically significant at all? Try to find questions in general for lisp, scheme, cl and variations on stackoverflow for example.
I know you have wrote a successful software where you raved about Lisp - and it's cool and all, but that was back then. I also see people talking about Clojure in volumes, but I haven't seen any real world usage of significance statistical merit.
However, when watching presentation at google from Dan Weinreb - http://www.youtube.com/watch?v=xquJvmHF3S8 or when looking at presentation from naughty dog (for their new engine): http://www.gameenginebook.com/coursemat.html you really have to think about it and say to yourself: 'Hell, why this isn't used more?'. That is all.
I guess we'll see another boom when the major libraries will have stabilized as well...
And if Clojure becomes a widely used language, so much better. :-) That's still a big "if", of course.
If you look at concerns other than langauge, you could easily say that "writing quality software is dead", since most programmers write low-quality programs that barely work at all. That doesn't mean that you shouldn't try to do a good job, even though quality in the industry is "dead".
Don't base what you do on the actions of an ignorant majority. If you like Lisp, use Lisp. You won't be the only one.
But my point is; ignore the industry -- it's a failure. Do what you want to do instead -- you can't do worse than the industry as a whole.
I essentially see two problems with the state of Lisps. First off, the one with major library support, Common Lisp, is often times lacking in good library support. You'll find 10 libraries to do anything, but 5 of them will be undocumented, 3 of them will be dead projects, and 2 of them are incomplete.
The second problem is community. Again, I speak of Common Lisp, because it's the most fully-featured Lisp out there right now. Where is the community of Common Lisp? Who do I go to ask for feature requests in the 'next version'? Do I get some form of support? (I mean, c'mon, just compare it to Python!)
I think that Clojure has solved the second problem, more or less, with it's BDFL. However, I don't think it has the breadth of features that CL has, and being on the JVM brings some of its own problems.
Clojure is getting there, but I think that if a group of, say, 10 people decided to 'popularize' Common Lisp, they could. Just create a distribution of Lisp (say, using sbcl or anything else cross-platform) and ship it with a few helper libraries - since Lisp has macros and reader macros, it can essentially be transformed into the clean language it sometimes isn't.
When I google about the same for whichever of the multitude of Common Lisp implementations exist out there, I have to filter down to which one as well as which platform.
That is a tremendous difference, especially for newcomers.
As for "actual" support, given the focus on a single implementation (which I understand and like that it is not possible with CL), when you need to hire some random consultant service, you don't have to do the same thing when filtering prospective people.
Currently the state of implementations like SBCL and CCL is really good. There are lots of further improvements possible and necessary as the computing landscape evolves. But from a Lisp user perspective, the free implementations have never been better than today. Users should be aware of that and take advantage of that. More work on graphics, development tools, etc. is welcome, though.
I understand that Clozure CL and SBCL are also good. Maybe not THAT good, but good. Thing that bothers me with that argument is if CL or Lisp in general had the appeal to the critical mass of programmers, it would be built - no matter if standards were the obstacle or commercial implementations. They would come and build a new one, no?
As for the libraries - there surely is a way to make bindings to other libraries. For example, we have that kind of problem in D, but we can easily bind to C libraries, so that is not that much of a hassle.
My deep thought on all of that is what bothers me the most. Here is my theory. Is it possible that the real reason is that there are basically two types of programmers:
1. Recipe / Cargo Cult programmers that are programmers by career inertia
2. Programmers that think more about the nature of the problem and hack their mindset to the code. You know, the type that thinks more than they code - code is just a splurt at the end of the process.
If that is true, and if Lisp appeals only to the second bunch (not all of them, of course). It might be a hypothetical disastrous indicator of the ratio in programmer population.
Or maybe I'm just thinking too much about it all.
1. because I seem avoid so much code that I could see myself having to write in other languages, and
2. because it gives me all this great dynamic functionality while still being extremely fast, especially compared to other languages.
Once I realized all the crazy things you can do with it and still run fast its been difficult to imagine using anything else where I have a choice.
This assumes lisp = Common Lisp.
Clojure for example is entirely Open Source and a delight to code in.
Ah, you've come a bit too late. Late 80s through early 90s Lisp machines were popular in the CG industry; modeling, animating and even rendering software were written in pure Lisp. After Lisp machines were taken over by unix workstations, those tools were ported on Common Lisp and as far as I know they kept some share until late 90s, used in quite a few popular titles on consoles such as PlayStation or Nintendo64.
Why have they gone? I think the reason is compound of many factors, and I've only seen the transition from just one corner of the industry; other people have different opinion. One thing I suspect is that Lisp camp didn't have enough resource to catch up the steep increase of demanded quality and quantity as graphics hardware quickly evolved.
Back in mid-90s I used 2k to 5k-poly model, made in the modeler written in Lisp, to make some demo game that run on big SGI machines; such poly count became norm in late stage of PlayStation, and when PlayStation 2 appeared the poly count increased order of magnitude, the trend which still keeps going. The architecture of the Lisp modeling tool back then wasn't suitable for the "next generation" graphics; it needed to be rewritten, but they had hard time to do so. Meantime, bigger players entered into the industry, using legion of developers to pump out authoring tools and middleware.
I believe Lisp can boost productivity, but it tend to work better in a small team tackling one hard problem. When tons of features and optimizations done by lots of developers are required, I guess the power of language probably matters relatively less, and the amount of circulating money matters more. (Naughtydog was probably an exception. The founder was Lisp guy, incredibly good one.)
I suspect that it may be a general trend. When a problem is very hard and only small group of enthusiasts are working on it, they choose tools that are most effective for themselves. Like explorers who go into the wild where nobody ever traveled. Lisp may shine in such circumstances.
Once the wilderness is roughly mapped, small towns are built, and dirt roads are created, then much broader development is required. Lots of developers come to the front and start expanding the envelope. Once the field enters in that stage, the choice of language reflects the proportion of their share in general; many use C/C++, for example, and Lisp becomes minority. (I don't intend to despise the developers in that stage; there are still hard problems and they are still doing incredibly cool things. It's just that the earliest stage, where even what's hard is not really clear and only extremely ambitious people are active, may have different demographics in terms of the choice of the language. I guess Viaweb was also one of such cases.)
In some other areas there was the trend to standardize on Java. Though Java failed (mostly) over time in graphics and simulation.
Still some people use Lisp in graphics related domains. For example CoCreate uses Common Lisp for its CAD package. Recently Lisp has seen increasing use by hobbyists and enthusiasts to write games (using SDL etc.).
I am still trying to use Lisp for these domains. A small piece of Scheme code is running on the server side of one of the metaverse applications currently doing beta.
Xach recently wrote about the use of Lisp in Games:
http://xach.livejournal.com/229485.html
But there is more, I remember for example that there is a relatively large online game that uses Common Lisp: neuroarena.com - a tutorial video is at http://www.youtube.com/watch?v=XFzP6Shxbbs .
This was still the era when PA/Maya was coming out, SI|XSI was on the horizon and packages cost were $15K+. Bad management ruined Nichimen.
There is a common motif amongst Nichimen, Naughty Dog, even Viaweb - they all had a system that let them react to and develop new stuff at levels and speed no one could've competed against. And if you look at the talk from Dan Weinreb, he is basically saying the same thing about them: http://www.youtube.com/watch?v=xquJvmHF3S8
I do remember I cursed a lot about Maya's scripting interface and C++ API, and wished to go back to the old days I could invoke REPL in Nichimen and hack almost everything in it. I ended up making a Maya plugin that allows Scheme to be scripting interface instead of MEL, and after that I used Scheme almost exclusively scripting Maya, but it wasn't as close as the feel in Nichimen where you could open the hood and mess around its guts.
I'm sure it wasn't the first one. LW had MetaNurbs before that, and Catmull had paper before that also - which is funny since Pixars RAT incorporated SubD's only later.
What Nichimen did was show the world a new way of doing things, that how we got things like connect poly shape in maya and other tools which didn't yet have proper SubD support.
Funny you've mentioned working on FF the movie (I still have nice rejection letter for being too young :) ), I still reference that kilauea parallel renderer paper now and then. It was a real feat. I also remember reading somewhere how you guys had only a handful of shaders in your pipeline - first time I've heard about someone actually making and using ubershaders in production.
Nichimen Mirai tech was bough by softimage and later some of the features made it into the XSI (poly bridge and other cool stuff).
Maya is still IMO a great platform, you can extend it to whatever you wish.
Wikipedia (http://en.wikipedia.org/wiki/Mirai_%28software%29) tells me that it
"traces its lineage to the S-Geometry software from Symbolics"
It's easy to get people to line up by ASP.Net. That's the Microsoft way like it or lump it.
It's easy to get people to line up by PHP. That's the popular choice.
It's easy to get people to line up by Jave. That's the non-Microsoft business option. It's the popular business choice.
It's not easy to get people who are dissatisfied with someone elses standards and who are all individully looking for the best option of features they like and want and minimal amounts of the things they dislike, and who are raised in a society of quiet agreement and noisy disagreement, to line up by any particular standard.
I think scheme/lisp is used inside a lot of big systems, such as AutoCAD, Emacs, IRIS Explorer and probably more. One of computer graphic pioneers Ken Perlin wrote a shading language for his own renderer in scheme. But unless you use his renderer, you will never know he put a scheme interpreter inside to process his shading language.
Anyone else find this comment motivating? It kind of sums up a moment of reflection I guess I somehow aspire to. Find an insanely difficult challenge, that you deem important enough to devote a significant amount of time and energy to, and then one day: "It turns out you could do it: I programmed it for the ${preferred architecture}." Thanks Doug^H^H^H^H hold on - didn't Steve Russell implement McCarthy's original Lisp on the IBM 704 - not Doug McIlroy. What am I missing that pulled me away from my moment of zen here?
Speaking of Steve Russell, does anybody know what became of him? It's extraordinary that he wrote both the first Lisp interpreter and the first computer game.
Not sure about Steve Russell, but I agree that is rather extraordinary.