A programmable programming language (2018) [pdf]
cs.brown.edu
cs.brown.edu
I know PG claims meta programming was the secret to ViaWeb's success, but I'm not sure how exactly that was the case, and even if true whether that is still relevant today.
I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable.
For example, there's a CL library, iterate, that introduces several new iteration constructs. There are a bunch of complicated macros under the hood, but as an end user I never have to think about them, and the provided interface is obvious, even without knowing anything about the iterate library. For example:
(iterate (for (key . item) in alist)
(collect key into keys)
(collect item into items)
(finally (return (values keys items))))
I'm fond of Clojure for that, but it tends to de-emphasize macros as a good way of doing things. Most things are just regular fns.
IIRC, PG said the advantage of Lisp was being able to add features much more quickly than the competition, so that (a) they could be adding features almost daily that the competition would take weeks or months to match, and (b) whenever the competition added a new feature that looked like a good idea, they could implement it within a day or two, so by the time the press came around to ask them if they had it too, they could say yes.
I also remember him saying that Lisp macros were about 25% of Viaweb's code, so a significant part of the advantage of Lisp was metaprogamming.
> even if true whether that is still relevant today.
It's probably not that much of an advantage any more if your competition is other startups, since they probably have access to basically the same metaprogramming tools that you do.
It might still be a significant advantage if your competition is large, mature businesses whose business model you are trying to disrupt.
> I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable.
I have found this to be true as well. But I also find that I do a significant amount of metaprogramming in Python, and one of the key things to transitioning to more rigorous and maintainable code is to make sure that whatever metaprogramming needs to stay there for the long term is packaged into a library or a tool whose internals won't need to change.
- Writing functions that make functions: often I'll find that I want a family of functions that all follow a the same general pattern. In Lisp this would generally be done with a macro; in Python it can usually be done by writing a function that returns a function. The "maker" function will generally have some parameters that fill in specific parts of the general pattern; the function that gets returned is therefore a closure over those parameters (i.e., it has specific values of them filled in). Doing this with a "maker" function is less error prone (particularly if there are a large number of variations on the general pattern) and follows the DRY (don't repeat yourself) principle, which I think is a very good one.
- Writing decorators that add a particular common behavior (such as logging or debug output) to a variety of different functions or classes or methods. This is a common case. (In fact one could argue that decorators in general are examples of metaprogramming.)
- Using data to construct code. Database programming is a common application of this: for example, I have a database library I commonly use (it's on GitLab here: https://gitlab.com/pdonis/plib3-dbtools) which constructs typed namedtuples specific to a database table's fields, and wraps rows returned from queries in them for easier field access. More complicated database ORMs use similar techniques.
Unfortunately I don't know of any good resources on metaprogramming in general.
It's possible LISP was more complex because you weren't comfortable with using it?
(defun make-incr-functions (numbers)
(mapcar (lambda (n)
(lambda (arg)
(+ arg n)))
numbers))
I can totally see this if you are alone or a very small tight team is working on the code. You will know what’s going on but once you start adding more people it will be very hard for them to work with such a codebase. I have done some really elegant stuff that made me super productive but looking at the code three years later I myself can barely figure out how it works or how to make changes. other people have almost no chance.
Once you grow it’s probably good to stick to paradigms a lot of people in the Industry are familiar with.
Could well apply to me and some of my old C++ projects. Well, except for the super-productive part.
PG would say that it's not that the abstractions afforded by metaprogramming are incomprehensible. It's just that, if you are a habitual Blub programmer, everything up the power curve from you is always going to appear weird and hairy.
This sort of argument is a terrible one to have with generalizations, though. You should look at specific cases where metaprogramming has been used and analyze them on a case-by-case basis. Sure, a bad, leaky, poorly-documented abstraction is going to lead to grief. On the other hand, umpteen thousands of programmers use the metamagic of Rails on a daily basis and barely bat an eyelash. (Or so it seems to me.)
I've found Swift to be extremely good for this. It might be hair slower than python to get a project up and running in that very early stage, but it has a similar feeling of "writing the intent of the program not the syntax of the language" and the type system and compile time checking help alleviate a lot of the growing pains you might run into when a project starts getting medium-sized.
An example of macros you often see is surrounding a block of code with open/close pairs to make sure you don't exit the code leaving dangling resources. File open/close, database connections, etc. Python has the __enter__ and __exit__ protocols to handle that. You can't do every bit of metaprogramming in Python, but over the years they've covered close to every contingency a developer comes across.
The difference between some Lisps and CPython is this: Lisp is its own implementation language and provides lower level control flow constructs, which can be compiled to efficient code.
https://dspace.mit.edu/bitstream/handle/1721.1/113144/101799...
What they do restrict, to various degrees, is the syntax.
So what we're talking about here is custom/customizable syntax to go with your DSL. That can be nice, but it comes at a high cost -- e.g., developers will have to understand the syntax, it will need to be maintained and extended as requirements grow or change. Of course it's worth it in some cases, and tools to help do it are great when it is.
But I just don't see that there's a general problem that we don't have enough custom language syntax to deal with.
Maybe it's just me.
Even with a mountain of constraints, there is still usually far too much subjectivity in software development. This language seems to me to be moving the needle in a very wrong direction. Having a stable basis that is standardized and everyone can (mostly) agree upon is the foundation for growing a technical team or community. Being able to type "LINQ GroupBy Select examples" into your preferred search engine of choice and receive actionable advice is clearly an advantage for productivity. Sure, in a purely academic sense there is almost certainly a more ideal solution. But, if you are seeking to actually get things done, constraints are the name of the game.
> I start to question whether I can trust anything I'm
> seeing on the screen to not be hiding crucial secrets from me.
What could go wrong? Try my cool optimizations!
#define while if // saves cpu cycles
#define struct union // saves memory
lol, thanks for the laugh!
Just built my first single-page website with React and I ran up against the new Javascript ES6 import system. It makes no sense that people keep introducing this kind of unnecessary complexity into their tools. What was wrong with good old "require"?
This is just one instance among many many many. It feels like I'm aging a year every day.
The users of Lisps are often very intelligent single-hero developers as well (why, Paul Graham's Lisp article is a perfect example of it).
And since documentation is ALWAYS terrible for anything except trivial things...
It feels incredibly liberating and joyous to be able to grab any part of your program and without any preceding ceremony just to "eval" it and see what happens.
And let me prematurely answer the comments that probably will be thrown here. No, other languages (most of them) don't have real REPLs. What they have, at best can be categorized as "interactive shells".
Younger folks who are learning, juniors, seasoned developers who for one or another reason, never had any exposure to a Lisp, please do yourselves a favor. Try Racket, Clojure, Clojurescript, Fennel, Chez. Give Lisp an honest, heartfelt attempt to learn it.
Follow the work of people like Harold Abelson and Gerald Sussman, Matthias Felleisen, Dan Friedman, Rich Hickey, Guy Steele, Richard Gabriel, Alan Kay, and many others.
Once you attain familiarity with a Lisp to the sufficient level, perhaps then you will get it. You will see that it is indeed a lifetime investment that would allow you to reap the rewards for the entirety of your career.
I am telling you all this, not from the top of some ivory tower. I never went to CS academia. Like most of you, I am altogether a self-taught programmer. I am honestly begging you not to make the same mistake that I did. For many years I have ignored Lisp due to my self-righteous ignorance. Unknowingly, I delayed my true growth, and I sincerely regret it.
> Try Racket, Clojure, Clojurescript, Fennel, Chez. Give Lisp an honest, heartfelt attempt to learn it.
Compared to actual Lisps, some of these have mostly interactive shells.
> or that title forever belongs to Common Lisp only.
Emacs Lisp, Visual Lisp, ISLisp (an ISO standard), various Common Lisps and their variants, Interlisp, Portable Standard Lisp, ... and a bunch of others.
You are setting your focus on languages which are more or less derived from those (for a reason, often with specific improvements or alternative features), but missing out on the core language implementation features. All the above have LISP in their name, which Racket, Clojure, Clojurescript, Fennel, Chez don't. There must be a reason for that. ;-)
Take Clojure and ClojureScript. Without Common Lisp those would not exist. I can remember when Rich Hickey was an active Common Lisp user (he used LispWorks and we were on the same mailing lists). He developed stuff in Common Lisp, worked on Common Lisp to Java integration, wrote a first sketch of Clojure in Common Lisp and so on.
Luckily he was open minded, learned a lot and created a language (with lots of lessons from Common Lisp), which a lot of people like...
The language you wish is dying, was literally the one where Rich Hickey learned from..., like many people before and others, open minded, will also do in the future.
My own opinion about CL (why I said "I'm kinda glad it's dying"), shaped through influence of other respectable Lispers quotes:
Guy L Steele notably criticized its standard for being over 1000 pages.
Daniel Weinreb criticized it for being Lisp2:
> ... It makes the language bigger, and that's bad in and of itself.
Richard Gabriel:
> “Common Lisp is a significantly ugly language. If Guy and I had been locked in a room, you can bet it wouldn't have turned out like that”
Paul Graham:
> A hacker's language needs powerful libraries and something to hack. Common Lisp has neither. A hacker's language is terse and hackable. Common Lisp is not. The good news is, it's not Lisp that sucks, but Common Lisp.
etc.
just the opposite. When there is lots of interest in Lisp derived languages, the core language stays important, since it's the fountain for many ideas.
> through influence of other respectable Lispers quotes
But you ignore the context. The only real one critical about Common Lisp is Paul Graham, and he got rich and famous through a Common Lisp application: he wrote an online-store system in CLISP and wrote two books on CL, one of which is a real classic: On Lisp. But later he was critical and designed Arc as a very different take on Lisp: small language, small programs, small identifiers, for web programming. It's the one which was used to develop this website here on Hackernews.
Guy L Steele, Gabriel and Weinreb came actually out of a tradition where they used Lisp systems which were MUCH larger than Common Lisp. Steele literally worked two decades with and for Maclisp and Common Lisp. Steele wrote the first two defining books on CL. Steele also has a much larger scope: he worked on things like Scheme, High-Performance Fortran, Parallel Computing with the Connection Machine (which started out with a parallel Common Lisp), C, Java, Fortress, ...
Weinreb was working at Symbolics on and with Lisp Machine Lisp. He co-wrote one of the first object-oriented databases, in Common Lisp. Later he took those ideas to a C++ based OO-database. Years later Dan was back working with Common Lisp at ITA where he spent lots of time... I met him once in Hamburg at a Lisp meeting and he was happy then, working with SBCL and Clozure CL. Unfortunately he died much too soon.
Gabriel worked on the Common Lisp design. He also wrote a book on benchmarking Lisp. Then published a critical paper on Common Lisp and THEN founded a company which developed a wonderful implementation: Lucid CL. He also was then working on the CLOS standard proposal.
You are totally missing on the background and what these people actually were working on. Since the broader Lisp community is very diverse and always was, there were and are always critics in all directions. Some people think modern Lisp is too static, other think it is not static enough, some think it's too interactive, others think its not enough interactive... but you need to understand the context.
Steele, Weinreb and Gabriel were actually the first designers of Common Lisp - the so-called 'gang of five' included also Fahlman and Moon.
It's like claiming that Tolkien was critical of The Lord of the Rings, because it has too many pages, took too long to write it and there are lots of new fantasy books. Tolkien wrote that defining book - like Steele, Weinreb and Gabriel developed Common Lisp and Graham wrote a classic book on (Common) Lisp programming.
There are lots of other things in the Lisp history and I'm happy that some people work to preserve those ideas - not wishing to ignore it and that they are dying.
But using a very dense language with too many features for app development, in practice, often causes more issues rather than solving problems.
Clojure is opinionated and that's a good thing.
Common Lisp was designed to be that on its own, and its actually not that large anymore - compared to similar options.
Everyone who wants to learn a Lisp which stands on its own feets - for example SBCL has only a relatively small C core and all the rest is Lisp itself - should have a look. This has deep effects generally for Lisp: better error handling, better interactivity, images, simpler debugging, less language compromises on the low-level, ...
Lisp is also not opinionated. Working in a non-opinionated languages is different. In Lisp the developer may need to develop his/her own opinions.
Right now you are looking at a shallow image of Lisp.
Lisp programmers tend to spend their Slope of Enlightenment and Plateau of Productivity working in other languages, applying what they learned from Lisp to Python or Ruby or Haskell or Rust -- or JavaScript. And honestly, JavaScript with Babel gets you 80% of the dynamism and metaprogramming capability Lisp has, but also better integration, tools, and libraries than any Lisp. So yeah, most people in the real world will use that.
And there are no "hype cycles" or "peaks of inflated expectations" as you put it. I believe many Lispers do not have any illusions about the real world; we do understand that for the majority of software developers today, Lisp has become a "hard sell," it almost vanished from the curricula of many universities. Now the industry is paying a hefty price for the decisions made a few generations of college students ago.
Lisp is so universal you can find a Lisp dialect for almost any platform today. Knowing Lisp makes it easier to work with any language. For example: whenever I need to solve a problem and for whatever reason, I am not allowed to use a Lisp-dialect; I would prototype in a Lisp and then adapt that to the language that it needs to be. Although it seems to be double the work, I am still faster doing it that way.
> And honestly, JavaScript with Babel gets you 80% of the dynamism and metaprogramming capability Lisp has
Well, I've seen both sides. I've written quite a bit of Javascript. I have used not all; still, many popular kinds of the *script languages that transpile/compile into Javascript - I have used Coffeescript, LiveScript, Typescript, Gorillascript, IcedCoffeescript, Fay, Haste, GHJS, Babel, Traceur, Google Closure compiler, I've looked into Elm, Purescript, and Reason. And without a doubt, I can tell you: Clojurescript today is an excellent option among all these (but that's my subjective opinion based on my empirical observations, YMMV) And your notion about "better integration, tools, and libraries" is a very subjective statement, given that Cljs has a very solid FFI to JS.
I can get into a rant (but I rather choose not to) of how Lisp influenced other languages (including Javascript). How destructuring, transducers, immutable data-structures, etc. first popularized in Lisp dialects and then later sought adoption in other languages.
One of the recent examples: React hooks feature, which gets so much praise by JS developers. Most of them don't even know that Clojurescript devs been similarly writing code for many years already, and it is still a better way to handle state within a component.
Anyway, I don't want to sound like I disagree with you completely, we seem to agree on one thing: first two sentences with which I started my previous comment.