Why Racket? Why Lisp? (2014) | Hacker News Reader

Why Racket? Why Lisp? (2014)

practicaltypography.com

242 pointsdgellow164 comments

It's about a month since I've started seriously diving into lisp.

The last couple of weeks I've spent my days reading 'The joy of Clojure', Structure and Interpretation of Computer programs, lots of tutorials and documentation, playing around in the repl + experimenting with all kinds of frameworks and libs in clojure (eg. Om).

I've spent today implementing the brainfuck interpreter in Racket.

I can't explain it, it's like something is calling me - 'learn lisp. now.'.

Given the amount of Lisp code read/written and the relative novelty of it, I'm dreaming lisp code and the arguments to functions are actual physical things, which are then mapped, reduced, recursed or expanded.

Literally, I think I'm going crazy.

I can fluently write x-platform C++, Javascript/CoffeeScript, Objective-C, Java, Pascal, VB and everything in between.

But never have I experienced this kind of mental strain/obsession as I do now with lisp.

But.

Seems like I'm not alone! Given the amount of lisp news lately on HN, I fell like more and more people are going through what I'm going.

Seems like we here on HN follow a common mental pattern and suddenly everyone's talking/learning lisp.

As interesting as learning lisp is, this 'group preference' thing is even more interesting to observe.

It's not so sudden. HN itself is written in a LISP [1], and Paul Graham's pre-YC claim to fame was Viaweb, a store platform written in LISP that became Yahoo Stores. This community has been above-average LISP friendly for a long time.

LISP's also has a long history of popping up all over the place because they are extremely implementation-friendly (if you want to write a simple language; if you want to make things fast they get trickier, thought by no means impossible) given the simple syntax.

So you'll also find a lot of people (like me) who don't really like LISPy languages, but who still end up dabbling with related technologies occasionally (my very-slowly-in-progress Ruby compiler uses s-expression syntax (though no LISP semantics to speak of) to express a tiny language to implement some of the lower-level plumbing to bootstrap the Ruby core classes, for example)

[1] http://arclanguage.org/

It's spelled 'Lisp'. There has been zero overlap between people writing "LISP" and people writing lisp code for about half a century.

That's really overstating it, especially seeing as Lisp has barely existed for half a century, and people like ESR call it "LISP" to this day.

AFAIK the allcaps "LISP" fell out of favour in 1984 when Common Lisp was introduced, and only the original Lisp implementation and Mac Lisp used the allcaps name.

Much like Fortran (no longer FORTRAN), the acronyms became proper names in their own right.

Someone pointed out that Common Lisp is case-insensitive by default, so really they have no grounds to object to the name LISP. ;-)

English is the relevant rule set here, not Lisp :)

The CL reader by default[1] converts unescaped characters to uppercase, but only when parsing tokens. The CL printer by default[2] converts everything it prints to lowercase. CL people seem to adopt these conventions as well, although they have been known to :capitalize Lisp on occasion.

[1] http://l1sp.org/cl/2.2

[2] http://l1sp.org/cl/*print-case*

    CL-USER 30 > (eq '|LISP| '|Lisp|)
    NIL

Common Lisp knows very well the difference between LISP and Lisp

You're wrong, and you need to get off your high horse. People use "LISP" when referring to the general family of LISP-like languages. And people that do this tend to know about Common Lisp. That's why they do this.

> LISP's also has a long history of popping up all over the place because they are extremely implementation-friendly

This is a LIE.

I get tired of hearing this over and over. Broken Lisps are easy to implement.

Try implementing a real s-expression parser in C. I'll wait.

For example, this library: http://sexpr.sourceforge.net/ takes 5000+ lines of code. And it doesn't even handle dotted pairs! (For good reason, handling dotted pairs without recursion is extremely difficult.)

I have written entire languages in that number of lines. So, have many others.

Lisp does not count as small.

I'll be the first to agree with you that most 'minimal' lisp implementations are going to be missing important features (efficiency, memory management, ..), but is the parsing of s-exprs comparatively a difficult task?

At just a glance, it appears that the library is slurping the expression and avoiding recursion instead optimizing for speed, memory usage, and error reporting, but it seems like you could implement a recursive descent parser if you weren't as concerned with those. If the author took their same approach to parsing s-exprs for a grammar like C, I'm not sure how successful they would be.

I haven't really looked at grammars / compilers in a long time, but I'm curious if I'm off-base in thinking s-exprs are relatively easy to parse. Some grammars like brainfuck or assembly would obviously be easier.

There is nothing difficult about writing an s-expression parser. None of the plethora of Scheme interpreters out there get parsing wrong. Handling dotted pairs is rather trivial compared to, say, getting hygienic macros right. Or implementing call/cc. Quasiquote handling is not even that difficult.

> There is nothing difficult about writing an s-expression parser.

Really? Try it in C.

The Scheme interpreters do very limited dotted-pair parsing until they bootstrap into scheme. There is a good reason for this.

And this is before we even start dealing with things like circular references.

Every once in a while, I like to sit with a cup of hot tea or coffee, in my bathrobe, in read about Clojure. Algorithms, ideas, data structures - it's so .... just...clean and well thought out. There's something really nice about LISPs, and it's drawing me into it as well.

I still do C#/Javascript/Java/whatever in my day job, but LISP is just...sexy.

And for those that hate parens - Emacs paredit. Nuff said.

>parens

If you include the curly braces, I'm not sure C/C++ has fewer parens than Lisp does, to be honest. Not much fewer, anyways.

It's the pre-fix parenthesis for scope that is the problem, I think.

function {} is more familiar mentally than (function).

Reminds me of this article: http://www.lambdassociates.org/blog/bipolar.htm

Yes. I've read that article several times. I fit the description perfectly and I think it's one of the reasons I'm learning lisp now. It might be of use for the huge project that I'm about to start in the near future.

I don't like the dark side, I hate to see all the beauty I create when I'm 'high' dribble through the impotent hands of the dark periods. But I've come to accept that this is how I am and always will be.

My only way out of the hole is to try and steer clear of addictions and learn new stuff.

Since I can't do anything of practical use during this period - nothing is exciting, the world is going to hell, I'm a failure and cannot do it, etc, I use the time to learn new stuff, read books on every possible subject and just try not to be too big of a drag on those around me.

I've been here multiple times and I know that each 'down' period is followed by a rush of energy and creativity and I have to be prepared with knowledge and health so that I can apply it during my 'mental spring'. This is how I've created great stuff before and I will in the future. At least this is my hope.

At the moment, learning lisp is a kind of psychotherapy that I apply to myself and it seems to be working, even though I have no idea what to do with that knowledge, it will soon all make sense.

The GP article seems like self-congratulatory fluff. So, just to be clear, what you're saying is that you're brilliant and this is a problem?

Whatever 'brilliant' is supposed to mean, when I try to apply that adjective towards myself I'm immediately reminded about the numerous times I've felt humbled by someone else's infinitely higher brilliance (be it a book, source code, mathematical expression, song, poem, movie, etc) and then I just feel stupid for trying to place myself even close to those I so deeply respect.

Brilliant is impossible to be, what we can hope for is moments of 'brilliance', where one can come up with original solutions to problems. Ideas, revelations..

I do have those from time to time and it feels great.

But as a human being, it does matter what and how you feel in between those moments of brilliance.

And during a 'low' phase (which can last months, day after day), all your moments of brilliance are replaced with mental blocks, melancholy, cynicism and pessimism about everything and everyone. It's a very bad place to be, it's painful, the feeling of being a fraud, the way you disappoint yourself and others by not following through, the way you see your dreams and hopes drown in a sea of confusion and frustration. When everything you've earned/gained is slowly erased and you have to start again from step 1.

Yeah, I think that's a problem. You either kill yourself, drink/drug yourself to death or find a way to cope with it. My solution is to just obsess on books/studying new stuff when the mental winter comes and try to stay away from the easy escapes.

When you start to reach to point where metalevel, evaluation, transformation, compilation all fit into your mind you feel even weirder.

>But never have I experienced this kind of mental strain/obsession as I do now with lisp.

That's why it's so valuable as a right of passage. If you've spent your career writing simple iterative imperative loops, suddenly you're a newb again with map/filter/reduce this, partial that, and recursion and lambdas and closures and finally after your 20th time staring, dazed at 3 measly lines of nested composed recursion - it all starts to flow, and boom, you have a new way of approaching every problem.

And even if you don't live in the lispy functional realm after that you'll always be better off for having crossed that threshold.

* rite of passage

Try typed functional languages next. You'll be a newbie again.

Indeed. The language that would eventually become Racket was my first true love, but my first encounters with the ML family were revelatory as well. I'm wondering, though, if I should go back to Racket again and see if there's anything new to learn there now that I'm older and more used to infix notation.

Knowing me, though, I'll probably just recklessly abuse hygienic macros until the sun burns out.

Very true. I got into Racket three years ago and had an experience similar to the OP. Then I did some Clojure and I loved the data structures and destructuring, but I'm not too fond of the Java bits, although I understand the benefits. Now I'm learning Haskell, and it's another world again. I'm glad I took the route I did because the Haskell is coming a lot easier than it would have without the Racket/Clojure background.

Right now I'm converting some old Clojure code from a couple of years ago into Haskell, and I much prefer the Haskell static type checking to the dynamic typing of Clojure. I still have a lot more to learn to get my Haskell powers up to what I could do in Clojure, but I think it's worth the work.

I went through a Lisp phase exactly like that. Currently I'm going through an extremely similar Haskell phase. I've learnt half a dozen languages before these two, but I've never experienced the satisfaction and obsession I've experienced with these two. My Lisp phase went on for the better part of a year, and I've been learning all the Haskell I can for 11 months now, but my obsession is still in full force. Haskell also turned me towards maths, and now I'm reading a book about category theory and a book about type theory.

Yeah, I feel that Haskell might also have this ability to energise a bored mind. The purity, the cleanliness, the power. Definitely on my list. This year, though, I feel will be the year of the 'great lisp project' for me.

My first exposure to Lisp was through a combination of Hacker News, SICP, and Paradigms of Artificial Intelligence. Also, there may have been a blog post by one of the Posterous (or a similar company) guys about getting started with SBCL for web services.

At the time, although I could read the material, it seemed to require too much mentally to actually Get Things Done. I think it was a deep misunderstanding about how to structure a program that is independent of state.

Later, after playing with Scala, amongst other languages, I came to understand that state exists in functional programs. It's not conjured up out of nowhere, spontaneously brought into existence, but rather, it exists as data, and it is the job of the program to direct the flow of data appropriately - like a plumbing system, so to speak.

I had this experience, too, years ago. I felt like my brain was getting rewired. I loved it. I also noticed that I was thinking in Lisp relatively quickly. To this day when I have a really hard problem I want to solve I write the code in Lisp because it makes hard things easier to understand. Then I implement it in the language I'm using.

You begin thinking "naturally" in Lisp because there's no syntax to memorize, it's all data and / or semantics. You can literally write the code in your head. Languages I've used for years and years don't do this to me like Lisp does.

It also made me feel more confident about programming in other languages and not to get distracted by "fancy" syntax or features that are just warmed over scraps from the floor of Lisp's feast on the table.

"Paul Graham’s programming language Arc, a dialect of Lisp, was built on top of Racket."

Since that statement occurs in a section about Racket languages, I should probably clarify. Arc isn't technically a Racket language—that is, it doesn't use any of Racket's facilities for defining new languages. It compiles to Racket (and so you're all using it right now), but has its own distinct implementation.

Historically, Arc wasn't built on top of Racket. I think pg may have started on Common Lisp, then went to Scheme 48, Mz Scheme, and so on. Probably the most accurate thing to say is that Arc was built in pg's head after years of thinking about Lisp. It doesn't have a particularly close relationship to any of those underlying platforms. Semantically it's closest to Common Lisp, but don't tell pg I said that.

I think it compiled to MzScheme first.

MzScheme came comparatively late. Here is a post from when it was Scheme48 and after it was Common Lisp:

http://www.reddit.com/r/programming/comments/bstl/arc_lisp_d...

A friend told me he used to track developments in Arc by noticing when pg had switched mailing lists. :)

MZ Scheme was a precursor to Racket. Same people, AFAIK same core codebase.

http://docs.racket-lang.org/mzscheme

Good discussion of the advantages of Arc versus Racket. http://stackoverflow.com/questions/8555440/the-advantage-of-...

"I wish they had implemented Arc as a Racket module language since then you could actually develop in DRracket, debug, and make executables."

That discussion doesn't seem very good to me, though there are some nice examples of Arc's notation at the bottom.

Our own kogir actually worked for a while on implementing Arc as a Racket language. It would have some advantages, especially around tooling. However, like all such platforms, Racket has a sweet spot (what's easy to build on top of it), and the further your semantics are from there, the harder it quickly gets. Redefining how null, true/false, lists, and macros work—as Arc does, relative to Racket—is not how a language platform is meant to be used! You end up fighting with it in a way that loses most of the advantages, and the difficulty seems to grow asymptotically as you approach 100%.

What you want instead is to stay in the sweet spot and let the underlying platform nudge the new language in all the directions that are easy to implement. The same is true of building transpilers—it's an order of magnitude easier when you can piggyback on the semantics of the underlying language. But this is not an option if your source language has already been defined elsewhere.

I think you're assuming that a language built that way would have to adopt Racket's underlying meanings for things, probably because Arc is, in the grand scheme of things, pretty close to Racket. But because Arc makes some fairly fundamental decisions that are different than Racket (such as how macros work), you can't keep those the same -- Arc macros can't be just Racket macros.

However, just because Arc can't be a layer of macros on top of Racket, that doesn't mean that the language can't be implemented as a Racket language. There are Racket language implementations of Algol 60, Python, and JavaScript, all of which are very different from Racket, more so than Arc is. And those implementations get the tooling benefits that Arc today misses.

It's easier to write a simple interpreter than a compiler, and Arc's current implementation seems to be working well for you, so far be it from me to tell you how to run your project. But certainly it could be implemented as a Racket language, no matter how far away it is.

Hmmm. The syntax stuff could easily be implemented using macros. I like using rackjure by Greg Hendershott:

http://www.greghendershott.com/rackjure/

Adding the if statements wouldn't be too much fuzz.

This is Matthew Butterick. I wrote “Why Racket? Why Lisp?”

As I allude there, Paul Graham’s writings about Lisp (mostly in Hackers & Painters) helped persuade me to explore Lisp languages. (Those writings have also persuaded many others.)

In particular, Arc's reliance on Racket persuaded me to take a serious look at Racket. So leaving aside quibbles about what “on top of” means — is Clojure not built “on top of” the JVM? Python “on top of” C? — Paul’s choice of Racket was influential in my choice too. (As it has been for many others.)

As for software being “built in [one’s] head,” that seems facially true of any software. The core thesis of “Beating the Averages” is that the tool you choose to get it out of your head and into the world matters. Having now had my own Lisp revelation, I not only buy Paul’s thesis, but I even think it could be strengthened: Lisp permits the implementation of a whole category of ideas that aren’t possible in other languages.

Moreover, Paul wrote that essay nearly 14 years ago. Since then, Lisps have gotten somewhat more popular (Clojure has led the pack). But as I say in the article, as a group, Lisps remain way behind the programming mainstream. So ultimately, my goal is not to evangelize for Racket and exclude other Lisps. I know Racket better because that’s what I use. But more people using all of them would be a great thing.

I'm curious what data supports the idea that Lisps have gotten more popular.

The total number of developers who know a Lisp may have grown, but the total population of developers is also rapidly growing.

Measured as a % of language popularity, I'd expect it is largely flat, despite awesome mass-media efforts like Seibel's Practical Common Lisp http://www.gigamonkeys.com/book/

How many conferences would you have seen ten years ago with a prominent lisp showing? IIRC even the primary (at the time) ILC conference was only held every other year.

These days with more regional conferences starting it's nearly in the twenties, and attendance at Strange Loop (which has had lots of keynotes by lispers) is several thousand and growing.

I couldn't agree more!

Previous discussion can be found here: https://news.ycombinator.com/item?id=8206038

Have noticed quite a bit more buzz about Racket and Lisp very recently around here. It's always been something I mean to explore.

I think it's because of this: https://news.ycombinator.com/item?id=9220468

Yes, if John Carmack built something with Racket and liked the results, it does pique one's interest. There are a lot of tools and languages to evaluate and so little time.

We seem to be on a Racket buzz lately, and I like it. I feel more inspired to dig further into Racket.

I'm already pretty good with Clojure, so can anyone who knows both languages well comment on the ups and downs of learning Racket after learning Clojure?

This is a blog post by technomancy, the original author of Leiningen, exploring Racket as a new language:

http://technomancy.us/175

EDIT: Wrong link. This one might be interesting as well so I've left it here:

http://technomancy.us/169

The TL;DR I would give is this: Clojure is preferable when doing server-side performance-intensive software or in situations where its JVM library reach provides a significant advantage. Racket is a lot nicer for client-side stuff where you need a fast launch, lower memory profile, a GUI, or distributable executables.

If you already have a solid development environment set up and have spent a lot of time learning project management tools, you might have an easy enough time getting started with Clojure, but getting started with Racket tooling is easy even for children.

Clojure is a lot better out-of-the-box for FP, but Racket has third-party tools (like Rackjure and fector) that do a lot to close the gap. On the other hand, Racket has pattern matching out of the box which is really nice, but you can get that in Clojure as well using external libraries. One problem that you can't fix in a library is Clojure's omnipresent nil; Racket does a much better job of ensuring sensible semantics in error conditions, partly due to the fact that pattern-matching is built-in.

  Using either still keeps you in the Lisp universe.

Also, Greg Hendershott's rackjure library emulates a few "Clojure-inspired" ideas in Racket https://github.com/greghendershott/rackjure

Here is a talk by David Nolen comparing Scheme and Clojure (and my summary of it):

http://www.wedesoft.de/clojure-for-schemers.html

Actually, Racket and Clojure share very little in terms of syntax and semantics. Lisps are like that: they all have parens and frequently that's the end of similarities. Semantics of Common Lisp, Emacs Lisp, Scheme, newLISP - not to mention Shen - are very different. You should treat them as separate languages, especially when learning. So, in short (I went Racket->Clojure), there are no problems if you remember not to write Racket in Clojure and vice versa :)

I go to my local Clojure meetup. Last time I sat next to a guy who uses Racket in his day job. He said "If I didn't know what the problem was and was asked to pick the language beforehand, I'd pick Racket. I still like Clojure a lot." Didn't get a chance to unpack that with him.

How production ready or mature are Racket libs for creating CRUD web services? skimming through the Racket website one can see Racket has the basics, a http server and a library for interacting with databases like postgresql, as someone relative new to programming/webdev, would I face to much "reinvent the wheel" versus Clojure or Javascript?

Depends what you're doing. It doesn't have the depth of libraries of the JVM or JS, so there'll be lots of "oh, can't use that technology" or "is this 3-year-old unmaintained library still going to work with Mongo?" moments.

Why hasn't XML been matched to Lisp very much? I think it's because Lisp programmers recoil in horror from XML. I mean, if you're already used to S expressions, XML looks disgusting. As verbose as Java, and less expressive. So I think most Lisp programmers look at XML and think "Ugh, don't get any of that on me..."

IIRC, it was Naggum that said the issue with XML is the useless distinction between elements and attributes, leading to countless discussion, bikeshedding etc etc instead of actually solving problems.

See the resources organized under XML and Scheme, by Oleg Kiselyov:

http://okmij.org/ftp/Scheme/xml.html

There is a Racket "package" (not quite a port) of SXML and SXSLT:

http://pkg-build.racket-lang.org/doc/sxml/

Tho in its infancy, Kawa scheme can "paper over" some of the XML in Android dev[0].

[0] https://www.gnu.org/software/kawa/Android-view-construction....

Has anyone tried to use Butterick's Pollen and written about it? The idea of debugging your website with racket as you write markup and styles seemed really interesting alternative to what I do; write styles, refresh, and then refine the stylesheet by incrementally adding changes with the inspector and then copying them to my stylesheet.

I'm working through the admittedly beautiful and very thorough documentation [1].

It's a very simple idea, really: you write your document and sprinkle racket s-expressions wherever you want. You can put your definitions at the top of your document, or in another file. You describe your templates in Racket also, because like Butterick says in his RacketCon video [2], "S-expressions and XML are the same thing". You can use tags that you haven't defined, they just get placed in the resulting HTML.

You still have to roll your own CSS and any JavaScript. As far as I can tell it doesn't help you out with things like keeping track of footnotes numbering or citations and such: you roll your own for things like this, Pollen is no LaTeX. This is partly why I say it's a really simple system. It makes a certain set of web authoring things simple, but it doesn't try to be a one-stop shop, which is excellent because this makes it a supremely flexible tool.

It also has some backend fancy sauce where you can save your file and refresh the page [3]. Also I admit I'm not really a big fan of DrRacket: I've just been using it for the tutorials because I have no idea what to expect from Racket, but I'm slowly moving to Emacs, and Racket seems to work fine there.

Definitely worth checking out Pollen if only to get a light introduction to the thoroughness of Racket documentation (they definitely do things differently in Racket-land!), but stay for the tools you need to roll your own ultimate static blog generator.

(Also, completely unrelated: cool video and cool title at another RacketCon 2013 talk: "Racket on the Playstation 3? It's Not What you Think!" [4].)

[1] http://pkg-build.racket-lang.org/doc/pollen/

[2] https://www.youtube.com/watch?v=20GGVNBykaw

[3] I've made a request for no-refresh updates: https://github.com/mbutterick/pollen/issues/35

[4] https://www.youtube.com/watch?v=oSmqbnhHp1c

> You still have to roll your own CSS and any JavaScript

True, though you can automate those files (and any other text-based files) with Pollen as well, so you can use common functions and data across all of them.

Also, here's the previous discussion of Pollen on HN:

https://news.ycombinator.com/item?id=7822057

The documentation is an absolute pleasure to read. I think I'm beginning to understand the use case for it more. I would definitely choose it if I had a large source text that I wanted to publish. I have to write the docs for a project I'm working on right now, maybe I'll get a chance to play with Pollen then.

I'm working on converting a fiction ebook to web-based with Pollen and may later launch a fiction magazine using it, although I'm undecided about that technology choice.

I think Pollen's biggest strength is that you can reprogram its markup -- as its documentation puts it, you can attach behavior to tags. I haven't taken much advantage of that yet, but you can do things like create a "TOC" tag that builds a table of contents by inspecting child documents and looking for h1 tags (or looking for, say, "chapter" tags, which you've defined to expand to "<h1 class='chapter'>"), or inspect the contents of paragraphs and subtly shift the first line margin to the left if the first character is a quote mark (which Butterick's Practical Typography does). You could replicate some of that with a template language that allows user-definable tags, but I don't think you could do all of it.

Its biggest weakness, at least for me, has been finding a pleasant workflow. Despite having a built-in web server it feels kind of clunky compared to other static site generators. You're largely on your own for writing a deployment script ("raco pollen clone" is not a valid substitute). The DrRacket IDE is virtually a requirement for Racket programming, but it sucks teabags for editing long prose documents; you'll likely find yourself working in one editor for Racket language files and another for Pollen source. This isn't necessarily a dealbreaker, but it's at the least annoying.

Thanks for the response. I might try converting some public domain text to Pollen sometime, any tips for getting around DrRacket's short comings besides separating racket and pollen source?

That's probably the main one, really. DrRacket really isn't a bad IDE -- it's not up to Emacs levels of programmable madness, but its keybindings are clearly inspired by Emacs and it has a lot of Lisp-aware functionality. But, its syntax highlighting engine seems to be very slow and hard to really turn off.

If you are an Emacs fan, you can use it for Racket instead of DrRacket using Geiser and Quack, which might solve a lot of problems. (I've never tried it -- I'm comfortable with Emacs, but I've never loved the way it handles prose rather than code.)

FWIW I go between DrRacket and Sublime Text as necessary. Sublime is of course faster for typical writing & editing tasks. But when working on code-related things, the DrRacket REPL is useful.

That makes sense. I think I was using Sublime as well -- I experimented with making the world's hackiest syntax highlighting file to show embedded Pollen commands. (IIRC, Sublime was unamused with the notion of ".html.pmd" as a single extension.)

As someone who thinks Lisp is pretty cool and uses Emacs to study it and is about 100 pages into SICP: I still don't get it. Common Lisp macros went entirely over my head too. A programmable programming language? Aren't all languages like that? I've only been programming for a couple years so I'm afraid I might not recognize the value of Lisp until I use more of the "less powerful" alternatives.

The principal data structure of Lisp is a list which is written like this: (a b c d). Like the article mentions, when you see Lisp code you're in fact reading the same list syntax, only that, when the list itself is not quoted, the "a" is evaluated as a function, which is why (+ 1 2) behaves the way it does. If you quote the list by saying "'(a b c d)" then it is a list of symbols, so "'(+ 1 2)" does nothing.

Hence, Lisp code is made of nested lists. When you think further about it, it means that Lisp metaprogramming facilities may do things just with plain list manipulation functions! So it's not about metaprogramming itself, it's that it's incredibly easy. This is what people refer as "code as data" and "homoiconicity" and so on.

I found the "Little Schemer" an enjoyable tour on exorcizing the power of lisp. Depending on how far down the rabbit hole you want to go there are a plethora of information in various AI memos as well as the READscheme web resource with the original LAMBDA papers (readscheme.org) which most certainly worth PRINTing if anything to get primed for SiCP.

Lisp "advocacy" often touts the power of macros in an abstract and unconvincing way. Persuasion by explanation isn't very effective. I'm not very interested in persuading others that Lisp macros are necessary, and there are many languages I like to use that don't have macros. However!

Yes, all programming languages are programmable. But Lisp tends to be unlimited in its programmability, in almost the same way that Unix is: if you have root, you can change anything. I'm pretty sure that in most Common Lisp systems, you can redefine large parts of the compiler at runtime. And so on.

Syntactic macros are just one part. They can be extremely handy. Lots of Lisp systems use them well. Random examples: the DEFSYSTEM macro of ASDF; the DEFINE-EASY-HANDLER macro of Hunchentoot (a Common Lisp web server); the (controversial) LOOP and ITERATE macros; etc.

Another somewhat random example: Movitz was (is?) a project to write an x86 kernel in Common Lisp. It actually included its own compiler. It defines a lot of macros for instruction definition [0] and uses them in code that I don't understand anything of [1] but is probably very clear to someone versed in assembly.

Recently I've been trying to add some nice logging to a JavaScript program, and it's a typical scenario where the normal syntax is just annoying enough to make the code ugly and hard to scan—if I had macros, I could invent some other syntax.

How to use macros is a tradeoff that I guess comes with Lisp experience, but they can be a powerful escape when the standard syntax is annoying, and a way to define your own DSLs without restriction.

[0]: https://common-lisp.net/viewvc/movitz/ia-x86/def-instr.lisp?...

[1]: https://common-lisp.net/viewvc/movitz/ia-x86/instr-add.lisp?...

Lisp doesn't have pattern matching. With optima[0] it does.

Lisp uses prefix notation. With mexpr[1] it uses infix notation.

And so on.

[0] https://github.com/m2ym/optima

[1] https://github.com/tmccombs/mexpr

> A programmable programming language? Aren't all languages like that?

How would you write a program to write a program? How can you use this generated program in your other code? It's really hard to do with a lot of programming languages, even those that have eval like JavaScript, so it never occurs to people that something like a template system or a code translator/DSL should be trivially easy to write.

I have a feeling people tend to over estimate the importance of language choice. Might be just me though.

Would you have a look at this?

http://www.paulgraham.com/avg.html

Many people agree with you but I have the opposite view. Of course you can write any program in any Turing complete language, but the programs people do write in a given language tend to have a family resemblance to one another. Therefore if we want different kinds of programs we should have different languages.

A given language makes certain ideas easier or harder to formulate, and thus easier or harder to have in the first place. So the language constrains not only how you think about your program, but what you think—and thus what the program itself becomes.

Platforms, communities, and culture also help determine these things, but they're not independent of the underlying language.

Some have also said that the human language you speak has an effect on how you think, but that's pretty controversial among linguists if not altogether discredited.

I have no doubt the difference among human languages and programming languages could be completely different, but is there any concrete example you could give of an idea that is easier to express in one programming language than in another?

IMO the analogy to human languages is a distraction and best avoided, since we're not able to say meaningful things about both.

> is there any concrete example you could give of an idea that is easier to express in one programming language than in another

Sure. Since this thread is about Lisps: writing code that works with other code is an order of magnitude easier in Lisp. Therefore people tend to do it a lot more.

Try to write Python style metaprogramming is something like Java some day. It's perfectly possible, but nobody would think about that style. Now try to write it in C++, just for the lulz.

Or, if you are not satisfied, try to create a non-synchronous TCP server in C, Java, or Python. Then write one in Haskell.

Try creating RAII containers in Java... Ops, that can not be done!

I don't know about "possible to express", but having learned Haskell, I see monoids everywhere.

In the SICP lectures, when Sussman teaches the class the word "predicate", he mentions that knowing the names of these things is important because "As any sorcerer will tell you, if you know the name of a spirit, you have power over it".[1]

If you only know C, you're unlikely to think in terms of map/filter/reduce. If you don't know Haskell, you're unlikely to think in terms of functors and monads. You may be vaguely aware of repeated patterns, but knowing you're looking at a monad gives you a lot more power to reason about it.

[1] https://www.youtube.com/watch?v=dlbMuv-jix8#t=12m50s

So, a sort of Sapir-Whorf hypothesis for the programming domain. It certainly feels true; living and breathing a language like Erlang would probably make one think very differently about what's possible than someone immersed in Fortran.

Think about this:

What kind of code do you think people will be writing 10,000 years from now? What will they be able to do that you can't? What will they have that you don't?

One of the most useful things I've gleaned from pg's essays is learning to think like that.

"Will people still be doing X in the same way Y years from now?"

If not, what will make that change possible? What will drive it? How can you help it along? Is that a startup idea?

In the long run it's true. The abstraction levels shift, you think in systems, data, bandwidth, maintenance, etc, which are outside of language scope (for now). Lisp is different though, it's not really a language, or it's just enough of a language to express ideas the way you need them to, so it does matter a little.

I find all his digs at javascript funny since, at least the way I use it, javascript is basically an ugly-looking scheme. Javascript is responsible for my 'lisp enlightenment' with the language using first-class functions and closures, and the wide variety of precompilers. It strikes me as fashionable to hate on javascript.

Java on the other hand, now that's a horrible language ;) ;)

I was thinking the same thing as I read it but I can conceded that its easy to overlook how much influence Lisp had on JS. I also find it mildly amusing to see all the Lisps written in JS that keep popping up lately.

Javascript isn't Scheme: http://journal.stuffwithstuff.com/2013/07/18/javascript-isnt...

If Javascript were only an ugly looking scheme...

Sweet.js brings hygienic macros to JavaScript and follows directly from Scheme R5RS mechanisms AFAICT.

Actually, sweet.js is based almost entirely on various macro innovations developed in Racket (such as the Honu approach to macros in languages without parens).

I too am amused by his opinion of JavaScript. A professor was moved enough to translate most of SICP to JavaScript and writes in its first chapter, "Above beyond these considerations, programming in JavaScript is great fun" [1]. I agree, and I'm enjoying going through this SICP/JS moreso than the original, as I can write solutions to the exercises in any old browser JavaScript console and paste them into a gist for safekeeping.

[1] http://www.comp.nus.edu.sg/~cs1101s/sicp/

> It strikes me as fashionable to hate on javascript.

Please name another programming language whose inventor has described it as having “a lot of stupid in it.”

It’s no longer a matter of fashion. It’s a matter of authoritative opinion.

I think the hardest thing for people to get for programming in Lisp is the immutability. Once you get over that hump, everything is downhill. From the outside, the parenthesis seem pedantic but it's completely superficial. You will stop noticing them in a few hours, if not days.

Are you perhaps coming from a Clojure perspective? Scheme is no more inherently immutable than, say, JavaScript. Many Scheme programs I've seen use mutation quite liberally. Immutability is a useful pattern in lots of languages and situations, but Lisp per se doesn't particularly enforce it.

Indeed. Common Lisp is similar with regard to the proliferation of mutation.

It in fact takes it to amusing extremes, allowing you to mutate even the strings associated with symbols.

    (setf (char (symbol-name nil) 1) #\U)

(Though I guess that could be taken as being the same thing as doing nonsensical pointer arithmetic in C -- you simply should know better.)

I'd point out that Common Lisp, kind of like Smalltalk, is a highly dynamic environment that has historically been used as an operating system in itself. That's one reason why the system is so mutable—you don't want to have to relaunch it even to modify core functionality.

I actually don't view mutation in Common Lisp as a downside. I find writing high performance (usually numerical, but sometimes not) code in Clojure to be painfully obtuse compared to Common Lisp, but I love Clojure otherwise. Writing highly optimized code can be quite elegant in Common Lisp, and I appreciate that when the problem at hand calls for it.

Actually, this expression is non-conforming: you must not try to modify the string returned by cl:symbol-name.

"non-conforming" means you will get different results in different implementation, perhaps something like a nasal dragon, or perhaps an error signaled, or perhaps some more benign behavior.

    $ clall -r '(progn (setf (char (symbol-name nil) 1) #\U) (quote nil))'
    CLISP                          Attempt to modify a read-only string: "NIL" 
    ECL                            Detected access to an invalid or protected memory address.
    Clozure Common Lisp            --> #|symbol not found in home package!!|#COMMON-LISP::NUL
    Armed Bear Common Lisp         --> COMMON-LISP::NUL
    CMU Common Lisp                --> COMMON-LISP::NUL
    SBCL                           --> COMMON-LISP::NUL

Some implementations will gladly and blissfully modify the symbol name, and your program will break because it cannot intern the original name anymore (not the best behavior for an implementation IMO); some implementation will detect the non-conforming access. Worse could have happened (remember, you are on the Internet so you can be located by geoip/gps/wifi, and missiles silos can be hacked by botnets).

But otherwise indeed in general, mutable lisp objects are mutable, and you can implement mutating algorithms as well as purely functional algorithm. Foremost, you can have an hybrid approach, using what's best to solve the current problem.

Racket also defaults to immutable cons and you have to require set-car! and friends. But I agree, most Lisps do not impose/favor immutability

Ah, right, but that doesn't mean the language is immutable to the degree that it would make for a steep learning curve. Mutating cons cells is arguably an obscure feature; you are free to mutate variables, vectors, structs, tables, and so on (even if you might have to specify that you want mutability).

My experience with racket is certainly not one of steep learning curve. Plus it is _thoroughly_ documented. If I had to recommend a Lisp implementation for someone that wants to learn a Lisp it would definitely be Racket as it is easy to install (and works on windows), has great documentation and a super friendly and interesting community.

The only thing I'm not fond of Racket is that compared to CL w/ Slime it feels way less interactive, a step up from Python, but still not CL (nor Smalltalk)

But lets focus on the positive, the OP is highlighting that the trite criticism of non-lispers about parens is bogus.

Perhaps I am but even Clojure is not "purely functional" in that respect.

Lisp is not an immutable language.

There's setf and setq for assigning; nconc, nsubst, and the destructive list operations; rplaca and rplacd; etc.

It's easy to write Lisp that doesn't mutate (let), or use constructs that hide it (dolist, dotimes, etc.), but mutable data has been in Lisp since the beginning.

That said, there are languages in the family that don't allows mutation, but they're a small minority.

Clojure isn't immutable either. It favors immutability, though, and has strong opinions directing the developer away from mutable state.

> What's in it for me? Expressiveness.

I appreciate the author's wanting a more explicit and practical answer to this question, but I don't think his answer here ("expressiveness") does any better job than the other explanations he criticizes.

> But [learning Lisp] also requires an investment of about 100–200 hours. That's asking too much.

No, it's not. That's how long it takes sometimes, or longer. Why do we all expect answers and understanding to come so quickly and easily? It'd be a nice world otherwise, but almost always enlightenment comes at a cost: patient focus and study.

Sure, that's true. But the problem is, there are dozens of things telling me that, if I invest that kind of time in them, they'll pay me back. Which one(s) should I pursue? In order to decide, I have to have some explanation of the benefits that I can grasp (at least in the abstract) before investing 100 hours. "Invest 100 hours and then you'll see why you should have done so" is not a sales pitch that will win with very many people.

Why do you expect things that really improve your life to come with sales pitches?

People are usually eager to recommend things that they feel really improved their life. If you, for example, read HN, you see "sales pitches" (in the broad sense) all the time.

Yes, that's true. But it's both naive, arrogant, and rude to require those "sales pitches" to be good. At the end of the day, it's up to you to decide what's better for you. If you try to outsource it, you'll discover that the best sales pitches around are for dead-end technologies.

At the other side of it, people giving you those recommendations really have no better way to explain it. PG has an well known essay about language power, but even that isn't very precise. Some kinds of knowledge you really have to know before you understand what they are good for.

I have so far stuck with Clojure instead of a Scheme primarily because of the nice literal syntax for maps, sets, vectors, and regexes that Clojure supports. I don't want to give that up and just use cons/car/cdr.

And also, last time I looked at Racket, it seemed very focused on academics. I just wanted to write some simple Scheme in a .scm file and try it out (a script), but I think I had to choose a language first (maybe a line at the top of the file? Don't recall) --- the Racket ecosystem seemed off-putting.

I'd encourage you to give it a longer try. When selecting a language, just pick R(5,6,7)RS and you're set.

Liking syntax sugar is hardly a reason not to use the language with the most powerful macros around. You can write macros to provide nice literal syntax for whatever you want - this is part of the fun.

Have not yet learned to use or appreciate macros.

The idea of everyone having to come up with their own syntax for such commonly-needed things as, say, hashes, seems counter-productive to me. That would lead to everyone's code looking like a different language.

I never said "with lisp, everyone has to come up with their own syntax for commonly-needed things". But it sounds your mind is already made up by hypothetical prejudices.

You said you stick with Clojure over Scheme on account of some syntax. I said you can have that syntax in Scheme. You then said the non-sequitur "the idea I created in my mind of each individual user coming up with their own syntax is absurd", which I agree with. I don't know what to tell you but you seem confused. What you said is not a response to what I told you. I eliminated the difference you complained about, and you complained that in Scheme it's possible to solve the issue you have with it.

There's no pleasing some people.

It might work better to tell your own story about using Racket. How do you work with data structure literals? Perhaps you've found that Racket's built in functions are good enough. In that case, maybe show an example demonstrating that fear is unwarranted.

Trying to please everyone—or convince everyone—is not a fruitful method of language advocacy.

I don't mind if they choose not to pick the language I like, but I do mind that other people will fall for this faulty reasoning, and not give the language a chance. That's why I pointed out where the fault in the reasoning is.

Tell me you wouldn't want to avoid this boilerplate:

    public class HelloWorld {
        public static void main(String[] args) {
            System.out.println("Hello, World");
        }
    }

and instead write a nice little main macro, so you may just write:

    (main (print "Hello, World!"))

Do that a few times in an application, and instead of 10 million LoC, you can write it in 100,000 LoC. This becomes much more maintainable!

Also, see: Alan Kay "Programming and Scaling" 2011

    http://www.tele-task.de/archive/video/flash/14029/
    https://www.youtube.com/watch?v=gZmcmdsoAXU
    https://www.youtube.com/watch?v=-UOmItPa4iA
    https://www.youtube.com/watch?v=QlPavndhYxQ
    https://www.youtube.com/watch?v=y9xLi0iJg1g

Also have a look at:

    https://groups.google.com/forum/#!msg/comp.lang.lisp/gsQJOGKYUw4/oLLHW0f4Ce4J
    https://groups.google.com/forum/#!msg/comp.programming/FiNIiSm5cJE/JkF5wa6Ke54J

Sure. And then imagine that you have to take over someone else's 100,000 LoC written with their own, unique, macros, some of which may invoke other macros, and interact with them in surprising ways, depending on what's passed into them. Now you have to be able to keep in your head what all those macros do (including to each other) while you're trying to read those 100,000 LoC.

At that point, you may not find the macros to be so much of an improvement on the boilerplate...

Imagine you have to take over someone else's 1,000,000 LoC written with their own, unique, classes and methods, some of which may invoke other methods, and interact with instance variables in surprising ways, depending on which method gets called first. Now you have to be able to keep in your head the flow of data (because the methods need not explicitly pass instance variables around) while you're trying to read those 1,000,000 verbose LoC.

At that point, you may not find Objects (aka copyable global scopes where instance variables are globally available to all instance methods of a class) to be so much of an improvement over symbolic computation.

Honestly, what you said never happens in practice (no one is abusing macros like that) but what I said is literally taught by Sandi Metz in her book Practical OOP in Ruby.

Your (common) reaction is just your everyday fear of the unknown. Try it.

This. I used to do all my programming in clojure. Now I am going back to python -- because I am not able to figure out a month after writing some code, what these deep recursive, reduce, apply things do.. But it is a good exercise writing it.

>commonly-needed things as, say, hashes

Hash tables are commonly used in imperative programming, but they aren't needed so often in Scheme where we prefer to use a purely functional style. We usually use alists because they are persistent. When I reach for a mutable hash table, it's usually part of some imperative process where I start from an empty table.

Depending on the Scheme you use, there could also be a persistent hash table implementation. Guile has vhashes, and there's an alist->vhash procedure, so you can still use the alist syntax.

Hash tables have nothing to do with imperative programming. It's just a coincidence that some MLs or Schemes don't include purely functional hash tables; Clojure and Racket both include have them.

I mentioned that Guile also has one, called a vhash. I guess my point is that special read syntax for them isn't that useful when you can use an alist and convert.

Sounds like a bit of a turing tarpit if you ask me. Sure there are technically ways to work around the lack of read syntax (I mean, Java didn't have literal syntax for any non-array collections for ages) but that doesn't mean it's a good idea.

This is wrong. There are no purely functional data structures analogous to hash tables in terms of performance. Hash tables are inherently stateful and cannot be implemented without state.

Most people are OK taking a small performance hit for the simplicity of working with immutable data.

When selecting a language, just pick R(5,6,7)RS and you're set.

Might as well use `#lang racket` and get the already-existing regular expression literals, "batteries included" libraries, etc.

There are also some practical reasons to favour Clojure over Racket, specifically if you work in Java shop. The inertia to move to Clojure but remain on the JVM is going to be considerably lower than moving to whole new infrastructure as would be the case with Racket (indeed the language interoperability offered by the JVM was probably one reason why Scala gained such ground). However, even with Clojure I'm going to get some damn funny looks at the next code review.

Yes, I think one reason for Clojure's popularity is that it allows you to escape Java while still remaining on the JVM and being able to interoperate with your existing Java codebase.

That said, for a variety of reasons, I would actually prefer to not be tied to the JVM.

Also, Clojurescript.

I believe Rackjure [0] gives some of the particular conveniences of Clojure back to Racket.

[0] http://www.greghendershott.com/rackjure/

> I don't want to give that up and just use cons/car/cdr.

Any somewhat modern Lisp has hash tables, vectors, etc.

Yes, but they don't always have convenient literal notation or a consistent sequence abstraction that encompasses all collection types. For instance, I love Racket, but the hash literals are very noisy and apply implicit quoting, so they almost never get used. (Though Racket does a good job in most places of having a unified collection interface.) Emacs Lisp has the same problem but nothing coming anywhere close to a collection interface.

What is a good resource to learn Lisp for an advanced (non Lisp) programmer? Most guides for most languages only cover declaration, simple functions and tiny applications. I want something that quickly goes through the basics and skips to advanced uses in bigger codebases.

Practical Common Lisp by Peter Seibel:

http://www.gigamonkeys.com/book/

PCL is a good book but too wordy for my taste.

Paul Graham's ANSI Common Lisp will get an advanced programmer up to an advanced level more quickly and efficiently.

http://www.paulgraham.com/acl.html

I'd say both are essential, but PCL should be digested first (no need to read every paragraph). On Lisp is IMHO a lambda power user guide and not a good general introduction to CL.

Note that I recommended the book "ANSI Common Lisp" for an advanced (non Lisp) programmer, not "On Lisp".

That aside, I agree with your description of On Lisp.

I'll second PCL. It really is incredibly good. Unlike some other introductions to Lisp, which tend to be of the algorithmic, CS side of things, PCL approaches things from more of a scripting/software-development side.

Common Lisp really is a great developer's language, and PCL helps show why. Peter Seibel's a great writer, too.

For Common Lisp there are two very good books: Practical Common Lisp (available for free on the web[1]) and Paradigms of Artificial Intelligence Programming (PAIP). For scheme, there's Structure and Interpretation of Computer Programs[2] although it misses some topics like macros.

[1] http://www.gigamonkeys.com/book/ [2] https://mitpress.mit.edu/sicp/full-text/book/book.html

I'd suggest going trough learn-common-lisp-in-y-minutes (1) just to get an idea about the syntax, read On Lisp (2) and Practical Common Lisp (3) and just starting using it to build stuff.

On Lisp by pg explains very well what lisp is all about in the first few chapters.

1: http://learnxinyminutes.com/docs/common-lisp/

2: http://www.paulgraham.com/onlisp.html

3: http://www.gigamonkeys.com/book/

Also, recently published, Clojure for the Brave and True: http://www.nostarch.com/clojure or http://www.braveclojure.com/

I also recommend this book/website. The pacing is good and it gets into some of the really nice parts of Clojure, such as handling concurrency, with clear explanations. Heck, it even has a small intro to Java so you can interact with the enemy to some degree. ;)

A more interesting and unusual section of the book is the intro to using Emacs with Clojure -- it's not necessary to use Emacs, but you will learn how to edit all those parentheses without worrying about editing parentheses. Even if you don't use Emacs, this feature (paredit) is available for many editors and IDEs, so it's worth checking out.

I love this site, it's so simple and beautiful

> (x + (if is_true(): 1 else: 2)) [is invalid in Python] because the if–else conditional is a statement, and can only be used in certain positions.

Point taken, but troll-mode pedantry: (x + (1 if is_true() else 2)) would be valid :)

...but, in that case, the `X if Y else Z` syntax is considered to be an expression in Python. You can't, for example, omit the `else Z` part. In Haskell -- again, in which everything's an expression -- conditionals are the same, just written in the "normal" order: `if Y then X else Z`.

    For more on the per-
    ils of taxing reader
           patience, see
            WHY DOES TY-
           POGRAPHY MAT-
                     TER

This quote, hyphenated as is, provides all sorts of opportunities for snark. Instead I'll just note that it seems Pollen needs work. It's far too eager to hyphenate; doesn't protrude hyphens, commas, etc.; and doesn't use TeX-style paragraph-level optimization. Not being able to identify the bounds of hyperlinks is also a strong source of irritation.

Where do you see this? On my machine it's not hyphenated

Chrome, OS X. Hyphenated exactly as shown regardless of magnification.

         For more on the
        perils of taxing
    reader patience, see
                WHY DOES
              TYPOGRAPHY
                 MATTER.

This is what I get on Firefox and Chome

I'd be more interested in a "Why Pollen over Scribble?"...

How does Racket stack up against Clojure? I like some of the special extended reader support for data structures, I dig the immutability, some of the concurrency stuff....I don't think immutability/concurrency is that typical for LISPs...for people who've tried both, which one do you prefer?

I think people tend to use Clojure more because it has the Java VM and platform as support, which is a nice thing to have given its extension.

The .Net world has IronScheme which is very stable and interesting too.

is there a tl;dr. After about 45 secs of reading author did not answer why racket or why lisp

It's simpler than using the language you're probably using now. I cannot think of a higher praise for any design problem is that it's solved more simply with Y than X.

Forth is even simpler.

1 - Everything is an expression so it's easier to snap code together.

2 - List based = flexibility, get more done faster because lists are simple and everywhere

3 - Strongly nudges in the direction of functional programming

4 - Rocking documentation, advanced libraries

5 - The Dr Racket GUI IDE

6 - X-expressions (don't know what they are)

7 - Hygienic macros

9 - Easy to write powerful DSLs

10 - Many low hanging fruits to contribute

Dr Racket is a horrible IDE. Routinely uses ungodly amounts of ram, ever try scrolling through your code because it spends 30 seconds in some paren matching loop.

I wouldn't say "horrible." I haven't used it for large projects, only for tinkering, but for that purpose, it's excellent.

Unfortunately I have to agree here. I abandoned DrRacket for Emacs racket-mode and geiser because waiting for DrRacket to finish whatever it was doing became too much for me. On the other hand, as an Emacs user, I'd probably do this even if DrRacket was much better optimized than it is :)

Don't waste your time with lisp. Learn lambda calculus and then see how these ideas are applied in actually useful imperative programming languages.

Pretty ironic you've been posting here for 4 years to a site built with a language that's not useful.

I'll believe that it's useful when unenlightened plebs like me start getting money thrown at them for writing lisp code. So far the industry seems to disagree with you about what's useful.

This site's backend could be built with dozens of different languages.

Reply on news.ycombinator.com