Why Racket? Why Lisp? | Hacker News Reader

Why Racket? Why Lisp?

beautifulracket.com

327 pointsalokrai122 comments

I think one of the most amazing parts of Racket, and one of the reasons someone should try it out if they haven't, is that the source code and evaluator support images and not just text. The following link shows one aspect of this:

https://docs.racket-lang.org/quick/

Just being able to put an image in a comment explaining your algorithm into your code would be huge improvement over the traditional syntax colored ascii you get with most languages and editors. There are many times I've written an algorithm which is difficult to understand without the associated diagram I drew on the whiteboard or in an image editor. Racket lets you insert that image or photo directly into your source, and I think this is a significant improvement over putting a link to that image in a comment.

Racket also goes a step beyond that by letting images be values which can be assigned to variables or returned from functions. The link above shows this clearly. I'm sure there is some value in this as a teaching aid, and I think that's why they did it, but you can also return mathematical plots from your functions and so on. This feature is similar to the various interactive notebooks people use for Mathematica or Python, so it's not really specific to Racket, but it is interesting to play with.

Obviously there are downsides to putting images in your source. After you do that, your code is no longer ascii, and it won't be something you can edit with vi, emacs, or any non-Racket IDE. Also I doubt it will play nicely with git any time soon. However, it's a neat feature of Racket whereas many of the other benefits in the article apply to any Scheme (Chez, Gambit, Chicken, Guile, etc...) or lisp.

I wish there was some reasonable standard (like a better version of Rich Text) that was commonly adopted so other languages could put graphical pictures in the source code.

> I wish there was some reasonable standard (like a better version of Rich Text) that was commonly adopted so other languages could put graphical pictures in the source code.

I have felt exactly the same for many years. It would need to be something that was totally open (to achieve wide adoption), and reasonably easy to support in IDE's for edit and preview. And git-friendly. Belly-flopping on an existing standard is probably the easiest way to make that happen.

One thought is to presume a block comment with a language-specific marker, and make the contents a restricted subset of Postscript. a) You get one or two open-source fonts. Deal with it. b) The rendering area is strictly limited, c) the operator subset is limited to something reasonable -- like just enough to do a monochrome version of 80% of the kind of graphics that PowerPoint-ish programs give you.

Of course, you would never want to write Postscript by hand (although it isn't hard) but an IDE should be able to support a graphical editor plug-in. Or in a pinch you could even use another tool that left the rest of the code alone, and just edited the graphical block comments.

How about 3 fonts (serif, sans, and fixed) and color instead of monochrome? You're too quick to compromise :-)

Vector graphics would be adequate and could discourage huge binary dumps in the code, but you know someone would try to jam a photograph in there by inserting a square per pixel, and I can't really blame them. Postscript could work, but if you picked something explicitly line oriented it would cause less confusion in revision control.

> How about 3 fonts (serif, sans, and fixed) and color instead of monochrome? You're too quick to compromise :-)

Minimal Viable Picture :)

> Postscript could work, but if you picked something explicitly line oriented it would cause less confusion in revision control.

Agreed. But Postscript has a lot of existing unencumbered infrastructure code littering the net. With a new syntax, you need to overcome an activation energy problem. You would have to make sure that rendering and editing code existed that could be munged into a plug-in for everybody's favorite IDE. EMACS mode, anyone?

Also, maybe an independent tool that ran in a local web browser (not uploading code to random servers) would be a way to jump start things.

SVG could work (if you can stomach XML). The great thing about standards is there are so many to choose from.

or base-64

There's a Visual Studio plugin that I use that does this. I have an images directory in my source tree and in comments I can refer to them with syntax that looks something like:

/* <img source="images/bubble_sort.png"/> */

If you have the plugin installed, the image is displayed immediately after that line.

You mean like the 'pic' language? http://floppsie.comp.glam.ac.uk/Glamorgan/gaius/web/pic-20.h...

Developed in 1982. Present on almost every UNIX-like system and integrated into the manpage generator.

Probably the main reason we've not seen things like this achieve wide adoption is the small but very influential subset of programmers who refuse to use anything that isn't a pure text terminal for development.

Interesting. I'm not familiar with "pic", but I'll check it out. I've never seen a manpage with that in it.

A terminal with graphics in it would be nice too. Maybe if Tek4014 (and the Tek emulation in xterm) had graphics and useful text simultaneously it would've been more popular than the modal approach. I've also heard of Sixel, but never seen it live.

This isn't that hard to keep in an editable form for you. Visualizing it as an image can just be a property of the editor, and the "on disk" representation can simply be a URI to a standard image format. This is basically what emacs org-mode does, and it works really well.

I should check out org-mode. It's been more than 25 years since I used emacs with any regularity.

Thanks for pointing out the image part. I also do this with both Python and Mathematica all the time just like you, but haven't touched Racket in 10 years.

Mathematica allows similar things to be done in its notebooks. Fun!

For Visual Studio, the Supercharger extension is capable of embedding images in comments. Those with Supercharger will see the inlined image, and those without will just see the markup containing the image path. It's not as powerful as what Racket is capable of with images-in-code, but gets the job done for most purposes.

Since Markdown is now the web's rich text, its image tag is probably the thing to try. Then, Emacs is guaranteed to be able to show images. Possibly Intellij Idea, since their MPS supports wacky stuff in DSLs' editing interfaces. Dunno about Sublime etc., but I hope Atom and company can make proper use of their platform. Even if not inline, an image can often be opened in a split frame.

BTW, Mermaid and such might be more suitable for embedding diagrams in comments―maybe even just one of the ascii art diagram languages (for which there are utils to convert to graphics).

> Just being able to put an image in a comment explaining your algorithm into your code would be huge improvement

i do this all the time.

If you dive into old Lisp Machines and Interlisp-D papers, it was already a feature back them.

Many FOSS devs don't have a full picture of Lisps, by constraining themselves to what Emacs+SLIME are able to, without experimenting commercial Common Lisp environments.

Comment Area Markup Method Literary Programming, Programming was the first, Literary was the second.

the main purpose of the Code comment area markup method is to live Preview directly in the Code Editor Preview panel without exporting or any preprocessing.

Just add a line comment character of the programming language before each line of Markdown.

In the comments of code, you can draw flowcharts, tasklist, display data visualizations, etc.

The method is to add extension instructions in any programming language comment area:

markdown manual eval code, live eval code, print result, display data visualization and other directives When previewing or converting a format, you only need to simply preprocess: delete line comment characters with regular expressions, example: sed 's/^;//' x.clj

Note:

line comment character of Clojure(Lisp) is ; line comment characters of the current file type can be obtained from the editor's API. when we edit the code, we can preview the effect in real time. Editing literary code has a live preview panel like most markdown editors.

[Markdown Literary programming that don't break the syntax of any programming language](https://github.com/linpengcheng/PurefunctionPipelineDataflow...)

> Obviously there are downsides to putting images in your source. After you do that, your code is no longer ascii, and it won't be something you can edit with vi, emacs, or any non-Racket IDE.

Emacs handles images.

I have written books on Common Lisp and Scheme. Their power feature to me is the combination of functional programming (functions without side effects) and how these small functions can be built bottom-up in an interactive repl. Other languages like Python and Ruby also support repl style development, but aren't sufficiently functional. A little off topic, but I am working on a commercial product (KGcreator, a tool for generating graph data for knowledge graphs) and started prototyping in both Racket and Haskell. It was a tough call but I chose Haskell. I think of Haskell as being another Lisp language that also supports repl style development.

Tell us more about your books. I'm addicted to scheme books :-)

Also, what made you choose Haskell for your project? Can you share some of the reasoning?

See Mark’s books here [1]

[1]: http://markwatson.com/books/

I think the deployment story for Haskell is better than Racket. It is easy enough make standalone Racket executables but with stack and cabal it is baked in to easily build multiple executables, separate libraries, keep everything tidy.

Racket is much better to get something done and working quickly. Same comment for Common Lisp.

Haskell has great support for strongly types web services (servant) and lots of great libraries. Racket has a very rich ecosystem of libraries, custom languages (like Typed Racket). Both are great.

EDIT: It takes me longer to get to working code in Haskell but once written the code has higher value to me because it is so much faster/easier to refactor, change APIs, reuse in other projects, etc. I just did a major refactoring/code-tidying this morning, and it was very simple to do.

> functional programming (functions without side effects)

CL has mutable data structures(lists are mutable by default) and setf etc. Scheme/Clojure makes a much pleasant functional programming without side effects experience.

RxRS Scheme has mutable conses as well. One of the reasons Racket isn’t called MzScheme anymore is that its primitive conses are immutable (iirc)

Immutable cons cells also lets Racket implement a constant time `list?`, which is kind of nice.

https://docs.racket-lang.org/reference/pairs.html?q=list?#%2...

This doesn’t require immutable conses if you just update the cache whenever the cdr is mutated.

If the last cdr in a list is modified, then you need to propagate that change up to the front of the list during that modification, or check the whole list when you run `list?`. Either way, you pay linear time for it.

Only if the new value makes it no longer a list (or changes a dotted list into a real list). My hunch is that this would, in normal usages, work out to constant time (setf (cdr foo) ...) with the occasional (setf (cdr foo) atom) requiring the change in state to be propagated in linear time.

It's not only if the new cdr is an atom that the whole thing becomes a non-list. It remains a list if and only if the new cdr is a list. There are non-atom non-lists that could render it a non-list, but more disturbing is the case where the new cdr is a list until the modification is performed. If the cdr is modified to create a cycle, the the new cdr is a list right up until the point that modification is actually performed. The new cdr only ceases to be a list after the modification. So that's a really hairy situation. Note also that the cons cell being modified could belong to any number of lists or non-lists, not just one, so propagating through a change could be much worse than linear time in practice depending on what's being done. (Not to mention that propagating state the wrong direction through a chain of cons cells is itself a mess...)

Immutable cons cells eliminates all of this. If a cons cell is created with a list cdr, it will always be a list. If a cons cell is created with anything but a list as the cdr, it will never be a list. Very straight forward. You don't have to worry about linear time or worse propagations, and you don't need to anticipate what some other programmer down the line will consider "normal usage".

For a much better experience programming functionally in Common Lisp, try the FSet functional collections library: https://github.com/slburson/fset

I love Haskell, but as far as GHC goes, its REPL story is awful: your environment doesn't even survive loading a new file! You can ask it to replay your bindings, but that's a pretty crappy solution, likely to produce unpredictable errors and, if you're doing any serious computation (such as when theorem proving), it's probably useless.

Compare this to a language like Ocaml, where I have worked at the exact same REPL, modifying the same environment, for months at a time.

This was such a standard thing to do in the Lisp and ML worlds that these languages traditionally include a save-lisp-and-die function which saves your entire runtime to disk so it can be reloaded later. (We used DMTCP in Ocaml for the same effect).

Not to moan too much. Haskell is still my favourite language. I just wish it had a decent REPL.

Isn't the lack of a stateful REPL a feature in Haskell? Don't get my wrong, I love image/state-based developing (having working knowledge with SmallTalk and just getting into emacs). But I think it is quite haskelly to introduce functions that create the state rather than create the state interactively yourself. This should be good for mocking and tests also. Yes, this is more work than in most other practical languages, but this is Haskell for you ;)

> If __ are so great, then it should be possible to summarize their benefits in concise, practical terms. It should be possible to demonstrate the power of __ in one hour, not 100. If __ advocates refuse to do this, then we shouldn’t be surprised when __ remain stuck near the bottom of the charts.

This paragraph seems like it would be equally true for almost any subject you filled in the blanks with.

As someone who had to learn a foreign (natural) language in school, I can see the benefit now, but there's no way I could explain it, and certainly not in an hour. It's a type of learning which causes a change in how you organize things you already know. How do you sell that?

(No, I don't believe "so you can talk to people in that language" is a realistic benefit. I don't think even my school thought that. The selection of languages offered is simply not useful. I've never met anyone this side of Stuttgart with which to use my German. Of the top 10 non-English languages spoken in my state, only 1 was offered as a class at my school. It almost looked like they went out of their way to find teachers in less common languages.)

(BTW, that's also the same answer as "If Lisp is so great why isn't anyone using it?" It works for any subject. If trig is so great, why aren't you using it? If music is so great, why did you stop playing after you graduated, and were no longer required?)

I'm not trying to downplay the importance. It's a real problem, for many fields. As a Lisp programmer, it's my nature to try to sell everyone on learning Lisp even if they won't use it, and also to over-generalize problems to nearly the point of absurdity.

How do you get someone to want to learn something when it may have no immediate and apparent practical value to them? Especially today when their whole "learning" slice is competing with Netflix and Facebook and all the rest. I guess the trendy answer right now is something in the neighborhood of "freemium gamification" and for reasons I can't explain that makes me sad.

The problem of people failing top provide empirical evidence for claims of things is indeed universal and common. But it is possible to do so, sometimes it requires a ton of work, but if we want to know things, we have to do that work. Until we do, we don't know things.

> It's a type of learning which causes a change in how you organize things you already know. How do you sell that?

That's potentially a good pitch all by itself.

> Of the top 10 non-English languages spoken in my state, only 1 was offered as a class at my school. It almost looked like they went out of their way to find teachers in less common languages.

This is a great example of language education being frozen in time for essentially cultural reasons. Pre-20th century, and probably up to about WW2, the common choices of foreign languages to study were determined on cultural grounds and largely for ""elite"" purposes. You'd learn German so you could read Goethe and Schiller, or the various scientific papers published there.

Wind forwards to the present day and the obvious choices for US language learners would have Spanish at the top of the list, but that would involve the complex politics of the relationship between the US, the rest of Latin America, and its immigrants. Much easier to carry on pretending that someone might want to read Goethe.

In my mind there is one killer feature in Lisp, and that is Compile-Time-Computing.

You have a macro system that isn't some lameass additional language with severe limitations like the C preprocessor or C++ templates. You have all the language available both at compile time and at run time. You can even use functions you already have (and tested) in both places.

This allows you to leave every single assumption you make during programming in one place. You don't have to splatter uncertainty about what the program is supposed to do into multiple places because you language isn't expressive enough, e.g. you cannot just redefine control structures to express an assumption. That is what leads to "changeable software", not just "readable software".

My writings on the subject: https://medium.com/@MartinCracauer/a-gentle-introduction-to-...

https://medium.com/@MartinCracauer/a-gentle-introduction-to-...

Example - using scientific units attached to literals in source code, but keep them at compile time and don't slow down runtime with unit checking: https://medium.com/@MartinCracauer/a-gentle-introduction-to-...

And speaking about early or late (static/dynamic) type checking. If you have compile-time computing you don't have to choose. How silly would it be to make a programming language that can only do one or the other. https://medium.com/@MartinCracauer/static-type-checking-in-t...

Finally, there is turnaround time during development: https://hackernoon.com/software-development-at-1-hz-5530bb58...

You should check out Zig. Here's the part of the documentation concerning compile-time code.

https://ziglang.org/documentation/master/#comptime

I've done some elisp (and a tiny bit of racket) and some Clojure and honestly I struggle to see why I would ever pick a racket/scheme/elisp over Clojure unless I desperately needed to do a ton of C ffi. Clojure is much more opinionated but I never find myself feeling like it's constraining or a straightjacket

When do people say to themselves "screw this, I need a more flexible tool like Racket"? Is it when you get super deep in the macro magic? (I'm not reallt sure how the Clojure macro system compares to the Scheme one)

I just find racket to be more fun. If I'm blowing off some steam on the weekend, racket is what I reach for. It feels clean like scheme, but has the "batteries included" feeling of python. Clojure is alright, but it feels more like work than fun. I don't really know how to justify that feeling, but that's how it feels. I guess it's little things like racket/match being an 'included battery', and having a ton of useful forms already defined out of the box (e.g. match-lambda and match-letrec-values.) Clojure's core.match is pretty spartan in comparison, last I checked.

As far as macros go, I greatly prefer racket's syntax-parse to anything else I've tried.

I think I was trying to articulate the same thing before, why Racket instead of more-practical-tool-here for "experiments/fun" and it really is just because it's more fun to work with, nearly everything you need is in there!

Also personally something I wish more functional languages would have is a little bit of Racket's maximalism with its forms like: (for/list ([e (range 10)]) e) [1] and for/fold, including the nested ones and the like that make mapping over things and even filtering a dream!

[1] https://docs.racket-lang.org/reference/for.html#%28form._%28...

I implemented racket's for loops for guile scheme (my server is down since a week, so my hosted documentation isn't up, but you can find it in the repo linked below). It is maybe not a trivial macro, but I didn't need to struggle much to reach about 90% feature parity with racket.

The code it outputs is almost always as fast as a hand-rolled named-let (just as with racket), and you can port it without much difficulty to other schemes. I support most of the sequence iterators (in-range, in-list etc) and support non-tco loops to create things like lazy streams.

https://bitbucket.org/bjoli/guile-for-loops/

I've done way more Clojure than any other Lisp, and I'm now transitioning to Racket. Primarily, because I want to get away from the JVM. One thing about Clojure that is not easy to put into words, and that I miss, is just how "smooth" working on it is. The Clojurisms, as they come to be known, are very well thought out, and the whole language fits together like a perfect puzzle, or at least in my experience. To come back to the JVM thing, I find that Clojure is applicable only serverside, where you can have the slow starting hundreds-of-megabytes JVM running non-stop, or in the browser, through ClojureScript, and it's really perfect for those environments. Though as I shifted more towards non-server environments, and I need to use FFIs, Racket is the right blend of expression and performance.

Have you played with the different schemes or just jumped on racket? If the former, I'd be interested to know why racket over gambit or chicken?

No, Racket is my first Scheme.

Yeah, I'm in the same boat. I mainly used Clojure, but have been doing more Racket lately. Mainly for short scripts, because it doesn't have the warm-up time of Clojure. Racket is a beautiful language, but some things in Clojure are just opinionated in a very nice way.

For example, the seq abstraction and the fact that most sequences and functions are lazy by default is really nice. It also makes more use of polymorphism, so you can e.g. call map on a hash-map, a vector or a lazy seq. Also the fact that maps and keywords act like functions is extremely convenient. Similarly, let is actually let* and binding forms allow destructuring. Racket has all this, but you have to remember to use e.g. match.

Racket macros are a work of art and I'm only just exploring them. However, the surface area is very large. In Clojure once you know the language and some of the standard library you can write macros. In Racket it seems there's a whole other language you have to learn. It's clear though that they're extremely powerful, so I probably just need to invest more time in it.

All in all my ideal language would be Racket with some of the conveniences and choices of Clojure. Gerbil is another lisp that looks interesting; maybe I'd steal some stuff from there too. Of course, Racket is extremely well suited to building languages, so I could actually build this ideal language if I wanted (and had the time and skill required).

Have you checked out the new aot compiler that can produce 10-20mb images that start about as fast as c applications?

https://www.graalvm.org

I find the usability and elegance of Racket to be far better than Clojure. Clojure feels messy, mostly due to the JVM. Racket feels clean and fun and expressive. The ecosystem is better integrated as well.

Clojure's performance is of course better, though.

While being a general purpose programming language, Racket is intended for Language Oriented Programming[1]. I was looking at Lisp Game Jam 2018's submissions when i stumbled on this game https://euhmeuh.itch.io/lisp-vs-blub-empire. The author made a throwaway language by wrapping Web Assembly intermediate language into racket using less than 250 lines of code[2]. This is what Racket is really about; an environment to quickly create a specific programming language for a specific problem[3].

[1] https://en.wikipedia.org/wiki/Language-oriented_programming

[2] https://github.com/euhmeuh/wasm-adventure/blob/master/src/wa...

[3] https://felleisen.org/matthias/manifesto/

The flexibility might not just be in the tools themselves, but in the culture of the language. Elixir for example has really powerful macros, but one of the main books about metaprogramming says "the first rule of macros: don't write macros". So you have some beautifully designed DSLs like Ecto.Query showing how powerful they are, but you rarely go trying to solve your problem by using that power for creating your own DSL or language (and mostly just use 'use' macros for code injection in your module).

So even if one language's power is enough for someone's programming ambitions, they might still want to go for an environment with more like-minded people that are more supportive of using (and in the perspective of some, abusing) that power.

I've done a bit of both, and while I probably prefer Clojure as a language, the Racket docs are more comprehensive. That made Racket more accessible for me.

Also, trying to figure out how to build (non-web) GUIs with Clojure and Swing wasn't a pleasant experience for me.

But Racket can be overly verbose sometimes.

That’s a longer read than I expected but there are some good, frank points like non programmers seeing some of the praise for Lisp called “unsubstantiated hoodoo”.

I like that the author addresses the “what’s in it for me” head on as well. Makes it a bit more clear what some of the immediate benefits are.

I've been using Racket for a few years now as my main "fun" toy language and I can't stop using it or trying to come up with new ideas for it. I'm currently writing my new website's publishing code in Racket, using Racket programs to create my static pages.

For anyone who enjoys programming language dives, I recommend Racket fully. It's simple and powerful in various ways.

I'm wondering if combining Racket with AWS Lambda (https://github.com/kpiljoong/aws-lambda-racket) would be a good way to play with Racket and do cool things with it. I've done a bunch with Emacs Lisp but always felt weird writing a server application with elnode.

I've read this article a few times and every time he takes beef with Python:

  x + (if is_true(): 1 else: 2)

I think he just got he syntax wrong, it's supposed to be:

  x + (1 if is_true() else 2)

And basically can't follow the rest of the argument. I have yet to learn a Lisp. Honestly, what am I missing?

The macro system

> The macro system

To add a little more detail to a spare but correct (I think) comment:

Outside of Beautiful Racket, some other helpful reading I've found in the past that covers macros:

> Creating Languages in Racket (Matt Flatt, 2011, ACM Queue) [0]

The details here are tantalizing, but there's a lot that goes unsaid. Code is available for download and review. As a longer-form example of something fun to do, it's good, and I've enjoyed it.

> Automata using Macros [pdf] (Shriram Krishnamurthy, date unk., Educational Pearl) [1]

This is a worthwhile read at only 14 pages. Krishnamurthy writes with clarity, iterating through several solutions to writing a finite state automaton.

[0]: https://queue.acm.org/detail.cfm?id=2068896

[1]: http://cs.brown.edu/~sk/Publications/Papers/Published/sk-aut...

Thank you. Although it is against the norms of HN, I deliberately did not go on writing an explantion of the Macro system, because in my experience, I did not fully understand it's power until I used common lisp.

Before that, I thought of the macro system to be like the #define preprocessor in C, nothing more.

Thank you very much, for actually expanding it with great links!!

The point of the "if" example is that there is a difference between expressions and statements... yes, Python has a conditional expression (since Python 2.5, before it didn't), but it had to be a different syntax for that reason; the "if" expression is a completely different thing than the "if" statement (it's what's sometimes called a "ternary conditional") it just happens to use the same keywords (but without the ":"). In LISP everything is an expression, the syntax is totally uniform.

Interesting, and thank you for the explanation!

More lines of prose praising lisp are being written than lines of lisp these days it seems.

I recently asked how to write a tree shaker in #sbcl because I thought it’d be cool. All I got was a “why would you do that?” and “ok fine your time to waste” and no substantial answers. The Common Lisp community is small and curmudgeonly. Who needs this?

Yeah they're kind of assholes. Racket has a much nicer community, though.

Yeah the racket community is super sweet. What can I say though, I like the Common Lisp experience. Slime/sly are appealing, as is the totally cross sectional view from high level code to machine code. There’s a lot to like.

I should give racket a second shot though. I love Chez.

FWIW, stylewarning in #lisp was offering to pay someone to finish a tree shaker someone had started for sbcl, so it’s not the case that no one is interested in your project.

https://twitter.com/stylewarning/status/1052006872587202560?...

Yeah I’ve gotta say that the fact that a new hot startup is embracing lisp is really cool. Kudos to stylewarning.

DWAVE, another company in the larger quantum computing area also has been using Common Lisp.

https://cjelupton.wordpress.com/2014/08/14/quantum-computing...

Some things, like tree shaking, trigger some, in my opinion justifiable, bitterness in the Lisp community. People said Lisp couldn’t take off unless it competed with C for speed and image size. Tons of resources were invested on that front, building tree shakers and type inference engines. Then, JavaScript comes along and proves that none of that was necessary. Individual web pages are as big as some Lisp’s entire core images. The language spec might as well have been written in crayon. While Lisps were developing sophisticated numeric towers, JS does not even distinguishing between integers and floats! Dynamic typing languages won, with a language that was inferior in every way to Lisp, except popularity.

I hadn’t seen it that way, but that completely makes sense.

It's a bit anachronistic to compare Lisp's goals from 40 years ago to current JS situation. The latter we're stuck with for political reasons, while the former really did need to overcome hardware limitations to be successful in that era.

Also with Typescript being adopted by most major JS projects, I wouldn't say that dynamic typing "won."

Python is also really really popular these days. Maybe they haven't so much won but they've definitely become extremely significant, and they haven't lost.

Python has an optional type checker these days so you can write fairly statically typed Python if you want.

Tree shakers are not written because they 'are cool', but because one wants to deliver small applications with it. But that's not the focus of SBCL development and never was. People can't tell you much about it, because there were only few attempts made in that direction for SBCL. Some Common Lisp implementations have tree shakers - especially the commercial ones. But it's a rarely used tool.

See for example:

https://github.com/roswell/roswell/blob/master/lisp/dump-sbc...

> Common Lisp community

#sbcl is not THE Common Lisp community. Best not to generalize from an IRC channel to a very diverse group of people using a dozen different implementations.

> Tree shakers are not written because they ‘are cool’

Yeah this is all very serious business. Also, one of the most curmudgeonly responses you could have given.

> #sbcl is not THE Common Lisp community

This is not my sample size of one opinion. I’m just throwing one in the pot for the very widely held opinion of the Common Lisp community being curmudgeonly.

> Yeah this is all very serious business

It's work and SBCL is maintained mostly by volunteers, who may have their own agenda. Sometimes there is funding from commercial projects. So far maintaining a treeshaker wasn't high on the agenda, even though the project runs for some years now.

If you shell out serious money for your 'very serious business' the commercial implementations Allegro CL and LispWorks provide maintained tree shakers and all kinds of fancy application delivery features.

> throwing one in the pot

not very motivating...

You might check out the link I've gave you above, instead.

I may not have made myself clear. I don’t want someone else to write a tree shaker for me, I want an intro project to understanding the internals of a performant Common Lisp compiler, and I want to do the work to write a tree shaker myself. I’ve seen a few implementations, but they all come with notes explaining that they’re sketchy and experimental. I mostly wanted to know how to push them past that point into being robust.

Tree shakers require that you look at every function-bound symbol and determine if that symbol appears in the call graph of the main function(s) or in an isolated call graph. The functions that follow the call graph exist in most Common Lisps but they are not standardized; they're implementation-dependent. That's part of the reason tree shakers aren't widespread. Another reason is that if you're using the compiler in your runtime, even isolated parts of the call graph can still be useful, so tree shaking becomes something of a policy decision rather than a pure algorithm.

It's also not that easy, given one can cross the data is code frontier easily:

    (let ((fs '(+ - *)))
      (mapcar (lambda (f)
                (funcall f 1 2))
              fs))

How do we know that the functions in FS can't be removed? They are just symbols in a list in the code.

Exactly. That's what I was getting at but your example is better because the compiler isn't really even being used at runtime here.

Yeah that’s super tricky... how does LispWorks deal with such things?

I have the impression that lispworks gets the gc to do the tree shaker's work http://www.lispworks.com/documentation/lw50/DV/html/deluser-...

Treeshaking is a separate phase during 'delivery' (aka creating an optimized application/library): it may use the GC in the treeshaking phase:

http://www.lispworks.com/documentation/lw71/DV/html/delivery...

...and just as a follow-up, once you've dug into the problem enough to discover it's surprisingly difficult to define what is "the right thing" for a tree-shaker in Lisp to do, you realize that even if you build it, it doesn't buy you much. That's why we tend to sound curmudgeonly about the issue. It's a valuable learning exercise, so feel free to dig in. You'll gain some valuable insights into what makes Lisp special.

Hmm, the usual treeshaker might not tell you much about the internals of the compiler itself (a compiler is just one part of a Lisp system based on runtime, interpreter, various libraries, etc.), since the ones I've seen work over already compiled code. It's an interesting project, but not that easy for a complex piece of software like SBCL - depending on what the goals are.

With a quick search, I found an example for SBCL below. I can't say anything about its quality cuz I dont do CL.

https://gist.github.com/burtonsamograd/f08f561264ff94391300

All the CL people I've worked with have been amiable. Though, I suppose you could say: what high-powered hacker wouldn't be in a good mood, if they were getting paid to hack Lisp.

I don't recall any curmudgeon behavior in-person, but a bit "critical" is often a useful role for an engineer to play, if they can back it up and discuss. A useful mode of engineering discussion involves people making assertions, thinking aloud, and being challenged, and together you improve the ideas and generate new ideas. Sometimes it's appropriate to suddenly look at each other and start jumping up and down and shouting, like you're in a movie, because you've just hit on a solution that has passed your preliminary tests of critical thinking. If you're jumping up and down all the time, I suppose that could turn into incestuous amplification.

The Scheme and Racket communities are also good. I've spent the most time in Racket, and have a few ideas about why the community is good:

* The original professor (Matthias Felleisen) and grad students were solid PL people who were interested in making CS education more accessible, which seems like a pursuit that would value nurturing and accessibility.[1] When some of the grad students also became professors working with Racket, they kept up accessibility, such as in the main forum (`racket-users` email list / Google Group). (`racket-users` seems to implicitly coordinate, with community members, including the professors, responding to most any question.)

* This will sound a bit cynical or funny, but I've seen it change other language communities before: one thing that helps the community be good is that there's no money in it. (Once there's money happening, you'll get more of less-desirable behavior from some individuals and groups: promotion of personal brands, posturing and jockeying for the opportunities, SEO games and huge amounts of Web search hit noise from that, marketing puffery rather than engineering straight-talk, non-sharing, sometimes commercial landgrab games with the platform itself, sunny-sociopath workplace cultures, etc. Not that a community can't be great even when there's money involved, but "really, there's no money in this -- it's only for the merits and community" seems to scare away a lot of behavior, and the people who are attracted anyway set a tone.)[2]

* Racket is one of those tools that many hackers really like to use, and people generally have good morale when using it.

[1] You also saw this with the SICP professors, who are some of the best-regarded.

[2] Not that I haven't tried to promote commercial use of Racket, despite fear of spoiling a good thing. One of my attempts, I tried to do it while shaking up some usual expectations/modes: https://www.neilvandyke.org/racket-money/

One of the benefits of Lisp based languages is that they usually come with powerful macro based meta-programming facilities.

I've use macro systems quite often in more or less complicated ways: as a professional assembly language programmer, in school when studying Lisp DSLs, when using my favorite editor Emacs and its elisp, in systems I've built similar to Moores TRAC programming language, M4, my extensive use of TeX and LaTeX for decades, C++ STL, sendmail configurations, etc.

Over the years, I've lost my enthusiasm for powerful meta-programming facilities like Lisp macros. The underlying languages are Turing complete and don't strictly need meta-programming, and most modern languages aren't lacking in abstraction mechanisms available to programming without meta-linguistic alterations.

Like operator overloading, sophisticated macro systems change the semantics of program source code in ways that are not obvious. They allow new variants of the programming language to be created willy nilly placing demands on me the reader, maintainer, or user of a programming language package to fully understand the implementation of the meta-linguistic features. Powerful macro systems encourage a thick frosting of magic to be applied on the implementation of complex systems.

Some systems, like Lisp or Scheme or TeX, would be difficult to use without macro extensions, but it seems to me that identifying a good set of built-in abstractions for writing programs and building the language around them is a better approach. I am so grateful for the TicZ graphics package for LaTeX, it's all built out of TeX's crazy flexible macro system, but I'm even more grateful that I've never had to touch the source for it. Take a peek at: [1].

[1] https://github.com/pgf-tikz

I think it all comes down to what kind of programs you're writing.

For example, without a powerful macro system, something like the nanopass framework [1] would not have been possible.

Sure, you could write an external code generator, but at that point you've just implemented a bad macro system.

[1] https://github.com/nanopass/nanopass-framework-scheme

I only learn new languages to be able to do something new. I learned BASIC to make my atari do what I want. Then 6502 assembler and ACTION to make it do it faster. Then Pascal to have complete flexibility of data structures. I learned C++ as a Pascal with dumb syntax but never used it before Pascal went completely out of style because there was nothing new C++ could do for me. Then I learned Java because you could do applets with that. Then HTML because you could make the browser display what you want. Then PHP and SQL because you could build websites with that. Then Python because PHP sucked for console programs and JS because you could make things happen without bothering the server. Then some XML and XSLT because that could process a lot of data fast on client side. Then C# because that was the comfiest way to make desktop apps since Delphi kicked the bucket. Then I learned Ruby because I was assigned to project written in it, but promptly forgot it since then because it could do only the things Python could already do. Then I got back to Java since you could make your phone do what you want with that but Java is (was then?) worst language I know so it was short lived. I still retain it though for occasional utiliy, like extrnding Solr or playing with Apache NiFi.

The only language that kind of breaks away from this pattern of necessary imediate empowerment was CoffeeScript. It just exactly mirrors my way of thinking and was just an inch away from pseudo code I used for my notes since primary school. But then ES6 came and gave me enough CoffeeScript to almost be fine without it. Final nail was TypeScript that gave me stuff I wanted, smart, fast code completion and typechecking for places where I wanted types. Now if I could just have an editor that could display curly braces as indented blocks (python and coffee style) I'd be perfectly happy with state of browser coding.

I tried Go, Elm, Haskel, Scala but nothing stuck or even went beyond simple programs. Nim was interesting because allowed you to run code at compile time to transform code (like Racket macros). I might use it for console programs that need speed (although I'll probably just dust off C++).

Rust so far has the biggest potential because it allows you to have code running concurently without crazy bugs by forcing you to specifically track who owns what and for how long. That might be useful for me to make programs faster at some point since multicore is now firmly a thing.

I love Scheme and Racket. Personally, I prefer Scheme over Clojure and Common Lisp. It supports multiple paradigms well but not too bloated as Common Lisp, having a really good optional gradual type system, really easy to use reader macro system, etc.

I always wanted to use Racket in a bigger project to have a deeper understanding of macro/language creating. I always believe to achieve real 'domain driven design' is to create a layer of real business language which could interpret to a software system.

However, every time I want to do this I found Clojure is actually a much better choice. I guess to be fully practical is not #1 priority for Racket right now. But I really hope Racket can improve some of the following:

1. Encourage efficient data structures by default. I know lists are the soul of lisp but it's not good to use lists for everything. Clojure by default let you use highly optimized persistent data structures -- namely vectors and hash maps. These two data structures are highly practical, performant in most of the cases.

On the other hand, lists are more like write-heavy data structure, with really bad reading performance. This is like, a plain file system writes faster than databases, but most of the websites use a database because most of the business has much more reads than writes.

2. ClojureScript. JavaScript is a big thing until WASM fully arrives. Clojure has several really solid ClojureScript workflow, which makes me feel ClojureScript is really a first-class citizen.

3. IDE and debugging. I use Emacs + Geiser for editing, but Drracket for debugging. Drracket is really good, but still not great for editing hundreds of files. For Clojure, Cider and Cursive are IDEs makes me feel solid and complete.

4. Frameworks. I guess if the other 3 points are really good there would be many good frameworks come out every day.

I have to type this as a quick stream right now.

> 1. Encourage efficient data structures by default. I know lists are the soul of lisp but it's not good to use lists for everything. Clojure by default let you use highly optimized persistent data structures -- namely vectors and hash maps. These two data structures are highly practical, performant in most of the cases.

Scheme has vectors, and Racket adds hash tables and structs:

https://docs.racket-lang.org/guide/vectors.html https://docs.racket-lang.org/guide/hash-tables.html https://docs.racket-lang.org/guide/define-struct.html

There's also some work on matrices and arrays (beyond Scheme vectors): https://docs.racket-lang.org/math/index.html

There are some older libraries where lists (or alists) are used, when today you'd probably use hashes or structs. We could consider this an educational opportunity: there are still times when knowing how to do old-school list-processing is exactly what you need, and it's pretty fundamental data structures (e.g., singly-linked lists, trees), so we could consider it practice. :)

BTW, one difference between modern Racket lists and Scheme's is that Racket's default pairs are immutable. This turns out to be useful for optimizations, as well as encourage a healthy amount of functional programming.

Regarding #2, good point. I raised the WASM issue a couple years ago, and my thinking then (and now) is to build it for the forthcoming Chez backend, while getting plugged into the WASM standards work in the meantime. There are various ways to do JS with Racket (a big HTML5 Offline app of mine does it by generating JS and HTML from Racket), but WASM seems most promising.

Regarding #3, if Geiser works for you, great. You might also take a look at Greg Hendershott's racket-mode for Emacs: https://github.com/greghendershott/racket-mode

I appreciate that DrRacket's student-emphasizing IDE is very different than most IDEs, and it's missing some things I'd like, for editing many modules at once. If you're up to adding some features you'd like, there's an extension mechanism, and you can also do git pull requests to the DrRacket source itself. https://docs.racket-lang.org/drracket/extending-drracket.htm... https://docs.racket-lang.org/framework/index.html

Regarding #4, there are Web frameworks, and the main Racket Web Server (which you don't have to use; you can also do things like SCGI or proxy another HTTP server implementation), and you can also whip up your own frameworks very rapidly in Racket unlike many languages. I'm hoping a couple startups use Racket to get to launch, and release the light frameworks that they make along the way.

Regarding reader extensions, Racket has those, as well as a ton of great syntax extension mechanisms: https://docs.racket-lang.org/guide/hash-reader.html

An addendum to point 3:

Racket has an AMAZING debug instrumentation & tracing library, which, unfortunately, most people don't know about:

First, there's Medic:

Source: https://github.com/lixiangqi/medic

Docs: https://docs.racket-lang.org/medic/index.html

Demos: https://www.youtube.com/playlist?list=PL_U7i0VKF_mh7Vh3o2Yyt...

Paper: https://www.cs.utah.edu/plt/publications/fpw15-lf.pdf

Li, Xiangqi; Flatt, Matthew - Medic: Metaprogramming and Trace-Oriented Debugging (2015)

Building on top of Medic, but unfortunately still not packaged (unlike Medic), Li & Flatt developed (the somewhat ill-named, due to that name overlapping with something from the cryptocurrency crowd) 'Ripple', which makes debugging Domain specific languages a lot slicker:

http://www.cs.utah.edu/plt/publications/sle17-lf.pdf

Li, Xiangqi; Flatt, Matthew - Debugging with domain-specific events via macros (2017)

There's a YouTube demo here:

https://www.youtube.com/playlist?list=PL_U7i0VKF_mjF_SMiPbz-...

The published version of the above paper sits behind an ACM paywall, however, the download of the 'artifact' is open/free…:

https://dx.doi.org/10.1145/3136014.3136019

…and currently unfortunately represents the only way one can acquire the Ripple source code - and the artifact consists of a 2.4GB VM! :| (I understand why, and I consider it good scientific praxis - but I'd still appreciate a public repository in addition.)

Direct link to the artifact, for the impatient:

https://dl.acm.org/ft_gateway.cfm?id=3136019&type=zip&path=%...

Note that the artifact actually contains two very slightly different versions of the Ripple source code, a diff of which I posted over here on Github:

https://github.com/vygr/ChrysaLisp/issues/5#issuecomment-424...

(it's a two line difference in main.rkt)

If you do any of the in Racket, I can most highly recommend giving Medic (and Ripple, if you need it.) a try!

> highly optimized persistent data structures

These are not "optimized" with regard to their ordinary mutable counterparts.

> These two data structures are highly practical, performant in most of the cases.

Immutable hashes are highly impractical when you need regular mutable hashes, which is most of the time.

I really recommend that book, not just this section. I really enjoyed it and it was crucial for me in creating a talk I gave after reading it about the fun in creating little languages.

As i understand, the beauty of languages like LISP and Racket, is that, you can easily compose functions. Basically, as i read source code, i'm reading a "programming composition sheet", just like music sheet.

The only annoying thing, is how to reduce brackets ( and ) from distracting content from its layout.

Of course Python is not the answer (due to its strict identation on space/tabs)

If i could teach newscomer about programming, i would say: Programming = composition of functions.

A couple prior discussions:

https://news.ycombinator.com/item?id=9268904

https://news.ycombinator.com/item?id=8206038

> 1. Everything is an expression.

I am not sure this argument is given.

Immediately, I see there is a mismatch between what I want to program -- an objective, an algorithm, a procedure, a series of side effects that is outside the axioms of the programming language can support -- a mismatch between these objectives and this confinement of everything being an expression. An expression is a value. So to program in a language where everything is an expression is to map our idea into a (list of) value. This may be natural for programs that is seeking a value, but often not so. Even for programs that is seeking a value, the bulk of the program is to control the process of finding this value. There is no easy way or even correct way to map a process and side effects into a value. Math is logic or equivalency. To establish equivalency is to discard the effect of path or side effects. Therefore, to map the desired process and side effects into value, we have to add back the implicit knowledge of how these values are actually transformed. In stead of directly stating the transformation of values -- an imperative programming style -- we express that with dependency and relying on the understanding how these dependency is being resolved. The latter is very hard.

Of course, in practice, we have to give up on the fine control of our program to some extent and relying on the compiler implementation giving us desired result (the path and its side effects), then we only need worry about the value. When it works, it works great; when it does not work, we need either lower our expectations or give up the language, or transfer the burden/blame to compilers.

The argument for everything being an expression is the composability(although I thought the reason was ease and flexibility of writing compilers for it). This is similar to the argument that: if every object is a lego piece, then building something is easy. Well, it depends. First we need accept that lego pieces are all what we have. Second, we have to contend that what lego pieces can build is good for our needs. There are amazing lego projects, but they are nowhere I would find easy.

Well, in languages like Scheme and Lisp, expressions are returning a value. But they can also be control structures. For example IF forms are an expression, but IF is also a control structure:

   (if (rocket-engine-running?)
      (start-rocket)
      (start-engine))

When expression is not pure value -- they also can be control structures -- the arguments for "everything is an expression" is being defeated, right? A control structure is for defining the path -- control flow. The values in a control structure is a side effect just as the control-flow is a side effect in an expression language. When control flow is the center of logic, won't a control-flow oriented language -- imperative programming -- be more straight forward?

In your example, after you started the engine, don't you still need `start-rocket`? The bug sneaks in due to discrepancy that while the syntax is all about values, the semantics is all about flows.

It gives us as a developer the freedom to decide: do I want control flow, return values or both?

Racket is great and fun. I find it truly amazing that after all these many years people are still trying to justify Lisp. Frankly - if it was going to be adopted in mass it would have happened by now and no amount of explaining is going to change that. There are many reasons why languages get adopted and "logic" is not the primary one. Take JS for example - it would have been in the dust bin if not for the fact it is the only language that runs in browsers.

Does Racket have a nice DSL for wrapping web APIs? I have a hard time believing it doesn't, but I can't find it. I'm thinking of something like API-Wrap.el (https://github.com/vermiculus/apiwrap.el), but any approach that drastically reduces the boilerplate is fine by me.

Riposte?

https://docs.racket-lang.org/riposte/

Riposte doesn't look all that useful outside of a test framework. I want to be able to use REST APIs in a full Racket program, which seems impossible in a DSL without control flow or functions.

https://github.com/greghendershott/wffi is what you want

Nice write-up.

Waiting for Racket-on-Chez effort to come with more optimizations and multi-core access

Just to be clear: multi-core already works with racket places(basically running a separate racket vm thread) :).

Chez-scheme has a pthread-style interface to OS threads, which has it's own set of problems with respect to concurrency(knowing which operators are thread safe). Not sure how racket would expose this.

Beautifully written post.

This guy makes some statements about python that are completely and utterly wrong.

He implies python doesn't have an "if" expression. It does.

He specifically claims this is invalid in python implying that "if" expressions don't exist in python:

   x + (if is_true(): 1 else: 2)

The following is written in python and is correct syntax:

   x + (1 if is_true() else 2)

Although it's not "pythonic" to use python functionally, Python is a multi-paradigm language and has the facilities to be used as a very powerful functional language. It's not just a "functional" library. Functional is built into python syntax. The following is correct python syntax that will pass a type checker process as well.

   #List comprehensions using map and filter:
   x: List[int] = [i+2 for i in range(100) if i%2 == 0]
   #anonymous functions:
   x: Callable[[int], int] = lambda a: a + 2

With mypy, type annotations or any other external type checker, python approaches the power and correctness of typed functional programming languages like haskell or Ocaml, though it is missing many features.

The OP isn't trying to criticize Python... he was just trying to give an example of what it means for everything (in LISP) to be an expression. He didn't say that Python doesn't have an "if expression"... just that you can't use the if statement in an expression context.

OP literally says this:

In Python, an if conditional is a statement, and can only be used in certain positions.

This is categorically wrong. In python the if conditional exists in statement form and expression form. The if-expression also basically exists in every other imperative language out there, usually in this form:

   x + (is_true() ? 1 : 2)

I know he's not trying to criticize python. Just want to correct his mistake and emphasize that python has intrinsic design features that can make it very very functional.

Yep. Also most c-derived languages have 'progn' as the comma operator. Expression-...ism? is not really a problem.

I think you just misinterpreted the point he's making. Yes, Python has an expression form for if, but it's a separate form. You can definitely write functional Python, but the point he makes is that Python differentiates between expressions and statements.

No I made no mistake. I get his point. Look at my reference. I literally posted the statement that is categorically wrong. He literally says that in python the if-conditional is a statement. This is False.

I also offer some proof as to why python is a bad choice for his examples because python is actually a highly functional language. It's like trying to use haskell to prove racket is more expressive. It's a bad choice.

Sometimes I don't understand people. I literally posted just factual errors, no opinions on his article and then people misinterpret everything I said as opinion and proceed to tell me I misinterpreted things, then vote me down. Seriously.

He mentions the expression form of 'if' right after the part you quoted. I'm not sure what you're trying to get at.

I don't understand, he seems to be dealing with that syntax like two lines later?

...

“But wait! Python has a ternary conditional expression!” It doesn’t change the essential point, but OK—you can indeed write this:

42 + (100 if 1 < 0 else 200)

But now suppose we want to use a conditional in place of the operator, not the right-hand value. In a Lisp, because everything is an expression—including the operator itself—this is easy:

((if (< 1 0) + * ) 42 100)

But if you try the same thing in Python, it will raise a syntax error:

42 (+ if 1 < 0 else * ) 100

...

Hmm, my mistake I stopped reading after I saw he said something false. Guess I should read the whole thing instead of dismissing something after I see a false statement.

I've long stayed away from Lisps, precisely because of what the author describes in the first section, that there is a huge amount of praise over all of it's wonders, without ever specifying what those wonders are or how they are actually good things. I've had similar issues with the cargo-cult following of Rust; yes I have heard it is great, but WHY!? Great article, it de-mystifies these vague praises and addresses them clearly and specifically. I'm gonna try Racket this afternoon =D

I'm no 'rustacean,' but rust's benefits have always seemed fairly straightforward. The borrow checker provides static guarantees for thread- and memory-safety, and allows for safe, performant low-level systems code without the need for a GC.

Now that is some good straightforward information! Much more clear than most other comments I've read, and a better aggregate of features than Rust documentation offers. Thanks for chiming in!

Reply on news.ycombinator.com