Hy – A dialect of Lisp that’s embedded in Python

i started migrating a project from clojure to common lisp so that i could move the ui from swing to qt. i should really give hy + pyqt a try as well.

That's fascinating. What is performance like?

If you are a lisp-enthusiast, Python comes with strong support for data science and web applications, if nothing else.

If you are a Python enthusiast, lisp-like languages can be used to write DSLs or cut down on boilerplate code in a way which isn't as easy in Python or similar languages.

But yes, in general, a lisp dialect on Python has the problem that Python itself is a great language.

Python has one of the biggest standard libraries there is, there is a lot of useful stuff in there already provided, plus there is a humongous amount of third-party packages available through pip. It also generally has a pretty good C FFI, so there's a lot more support of C-libs in Python.

It's not so much about any specific libraries as it is about the volume of support one gets 'for free,' and as well I think Python syntax and semantics alike really get along better with Lisp than some of the other dialects that rely on FFIs (like Clojure to Java/JavaScript, or the number of C FFIs available for Schemes).

Lisps in general don't tend to include much of a standard library, opting for a 'batteries not included' model, with Racket being perhaps the most prominent (and quite deliberate) exception. So the idea of getting the whole Python ecosystem 'free' is potentially very appealing, especially to someone with a Python background (and there are a lot of those someones around, especially with it increasingly being a more popular teaching language than anything else except JavaScript).

This comes with any Python, out of the box: https://docs.python.org/2/py-modindex.html

What stands out will be different for different people. I use the ftp and telnet libs a lot, I've used the imap and email libs.

Here's a repackaging of a large number of 3rd party libs, packaged as an alternative python distribution, which I use at work: http://docs.continuum.io/anaconda/pkg-docs.html

I've explored twisted (networking), numpy and scipy (math and scientific), and sympy (symbolic math).

IPython's shell and notebook are great work environment enhancers. http://ipython.org/

Here's Python's larger set of 3rd parties: https://pypi.python.org/pypi?%3Aaction=browse

After a quick IRC discussion; you wrote it wrong :D! Using hy --spy so you can see the equivalent python code:

    => (list-comp (, x y) [x (range 5) y (range 5)] (> 2 x))   
    from hy.core.language import range   
    [(x, y) for x in range(5) for y in range(5) if (2 > x)]    
    [(0, 0), (0, 1), (0, 2), (0, 3), (0, 4), (1, 0), (1, 1) .....]

Can submit a PR and fix the docs so this is better displayed!

For me, one of the simplest explanations of the difference between macros and functions is the short-circuiting and.

In many languages, and does not evaluate all of its arguments, only as many as it needs to until it finds #f. This sounds like an easy thing to implement, but in most languages it actually isn't, because order of nested operations means that without being asked, it will evaluate it's arguments before our and.

So we can write this broken-and easily enough:

  (define (broken-and . rest)
    (cond
      [(null? rest) #t]
      [(car rest) (apply broken-and (cdr rest))]
      [else #f]))

But give it the following, and you get an error:

  > (broken-and (zero? 0) (string? "dave") (null? '(is fat)) (/ 5 0))

  . . /: division by zero

And that will work so long as all your args are valid. But what will work better is to write a macro. Macros aren't evaluated, they merely splice in their syntax, but we can write them a lot like code, using all our control structures and so forth, and in Scheme/Racket, also some cool pattern matching stuff):

  (define-syntax my-and.v2
    (syntax-rules ()
      [(_) #t]
      [(_ a) a]
      [(_ a b ...) (if a
                       (my-and.v2 b ...)
                       #f)]))

Now we can try our input from before, and get what we want, because the macro expands recursively and won't ever get to our divide by zero, at least unless all preceding args are true:

  > (my-and.v2 (zero? 0) (string? "dave") (null? '(is fat)) (/ 5 0))
  #f

This property of only expanding, not evaluating, is why macros can write all kinds of clever control structures, because you're mangling syntax, not evaluating code.

Fundamentally it's macros and homoiconicity of code and data that give lisp it's expressive power. Combined they grant you the power to inject new syntax / domain specific languages (DSLs) directly into your code. Any new syntax you create can look like normal lisp or can have different syntax evaluation rules like infix math notation [1]. A number of years ago I did a short write-up on stack overflow giving a basic explanation of how macros work in common lisp which may be worth a read [2].

Macros in general work by being a syntax tree -> syntax tree transformation. The neat trick is it's not a static pattern like C - you can leverage functions, conditions, etc to build that syntax tree.

As for comprehensions themselves, if you want to see the power of lisp look at the loop macro. It's a single macro yet gives you the power of normal control loops, list comprehensions, for-each all rolled into one package. [3]

[1] http://data-sorcery.org/2010/05/14/infix-math/

[2] http://www.gigamonkeys.com/book/loop-for-black-belts.html

[3] http://stackoverflow.com/questions/267862/what-makes-lisp-ma...

You can create a list comprehension using a macro. In common lisp, lisp comprehension is implemented by the loop macro. http://www.gigamonkeys.com/book/loop-for-black-belts.html

It is a little more involved, but basically you define a syntax for list comprehension and then parse it using the macro to generate the final result.

Python natively supports this sort of list comprehension (https://docs.python.org/2/tutorial/datastructures.html#list-...), and the Lisp source is being translated into a Python AST, so that snippet is probably being directly mapped into whatever AST form is necessary to invoke the list comprehension.

You can create macros that look like functions, but the macro actually has access to the the "program code", so you can do arbitrary transformations.

The implementation is easier if you assume that you have exactly two generators. Here is an example in Racket. It uses define-syntax-rule that is the method to define simple macros (there are more advanced options).

(The program also define string->list/string that splits a list in a list of one-character string, because the result of string->list is a list of chars. You can ignore it safety.)

  #lang racket

  (define (string->list/string x)
    (map (lambda (c) (list->string (list c)))
         (string->list x)))

  (define-syntax-rule (list-comp2 (proc v ...)
                        [u1 gen1]
                        [u2 gen2])
    (map (lambda (u1 u2) (proc v ...))
         gen1
         gen2))
  
  (list-comp2 (list x y)
    [x (range 8)]
    [y (string->list/string "ABCDEFGH")])

  ;==> '((0 "A") (1 "B") (2 "C") (3 "D") (4 "E") (5 "F") (6 "G") (7 "H"))

There's an excellent paper on how to build list comprehensions (in Common Lisp) on top of simpler iteration constructs:

http://www.ai.sri.com/~latendre/listCompFinal.pdf

I have a (very loose) implementation of this approach at:

https://github.com/TBRSS/cl-lc

Frankly, though, once you've learned the trick, list comprehensions lose a lot of their appeal.

How to implement list comprehensions in Lisp is an open-ended topic.

For instance, I did it once in a "researchy" way by implementing monads. With the help of CLOS, I was able to define various monads such as the list monad, identity monad, or state transformer monad (the map, join and unit being methods dispatched on the monad type).

Then a generic monadic comprehension macro provides different functionality based on which monad is selected. The list monad gives rise to a list comprehension. The identity monad gives rise to just sequential variable binding. And the state transformer monad ... to something like a pipeline of succcessive state transformations, I think.

http://www.kylheku.com/cgit/lisp-snippets/tree/monads.lisp

If you just want a list comprehension, it's not very difficult. Basically it has to expand to code which (for example) expresses a nested loop that steps every variable over its respective list. The expression being collected by the comprehension is stuck into the middle of this loop, inside a piece of code which collects its value into a list, which is stored in a hidden local variable. (Perhaps two local variables are used, to keep track of the head and tail of the list for the sake of efficiency.)

ANSI Lisp has functional applicators that process multiple lists. For instance if you want to add together values from two lists, you can do this:

  (mapcar '+ '(1 2 3) '(10 20 30)) 

   -> (11 22 33)

If you wanted to form a cross product of the two lists instead, the obvious thing would be to write a mapcar-like function:

  (mapcar-cross '+ '(1 2 3) '(10 20 30))

   -> '(11 21 31 12 22 32 13 23 33)

This function could be used as the target syntax for a comprehension macro so that, say:

  (list-comprehend (+ b a) (a '(1 2 3)) (b '(10 20 30)))

   -> '(11 21 31 12 22 32 13 23 33)

is macro-expanded into

  (mapcar-cross (lambda (a b) (+ b a)) '(1 2 3) '(10 20 30))

Translating (transliterating, really) the list-comprehend syntax into mapcar-cross is not very difficult: it's just some very straightforward nested list manipulation. The macro has to collect the list of variables from the trailing arguments after the expression, to form the (a b) argument list of the lambda. It has to remove the variables from those trailing arguments to produce the list expressions like '(1 2 3) and '(10 20 30) that will be applied to mapcar-cross. Then it has to assemble the mapcar-cross syntax.

Of course, you also have to write mapcar-cross, which is your "run-time support function" for your list-comprehend syntax (usable directly without that syntax, too).

"Real" List Comprehensions in 24 Lines of Lisp[0]

[0]: http://lisp-univ-etc.blogspot.com/2013/01/real-list-comprehe...

Macros (the general Lisp concept) are really interesting. I'm still trying to grok them myself, so if I made any mistakes please downvote and correct away.

A normal Lisp function call might look like this:

(myfunc arg1 arg2 arg3)

where arg1, etc can be literals, variables, expressions. Like in most programming languages, each of the arguments is evaluated and then the function is called with the evaluated values of those arguments. When it's done, a value is returned.

A macro is a lot like a function call:

(mymacro arg1 arg2 arg3)

However, it works differently. First of all, the arguments are passed straight into the macro - if arg1 is something like (+ 1 2), the macro will see (+ 1 2), not 3 (as the function would).

Secondly, the macro is intended to return a list, which can be a function call or another macro call. (The macro could also return a final value.) The list can then be evaluated to either expand another macro, or to call a function to produce a final value. So you can think of a macro as a special sort of function that, rather than taking in values to produce another value, takes in literal code to produce transformed code (which then can be evaluated to produce a final value).

Now, one of the uses of macros that people cite is, for example, implementing your own control flow structures. Why are the two related? Lisps have something which are known as special forms, which look like function calls but don't have their arguments evaluated before being passed in (like how macros work). The reason that matters can be described by a simple example: let's say you wanted to implement a very simple version of 'and' as a function (Clojure syntax):

(defn my-and [left-expr right-expr] (if (not left-expr) false (if (not right-expr) false) true))

However, the problem is that left-expr and right-expr will both be evaluated before my-and can even begin executing, along with the attendant side effects, etc. The problem is that functions can't choose whether or not their arguments should or shouldn't be evaluated - they just get the values of their arguments after they've been evaluated[1]. It's not really possible to implement the traditional short-circuiting 'and' using functions, or many control flow structures where you do not want to evaluate every possible branch unconditionally.

However, macros receive the literal code comprising their arguments, and they can choose to evaluate one, none, some, or all of their arguments depending on their internal logic. This makes them better suited for implementing behavior that resembles those of the Lisp special forms.

[1] This is the difference between call-by-name and call-by-value semantics. Most languages are call-by-value, as described above, but (e.g.) Scala has '=>' and Swift has '@autoclosure' to allow for some sort of call-by-name support.

It's a kind of rite of passage, I think, to implement a Lisp. I wrote half (the easy half) of ISLISP that ran on the Python VM back in 2005 :)

Serendipity :)

It dosn't stop there. Why use the decorators directly when you can get Hy on Meth? https://github.com/hylang/hy/blob/master/hy/contrib/meth.hy

Well, that most likely won't change. Hy's goal is to be bidirectional, so there are a few things here and there that doesn't feel as lispy as it should. Like having data returned when you use assoc as an example. The scoping is an issue aswell, like let-over-lambdas doesn't work, and iteration over maps isn't as fluid as it is in Clojure.

The entry is misleading. We don't really do TCO. It's just a trampoline calling the loop/recur form until there is nothing left to produce.

Same solution clojure has, as the JVM dosn't allow for TCO.

Can you give an example of where you find Clojure to be implicit?

I don't find the Clojure that I write to be implicit.

What about Clojure seems dense or implicit to you? Clojure and Python have a similar feel to me. Both are well-designed in the sense they strive for a small set of primitives useful in solving a broad range of problems.

You don't need to translate it -- you can import modules both ways just fine -- THAT's the killer feature.

Being an Emacs head, I like sexp based syntax for edition and navigation. Now I can edit hy like any lisp and push it to python hosting servers.

my 2 cents.

Helluw, one of the core devs here. This was originally an idea just to display whats possible with the Python AST. It's pretty neat. The result is an kinda fancy lisp variant with bidirectional interop, macros and all the fancy lisp stuff.

It will import cython stuff, and vice-versa.

By python-sh do you mean http://amoffat.github.io/sh/ ?