How Hy backported “yield from” to Python 2 | Hacker News Reader

How Hy backported “yield from” to Python 2

dustycloud.org

120 pointsaroberge22 comments

They wrote a Lisp, and added a feature to the Lisp, which they can do really easily because Lisp has macros.

Their choice of runtime environment (Python 2) doesn't impact anything. They didn't port "yield from" to Python 2 the language, they ported it to Python 2 the bytecode interpreter.

You don't need macros in Python. In Maslo's Hierarchy of Programming you can get by with first class functions and an AST module just fine. But it is a fine day when you can generate code using the same primitives you would use on plain old data.

It's not the most interesting example of macros but it does illustrate one small point: we can use yield-from in an expression as if it were any other symbol in the language. This is what Lispers mean when they say that you can program the language with macros. They're mini-compilers. It's when you start layering them together that things start to get interesting. You can use the yield-from symbol in any valid place within an expression... and since Hy is all expressions and has no statements it gets expanded out and put in the right place and just works.

In languages without macros you have design patterns. The lack of facility for linguistic abstraction forces us to use primitives to describe more sophisticated phenomenon than our language provides by default. So Twisted has had sub-generators in Python 2 for a while and they work rather well via its Deferred abstraction. It works well and I was glad to have it... however its implementation is much more complex than the macro described in this blog post. We had to wait for Python 3 to have them integrated into the language. Macros can bypass that which is the point the OP was trying to make.

The point I'm making here is that you can do pretty much everything you want to do without macros. You'll just have to write quite a bit more code and come up with many more metaphors and analogies in order to get it to work. Macros just let you get straight to the point by extending the language.

The point you're making here is that having more tools for abstracting things results in easier time abstracting things. Which is of course true, and Lisp usually provides at least one tool (syntactic abstraction) more than other languages.

There are however - perceived or real - downsides to providing too many such tools. Personally I don't believe they outweigh the benefits of having more powerful tool in hand, but if you ask Java, Python or Go designers you'll hear a very different story.

It's not about need - you really only need Brainfuck to do anything doable with any other language - it's about a set of trade-offs language designers and programmers make. Personally I'm ready to sacrifice much for expressive power, however I understand that people working in different environments are more interested in less expressive but safer tools.

I wasn't talking about whether macros are useful to everyone.

The grandparent comment was dismissive without providing constructive criticism.

Of course you can implement yield-from in Python 2. Of course Hy generates Python AST. We know that. It doesn't make the yield-from macro any less interesting as I hope I've pointed out.

Update Perhaps it was the title that is misleading.

I was only referring to your last point, specifically this:

> The point I'm making here is that you can do pretty much everything you want to do without macros. You'll just have to write quite a bit more code and come up with many more metaphors and analogies in order to get it to work. Macros just let you get straight to the point by extending the language.

This is all true, but it's also very one-sided. It makes macros look like a universal solution to all abstraction problems and without any downsides at that. Now, as I said, I don't really believe that the "downsides" are that important, but I wanted to add to your comment by pointing out their existence.

I also didn't read the GP comment as dismissive, it seems a bit pedantic at worst.

> This is all true, but it's also very one-sided. It makes macros look like a universal solution to all abstraction problems and without any downsides at that.

They are a universal solution to the problem of abstractions: they let you use fewer abstractions!

> I also didn't read the GP comment as dismissive, it seems a bit pedantic at worst.

I can see that. I wasn't being very charitable with my choice of words. Apologies to GP.

We can look at it like this.

You essentially have macros in Python: they are hard-coded in the form of phrase structure rules in its grammar.

Macros are essentially this: new phrase structure rules in your interpreter/compiler that you program yourself and stick into your application.

What you're really saying is actually that you don't need any new macros beyond the phrases which the benevolent dictator for life has bestowed upon you in the parser.

Whenever a programmer upgrades to a newer Python which has new syntactic transformations in its updated grammar, and then uses these new things, that programmer contradicts his or her earlier claims that no new macros were needed.

I don't need everyone else in my shop making horrible sloppy messes at an unnecessarily high level of abstraction using macros. I'll be picking up after their cute shit at the end of the month, under emergency conditions. There's currently no problem with them recompiling Python to make weird one-off syntax because that's clearly crossing a line. I could add my own syntax and recompile, but I legitimately don't need or want to. No, I don't need a general runtime macro capability in Python. It's not a good idea and it doesn't buy anything I actually need.

Take OCaml for comparison: It has a more or less reasonable way of extending the language with an extensible parser camlp4 that ultimately yields a fixed AST. No one would get the idea to use it just for fun, or one-shot abstractions, but it is integrated into the toolchain well enough that you can reliably distribute libraries that use syntax extensions. The general attitude that "coders are dumb, we need to protect them from themselves" lead to the development of mediocre languages like Java. Compare that to something like Racket, where despite some flaws most of the macros provide an obvious and very real benefit, without overcomplicating the compiler (for example the match macro, or the macros that constitute the C-FFI). Ideally in my eyes a language should provide the programmer with maximal possible expressiveness without compromising a core simple design.

So... don't use it? Nobody is suggesting we add macros to Python.

Macros do not make code unmaintainable by their virtue alone any more than pointers, references, or any other language feature.

I suggest you look into code review and use some code checking tools like style checkers, linters, and test coverage report generators.

An excellent way of looking at it.

Hy is essentially a self-evaluating macro that spits out Python AST.

So you can use map, filter, and friends just as easily on Hy code as you can elements in a list or tuple.

In the end you get Python AST. We generate all of the boiler plate instead of writing it out.

You also do "have" macros in a very obtuse way in Python. You have eval, compile, and the ast module. You can write code, in Python, to modify or generate Python code. It's just tedious and cumbersome in a language where the code is not also data.

> You have eval, compile, and the ast module.

Using this to write normal code is also a bad idea and a maintenance nightmare. The issue isn't that it is tedious and cumbersome to do so. If it's tedious and cumbersome to make my life hell, that's a feature.

Oh... you must ban the use of the namedtuple function in the collections module then. How unfortunate. It's pretty useful.

Yes, and that really highlights the limitation of using the Python syntactic interface to its implementation, compared to a Lisp interface.

However, even if the Lisp approach cannot be used directly, it still provides proof-of-concept evidence. If a macro in the programmable-syntax interface to the machine can achieve the feature, then in all likelihood, the feature could be also implemented as a similar hack in the Python parser where the equivalent of that macro is implemented as a hard-coded phrase grammar rule.

Of course, deploying such a change is cumbersome because installations have to update their Python images to the patched ones. Oops! Whereas the Hy installation doesn't have to be upgraded in order for a new macro to be used by a program.

The Python interpreter has support for custom code loaders, so you can use the new macro using plain Python, as long as the macro machinery is loaded before the other code to set it up.

See importlib: https://docs.python.org/3.1/library/importlib.html

Chris gave a great talk about Hy at ChiPy (Chicago Python) last year: http://pyvideo.org/video/1869/hy-a-lisp-that-transforms-itse...

You should also check out his work on MediaGoblin. "MediaGoblin is a free software media publishing platform that anyone can run. You can think of it as a decentralized alternative to Flickr, YouTube, SoundCloud, etc." http://mediagoblin.org/

I really should play more with Hy someday. I discovered it when I was still too new to Lisps to make sense of such a new project (and such new and experienced-user-oriented documentation).

Nothing to see here. Even the PEP of yield from contains code simulating a 'yield from' with plain 'yields'. The thing is, you are just simulating it with the expense of switching generators back and forth.(like a trampoline)

inspect module offers isgenerator and isgeneratorfunction. the author should have used that.

https://docs.python.org/2/library/inspect.html#inspect.isgen...

Which does exactly the same as what the code in the example does.

Looks like they use defmacro/g! from let over lambda.

Reply on news.ycombinator.com