Parinfer – Simpler Lisp Editing
shaunlebron.github.io
shaunlebron.github.io
I don't think this is true. I develop Cursive and I use it, despite being competent at paredit. A ton of my users do too, even experienced ones. It's just easier, and it turns out that a lot of editing is actually line based, even in a lisp. And when it's not, I can still use the paredit commands in parinfer mode, at least with Cursive.
It's referenced in an edit to noctuid's document, but many of those criticisms are no longer valid since smart mode became the de-facto default for parinfer. It's now really intuitive. Parinfer's worst problem now is its difficulty inferring parens when there are multiple expressions on a line (i.e. when there's no indentation to infer from), but it's surprising how infrequently that's actually a problem.
Now, Cursive is used by something like a third of Clojure users. Half of them use Emacs, and in Emacs land there are rather few Clojurians who seek something like parinfer. Even when they do, it's a usual "how we can make this more palatable to outsiders" type of quest rather than "I'm really frustrated by how Emacs does Lisp editing. I'd like something like parinfer"...
FWIW Emacs users do ask about parinfer in #parinfer on Clojurians because they actually want to use it. There currently isn't a smart mode for Emacs though, and that's what I consider the game changer. I suspect that uptake will be good once that's available.
If brackets can be inferred from indentation, doesn't this imply that they're extraneous — that indentation only would be sufficient to write many Lisp structures?
Alternative notations were originally planned for Lisp when it was designed - some books were published using it, but for actual programming it wasn't used too much - especially since most implementations actually a) didn't support it or b) had s-expressions as a default/internally.
Then several attempts to modernize Lisp with a different syntax mostly failed to get traction or failed big (like the Lisp 2 effort).
Then some efforts failed to provide more than one syntax and also failed to appeal to people (Dylan).
What we saw is a bunch of languages derived from Lisp with different syntax (ML, Logo, ...) and a bunch of Lisps for mathematics with different syntax (Reduce, Macsyma, ...).
But the core Lisp programmers were never willing to give up the s-expression-based syntax. It has some clear advantages when manipulating code, especially in interactive programming.
I can see the value of homoiconicity / code as data from the perspective of someone writing a parser.
I've asked what writing code as s-expressions gets you as the end-user of a language before. People usually say, “macros” or “it's easier to manipulate code”. But macros exist in non-homoiconic languages, like Nim and Elixir. And in my short time using ClojureScript I'm not sure what manipulations Lisp syntax gave me that I don't already have with other languages in a modern text editor.
I'd love to see examples of accomplished Lisp users on YouTube or elsewhere, to see what those code manipulations look like.
I think I keep lisp syntax~ at heart because its nature made me learn and think recursively which is something of a rare value (and I'm not saying this as a cult, it really made me solve problems I couldn't in ways I was confident it would either work or be clear that my perspective was wrong, very very nice feeling). Another example is that I still prefer car/cdr even though they're laughably cryptic .. but they embody the whole culture (and have a tiny symmetry to it).
One reason Lispers find macros so beautiful is the simplicity of writing macros using the exact same list processing functions. The code-is-data thing is reinforced through the fact that the AST is one-to-one with the syntax.
See for example this program from McCarthy, 1960. On top is my Common Lisp translation and below you find the original code. The original had to be hand-translated into s-expression syntax to run it in Lisp - in 1960.
https://gist.github.com/lispm/e44d81c3bb9b86d4313763647e058a...
For a Lisp 2 (Lisp 2 as the successor to the original Lisp language - with the long promised notation) example see: http://www.softwarepreservation.org/projects/LISP/lisp2/SP-2...
> But macros exist in non-homoiconic languages, like Nim and Elixir.
Lots of languages have some kind of macros: C has textual transformations and Elixir has AST transformations.
Lisp macros are different, since they are not AST transformations but token tree transformations. That means Lisp accepts arbitrary token trees (give that the first item is a macro identifier) and one can write programs to transform them.
Languages which support macros for AST transformations, need to have the code parsed into an AST first. That means the expressions it can transform need to be already parsed into an AST - and the language accepted by the parser limits the expressions - unless one can add syntax changes to the parser, too.
ASTs are special purpose data structures, while s-expressions are general purpose data structures and thus are not limited to expressing programs in a particular syntax and are not limited to express something which looks like a program.
Lisp offers a bunch of macros: macros for transforming Lisp code, macros for transforming symbols, compiler macros for optimizing code, macros for transforming s-expressions, ... Additionally the macro has access to a full Lisp - which means it can do arbitrary computations: primitive transformations, complex language transpiling, full parsing&transformation, doing side-effects in the development environment, running at runtime in interpreters, ...
That means for me as a programmer I can automate all kinds of code manipulation. There are many simple and many complex examples.
Another advantage is that the language core can be reduced to a minimum set of syntactic forms and every other syntax can be built on top of it - and the developer uses the same mechanism as the language core: macros. This makes macros very pervasive - a programmer will use a lot of macros and may also program macros.
One of the disadvantages is that macros also cost us: the code we see can be very different from the code which executes and there is an additional programming paradigm to understand: using syntactic abstractions with macros. One can learn some Lisp without touching macros (see for example SICP, which does not use macros), but in real world programs one might see a lot of macros.
Paul Graham wrote a book, which explained use of macros in Lisp: On Lisp. Free download at: http://www.paulgraham.com/onlisp.html
There are more disadvantages and also attempts to improve macros. See for example: http://library.readscheme.org/page3.html
An example for a typical use of macros is CLOS, the Common Lisp Object System. It has a three or four - layer architecture:
layer 0 : meta object layer for CLOS - here CLOS is implemented in itself
layer 1 : object layer for CLOS - here CLOS deals with simple representations of itself
layer 2 : functional layer for CLOS - here have functions operating on CLOS objects
layer 3 : macro layer for the CLOS programmer - here we have macros providing ways to configure CLOS programs
Take for example defining a CLOS class:0) on layer 0 CLOS has CLOS functionality to represent and create instances of classes and metaclasses, he were can also write extensions to new kinds of metaclasses
1) on layer 1 CLOS classes are objects: which have slots which are objects and which have superclasses which are objects
2) on layer 2 CLOS provides functions to create all that stuff: classes, slot objects, ...
3) on layer 3 CLOS provides the DEFCLASS macro which at macro expansion time - which typically will be triggered by compilation - can do all the transformations and can interface to the development environment at compilation time.
The programmer will typically use layer 3 to define a class using the macro DEFCLASS. For more advanced use one could use the function ENSURE-CLASS
Common Lisp originally in 1984 did not have any syntax for defining classes. The first CLOS implementation was a Lisp program which implemented all the layers above and provided the necessary syntax to define the new language objects: generic functions, methods, classes and some others. So it was mostly a user-level program which added a full complex object system - with only little code for implementation specific stuff.
Thus the DEFCLASS macro could introduce any syntax it wanted as a user level program - since it is not constrained by a fronted parser. So the CLOS developers came up with something that was convenient for them to define classes. This could mean that it does something trivial by providing a different order to define things, make things shorter to write or that it does something more complex by actually checking the provided class configuration and doing some code transformations.
The whole story of CLOS in CLOS itself is written in AMOP: https://en.wikipedia.org/wiki/The_Art_of_the_Metaobject_Prot...
See also the pointer there to the first and portable implementation of CLOS: PCL.
For one example, see https://github.com/abo-abo/lispy/
The author has a similar package for python, but it is not as powerful because of python's syntax.
The author has demos of some of the functionality on his youtube channel: https://www.youtube.com/user/abo5abo/videos
Yes; this deduction is valid.
However, can brackets be inferred from indentation.
Let's start with a trivial case:
[indent]15
Is this object 15? (15)? Or ((15))? How do we apply indentation to distinguish them?Do we count the spaces? Indentation is two-space so that 15 preceded by no extra indentation relative to the current level is just 15, then 15 preceded by two space is (15), and so on?
And is that really going to be readable?
What if there is whitespace?
material material
;; .. vertical break
15 ;; this is (15) but could be 15 if I don't see the preceding material.
There is ambiguity in indentation. If I see this: (1 2 3
4 5 6
where 4 5 6 is the last line in my edit window, I know that this list is not well-formed: zero or more items must follow, and then a closing parenthesis. If indentation is used, I have no idea: I could be looking at the complete thing.I think the example I typically go to is to ask how to automate cutting/moving text around the editor. Having had YAML blow up on my team enough in the past few months, I've grown quite weary of any inference schemes.
Gets even worse, because as soon as you get into the indentation game, you open up the debate about how many spaces. Such that now some of our YAML files are using two spaces, and some more. Portability of the code is basically nil. Worse, it will look right, but won't mean what someone meant for it to.
I really like Haskell as a language, but the fact it isn't based on s-expression is the biggest issue I have with it. Whitespace, braces, etc. are very minor quibbles: they're basically debates about language as a UI. The problem is, that same language is also the API we must adhere to when manipulating code programatically, e.g. writing interpreters/compilers/documentation generators/code formatters/linters/renderers to HTML or LaTex or whatever/static analysers/verifiers/model checkers/IDEs/refactoring tools/etc.
I think it's telling that a language so heavily focused on language implementation (DSLs, etc.) effectively has a single usable implementation (GHC), and a mountain of dead/niche implementations ( https://wiki.haskell.org/Implementations ).
Imagine we're given the path to a file containing Haskell code, and we want to transform it in some way (e.g. replacing calls to one function with another function). The most basic thing we need to do is parse it, but my work on code analysis tools over the past few years has taught me that we can't even manage that reliably.
We might naively reach for the GHC API, but that requires that we set a whole bunch of configuration options that we may not know (language extensions, package databases, commandline flags, etc. collectively referred to as "dynflags"); if we get those wrong, our program crashes saying `the impossible happened!`. We can't, in general, figure out what these should be; the only real solution is to invoke our program via Cabal, and read in the needed values by emulating the commandline flags of GHC. This solution is useless if we have a string of code, without any associated Cabal project.
We might instead opt for a standalone library, like haskell-src-exts. The problem is, those libraries typically can't parse Haskell code found "in the wild". Language extensions are one problem, but another major blocker is widespread use of the C preprocessor (an unhygenic, unsafe macro system based on string substition; which would be largely unnecessary if Haskell used an easily manipulated format like s-expressions instead).
Note that even GHC can't re-use its own ASTs: the Template Haskell extension provides its own AST representation, entirely separate to that used by the compiler's frontend!
Whilst there are layers on top of Haskell like "liskell", which accept s-expressions and produce Haskell code, they're solving the opposite problem: it's easy to convert from s-expressions, since they're so trivial to manipulate. The hard problem is being able to do anything useful with the mountain of existing code which isn't in a nice s-expression format, other than compile it with (some versions of) GHC, if invoked with the right options from (some version of) Cabal; or maybe Stack; maybe after running Hpack; or who knows what else!
I wrote up some of this at http://chriswarbo.net/blog/2017-01-31-syntax_trees.html
Isn't that about the expression problem, though? How would changing the syntax help?
If we think of the current GHC implementation in this way, we find that the only de facto representations of Haskell code is `ByteString` (or `Lazy.ByteString` or `String` or `Text` or `Lazy.Text`, of course ;) ). Since these are generally unparseable (e.g. via haskell-src-exts and friends), it's hard to do much with them (hence the tendency to delve into GHC's own representations).
Note that for my own work I ended up abandoning Haskell syntax altogether in favour of GHC Core. I use Cabal and GHC to do the parsing (since that seems to be the only way to handle real world code), and a GHC plugin to dump out Core as s-expressions http://chriswarbo.net/git/ast-plugin
tl;dr the killer feature of s-expressions is that they're trivial to parse and generate, which makes it easy to read/write some other "front end" instead (i-expressions, sweet expressions, wisp, etc.).
Lisp's syntax is just a textual representation of the internal abstract syntax tree, so you could represent it any number of ways: s-expressions, indentations, even something like json.
I think s-expressions were just the absolute simplest way to represent them with the smallest parser required. It's really really not much.
https://www.patreon.com/sindresorhus
Most JS devs know sindresorhus by reputation. You can't help but knowing him through his work. He even put up a separate twitter solely because people kept wanting a way to see whenever he made a new github repo.
Patreon makes him $3300/mo. So that appears to be the ceiling for all of us.
And closed source isn't either when it comes to productivity tools (because there are so many free alternatives).
Hopefully it becomes an average, not a ceiling. We're not all going to make the next Vue, but that guy is getting $16k/mo https://www.patreon.com/evanyou .. that's what I consider the ceiling.
I'm sure there are people on HN who can do just that, and much, much more. Some of them may even use parinfer. But whether they can and whether they want to contribute are two different things.
For me the more interesting question is how the tiny minority of people who have gotten their hands on the overwhelming majority of money, power, and resources in this country and around the world can be persuaded to use what they have in socially constructive ways rather than for the benefit of the tiny, exclusive minority they are part of.
- looks at your Github repos
- looks through all your dependencies
- lists all the projects you depend on that support donations
- shows you how much (or rather little!) they're making now
- shows you the (Patreon) rewards you can get for supporting them
- allows you to support them right there and then!
I honestly think this is a legitimate idea and kind of want to pitch it to Y Combinator right now. It would be like something that ties Patreon and Github together.
---
Okay I just posted this idea to Hacker News, find it in my submissions if you're interested :)
Direct link: https://news.ycombinator.com/item?id=17634117
Does that imply that you see it making money too? How would it do that?
Income, corporate, payroll, VAT, property and capital gains taxes are great for this.
That's what taxes are for. "Give up some of your money to be used for the collective benefit, or we will lock you in a box and then take your money away anyway" is pretty persuasive.
That's not a valid reason. They can also pick and contribute to their 3-4 more important, or to their 3-4 favorites, or to those that look more in need (eg. not much need do donate to Mozilla for example).
But how many Lisp programmers are out there anyway, and use this? The intersection should be quite small, compared with most famous FOSS projects for languages like Java or Python (and even those projects don't get much).
Additionally, it's a niche market in the first place. People aren't exactly clambering for more lisp dev tools except in recessed corners of the academic universe and places where clojure is used. Aside from those users, there's little demand for a tool like this, and that market is like I said, further eroded by significantly higher quality tooling that's already available.
Not every piece of code someone writes is worthy of being paid for. Not every piece of code written deserves to have been written in the first place, unfortunately.
The only problem is working on a shared code base where people don't bother to indent code properly.
Parlinter was designed for this purpose. Maybe you could suggest it to your team? https://github.com/shaunlebron/parlinter
But ultimately Lisp taught me not to overvalue technical superiority, so I eventually stopped using it.
edit: better video https://i.imgur.com/4BsYQ3e.gif
Exposing newbie end users to modal editing would seem to be a non-starter, i think smart mode needs to work
I'm wondering if paren mode might be what newbies actually want, and they can go and learn about indent mode later?
Parens are only useful if everything is on a single line. With new lines and indentation, things can become a lot cleaner like python style.
I strongly believe python got so popular because of its very simple to read, almost pseudo-code like syntax that anyone could learn.
List had great ideas, everything is a list, but dealing with parens hell makes me think it could be a lot more elegant.
Lisp programmers don't deal with "parens hell". That's one of the misconceptions that come from outsider looking in. Parens are for the editor. Lisp indeed uses indentation and not parens as you said. It's very python-like that way."You have reached Lisp enlightenment when parens disappear"
I think it would be real helpful to have editor for beginners that shows parents as transparent and low contrast, separately from the rest of the code. It would help people to understand they should not be trying to match them visually.
Your example expressed in Python is something like [[[]],[],[[],[]] ...
When I finally got around to learning Python, I found that its meaningful whitespace usually wasn't as bad as I'd feared, but it's still a pain in the ass in certain circumstances.
For instance, there are plenty of places online (like some blogs) where python code is published that screws up its whitespace so that copying and pasting from there totally screws up the program and I'm forced to fix it by hand. This is an unnecessary pain that explicit delimiters like Lisp's parenthesis solve.
Python's meaningful whitespace is also not enough for editing software to easily infer and manipulate the program's structure without actually knowing python. That's a burden that editors that are editing Lisp do not have, as they're always just going to be manipulating sexps, without needing a deep understanding of the underlying language. So that usually makes sexp-aware editors (and plugins like pariner) much more powerful and flexible when editing Lisp than your typical python editor.
Python also has way too many other types of delimiters. Though modern Lisps also don't have just parenthesis, most of their delimiters are parenthesis and that makes thinking about which delimiter is appropriate in which context a lot simpler. You have a lot less rules to remember, and that's a blessing if you're a polylingual programmer who's bouncing around from one language to another -- you mostly don't have to remember which delimiter has to go where, as it's mostly just a parenthesis with Lisp.
Instead of trying to improve Lisp by taking away parenthesis and adding meaningful whitespace, I'd rather improve Python by taking away its meaningful whitespace and using parenthesis.
I love indented languages. Python, make, coffeescript etc indentation communicates hierarchy. It’s a very simple idea.
All I’m saying is Lisp is amazing but parens aren’t for everyone.
Lisp is essentially list of lists. There are other ways of representing that
> All I’m saying is Lisp is amazing but parens aren’t for everyone.
It's trivial to write Lisp using indentation and no parentheses if you want to. The same goes for any language which uses s-expressions. Just pick an alternative syntax, e.g.
- Readable http://readable.sourceforge.net
- Sweet expressions https://www.dwheeler.com/readable/sweet-expressions.html https://srfi.schemers.org/srfi-110/srfi-110.html
- I-expressions https://srfi.schemers.org/srfi-49/srfi-49.html
- Wisp https://srfi.schemers.org/srfi-119/srfi-119.html http://www.draketo.de/english/wisp
These can all be converted back and forth to s-expressions, such that the programmer doesn't have to care about s-expressions (found some code online? send it through a converter before reading it), and the language implementation doesn't have to care about the syntax that the programmer is using (set up the relevant converter as a preprocessor).
Two things to note:
- The reason we can do this is because s-expressions are such a useful language format. Using an alternative syntax for Python, make or coffeescript would be a time consuming, maintainence heavy burden. Anything using s-expressions gets all this "for free".
- The fraction of Lisp programmers who do this is vanishingly small. This is empirical evidence that parentheses are not a problem worth caring about.
>I strongly believe python got so popular because of its very simple to read, almost pseudo-code like syntax that anyone could learn.
Agreed. And python programmers are asymptomatically approaching Lisp's features. I had previously thought that only Lisp dialects could do symbolic differentiation and integration on code written in the same language, but I was wrong. http://www.sympy.org/en/index.html
> List had great ideas, everything is a list, but dealing with parens hell makes me think it could be a lot more elegant.
Lisp is already the most elegant. In John McCarthy's 1958 paper, he wrote a Lisp interpreter in Lisp, and the interpreter was only a page or two. That is elegance.
http://www.hevanet.com/cristofd/08.html
(This is not to belittle the point you are making, I just can't resist being That Guy. Lisp manages to be both a usable language and pretty easy to interpret, whereas Brainfuck is... something else.)