TXR Lisp | Hacker News Reader

TXR Lisp

nongnu.org

152 pointstosh27 comments

Every once in a while I come back to this, and walk away feeling like it's not worth my time to learn.

The language might be "objectively" simple, but it's not at all a simple language. There is no user guide, nor is there a nice readable language spec like R7RS. There's a reference manual, and a bunch of StackOverflow examples that look like Perl with S-expressions. I'd love to sit down and work through a reference manual, but who has time for that nowadays?

I love the idea of a Lisp purpose-built for data processing, but practically I don't know what I'll gain by learning this that I don't already have with Python, Zsh, and Gauche Scheme.

Can someone convince me otherwise? What's the killer feature of TXR that makes it worth the effort of learning it the hard way?

Also, what's with the leading @-signs in some of the examples? Does that (try) form delimit some kind of block or region? Why are there some lines that look like bare string literals? It's clever and impressive to manually build a patch that can be applied with Git, but is this solution any cleaner or faster or easier to write or read than the equivalent in Python or Perl or Gauche or Guile or Common Lisp? https://stackoverflow.com/a/37336801/2954547

You should learn Prolog sir: for the man that has everything.

I actually started down that path last year, with the incredible Power of Prolog book and video series. It's one of the more impressive learning resources I have ever come across.

Prolog is only for the brave and true: when the student is ready, the teacher will appear :-)

I think a lot of the difficulty in Prolog comes down to the fact that you can read Prolog programs three ways: as a restricted form of first order logic (Horn clauses), as a procedure, and as an execution model. The last is double confusing because almost all programmers are used to controlling the execution model and Prolog demands that you relinquish a lot of control there ("Algorithm = logic + control" by Kowalski talks about this a bit). You also cannot ignore any of them because any non-trivial Prolog program exploits all three.

Yeah I think that's right. I'm doing Advent of Code in Prolog at the moment and checking other people's solutions -- I've yet to see 2 Prolog solutions which are mostly the same. The admixture you described results in code that aligns very closely with the thinker, and less closely with a specific procedure. Its very weird to see.

Having said that -- once you get Prolog and use it for a while, your brain (at least my brain) wants to replace EVERYTHING with Prolog.

It should be easy to implement miniKanren in txr lisp.

I would consider something like that for inclusion in stdlib/.

> The language might be "objectively" simple, but it's not at all a simple language.

You could say that it's simple but not easy

I got stuck into txr lisp a few years ago, and wrote some code for parsing bank statements. It has a builtin DSL (the @ syntax) which is great for text processing (better than perl IMO). It's also a fun language to learn, with lots of cool features such as combinators, continuations, macros, pattern matching, range objects, JSON objects, etc. It's more powerful than python, R or julia and would make a great successor to those languages for data science, but there are very few libraries ATM so that's a long journey. Here are some I wrote (including a zsh script for browsing the manual): https://github.com/vapniks/txr-libs IMO a good first step would be to use the txr FFI to write a library for Apache arrow: https://arrow.apache.org/

Thanks for that! Do you have advice for learning it? How did you get started, and what made you decide to stick with it instead of other languages?

I'm happy to try to use it for Advent of Code 2024, but it would help if I could at least get oriented so I'm not going in totally cold (and thereby likely to fall off the wagon after 3 days because each one takes 2 hours in an unfamiliar language).

The data science use case is particularly interesting and not something I had considered here. Arrow bindings help immensely. Of course at work I'll probably never get a chance to use anything other than Python and R for many years to come; maybe Julia is the most exotic I think I'd ever be able to go, and that's only if I ever hit a performance bottleneck in Python that can't be remediated with Numba.

There are some Advent of Code 2021 solutions in TXR:

https://www.kylheku.com/cgit/advent/tree

That gives a flavor of how those kinds of problems can be approached.

The solutions are all self-contained; they don't share any AoC-specific library or anything, and were developed with a throwaway program mindset, where we just solve that problem and don't try to produce anything that is reusable other than by copy paste into the next program.

If you already know a lisp such as scheme, much of it should already be familiar to you. You can program in a lisp-1/scheme like way using square brackets, and a lisp-2/common lisp way using round brackets. The thing that really makes txr lisp different is the pattern language DSL, and that was what motivated me to learn it because I had a specific use case (processing my personal bank statements). The manpage is huge due to the large amount of builtin functions and features to make up for the lack of external libraries, but there is a lot of overlap with other lisps. I learnt by reading the manpage, looking at examples on stackoverflow and rosetta stone, and playing around with it in the REPL. The pattern matching language takes some getting used to, and it helps to have a good understanding of what its doing under the hood, otherwise you can get very frustrated. Also it uses non-standard regular expressions which lack features such as repetition operators and capture groups, but you can work around this (e.g. by using the pattern language). I recommend finding some non-time-critical use case to motivate you, and work on it in your spare time. Learn how to use the txr repl & command syntax, and create a directory containing some example text files to quickly test things and get a better understanding of how it works. Also create a file for keeping notes about txr & its quirks which you can easily refer to.

Just to clarify, the pattern language is a more powerful alternative to using regexps (but you can mix them). My bank statements are pdf's which can be converted to ascii using pdftotext, however this destroys the structure of the documents which makes extracting data using regexps (even pcre's) very difficult, but much easier using txr.

Nice to see you here. I was just casually looking through the txr-libs there and just randomly spotted this, the CSV library, in a let form:

  (fieldrx (regex-compile ;; using a regex object here messes with emacs syntax highlighting
             "([^,\"\']*(\"([^\"]|\\\\\")*\"|\'([^\']|\\\\\')*\'|)[^,\"\']*)+"))

I understand that here using the #/.../ regex literal object didn't look right in your Emacs, so you dynamically compiled from a string literal.

But that now happens each time you call the function.

Lisp has your back here; we can hoist the compilation of the regex to load time using load-time:

  (fieldrx (load-time (regex-compile "...RE...")))

load-time makes a difference in interpreted code, not just compiled files. Even in interpreted code, the regex-compile expression will be evaluated once only and then the cached value used:

  1> (dotimes (i 5) (prinl "hi"))
  "hi"
  "hi"
  "hi"
  "hi"
  "hi"
  nil
  2> (dotimes (i 5) (load-time (prinl "hi")))
  "hi"
  nil

Of course, you can also just initialize a top-level variable with the object returned by regex-compile and use that.

Cheers Kaz, I'll make the changes. There's still quite a few TODO's in that code that I intend on fixing up at some point.

> that I don't already have with Python, Zsh, and Gauche Scheme.

Those times when it would be nice for the Python-Zsh-Scheme solution to be in one language and program.

> is this solution any cleaner or faster or easier to write or read than the equivalent in Python or Perl or Gauche or Guile or Common Lisp?

I believe so because we can see the salient parts of the diff hunk structure appearing literally in the code, so we know at a glance where it is pulling what. I've not looked at that since 2016 but it's transparent.

On top of that, any Python, Perl, Guile or Common Lisp solution can be transliterated to decent TXR Lisp. There are multiple ways to attack that same problem in TXR, not necessarily using the TXR Pattern Language.

There are "spiritual ties" between TXR Lisp and Common Lisp. There probably isn't another language distinct from Common Lisp that is as similar to Common Lisp as TXR Lisp. The TXR Lisp reference manual mentions ANSI CL numerous times in various Dialect Notes.

There is a translation to TXR Lisp of the CL-WHO HTML generation library called TL-WHO. It's a function-by-function translation that retains most of the structure of the original code. https://www.kylheku.com/cgit/tl-who/about/

If you have CL skills and knowledge, a lot of it will transfer over to TXR Lisp, and vice versa. There will be things you will miss in either direction.

By the way, here is another TXR program that matches unified diffs: diff2err:

https://www.kylheku.com/cgit/diff2err/tree

This reads diff material from standard in and converts it to compiler error format, stored in a file called errors.err (implicitly used by "vim -q" quickfix mode). This is useful for visiting differences in the actual code, with the editor: you can jump to all the places that were changed, and at every location, the diff info appears in diagnostic format: how many lines were changed, and what the original lines are.

R7RS is readable because it's small; you could read it in one sitting.

The writing style isn't particularly better than the one in the TXR manual.

Both are specifications, and not tutorials.

The Scheme world depends on R7RS for giving the requirements for the portable, common language core.

Since TXR isn't a fragmented language family of incompatible implementations, there isn't any need to have a small document laying out what is common.

Where is the hyperlinked, HTML version of R7RS? Every time I need to look up something in that, I have to use a clunky PDF.

A search "HTML R7RS", shows mostly old posts in mailing lists and forums asking the same question. That and someone's github repo containing what seems to be source for generating a HTML-ized R7RS.

The TXR manual is available in usefully (though imperfectly) hyper-linked HTML, PDF as well as a man page: just "man txr" and /search. When you type (doc 'symbol) in the listener, it jumps your browser to that section of the manual (the externally hosted version, easily retargetable to a local copy by changing the value of *doc-url*.

TXR Lisp – an innovative, original dialect from the Lisp language family - https://news.ycombinator.com/item?id=24449511 - Sept 2020 (1 comment)

TXR Lisp - https://news.ycombinator.com/item?id=22683735 - March 2020 (9 comments)

TXR: An Original, New Programming Language for Convenient Data Munging - https://news.ycombinator.com/item?id=21699927 - Dec 2019 (2 comments)

TXR – A Programming Language for Convenient Data Munging - https://news.ycombinator.com/item?id=19908197 - May 2019 (73 comments)

TXR: An Original, New Programming Language for Convenient Data Munging - https://news.ycombinator.com/item?id=12338441 - Aug 2016 (2 comments)

TXR: A Programming Language for Convenient Data Munging - https://news.ycombinator.com/item?id=8409391 - Oct 2014 (38 comments)

Show HN: Munge your data with TXR - https://news.ycombinator.com/item?id=8020106 - July 2014 (1 comment)

> remove verbosity from Lisp programming

Common lisp verbosity is a feature. It make's it very clear what is happening. Everyone understands.

Yeah. I personally agree, though maybe we're not the target audience. If I saw some examples of exactly how it does this I might change my mind, but with none right there on the page, I've lost interest before I can muster motivation to go look for them myself.

Apparentlt the author hangs out here, so I'd recommend the author stick in some links to examples there if they don't want to muddy up the introduction page, because I think plenty of people that like lisp (not all, but plenty) will see this as an anti-feature without proof to the contrary.

The author, Kaz Kylheku, hangs out on HN as kazinator.

If you're using the TXR Lisp REPL, be sure to make a ~/.txr_profile file that has this:

  (if (real-time-stream-p *stdin*)
    (put-line (quip)))

Then you will see a random humorous quip on startup.

The stream test around the put-line is recommended, otherwise you will see the quip even if the REPL is used in a pipe:

  $ echo '(+ 2 2)' | txr
  4

This looks like an interesting language. It tries to address many shortcomings in old lisp by introducing new semantics and new syntax. It looks like a pragmatic language aiming to get out of the programmers’ way and let them focus on solving problems.

  (mapdo .start-time.(set 42) obj-list)
  (swap [a 2..4] [b 0..2])

This looks gross. Why does no one seem to understand that lisp should not have infix operators?

I respect the idea that Lisp syntax should only be based on dispatch based on looking at the leftmost input character.

In TXR Lisp, I decided to experiment with a small number of notations, to see whether they could make Lisp coding a bit more ergonomic, similarly to how 'x has helped with (quote x) for over sixty years.

There are rules: the notations have to absolutely fit into the surrounding Lisp.

The notations all have an underlying equivalent tree structure: list headed by a specific symbol.

There must be print-read consistency.

Not all combinations of the underlying structure need to (or should) map to the notation.

  1> '(qref a b c d)
  a.b.c.d
  2> '(qref a b (qref c d))
  (qref a b c.d)
  3> '(qref (qref a b) (qref c d))
  (qref a.b c.d)
  4> '(qref 1 0)
  (qref 1 0)
  5> '(qref 1 a)
  (qref 1 a)
  6> '(qref a 1)
  (qref a 1)

Not every (qref X Y) is blindly printed as X.Y, which would break print-read consistency. Multiple different qref nestings would map to the same notation, and (qref 1 0) would produce the floating-point token 1.0.

Of course, some of these notations complicate parsing. When we see a symbol token like a, we cannot just return the symbol, because it could be followed by a dot, in which case we are in (qref a ... syntax, or other possibilities.

Be that as it may, working with this stuff is fine; all in a day's Lisp.

Reply on news.ycombinator.com