LuaTeX Comes of Age (2017) | Hacker News Reader

LuaTeX Comes of Age (2017)

lwn.net

134 pointsgoranmoomin41 comments

LuaTeX is what TeX should have been from the start (and would have been, if the technology of the time had permitted it). Apart from modern font-handling and Unicode, the main thing IMO is the hooks it provides (in the form of callbacks).

With other TeX engines (mainly: pdfTeX, XeTeX, or the original Knuth TeX), the only "programming" facility is in the form of macros, which were originally added by Knuth only for some simple text substitution to save typing. He never intended to add programming features into TeX, but of course, it doesn't take much to become "accidentally Turing-complete", and that's what TeX macros became. (There's also some rudimentary support for counters/registers, which IIRC Knuth added after he found users were (ab)using macros to encode numbers using unary or Church numerals.) So the only way to influence anything TeX does automatically, whether it's hyphenation or line-breaking or page-breaking or whatever, is to set up some macros whose blind expansion will ultimately—at a remove of several steps—result in the outcome you want, without affecting anything else (e.g. even an accidental space might get typeset, with undesirable results). This of course is cumbersome and error-prone.

With LuaTeX you can program these things at the level you actually intend, e.g. while you can still use macros (or use the `process_input_buffer` callback in Lua), you can now also directly, say, influence the page layout, in a Lua callback that is actually aware of data at the relevant level of abstraction (the vboxes on the page, say), rather than trying to do everything at the lowest (text expansion) level.

Knuth has said that he never intended for TeX macros to be used as a full-fledged programming language (he expected people would directly edit the SAIL/Pascal code of the TeX program for anything nontrivial), and that he dislikes each tool coming with its own Turing-complete programming language, and that if a standard embeddable programming language had been available he'd of course have used it in TeX — I think Lua would count.

You can see "TeX without TeX" page on the LuaTeX wiki for an example of the power of LuaTeX, typesetting with TeX while completely bypassing TeX syntax: http://wiki.luatex.org/index.php/TeX_without_TeX — a more elaborate production system is Speedata Publisher: https://github.com/speedata/publisher

As for myself, here is the most fancy thing I did with LuaTeX: https://tex.stackexchange.com/questions/401604/book-on-a-sin... (a "poster" of Kafka's Metamorphosis done just for that answer, never "productionized" into a package or whatever). Also, something possibly illuminating is this TeX-vs-LuaTeX comparison of something hyphenation-related (https://tex.stackexchange.com/questions/378704/how-to-avoid-...), and in this answer (https://tex.stackexchange.com/questions/398310/why-in-2017-d...) I link to some other times I used LuaTeX, though some of them just use the Lua part of LuaTeX rather than LuaTeX hooks, such as computing digits of pi for generating pretty pictures (https://tex.stackexchange.com/questions/367902/tikz-color-op...).

> if a standard embeddable programming language had been available he'd of course have used it in TeX — I think Lua would count

I agree with your general idea, but it is really a shame that Lua was chosen as the language to embed in TeX, instead of something sane like a Scheme, Common Lisp or at least Python (today, maybe Julia). Furthermore, the effort to rewrite as much of the TeX implementation as possible in Lua is perplexing.

> Furthermore, the effort to rewrite as much of the TeX implementation as possible in Lua is perplexing.

Very little of the TeX implementation has been rewritten in Lua. (Assuming you're talking of LuaTeX itself and not the "TeX without TeX" wiki page, which is just for demonstration.) LuaTeX starts with the WEB (Pascal) code of TeX, manually translated into C (see e.g. https://tug.org/TUGboat/tb30-3/tb96hoekwater-pascal.pdf), and, additionally, hooks are added: callbacks that can be written in Lua, and which have access to the internal data structures.

(By the way, SILE is a separate typesetting system (unrelated to TeX) that is actually written in Lua, and re-implements some of the TeX algorithms in Lua, I think. You can see this talk by its author about it: https://www.youtube.com/watch?v=5BIP_N9qQm4 )

Julia is a wildly inappropriate choice for embedding, and Python is not much better. Some implementation of Scheme might make more sense (isn’t that what Guile is about?). Except that most people are put off by s-expressions; Lua is comparatively “normal”.

> Except that most people are put off by s-expressions

This triggered my obligatory rant that Lisp doesn't require s-expressions, there are a bunch of equivalent alternatives, and indeed the entire point of s-expressions is that they're trivial to generate, parse and manipulate automatically (if you don't want to write them manually).

http://chriswarbo.net/blog/2017-08-29-s_expressions.html

The suggestion was to use Scheme or CL.

If you're not chasing market share for commercial purposes, do you really want to cater to those who are put off by s-expressions if you personally enjoy them?

Probably not. But I’m guessing that the people who created LuaTeX prefer Lua. And maybe, if you wanted to create a scriptable TeX that would be usable by the typical TeX user, a language like Lua makes more sense than a Lisp. Those users tend to be scientists and mathematicians, and just anyone who needs to make good-looking documents; most are not familiar with Lisp and would find it more difficult to learn than Lua.

If you're making a tool for people to use, it's probably best to use a syntax that most people are comfortable with. UX doesn't only matter to corporations.

it is really a shame that Lua was chosen as the language to embed in TeX, instead of something sane like a Scheme, Common Lisp or at least Python (today, maybe Julia)

I personally find Lua to be a completely sane and easy to program language. But, to each their own.

Let’s look at these options and some others:

Scheme/Lisp: It’s true that Scheme and Lisp-like languages are pretty compact and easy to embed, but the syntax of Lisp is, for most modern programmers, pretty esoteric. Asking today’s programmers to program in a Lisp is, for better or worse, like asking secretaries to use Vi to edit documents.

Python: Python is a big, bulky language and one which is not really suitable for embedding inside another program. See this recent discussion for more on this: https://news.ycombinator.com/item?id=29700947

Julia: A somewhat non-mainstream language, and, like Python, it doesn’t look to be a language designed for embedding the way Lua is.

Forth: Like Lisp dialects, Forth’s completely stack-based syntax is not intuitive for today’s generation of programmers.

Lua needs batteries, which I know is off putting to some programmers. It turned me off from Lua for a while until Roblox made me give Lua a second chance. Here is my own set of public domain Lua batteries: https://github.com/samboy/LUAstuff (stuff like a Lua copy.deepcopy(), sorted table/dictionary iteration, split strings based on a regex, etc.)

Python is more like something that should be run alongside LaTeX, not embedded in. PythonTeX already does this

See Lilypond, the music typesetting project, if you want to see how Scheme embedded in a TeX-like language looks. From my experience, you're generally evaluating to actual AST-like objects instead of sprintf-ing something TeX-like – I suppose this has pros & cons.

As much as I like all the alternative languages you mentioned, and as much as I hate it, I think JavaScript (or wasm) would/should be the language of choice were TeX to be extended today.

Holy cow, you have some amazing things there. The Kafka poster example is worthy of some deep study, as is the article you wrote around it. Thank you.

Your examples of LuaTeX have me wondering if I could use it to make a package that internally works by abusing the \minipage environment to make documents that would make Mark Danielewski proud. Thank you for the inspiration.

"Modern" TeX processors (LuaTeX or XeTeX) are sadly held back because publishers still cling to PDFLaTex. For instance, ArXiV "fully supports and automatically recognizes PDFLaTeX" and makes no mention of other backends [0].

So, LuaTeX is perfectly fine for personal stuff that you can compile in-house, but basically nobody wants to risk using it in any scenarios where sources are to be sent to a 3rd party for compilation, only to find out that they need to make them PDFLaTeX-compatible.

And after all LuaTeX came out of beta in 2016 [1], which is pretty recent in TeX-years. So it kind of makes sense that publishers still prefer the quite battle-tested PDFLaTeX...

[0]: https://arxiv.org/help/submit_tex

[1]: https://tug.org/pipermail/luatex/2016-September/006201.html

ConTeXt[1] has had Lua integrated since around 2009[2]. KeenWrite[3], the text editor that I'm developing, converts Markdown documents to XML, then can call ConTeXt with a given theme[4] to typeset the document. This allows for complete separation of content from presentation, as described in my Typesetting Markdown series, in which part 8 briefly touches upon Lua[5].

[1]: https://wiki.contextgarden.net/Installation

[2]: https://tug.org/TUGboat/tb40-1/tb124hagen-lmtx.pdf

[3]: https://github.com/DaveJarvis/keenwrite

[4]: https://github.com/DaveJarvis/keenwrite-themes/

[5]: https://dave.autonoma.ca/blog/2020/04/28/typesetting-markdow...

IIRC context is "just" a macro package on top of luatex (and pdftex before). so whatever luatex has, context has.

The linked LWN article mentions that it’s difficult to put comments in inline LuaTeX macros because those macros remove newlines.

To add a comment to the middle of a line in Lua, use the --[[ comment ]] form:

  a = 1 --[[ Set a to 1 ]] print(a)

If we have a comment that has ]] in it, we can do this instead:

  a = 1 --[=[ We love [[brackets]] a lot ]=] print(a)

The --[[ (or --[=[ ) form also works for multi-line comments

  a = 1 --[[ Lorem ipsum dolor sit amet, consectetur
  adipiscing elit, sed do eiusmod tempor incididunt ut
  labore et dolore magna aliqua. Dictum fusce ut placerat
  orci nulla pellentesque dignissim. Quam viverra orci 
  sagittis eu volutpat odio facilisis mauris. ]] print (a)

Here's something better that doesn't use TeX and is fully WYSIWYG:

http://texmacs.org/tmweb/home/videos.en.html

TeXmacs doesn't use TeX?

> Documents can be saved in TeXmacs, Xml or Scheme format and printed as Pdf or Postscript files. Converters exist for TeX/LaTeX and Html/Mathml. Notice that TeXmacs is not based on TeX/LaTeX.

From http://texmacs.org/tmweb/home/welcome.en.html

It doesn’t use emacs either!

In my judgement, it’s worse.

Without any arguments this is an empty statement. And to be fair so is parent comment (the "better" part).

Indeed. What was it the late Hitchens liked to say? Something that can be asserted without evidence can be denied without evidence. Here’s a clue: TeX has no bugs.

"dismissed", not "denied". There's a subtle difference. The exact quote:

> That which can be asserted without evidence, can be dismissed without evidence.

The current TeXmacs release is stable. Requiring no bugs is unrealistic and impedes progress.

Why?

LuaTeX is another way to create LaTeX macros, but using Lua as the language to manipulate text. I think it helps create macros that would be too complicated in TeX, but on the other hand it is another language for LaTeX users to learn. I haven't see much advantage of their approach other than in specialized niches such as creating complex figures.

I was tickled to see my 2017 article pop up here.

Lua is a somewhat odd but perfectly sane language that you can pick up in a day. With a few hooks into TeX you can be writing complex programs that manipulate your document, insert the result of calculations into the text, etc. Writing these things in TeX is insanely difficult. I’ve been using TeX literally since it became available, and I wouldn’t dare try to write anything complicated in TeX the programming language. But Lua is a breeze.

Because of this I’m a bit surprised that I haven’t seen more. One big application is the layout algorithm for network diagrams in tikz.

I agree that Lua is much easier to learn, however the kind of thing that you can do with such macros purely in Lua has limits. To do anything really important in LaTeX, you have to understand the LaTeX macro system, and you also need to have at last some understanding of the TeX language. You could do such things in Lua, but in that case you'll have to interface with LaTeX anyway. So in fact you're increasing the complexity of the task, because now you need to understand LaTeX and Lua.

On the other hand, if you're just manipulating text that doesn't interface with LaTeX, then you could really do that from outside LaTeX, using any kind of script language such as python or perl. That's why I think that Lua scripting is not being used outside of some niche areas.

In practice this is simply not the case. Look at my first example in the article. With just a few lines we can set a paragraph to follow the shape of any mathematical function. I didn’t have to know anything about TeX macros to do this. In fact I have no idea how to do this with LaTeX or TeX macros, and I don’t think they have nontrivial mathematical functions available. The same holds with the other examples. Almost trivial using Lua scripting (not “macros”), don’t even know where to start with TeX macros.

LaTeX isn't just for users, there are also people who are developing packages or classes. And since LaTeX doesn't even offer the luxury of arrays or hash tables natively, LuaTeX is very a refreshing change.

My company is using their own LuaLaTeX class extensively to produce hundreds, if not thousands of reports per year, and features like automatically computing the CVSS base score of a CVSS vector will take a couple of minutes in lua, while it will cost you several days and a bunch of gray hairs to do in TeX. It's only the developer who has to know any lua though. It's completely transparent to the user.

Appreciate you giving an example of a real world use for the Lua in LuaLaTex. Any others appreciated! Like the parent, i've never much come up with a possible application.

I took a class in TeX a long time ago and got marked off a point because I used self-modifying macros to save a variable.

Sounds like it should have been extra credit.

Discussed at the time:

LuaTeX Comes of Age - https://news.ycombinator.com/item?id=15146300 - Sept 2017 (28 comments)

LuaTeX has some small differences from TeX that are not present in pdfTeX or XeTeX that I find really annoying. Most notably, it will not break lines after – or — (whether entered as Unicode or with traditional TeX input using -- or ---). And the slowness is on pretty much all documents. It also does some weird mapping of character codes in generating the pdf files that should be invisible to most users but is evident if one digs into the output files. It could cause issues in malformed pdf readers although as far as I know this isn't an issue. Unless someone specifically needs Lua, it's hard to recommend LuaTeX over XeTeX.

LuaTeX does break after -- and ---, see e.g.

    \hsize 0pt
    abc--def
    \bye

I also don't know what kind of weird remapping you are talking about, if you are referring to the Unicode mapping then they are required to get proper copy&paste (and are also generated by other engines these days).

The article is from 2017.

Would be great if that could be added to the headline here.

Reply on news.ycombinator.com