Hacker News Reader

Comment by ubavic | Hacker News Reader

I use some form of TeX (LaTeX or XeTeX) almost daily, and I often think about the future of (La)TeX and potential replacements.

I think that we desperately need some modern language for document preparation, which will replace TeX and its derivatives, but I also don't see anything like that on the horizon. The biggest drawbacks of TeX are lack of proper Unicode support; lack of support of modern font formats; and outdated programming practices. Tools like XeTeX, LuaTeX and ConTeXt tackle these problems with some success, but they are still not perfect. Also, they are too similar to TeX to bring drastic changes to programmers. We need something that is designed from zero specifically as a TeX replacement (including compiler, package manager, popular packages, bibliography manager, converters, etc...). I can only hope for better future...

BTW, fun fact: TeX is still actively developed. Knuth fixed some bugs in the kernel last month (after few years of pause).

View on HN

When I'm OK with the defaults, LaTeX is a breeze.

When I want to change something nontrivial about the formatting or style, it's a nightmare. For example, there are zero mature fonts besides the default computer modern that also have math rendering that's aesthetically acceptable to me. And I forget the details, but typesetting is handled in such a way that the kerning around things like hyphens is broken. I've fixed it manually before in places where it was particularly egregious.

Also on this point: an ungodly number of hours are wasted every year by academics figuring out how to contort LaTeX so the resulting document complies with NSF grant specifications, which you must meet or be auto-rejected. In particular, NSF requires one-inch margins, but there's no way to force the typesetting engine to strictly respect the margins you set. So at the end you're left to turn the visual margin guides on and check there are no margin overruns, or you risk getting your proposal set to the trash (theoretically, at least).

I've also heard that LaTeX editors are development disasters with a bus factor of one. The NSF really ought to fund them, because if we wake up one day after the maintainers of all the popular editors die and Apple/Microsoft force an operating system update that breaks compatibility, paper writing in the physical sciences will grind to a halt.

"In particular, NSF requires one-inch margins, but there's no way to force the typesetting engine to strictly respect the margins you set. So at the end you're left to turn the visual margin guides on and check there are no margin overruns, or you risk getting your proposal set to the trash (theoretically, at least)."

TeX should always give you warnings like "Overfull \hbox (0.43386pt too wide) in paragraph at lines 42--45" and the text of theaffected line when it "disrespects your margins", so I don't see why you have to check for them visually?

Oh, I forgot about the console. I still find it annoying I have to worry about this at all and can't enforce it.

But you can enforce it in many ways; for example you can set `\emergencystretch=\maxdimen` (or any reasonably large value) and you'll never have overfull boxes (what you called margin overruns), unless of course you have a single unbreakable "word" longer than a line by itself. (On the TeX.SE website here are a couple of answers of mine about related stuff and alternative approaches: https://tex.stackexchange.com/a/422652 https://tex.stackexchange.com/a/401315 though the canonical question with good answers is probably this: https://tex.stackexchange.com/q/50830 .)

So this complaint of "I can't enforce that there are no margin overruns" is (to use the same word) actually something like "If I don't take any steps to avoid overruns, and moreover ignore the warnings that are specifically about overruns, then there are overruns". This complaint is not specific to you and is surprisingly common actually, which brings me to my main point, which is that the majority of people seem to use TeX/LaTeX without ever having read the manual. Some of them even joke about "incomprehensible" messages like "overfull hbox". But the whole point of TeX, what TeX is at its core, is this breaking-paragraphs-into-lines (in my view, the rest of TeX is just a large wrapper around this basic function of setting boxes).

The preface of the TeXbook describes it as "TeX, a new typesetting system intended for the creation of beautiful books", and for its initial design goals, it does make sense that it chooses by default to declare defeat in impossible situations and overflow margins with a warning to the user so that they can take appropriate action, instead of silently producing an "ugly" paragraph with lines subtly looser than whatever tolerance settings the user has specified. What is remarkable here is that a lot of people are using TeX who do not care about meticulous typesetting of "beautiful books" at all, and will blithely ignore warnings related to such matters.

The lesson may be that such a state of affairs is inevitable, so perhaps a better default for similar software today would be to produce loose lines to fit in the margin, so that results are good even for the users who don't read the manual, or even the warnings, or even proofread the final result.

> What is remarkable here is that a lot of people are using TeX who do not care about meticulous typesetting of "beautiful books" at all, and will blithely ignore warnings related to such matters.

Is this really remarkable? It's the standard tool for writing papers in physics. Get result, put into LaTeX, put on arxiv, repeat. The fact that these people have to constantly Google for obscure LaTeX incantations (faster than TeXbook) is not ideal for them. Indeed, it would be better if it were easier to use without reading a 500 page book.

Anyway, thanks for tip. When I googled for this I found only \sloppy, which did not produce reasonable results.

I should have been clear: I don't think people's behaviour is remarkable here (I've done the same, with LaTeX years ago, and with many other tools); what I found remarkable is the history / state of affairs, where a finicky tool primarily concerned with the finer points of typesetting (TeX) became the standard tool for dashing off papers (thanks to Lamport and LaTeX I guess) even for people not concerned so much with typesetting. It's as if, say, a samurai sword had become the standard tool for chopping vegetables in the kitchen, and everyone was complaining about the awkward shape, the unnecessary sheath, etc. It's neither the tool's fault nor the users', but… something has gone wrong somewhere.

> there are zero mature fonts besides the default computer modern that also have math rendering that's aesthetically acceptable to me

The only one I like is URW Garamond but it's a breeze to install and use and I think it stands head and shoulders above everything else. It also matches the math pretty well imho. I feel you though: Computer Modern is pretty ugly and most other fonts really clash with the math rendering by default.

https://tug.org/FontCatalogue/garamond

I've been extremely happy with the TeX Gyre family. Its Pagella font comes with a math sibling and both together cover a wide range of symbols at high quality.

“typesetting is handled in such a way that the kerning around things like hyphens is broken”

Can you give more details about this? I’ve never heard this complaint before, nor noticed this myself.

Also, I’m not sure I understand what you are saying about “LaTeX editors”; I use Vim, and it works quite well.

The issue that immediately came to mind was that if you change the kerning settings, then you can't input em dashes as --- anymore. See, for example, this Tex.SE question: https://tex.stackexchange.com/questions/315822/unwanted-kern....

But after a while I remembered the more fundamental point that bothered me:

> one might add that it's not just microtype that doesn't offer optical kerning -- it's structural limitations in TeX's typesetting that prevent it. To TeX, each glyph is contained inside a (I guess: black) box. TeX knows the outer dimensions of those boxes, but has no clue (and doesn't care) about what the glyph looks like that's inside it.

https://tex.stackexchange.com/questions/139926/optical-kerni...

And this led to bad spacing for hyphens, which led to me experimenting with manual kerning, which led to the issue that I recalled first.

Also, I am not really convinced of the merits of Vim for LaTeX. Most of my time writing is spent revising, which requires a lot of "random access edits," and I'm at least twice as fast at jumping to arbitrary locations on screen using a mouse as I am with vim (even with the usual tricks).

(I tend to think of vim as something that was purpose-built for programming in C and with usefulness inversely proportional to how far a given task is from that purpose. While I realize that's not entirely accurate, I've found it a good heuristic. Programming: vim works decently for me. General purpose writing: not so much.)

Further, the ability to control-click .pdfs and skip to the corresponding place in the .tex file, and vice versa, is an amazing convenience. I understand that there are technically ways to do this in vim, but then at that point I'm using the mouse a ton anyway and I don't really see the point of vim (which is usually proposed to be "hands don't go off keyboard = write faster" in my experience).

So, yes, I could survive if forced to use vim. I just wouldn't like it.

Given this, and given I feel I'm much more willing to experiment with these things than the average person writing papers in the physical sciences. I can't even imagine how the conversation would go if I tried to tell my 72-year-old advisor that TeXShop's not going to work anymore because he updated his computer and it's time to get friendly with the command line. These people want to do science, they don't want to learn vim shortcuts.

TeX finds out about character widths and kerning pairs and ligature replacements from outside of TeX itself. TeX is perfectly happy to ligature together "--" into an en-dash, and "---" into an em-dash, and to then kern either one of them with whatever characters you like, if instructed to do so by these externally-supplied per-font instructions. So, it seems like you're being troubled by the externally-provided font info rather than any limitation of TeX per se.

Boring details: TeX, as it comes from Knuth, reads TFM ("TeX Font Metric") files to supply this per-font information. Knuth's Computer Modern fonts do not contain any kerning pairs that contain dash characters, which one may be pleased or displeased about. Various enhanced-TeXs get font info from the OS or perhaps directly from, say, OpenType font files. If the ligature and kern info isn't passed into TeX from there properly, or doesn't exist in the font itself, then of course you won't get the results you're looking for.

Yes, but there's no way to get optical kerning, right?

Arguably, there is no algorithmic way to compute high-quality kerning based on character shapes (just as there's no way to determine, say, character width just from an outline). If there were, then Apple would build it into MacOS and iOS, and Microsoft would build it into Windows, and Google would build it into Android, and then all applications from TextEdit to Word, Safari to Chrome, and troff to TeX would automatically get told about these synthetic kerning pairs, and use them by default.

That said, Adobe does have a proprietary optical kerning scheme that they use in their applications. But it's turned off by default (hint, hint), and their documentation is a bit defensive about it: "Optical kerning adjusts the spacing between adjacent characters based on their shapes, and is optimized for use with Roman glyphs. Some fonts include robust kern-pair specifications. However, when a font includes only minimal built-in kerning or none at all, or if you use two different typefaces or sizes in one or more words on a line, you may want to use the optical kerning option for the Roman text in your document." Loosely translated: Fonts from real type foundries have kerning pairs carefully created by their designers; if you have slap-dash one where they didn't bother, we'll do the best we can if you want. YMMV.

I think this information would be built in to any high-quality font. As drfuchs tried to explain, TeX has no information about letterforms; that’s not its job.

> Also, I am not really convinced of the merits of Vim for LaTeX. Most of my time writing is spent revising, which requires a lot of "random access edits," and I'm at least twice as fast at jumping to arbitrary locations on screen using a mouse as I am with vim (even with the usual tricks).

> (I tend to think of vim as something that was purpose-built for programming in C and with usefulness inversely proportional to how far a given task is from that purpose. While I realize that's not entirely accurate, I've found it a good heuristic. Programming: vim works decently for me. General purpose writing: not so much.)

I don't want to comment on the kerning stuff (I don't really know much about it). However, I found your comment about the merits of vim when revising interesting. Like you, I spend most of my time writing revising/correcting (often other peoples work), however my take on vim is just the other way around.

If I write a large text myself, it doesn't really matter what editor I use, it's when I revise/correct things that I love using vim. While you might be correct that you would be quicker to move to the appropriate spot using a mouse, I pretty much guarantee that all the gain is gone when you have to switch back to the keyboard to do your change. That's why I love vim for these kind of tasks, the moving and changing of specific text is such a seemless experience without even having to change the hand position on the keyboard. My productivity absolutely plummets every time I have to use word or some other editor for these sort of tasks.

> While you might be correct that you would be quicker to move to the appropriate spot using a mouse, I pretty much guarantee that all the gain is gone when you have to switch back to the keyboard to do your change.

I see! What I didn't say is that I'm working on a laptop, so the trackpad to keyboard distance is negligible – it's essentially just another big key. All I have to do is pivot my hand.

> I've also heard that LaTeX editors are development disasters with a bus factor of one.

In case anyone else is wondering about this:

> The bus factor is a measurement of the risk resulting from information and capabilities not being shared among team members, derived from the phrase "in case they get hit by a bus."

(https://en.wikipedia.org/wiki/Bus_factor)

Fixing bugs isn't the same as actively developing. The bugs that are getting fixed these days are in increasingly obscure edge cases (in some cases no actual implementation in use can reach those edges).

That said, I've started working on such a project as you propose. The document preparation language will be almost identical to LaTeX, but no macro language and no category codes, UTF-8 as the standard input coding, first-class support for multiple back ends (e.g., use the same source to produce PDF, HTML or ePub), command definition through an embedded scripting language (I'm still debating between Python, Lua or something else entirely), formatting done declaratively, most likely through a YAML file, etc. My musings along the way are at http://finl.xyz

While the original TeX engine only get bug fixes these days, that's not really relevant for LaTeX since LaTeX doesn't even support that engine anymore. While pdfTeX development is rather slow too, it does still see feature development. That being said, for modern documents LuaTeX is really the only engine worth looking at and it is quite actively developed. Of course, it doesn't see much adaption but that problem will also appear for any new system.

That brings up the other big problem with contemporary TeX. LuaTeX has handfuls of subtle incompatibilities with pdfTeX and XeTeX (and a weird way of handling character setting at the low level where it maps characters to their index in the font file rather than unicode code points). It's also noticeably slower than XeTeX which is slower than pdfTeX. It would be nice if it were true that pdfTeX ⊆ XeTeX ⊆ luaTeX, but unfortunately, it's not.

> but no macro language

Any reason for that? The rest of your description looks very attractive, but why would I prefer a language without macros? I need conditional compilation, setting variables, parametrized notations, etc. This is extremely useful when preparing a scientific document. If anything, I don't really care about the document preparation language itself, but it must have macros to be useful!

To clarify, there would still be programmability, just not via macro expansion. Instead of macros there would be first-class variables instead of TeX's mix of registers and macros to emulate that. Instead of having to deal with the timing of macro expansions and all the pitfalls therein, an embedded scripting language would be available to handle complex settings (I'm leaning towards Python at the moment, but I'm far from having to make a decision on that topic). First-class flow control will also be available (right now, things like \foreach and \ifXXX are handled via macro expansion which causes a lot of pain beyond simple use cases).

OK, that's reassuring! What you describe sounds pretty much like a "macro language" to me (i.e., a separate language to describe your document). But this is just a question of terminology, somewhat arbitrary. Being able to generate the text of your document is a very powerful tool and, according to what you say, it will be even more powerful.

> we desperately need some modern language for document preparation, which will replace TeX and its derivatives

Do we? Sadly, appreciation for excellent typesetting has declined now that most people are consuming writing through a screen. And even for printed material there is pressure to cost costs: venerable and respected publishers with long traditions of obsessively good typesetting, realized they could outsource their typesetting to developing-world shops using Word or similar software, and few readers would care. (Similarly, and to the dismay of bibliophiles, even prestige titles these days are likely to get a glued building instead of a sewn one, and fairly low-resolution digital printing).

> lack of proper Unicode support; lack of support of modern font formats

Is that really true? I had no trouble writing UTF8 documents using regular TrueType-fonts in LaTeX almost two decades ago and with no problems with output quality.

The only thing was you couldn't go through the dvi backend so any tools which manipulated dvi was not possible to use. But I believe all this is default now.

> Is that really true?

Oh yes. For instance, LuaTeX doesn't print combining diacritics correctly unless you painstakingly swap the underlying codepoints around in some unspecified order (to be determined through trial and error) that doesn't match _any_ of the Unicode normalization forms.

I would be interested in how you did that? I had a lot of problems with Cyrillic scripts until I started using XeTeX three years ago. I think that LaTeX3 made some progress on UTF-8 compatibility in the last two years, but font loading is still a mess.

I think this is not a good reason to replace LaTeX. The big advantage of TeX over other systems is the extensibility of the language. Yes, the language is strange, but you get used to it, and it does what it's supposed to do. So I think that, just like UNIX, replacing TeX will never be completely done, unless something really great shows up.

Matthew Butterick is working on a new document processor that borrows some ideas from LaTeX and is built in Racket: https://docs.racket-lang.org/quad/

> I think that we desperately need some modern language for document preparation

See my comment below https://news.ycombinator.com/item?id=26498409 :-)

TeXmac looks very interesting. I will maybe give it a try. But I must say that TeXmac workflow seems a little bit dependent on GUI. That is not a bad thing per se but seams different form TeX.

Thank you for bringing it up :)

In case, for Linux I am using their static binary, if I recall correctly the Ubuntu package wanted to change my Guile to 1.8 and this wasn't ok for me. I have learnt today that Fedora has it in the repositories. The Windows installer works and as far as I know the one for Mac OS too.

Reply on news.ycombinator.com