Typst: A Programmable Markup Language for Typesetting [pdf]
user.tu-berlin.de
user.tu-berlin.de
As SGML pedant however, I can't resist commenting on the following:
> The second offspring of SGML is XML, specified by the World Wide Web Consortium (W3C) in 1998. It has a reduced feature set compared to SGML (for example, it forbids unclosed tags and concurrent markup). But it retains the most important aspect of SGML, one that HTML is lacking: The ability to define custom structural elements. This lets XML represent documents with much more semantic detail than HTML.
As the SGML vocabulary HTML was once envisioned, HTML itself doesn't need extensibility. When used as an SGML application, defining your own elements in HTML is as easy as declaring those in the "internal subset" or in a custom DTD right away. Assuming any wellformed element is accepted as of ISO 8879 Annex K's FEATURES IMPLYDEF ELEMENT rather than rejecting undeclared elements, that's actually only necessary if you want to validate/infer custom content models, or use any of the other things markup declarations provide, such as custom SHORTREF syntax a la markdown.
Arguably, HTML5's "custom elements" do provide a facility to define new elements, if incredibly lousy; ie. custom elements can't have content model restrictions (see above) and can't be used with tag omission/inference (important for customized elements), aren't integrated with DOM parsing, and need JavaScript for declaration - the latter point making them completely pointless as a markup feature.
What would you recommend instead? From my experience of the beta it is vastly nicer to use than latex, and I'm not really aware of any other competition.
And in fact, in practice you could just use your own custom elements without worrying about validity and it’d mostly just work. This wasn’t even particularly rare. (There was the whole “CSS doesn’t work on them until you call document.createElement("…")” bug in IE, but that’s the only problem I can think of, and it was easily worked around.)
What browsers made out of it isn't the matter here, but even if it were, the "practical, real-world HTML out there" argument is mostly used to pull up the ladder by an ad company/browser cartel made worse day-in day-out through an atrocious and absurdly voluminous HTML spec (and by CSS, of course).
Even though Ian Hickson, of WHATWG, wanted to capture HTML as it was understood by browsers, he couldn't help but added additional elements of his own - such as for marking up ads as "aside" lol plus the alien sectioning elements concept that gave rise to the flawed "outline algorithm" and misuse of heading elements (and earlier failure to understand SGML's RANK feature), a problem that was only fixed last year [2] by an incompatible change to HTML invalidating documents using hgroup as originally advised.
In practice, very few changes to the HTML syntax brought HTML outside SGML - for the most part, ad-hoc and basically unnecessary commenting rules for the script and style elements to keep legacy browsers from rendering JavaScript and CSS, resp., when those where introduced.
You seem to always start with the assumption that SGML was (and perhaps is) an end goal. I deny this.
HTML was designed as an SGML vocabulary, but, where it mattered, never implemented as an SGML vocabulary. If Tim Berners-Lee ever even expected it to be treated as SGML very much, I suspect he hadn’t thought things through well enough (though that could also just be hindsight bias on my part).
There has never been any particular virtue in HTML being an SGML vocabulary. No one that mattered (which mainly means browsers) cared about SGML, then or now, and no web developers or end users care about SGML, so being SGML is just needless complication and potential for confusion (due to that implying different behaviour from reality). SGML is a hideous, complex beast that no one wants to work with, and which almost everyone that has heard of it is glad is dead.
Yes, SGML had some nice ideas. Yes, we keep on reinventing parts of it. Yes, a variant of Greenspun’s tenth rule applies. But SGML was just too flexible/generic, large and ugly. It doesn’t actually solve things. And the current HTML parser is the best thing since sliced bread and my favourite popular file type spec by a large margin despite its size, because it’s clear, unambiguous, and implementable.
You can care about browsers, I care about documents and that they can be read and understood in a couple decades still. Preferably without kissing the ring of an ad company.
Defining your own vocabularies and SGML is also directly mentioned in the paper being discussed. SGML lets you define your own custom language and mapping to HTML as output/rendering language without further tools.
> SGML is a hideous, complex beast that no one wants to work with
As opposed to what? The web platform specs covering all of HTML, CSS, and JS roughly a thousand times the size of the SGML spec? Have you actually studied SGML or implemented a parser for a markup language, or are you repeating what you've heard elsewhere?
> And the current HTML parser is the best thing since sliced bread and my favourite popular file type spec by a large margin despite its size, because it’s clear, unambiguous, and implementable.
Which version of WHATWG HTML5? Oh, WHATWG don't bother versioning their phone-book sized specs. And parsing breaks all the time; eg. current head doesn't contain the param element anymore (as content of the object element still in the spec) which however requires that no end tag is specified, hence a parser for current WHATWG HTML will fail hard in the presence of param elements (similar story with legacy elements such as keygen). Then there are new "boolean attributes" being introduced all the time requiring special rules/markup declarations ...
With respect, the argument isn't particularly relevant anyway as those specs aren't aimed at folks having difficulties following a formal language spec/grammar but need procedural step-by-step instructions instead.
Random example:
https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2F...
Calculating stat-blocks by hand is a nighmtare for one author, so I need a typesetting language so I can type `\elf`, and have a random elf, then generate all its derived stats correctly (like Attack, Defence, et c.).
My example: https://i.redd.it/ng82unzqxru41.png
Each book has different versions (minimal/ full) so getting reliable output has been a bit of a chore.
LaTeX is the only tool I know which can do the job, but it's still a bit of a nightmare at time. Ugly formatting means my code looks whacky, images can't float cleanly around a multicolumn environment, packages conflict, words overlap and it needs constant hand-holding.
Typst looks like just what I've been after - something like rmarkdown, with more power. Tables and layouts, and best of all - no packages.
What licence is typst under?
Will it be able to reference page numbers from a different book dynamically? (so if book-a and book-b are in the same directory, one can reference another's sections)
Will there be support for intelligent floating images?
Will it accept 'every-page' commands, so I can use that #rect command to show a chapter's name on side-tabs?
Will error messages tell me hbox underfull badness 10000 at least 3000 times per compile?
In relation to formatting, I cannot tell what could be the issue but this does not correspond to mine and others' experiences.
Typically you would typeset the book with the generally intended text areas with LaTeX, and then generate the preprint with previously not included graphical elements and fine-tuned text layouts with Adobe InDesign. This is what publishers typically do, again, from mine and others' experiences.
[1] http://mirrors.ctan.org/info/luatex/lualatex-doc/lualatex-do...
We will open source in March, probably with a permissive license, but that's not yet definitively decided.
> Will it be able to reference page numbers from a different book dynamically? (so if book-a and book-b are in the same directory, one can reference another's sections)
This is not currently implemented, but also shouldn't be fundamentally impossible.
> Will there be support for intelligent floating images?
There will be floating containers (also with text flowing around). We will probably keep the amount of "intelligence" low for more predicatability. So you would specify top or bottom and it would be placed on the next page with free space.
> Will it accept 'every-page' commands, so I can use that #rect command to show a chapter's name on side-tabs?
Yes, every-page headers, footers, foregrounds, and backgrounds are already available and a way to query for the current chapter name is coming in a future update.
> Will error messages tell me hbox underfull badness 10000 at least 3000 times per compile?
No!
> There will be floating containers (also with text flowing around). We will probably keep the amount of "intelligence" low for more predicatability. So you would specify top or bottom and it would be placed on the next page with free space.
If it's not too 'intelligent', I hope there is - at a minimum - a floating image command that basically says 'place this floating image so it stays on the same page as this current line'. Currently I'm placing every float about 30 lines before where I want it and hoping for the best. It'd be a lot easier to just say 'place this float 50 lines back, typeset the page, and if the image is not in the right place, then start again 20 lines back', then loop until it's at no lines back or until it's with the right text.
That's ugly coding, but less ugly than the current hacks I do with LaTeX.
I hope the project goes well, and hope to see it in the standard repos before long.
I've been recently working on a book project, and I've reviewed all the tools available. My current conclusion is that I found nothing that would combine the simplicity of Markdown or Asciidoc with the typographical control of LaTeX.
This project seems to hit the sweet spot.
Is the source code of the thesis available somewhere? Of other works created with Typst? That would help making a more educated bet.
Edit: this doesn't seem to be an open source project. So not what I'm looking for.
There isn't much Typst source code out there yet, but you can find some examples and discussion on our Discord server [1] and in our documentation [2].
This would need various compiler backends (perhaps via pandoc) to be that useful. Certainly it would help adoption if you could emit, eg., markdown/latex/etc. and others you're working with wouldn't need to adopt your tooling.
It might be possible to have a subset of typst output to latex though, and then grow that subset. Like you could just evaluate all the functions to get raw text/markdown and then translate that to latex, and at the beginning you just wouldn't support all possible styling choices.
Does this system uses globally-optimal algorithm for second step (page composition)? Does anybody know globally-optimal formatting system?
All of these crates are actively maintained. Are people re-inventing the wheel?
[1] https://gitlab.redox-os.org/redox-os/rusttype
[2] https://github.com/alexheretic/ab-glyph
[3] https://github.com/mooman219/fontdue
I see no net gain in implementing or in using subpar nonreal-world-tested libraries just to do so in Rust.
They do not. What they solve is almost a toy problem compared to the size, scope and breadth of these libraries.
Just because some project is implemented in Rust does not make it comparable never mind superior by default.
There is a world out there and it is not homogeneous format and standards-compliant Latin fonts in English LTR text in linear disposition with some generic rectangular subpixel rendering on a regular rectangular grid.
I warmly welcome you to browse closed issues of FreeType [1] and also the closed issues of HarfBuzz [2]. If you feel inspired please do also look into mailing lists and discussion pages related to the development, building, tracking and patching of packages of these projects in any of the numerous places it is used.
The only argument Rust people have is in relation WASM but if you insist in targeting WASM why not fork FreeType, strip it to the strict subset of features your application needs and target it?
Why do it in the first place? Why reinvent the wheel?
As such I will restate my view: I see no gain in using any of these subpar libraries.
[1] https://gitlab.freedesktop.org/freetype/freetype/-/issues/?s...
[2] https://github.com/harfbuzz/harfbuzz/issues?q=is%3Aclosed
Although Typst uses pixglyph for rendering, it does support BiDi, complex script shaping, etc. For shaping, we use rustybuzz, which is pretty much a 1-1 port of harfbuzz to Rust. Although we would have gladly used harfbuzz, linking C and Rust in WASM is unfortunately not really possible. So we went for the practical choice of helping finish this port and using it.
> it does support BiDi, complex script shaping
Hey, that is indeed quite good. Would you mind if I ask you how well is the support for popular Asian languages?
> linking C and Rust in WASM is unfortunately not really possible
Damn. I am not very experienced in Rust but I would not have guessed that. I apologize if I misrepresented difficulties related to targeting WASM.
[1] https://github.com/apple-oss-distributions/WebKit/tree/WebKi...
[2] https://github.com/apple-oss-distributions/WebKit/tree/WebKi...
[3] https://github.com/chromium/chromium/tree/main/third_party/f...
[4] https://github.com/chromium/chromium/tree/main/third_party/h...
[5] https://searchfox.org/mozilla-central/source/modules/freetyp...
[6] https://searchfox.org/mozilla-central/source/gfx/harfbuzz
[7] https://cs.android.com/android/platform/superproject/+/maste...
[8] https://cs.android.com/android/platform/superproject/+/maste...
[9] https://www.amazon.com/gp/help/customer/display.html?nodeId=...
As someone who studied typography and worked in that field professionally I understand the complexity of these subjects.
If you re-read my question it was about rasterisers. I.e. something that takes a bunch of lines, quadrics and cubics and draws a filled shape.
Everything else you read between the lines of my question but it wasn't there.
I have just taken a look to the syntax. It is pretty close to djot [1]. Of course Typst is more powerful than djot. One noticeable difference is the chosen markup for headers (`=` instead of `#`).
This could be nice to reduce differences and make Typst a close-to superset of djot.
https://pldb.com/languages/asciidoc.html
https://pldb.com/languages/creole.html
As someone who writes a lot of math, I'm not sure I would want / as stacked fraction to be the default choice, as it is in Word (but not Latex). Many mathematical (sub)expressions, especially short ones, are more readable when written inline with the slash.
$$\frac{ x^{2/3} + 7 }{ 4y/z + 9 }$$
This makes it easier to read: nested fractions kinda suck no matter what, but with bars on bars on bars they're even worse.
Anyway, there should at least be a way to specify which fraction notation you want. I would even suggest // for stacked-frac, but that's a comment. Microsoft Word and powerpoint have copied quite a bit of Latex in the past several years which have made them more usable for math, but their lack of fraction rendering control is one of my pet peeves.
Edited to add: Also the slash is used for things other than fractions, like group quotients, which are not conventionally written in the stacked notation.
See eg https://www.math.leidenuniv.nl/~psh/ANTproc/04psh.pdf
Markup-based DTP was my interest for many years, and I happy to see new full-featured players on this field.
MarkDown, AsciiDoc and others github-README formats are nice but disappointingly limited (and under-specified).
Edit:
> the table of contents is typically at the start of a book, not the end.
In my culture it is typical at the end, though in last 20 years a sheer amount of translated literature which layout is copy'n'pasted form originals (by people who was not trained in classical typesetting, but learned DTP on-the-fly) somewhat change this.
If authors want to continue to support & expand their product, they need to study very diverse typesetting traditions of different countries and languages.
1) Is it possible to create new structure elements and rules how they can be nested? Now there are headers of several levels, and it's all?
2) What's about tables? 2.a) Multipage tables? 2.b) Multipage tables which have alternative orientation - like landscape table in portrait book?
3) What's about inserts, which must be placed in exact break in the main text?
4) What's about marginalia? Marginalia which must be synchronized with text?
5) What's about multicolumn layout? 5.a) What about multicolumn layout which is interrupted by single floating object (which belongs to page, not one column) - there could be two styles, when columns above and under floating object are independent and when they are continue after interruption.
6) What's about multiple flows of text, synchronized? Think about bi- and triple-lingual texts, like translations of legal documents or poetry. Different flows could go to different pages (odd-even) or different columns on one page, but each page/column must contain same text, even if one language is more compact than another and one page/column will be systematically underfilled.
7) What's about poetry, and modern poetry especially, where each line can require its own indentation (look up Mayakovsky's poetry).
8) What's about footnotes? 8.a) Long, multi-page footnotes? 8.b) Nested footnotes, more than 2 levels of them? 8.c) Nested multi-page footnotes?
9) Is it possible to create "Problem / Solution" object typical for textbooks? I want to type problems and solutions together and automatically place all problems where they are typed, but move all solutions in designated place (like, last chapter)?
10) Is it possible to have Initials automatically?
11) What's with custom alignment rules? For example, in some typographic traditions last line of aligned paragraph must be not shorter than 1/3 of paragraph's width, and must be aligned too if it is longer than 2/3.
12) What's with layout engine in general? Is it TeX' box-and-glue model? Is orphan/widow control customizable?
13) What's about non-rectangular images? 13.a) Alpha channel support for runaround? 13.b) Additional vector path for runaround? 13.c) Choice should image be trimmed or become background?
14) What's about images and other objects on the double-page spread?
15) What's about layouts typical for magazines, like "Cosmopolitan"? It is weakest point of TeX-like DTP systems :-(
> 1) Is it possible to create new structure elements and rules how they can be nested? Now there are headers of several levels, and it's all?
At the moment you cannot define custom structural elements, but you can define a function that produces some layout and the result is very similar. The structural aspect becomes more important when different parties need to agree on the meaning of something, i.e. you and Typst's standard library or you and a package author, so that the styling system also works with for those elements. We will add support for defining custom structural elements down the road.
> 2) What's about tables? 2.a) Multipage tables? 2.b) Multipage tables which have alternative orientation - like landscape table in portrait book?
Tables exist, also multipage tables. The table sizing is already quite flexible (and inspired by CSS grid), otherwise the customization is still quite primitive, though. We will improve tables in the future, for sure. See also here: https://typst.app/docs/reference/basics/table/
> 3) What's about inserts, which must be placed in exact break in the main text?
Do you just mean a block between two paragraphs? If so, that's possible, yes.
> 4) What's about marginalia? Marginalia which must be synchronized with text?
Not implemented at the moment, but planned.
> 5) What's about multicolumn layout? 5.a) What about multicolumn layout which is interrupted by single floating object (which belongs to page, not one column) - there could be two styles, when columns above and under floating object are independent and when they are continue after interruption.
Typst does support multicolumn layout. Floating objects that interrupt both columns are not possible at the moment, but we're aware of the requirement.
> 6) What's about multiple flows of text, synchronized? Think about bi- and triple-lingual texts, like translations of legal documents or poetry. Different flows could go to different pages (odd-even) or different columns on one page, but each page/column must contain same text, even if one language is more compact than another and one page/column will be systematically underfilled.
This is obviously a bit more complicated. I can see supporting something like this, but only in the long term.
> 7) What's about poetry, and modern poetry especially, where each line can require its own indentation (look up Mayakovsky's poetry).
You can create manual line breaks and insert horizontal spacing. Do you imagine a more automatic way?
> 8) What's about footnotes? 8.a) Long, multi-page footnotes? 8.b) Nested footnotes, more than 2 levels of them? 8.c) Nested multi-page footnotes?
This is one of the basic things that is still missing, unfortunately. But it has relatively high priority and we want to ship them before the public beta launch in March.
> 9) Is it possible to create "Problem / Solution" object typical for textbooks? I want to type problems and solutions together and automatically place all problems where they are typed, but move all solutions in designated place (like, last chapter)?
This will be possible with Typst's introspection system (which is already implemented, but not yet fully exposed to the user).
> 10) Is it possible to have Initials automatically?
Not yet, but planned.
> 11) What's with custom alignment rules? For example, in some typographic traditions last line of aligned paragraph must be not shorter than 1/3 of paragraph's width, and must be aligned too if it is longer than 2/3.
Exposing more control over paragraph layout is something I've wanted to do, but we haven't designed such a system yet.
> 12) What's with layout engine in general? Is it TeX' box-and-glue model? Is orphan/widow control customizable?
It's not boxes and glue, but we use TeX's optimal paragraph layout algorithm. Orphan/widow handling is currently baked-in, but similarly to exposing more control over paragraph layout, we also want to expose more control over flow (i.e. page or subbox) layout in the future.
> 13) What's about non-rectangular images? 13.a) Alpha channel support for runaround? 13.b) Additional vector path for runaround? 13.c) Choice should image be trimmed or become background?
Not prioritized, but planned.
> 14) What's about layouts typical for magazines, like "Cosmopolitan"? It is weakest point of TeX-like DTP systems :-(
This is a hard problem. We want to support content flowing across multiple boxes, which is of course a requirement for such a layout. But overall, I think the hardest part is not making it possible, but making the creation of a magazine in a markup-based system actually hassle-free. When creating a magazine, you just benefit much more from dragging stuff around with the mouse than if you're writing a research paper.
I understand, that Typst is very young now, and can not support all spread of typographic features, questions are about flexibility of internal architecture, which can (or cannot) allow to support all these features theoretically without global re-design.
I've started to plan something like this several times in last 20 years, but always dropped project (as it was pet-project for me) due to understanding that it is very hard design problem.
Good luck to your project! IMHO, we need something like this!
(and, yes, (14) was tongue-in-the-cheek question, sorry).