Elements of a great markup language

I think inspiration in this area should come from an unusual place: RFC documents (like https://www.rfc-editor.org/rfc/rfc6877.txt). These documents are fully readable in their standard text form. The only things missing are conveniences like clickable links, which could be supplied by a viewer application without transforming the text of the document.

Markdown comes the closest to being directly readable, but it makes a number of sacrifices to syntax in view of its intended transformation to HTML. It's also limited in scope, patterned after ASCII email syntax. So you miss document-oriented conveniences like standardized headers, proper citations, and pages - all of which RFC documents have!

That's in part the reason that a bunch of people have come behind and written extensions to Markdown. IMO someone needs to come back and gather the best of these into a consistent and simple syntax that is oriented around the needs of plain text readers.

This isn't really a fair criticism. True, the original Markdown.pl did not produce a generic tree structure, but that's a fact about the program, not the syntax it parses. Many Markdown and Commonmark implementations do support creation of an abstract syntax tree. Pandoc has done this for the last 17 years. It also provides nestable, generic containers as a syntax extension.

> It feels like there’s a smaller, simpler language somewhere

Here's my attempt: <https://djot.net>.

Today bold, italic, tables don't fit in plain text, but carriage returns and tab characters do, as well as Halloween pumpkin emojis. There is no good reason for this, this is just where we kind of ended up. You can't put a floating point number in text, or a price, or a phone number, or date. And when you don't have standards that support this you can't have any meaningful kind of data exchange. So forget about copying a table from a spreadsheet anywhere. Everybody ends up shoehorning this necessary functionality in text editors and it won't ever work.

How can computing still be at the stage where if you want a table you have to conjure up a bunch of pipe and dash characters that hopefully compiles into a table that looks presentable? And then you have to switch between "code view" and "presentation view" to check if you did it right. It would be funny if this wasn't so tragic.

The point of markdown is to be readable by humans, using similar annotations that people have been using in ASCII text emails for decades.

I have to disagree. Often the abstract nature is hard to describe and/or the constructs for it either don't exist, or need cleaning/updating. Often I find myself saying, "I don't know why it's more legible to format this thing such and such way, it just is." Creating a good abstract language or category set for a given domain is not easy. It's good to use abstraction where it's practical, but often you just have to tell them system "just format it like this!"

For example, the difference between a button and hyperlink is often blurry. We could abstract it something like: [Action FormatType="Button" Label="Send email to Mom" ActionType="mailto:mom73@moms.sample"/] so that FormatType could be changed to "Hyperlink", but most find this goofy and perhaps long-winded.

As far as compact wiki-like shortcuts versus XML, the first often has more escaping issues or confusion. It's not a free lunch, but about best fitting intended use.

Isn’t using a basic list structure for documents a feature, not a bug? For example, the article itself does not need or use any hierarchy. Surely keeping documents simple is part of the raison d'être for markdown?

Hierarchical structures for documents usually need more software support, which does provide some nice features such as nested formatting, table cells containing complex elements (including other tables), and auto-numbering of headings/outlines.

However a hierarchy is more fragile when text documents are edited as plain text or using text based tools. Surely a prime purpose of markdown and asciidoc is to be simple plain text.

A hierarchy also seems to lead to either (a) hard formatting requirements like strict XHTML, or (b) loose requirements that cause many corner case bugs between implementations (like HTML used to be when XHTML was created).

> Great markup format unambiguously interprets an input string as an abstract tree model of a document. It doesn’t ascribe semantics to particular tag names or attributes.

>

Yes!

> Markup language which nails this perfectly is HTML. It directly expresses this tree structure. Various viewers for HTML can then render the document in a particular fashion. HTML’s syntax itself doesn’t really care about tag names and semantics: you can imagine authoring HTML documents using an alternative set of tag names.

But it's a bit weird to uphold HTML when it seems to really be valuing the underlying syntax and not the big heckin' standard that details the valid elements, attributes, and their semantics. :)

> Great markup language defines the semantics of converting text to a document tree

This seems to conflict with the first one I quoted. Maybe I'm missing something? Maybe the author is using semantic in different ways, here? Markup languages tend to mix different kinds of elements, and it can make thinking and communicating in this space tricky! I started trying to untangle this knot recently in https://t-ravis.com/post/doc/what_color_is_your_markup/.

(I also found banning "semantic" from my writing helped me untangle by making it easier to notice when I was letting it do too much work. https://t-ravis.com/post/doc/semantic_the_8_letter_s-word/)

* It's not at all lightweight, I'm pretty sure it's Turing complete up the wazoo.

* You can reprogram the f'ing grammar, down to the interpretation of individual input characters...

* In fact, I'm not sure you can even call it a markup language. It's a macro-based programmable system on top of a typesetter.

They go from this, which I agree with:

> A good markup language describes an abstract hierarchical structure of the document, and lets a separate program to adapt that structure to the desired output.

> Great markup format unambiguously interprets an input string as an abstract tree model of a document.

To this:

> [In Markdown] there's no way to express generic tree structure, conversion to HTML with specific browser tags is hard-coded.

I understand why you'd want the markup to be turned into a tree, but that's different from saying you should be able to build any tree with the markup. Isn't the point to encode text documents? The fact that you can't build the entire layout of your HTML result in the markup language doesn't seem like a sensible requirement. Why do you want or need infinitely-nested structure in documents?

First requirement want to nest and organize everything, second one relies on some "natural" (highly contextual) way to do it.

What looks natural for list of strings will not be natural for lists of tables of lists of maping between location snippit and some text.

I'd really like to see a new API to allow creators to create their own markup languages directly in the browser with native support to map the markup to the underlying renderer.

  enum Element {
    Text(String),
    Node {
      tag: String,
      attributes: Map<String, String>
      children: Vec<Element>,
    }
  }

  fn parse_markup(input: &str) -> Element { ... }

The reason HTML nails it perfectly is because this is modeled after HTML.

If I were to make up a markup language, I wouldn't follow that model.

In particular I would get rid of attributes which to me are a restricted kind of children with a specialized syntax.

This is both unnecessary and undesirable in many cases.

The major problem with attributes is the <String, String> mapping. Once something is defined to be an attribute, it cannot be sensibly extended without creating an unnecessary problem.

For example the `class` attribute in HTML looks like it was originally designed to hold a single class name. Then people realized that it would be desirable to have multiple classes per element.

So, to keep things backwards-compatible, the value of the attribute was extended to hold a space-separated list of classes instead. Essentially creating a little DSL inside of the attribute's value.

If `class` was instead a kind of child, initially limited to a single instance per element, extending to multiple instances in a backward-compatible manner would not require introducing the DSL. It would be natural. Just allow many `class` children.

A valid argument in HTML in favor of attributes is conciseness. But that's an artifact of the syntax.

We could make up a language with syntax for nodes as concise as HTML attributes, eliminating that argument.

> It feels like there’s a smaller, simpler language somewhere

Certainly.

I have experimented with many different designs for such a markup language on top of Jevko[0]. One interesting design that trivially maps to HTML looks like this:

  h1 [Title]

  p [
    paragraph
  ]

  p [
    [paragraph with a ]
    a [
      href=[...]
      [link]
    ]
  ]

  [h1][Title]

  [p][paragraph]

  [p][
    paragraph with a
    [href[...] a][link]
  ]

[0] https://news.ycombinator.com/item?id=33287620

This is why I like FB2 and dislike ePUB.

[1]: https://github.com/civboot/cxt

And schemas and schema validation on top!

The `include` is a problem, though, and the devs have known this since at least 2017. Two directives under work here. The `ainclude` is an extension that works in the subdoc direction, and `subdoc` is a directive under development that's on the milestones to go into core. Generic blocks is also on the roadmap, and it's available via extension.

However.

There's deeper problems, I feel, inherent in the DNA of component content systems themselves, as a basic concept. It plagues every CCMS (component content management system), whether it's DITA, S1000D, DocBook (with xincludes), ReST, or otherwise. It's not a markup problem. In S1000D, I call it "the applicability trap".

To put this very briefly, you incur risk when you replace any component of natural language with a constructed language. In the case of CCMSs, you have an implied content architecture doing the work of NL in between content "chunks", but not everyone architects content, or, hell, even their product. The writers won't know this until they start, and by then it's too late.

Without architecture, you have a bunch of chunks that used to be structured linguistically, and now they're not doing much of anything at all.

I can't tell you how many S1000D systems I've seen built around a product that existed as little more than a twinkle in a salesman's eye. You build out an SNS and an applicability model[1], all from a bunch of emails or a spotty CAD diagram, and tell the writers to get cracking. Then, N months and N millions of dollars later, scrap everything because, lo and behold, the actual possible product completely invalidates the content architecture. "How did the doc set cost 19 million dollars?". Well, it's not the markup, it's not the vendor, and whatever miracle editor your salesmen buddies are pushing won't do jack to fix it, either.

What the writers need is a quantifiable test for architecture before they start, or else, sooner or later, the Applicability Trap will get them. Either that, or write individual BDMs[2] for every single product variant that spews out of the pieholes of business development.

I feel a little dumb getting worked up here. All of this crap is going to be glitter in the river come 2030 anyway, because we'll be training our pocket AIs for most of this.

[1](sort of a structure of conditions to filter content)

[2] Big Dumb Manuals, like mamma used to make

The most important thing in a markup language is the name. A language will not succeed without a good name. I have recently invented a very good name, and now I am looking for a suitable language.