Hope is an alternative to markup languages | Hacker News Reader

Hope is an alternative to markup languages

poef.github.io

49 pointsgjvc40 comments

The technical term for this kind of markup is "stand-off markup" or "stand-off annotation".

There is wealth of research done on stand-off markup since the SGML times, with a renaissance in 2005-2010.

Historically, the main drawback (as highlighted by other comments) is that stand-off needs a way to refer to the content, i.e. a pointing mechanism. Pointers in a pointing mechanism can be either a) hard to write but easy to maintain or b) simple to write but prone to breaking after a change (and thus requiring support from authoring tools). Nobody has found a viable ergonomic balance yet.

Agreed, you cannot edit one part without the other. But there are many situations where this is not a drawback. Nobody is writing Word documents in a text editor for example.

But I'm looking at a way in which you may be able to add annotations to a document that isn't under your control. If the document is either guaranteed not to change, you won't have a problem. Documents hosted on IPFS or a versioned URL to a Hyper document fit this bill.

If you want annotations to automatically change when a remote document changes, you need to be able to 'follow' the changes. Between when you originally 'linked' to the document and now, what has changed in the document. Which parts are deleted, which parts are added. If you know that, you can update the offsets in your annotations. So I'm looking into making a versioned document format to allow just that.

One of the my favorite apps, Scapple (from the same company that produces Scrivener), stores their data using just this approach (plaintext + character ranges), but in XML. It does make the data file much more readable, and it makes it possible to grep for text without having to worry about regex to skip through tags, but IMO the convenience is outweighed by the fact that there is no way to do mass search and replace (that is, using external tools) without totally messing up the formatting.

IMO the best approach is inline makeup but for the text editor to have the ability to hide tags. An example is PowerGrep's docx editor. You can open an .docx file and it will be editable entirely in plain text, but when you save it, all the formatting is preserved. In some way existing text editors (e.g. Sublime) are already capable of doing this via some kind of fold by regex functionality

I've toyed around off and on with FUSE drivers for various purposes, and hiding information like tags and then preserving those tags on save is something you can delegate to your filesystem. The advantage is that tag-hiding now works automagically with ripgrep, your favorite editors, and any other dev tool.

You _are_ limited in that you can only detect edits during interactions with the filesystem, which is strictly less information than you'd have in an editor tracking every keystroke, but a good diffing algorithm still mostly gets the job done.

The problem with this is that if you want to make any changes to the plain text you have to edit the markup in unison if you want the result to look coherent. So this gives up one of the principal benefits of markup languages: you can edit them with a simple text editor in a straightforward manner.

To be fair the author acknowledges this. To make an analogy, this seems to me more like the equivalent of ASM than the equivalent of C, the only reasonable way to make this work is to have a high-level language that "compiles down" to Hope.

Sure, but now you've come full-circle: what is that high-level language going to look like, if not a traditional markup language? And if Hope is intended to be object code rather than source code, what's the point of making it a text format at all?

If I have to be honest I don't see much from this either, but a potential upside is structural separation of text and presentation in a way that's different from HTML/CSS, in this case the content would be pure ASCII text as opposed to a tree structure/object model.

I'm not sure I agree with this premise either, the hierarchical structure seems inherently semantical to me, but the argument of the OP is it isn't the case.

> what's the point of making it a text format at all?

The text part is ASCII, so it make sense to represent it as such. The "presentation" part could be binary.

The only reason I used a text format is because I wanted to keep the code as simple as possible. Only introducing extra complexity when I hit a wall somewhere. I don't know if the current format will scale to more complex use cases, but I really like the fact that you can read HTML and reason about it, so I kept to that in Hope as well.

As for the text part, that is simple. I like to be able to manipulate text as text. There are many many tools to work with plain text. I can use regular expressions to search through text, get ranges of the result and apply an annotation to those ranges. You lose a lot of that in any other format.

> one of the principal benefits of markup languages: you can edit them with a simple text editor in a straightforward manner.

That wouldn't be much of a benefit for me because I don't use Nano. I don't think many people do either.

I can't put my finger on what alternative you're alluding to here. Writing HTML/Markdown/whatever feels mostly the same whether I'm doing it in Nano or Notepad or VSCode, and I probably use all three of those on a daily basis depending on what I'm editing/how long/what privileges I need.

> Writing HTML/Markdown/whatever feels mostly the same

That's because those languages are designed to be edited by Nano. docx files aren't.

Buffers and strings with range-indexed metadata is also how Emacs's text properties work. For example, this shows the print syntax for strings with properties:

    (concat (propertize "foo"
                        'face '(:weight bold))
            " "
            (propertize "bar"
                        'link "http://bar"))
    ;;=> #("foo bar" 0 3 (face (:weight bold)) 4 7 (link "http://bar"))

My mind's been picking at related ideas from different angles for at least a decade now, I guess.

I'm not sure how I feel about this one, but one thing I like about separation approaches in general is that it's easy to imagine how N different annotations can all exist for the same underlying document, so you could use something along these lines to share/exchange/discover/crowd-source annotations.

That said, my general yearning isn't to kick all markup out of my documents--but to kick presentational markup out. Semantic markup can add a lot of affordances to your documents, and if I'm already committing to working with something that requires a toolchain--I suspect I'd rather just generate plaintext or presentational markups from there?

But why?

Why is a plaintext unformatted file with just words and no markup any improvement? They're both editable, readable in a raw form, and probably not how you want them to actually be consumed.

So why not just keep it in one file?

Markup und plaintext are seperate entities by nature, and by seperating them physically, tools which don't understand the markup would work better with the pure plaintext.

One thing that Hope doesn't seem to support are contentless elements. E.g. some drawing may look so:

    100:100:svg:svg
    100-100:svg:g /* group */
    100-100:svg:rect
    100-100:svg:rect

These elements have no text content, so their extent is 0 and offset is same. Yet they also have hierarchical relationships and you need to store extra data to express those. E.g. in this example you can guess that 'g' and 'rect's are descendants of 'svg' and the first 'rect' is a child of the 'g' but only because you have some preconceived ideas about the meaning of these elements. There's not enough data to tell if the second 'rect' is a part of 'g' or not. You need to store at least the level in the hierarchy. If you stored the data in a database (i.e. without implied order) you'd also need the relative position of the element, but here you have the order built-in.

Original author here, this is exactly right. Hierarchy and contentless annotations don't mix well. This is the main disadvantage when compared to xml/html and the like. On the other hand, using annotations with offset pointers to the main text make things like overlapping annotations and multiple sets of annotations possible, which are difficult to pull of in xml/html.

Hi, original author here, I just found out my old project featured here and love the thoughtful responses. If you are interested, I've been working on a new version of and on, called cobalt (https://github.com/poef/cobalt/). Hope was the proof of concept, cobalt is a more rigorous implementation. Specifically the range algebra is complete there.

It's still a hobby research project. There's a lot I've learned building and using it, about the things that HTML is good at, and what it isn't good at.

What I'm trying these days is to see if I can get a WYSIWYG editor with a user experience that comes close to the user experience of writing markdown. This isn't easy and its interesting to explore what makes that so.

Reminds me of John McCarthy's lament at the W3C's choice of SGML as the basis for HTML : « An environment where the markup, styling and scripting is all s-expression based would be nice. »

Could this project be the beginning of an answer : http://lambdaway.free.fr/lambdaspeech/

Alain Marty

> On the left you can edit the content and below it the markup. On the right is the rendered output

I am not sure if I can see anything on the right/left. this is what I see on chromium/firefox https://i.imgur.com/quySrXT.png. No rendered veiw!

Original author here, hope is an old experiment and I hadn't updated it to avoid mixed http/https content. This solved now in the online demo.

Where do you draw the line between "content" and "mark-up"?

* Word break nearly always gets treated as content, denoted by the space character.

* Line break can be either: '\n' or <br>.

* Paragraph break is usually treated as mark-up: <p>.

Is there a good reason for that, or is it just historical accident?

This reminds me of how Nisus (the word processor program) styled text. The document, a Classic-Mac-OS file, contained plain text in the data fork and styling information in the resource fork. If I recall correctly!

Yes, this was a rather good way to deal with it. Even if the content wasn't rich text, it still may have things like current position, folded sections and so on. It totally pays to keep this stuff separate from the main content as the latter stays readable by any other program that supports text.

In general the whole business of file type plus creator type felt much better than the single file type of now. E.g. in desktop publishing Illustrator EPS were very different from Photoshop EPS and it was very convenient to be able to tell them apart.

XML is a serialization format. It's trivial to mechanically separate the markup from the content when reading and to merge them back when writing. (It should even be the preferred approach for many usage scenarios as it's rather efficient.) The description implies content and markup are physically separate; it can be done, but why? Although there is at least one upside: the same text can have multiple different markups attached (e.g. syntax and revision control or syntax and extra highlighting).

Why is you want revision control of both the text and the syntax markup? This seems incredibly brittle.

No, I meant that the same text may have two or three kinds of associated markup runs: syntax, revision control, and highlighting so that we can nicely publish a snippet.

Right, so revision control is one of the types of markup of the text. But what happens when you want revision control of some of the other types of markup as well as for the text. Or when you want highlighting of different revisions of the text.

Revision control is not exactly markup. It is metadata and so is the syntax information. It's derived from text or from something else (comparison of two text by VCS), but not added there manually. Markup is normally metadata added manually, although a markup language can be used for both manual and automatic metadata.

So we have a source and some associated metadata that is crafted manually. I don't see this to differ much from other cases when we need to manually sync two distinct files (e.g. C source and header files), so I don't quite understand the issue, I guess.

I suppose what I'm thinking is that is many cases markup is not metadata. It's semantic and part of the meaning of the text in the same way and to the same extent that punctuation is markup.

Nobody would put punctuation in a metadata file. But why not? It's markup in exactly the same way that a lot of HTML is. In fact formats such as Markdown can be considered extensions of punctuation just as much as they are simplifications of markup.

I remember reading a discussion of word processor document formats back in the 90s, concerning the representation of layout and embedded images and tables. One participant was saying that conceptually the correct way to treat such inclusions is as a character. It's part of the text, not some external thing imposed on the document from outside. For a page layout system such as DTP you might make a different choice because the focus is different, but for word processing I thought this was a very insightful point. A lot of markup is part of the data as well and best treated as such.

I agree, all markup added manually is same as content. If it cannot be derived automatically from the rest of the content, then it is on the same level as content and what we call "markup" becomes simply a special notation to describe these parts of the whole content. A drawing, a formula, a form, and so on. Or even punctuation; it may be important to tell the difference between the period that ends a sentence vs an abbreviated initial :)

I myself don't think it's worth to actually keep content and markup in separate files, so for me the only possible use case was conflicting markup, i.e. markup that exceeds the limitations of our tools, something like two overlapping tags in XML. Yet even this case doesn't require actual physical separation; it can be solved with a different notation or maybe a different tool (like SGML, which, I believe, supported overlapping tags).

Interesting idea. I've never thought of separating the markup and the text into two different files. To me, it suggests a more machine friendly way of keeping track of markup. Directly writing sounds very painful.

Yes, this is looking in the direction of a content API for documents, and is not meant to be human-generated.

Since my project doesn't use this, you could say... it has no Hope.

repository at https://github.com/poef/hope/

This is 6 years old with no progress...

Yup, it was an experiment. There is a follow up, called cobalt: https://github.com/poef/cobalt/

But its not meant as production ready code, instead I'm exploring the concept and what it means for editing, rendering, etc.

What problem does this solve?

Hi, original author here. When I started out I wasn't trying to solve a problem. I was exploring a concept I was curious about. But in the end, we (me and a colleague) ended up building it into our WYSIWYG HTML editor as an alternative to the browsers execCommand editing API. When editing rich text, an API that is build around ranges for everything is a perfect fit.

The editor (https://simplyedit.io/) converts a contenteditable area's content to hope, we apply some markup (annotations) and convert the result back to HTML. I wrote the Hope-HTML conversion routine once, and all the range based manipulations are now one-liners instead of treewalking horror shows.

Reply on news.ycombinator.com