Hacker News Reader

Comment by zokier | Hacker News Reader

The key observation to me here is that the character cell grid abstraction that terminals are built upon is poor fit for modern general-purpose multi-lingual text rendering. Works well enough with plain ASCII but anything beyond that gets increasingly messy

View on HN

I am working on a re-write (https://github.com/xtermjs/xterm.js/issues/4800) of the xterm.js "BufferLine" data structures that (among other benefits) could potentially support variable-width fonts in the terminal. Instead of a 2-dimensional grid of character cells, the primary data structure will be a list of logical lines, each consisting of characters and attributes.

A rather modest enhancement to the vt100/xterm escape sequences would be a mode where screen addressing would be in terms of logical characters and logical lines. The application should not need to know the width of characters or how lines are broken into screen rows. This makes use of variable-width fonts practical - which is necessary for decent rendering of many scripts.

DomTerm's display engine (https://domterm.org) does support lines containing a mix of monospace and variable width characters, and know how to wrap such lines. I'm working on long-term replacing DomTerm's native display engine with xterm.js (for speed and other reasons) which requires making xterm.js more versatile.

It fails miserably even with ASCII, works only with a monospace variety of it, and non-monospace isn't modern...

That’s an interesting thought, IMO. These flexible width fonts are often thought of as localization-assistance features, to help fit other languages in the terminal world. But English also had to be modified to fit into fixed-width fonts. I wonder how the discussion would change if this technology had been popularized originally in another country.

I would argue that strings as an array of individually addressable 'characters' is a bad fit for modern multilingual text handling.

Why? How does it fail?

In utf-8, bytes (uint8_t) may not represent a whole "code point". A code point being an individually meaningful element in utf-8 like a space an 'e' or a modifier code point like an accent or a ZWJ. Most utf-8 libraries will let you address individual code points but it might still garble the text if you split between an 'e' and a '`'. To prevent this, splitting should be done in between graphemes (sequences of code points that render like a single unit*). And even graphemes have their problems.

Very interesting blog post about graphemes that parallels my experience writing a terminal text editor: https://mitchellh.com/writing/grapheme-clusters-in-terminals

* Event that is not a proper description of a grapheme

And don't even get me started on regexes

Yes, I understand a little about Unicode in this kind of problem, but a code point is an individual logical item even if it is composed of multiple bytes; being a kind of 'string' in itself. I should have asked more carefully, what would be a better system in your view?

Thanks for the link, will check it out after Christmas.

I personally believe that Swift's strings where graphemes are the smallest indexable unit are the gold standard for writing logic that might truncate multilingual text. It's still not perfect though, they add overhead and updates to Unicode might change behaviour so there's that but it should handle most cases gracefully.

Reply on news.ycombinator.com