Hacker News Reader

Comment by segfaultbuserr | Hacker News Reader

Indeed, GNU Unifont looks awful to many people, and even we only talk about bitmap fonts, Unifont is not the best-looking one. However, it's a font designed for practicability, not aesthetics. It's one of those few fonts that covers the entire Basic Multilingual Plane, and is actively maintained. It's a perfect choice as a fallback font.

And in fact, the font actually looks pretty good under a CLI console. I once tried a Linux kernel patch that embedded the 10 MiB+ Unifont to the native Linux VT console, and I got a perfect multilingual console (without using a 3rd-party framebuffer-based or KMS-based console), including all the CJK characters. It's also a good choice for a dot-matrix LCD/LED display.

View on HN

> I got a perfect multilingual console

No, you didn't. If it was using Unifont, there are many scripts and languages that it was unable to display correctly because they require "shaping" that Unifont cannot support; it lacks the necessary glyphs, let alone the OpenType (or equivalent) tables to control the rendering.

Displaying one nominal glyph per Unicode character does not result in readable text for many languages. Just because it seems adequate for Western alphabets or for CJK doesn't mean it is sufficient for true multilingual support.

> Displaying one nominal glyph per Unicode character does not result in readable text for many languages

Completely unreadable, even to people with a basic understanding of the inner workings of Unicode and some practice with the font, or just not matching the norms of correctness?

Decipherable but clearly not correct would mark the sweet spot between being just as bad as placeholders and being so good that it undermines efforts for having a proper font for a given language.

Unreadable.

Arabic has a strong difference between letters of the same word, and different words. There is no exact equivalent in English...

b&u&t&i&t&m&i&g&h&t&b&e&s&o&m&e&t&h&i&n&g&l&i&k&e&t&h&i&s

If that example is representative, I'd definitely put it in the "readable" bin. Leaps and bounds from what you'd want to read, but infinitely more useful than a sequence of identical placeholders. Even just being able to tell wether two strings might be identical or not would be an improvement over blank placeholders.

There's also complex text layout (vowel-sign reordering) for Indic scripts, where some (not all) vowel signs require repositioning from phonetic order to visual order. Adding just that aspect to your example (without other complications like positioning of matras or conjunct consonant glyphs) makes it something closer to:

b&t&u&t&i&m&g&h&t&i&b&s&e&m&o&t&e&h&n&g&i&l&k&i&t&e&h&s&i

The degree of badness varies among languages/scripts. In some cases, yes, most readers would be able to decipher the intended content. In others, no, it really would be unreadable, except to those geeky enough to also comfortably "decipher" things like \x74\x68\x69\x73. Would you be happy routinely reading your terminal text in that form?

> it really would be unreadable, except to those geeky enough to also comfortably "decipher" things like \x74\x68\x69\x73

Can you give an example?

Try something in Devanagari script such as "क्षत्रिय" and see how it looks with Unifont. The results may vary, depending whether some kind of shaping engine (like Uniscribe, Pango, HarfBuzz, etc) gets involved in the process, but I'm confident it will be far from correct, and most readers would struggle to recognise it.

Try online here: https://fontlibrary.org/en/font/gnu-unifont

It does look different (likely incorrect), but is definitely better than blank boxes and question marks.

> definitely better than blank boxes and question marks

I disagree, I think it's better for it to be apparent when text is wrong or unreadable, even to people not literate in the language.

I wondered if the problem had to do with Unifont being monospaced but I pasted "क्षत्रिय" in on the Inconsolata page and it appears correct (i.e. the lines correspond to how they look in the font on this page).

https://fontlibrary.org/en/font/inconsolata

Inconsolata doesn't support devanagari at all, so when you try that, you're seeing whatever fallback font your browser chooses.

(IMO, the fact that fontlibrary.org doesn't indicate this is a serious defect in its presentation.)

Good point. fontlibrary.org lists each font's language support in the sidebar but since the page emphasizes the display of the font, it would be better if they prevented such fallback.

Its idea of "language support" is also less than ideal, as it just considers the character repertoire and not whether the font can actually render the characters in a typographically correct way. Thus, it claims Unifont has "support" for various South and South-East Asian languages where exactly this issue applies: the text is simply wrong when displayed linearly using a 1:1 character/glyph mapping.

How is gibberish or incorrect better than blank boxes and question marks? At least with blank boxes and question marks it is obvious that there is a problem. Gibberish would be useless to a reader who knows the script, and might cause someone (such as a developer) that doesn't know the script to think that there is no problem.

Exactly! In the worst case, some hapless person might be using Google Translate to show a tattoo artist what he wants to be memorialized on his chest. With this font. Perhaps with the blank box he'd know something was wrong.

On the other hand, if you're taking something from Google Translate and tattooing it on your chest, whether or not GNU Unifont is 100% accurate in its glyph rendering is the least of your problems, surely.

I think your example can be illustrated by adding spaces between the characters like so: क्षत्रिय vs क् ष त्रि य (note how the first "glyph" decomposed into two drastically different characters)

Actually, your "decomposed" version still has clusters that are different from the individual characters making them up. There's क्, which is क plus ् (OK, that one's easy); more significantly, you've left त्रि as a single unit whereas it is made up of त + ् + र + ि.

So "spelled-out", क्षत्रिय is made up of क ् ष त ् र ि य . I don't know if Unifont tries to render the combining characters as zero-width overstrikes, or what, but much more than that is needed for a readable result.

That looks plenty readable. Ugly as hell, sure, but not any worse then 3N6|_15H \/\/R1773N 45 1337 5P34|<, for example. It's a vast improvement over [0915][094D][0937][0924][094D][0930][093F][092F].

The problem is that the vowel ` ि` is in a misleading position in the decomposed form. It implies a spelling of kshatrayi instead of kshatriya, unless you happen to know that Unicode requires it to be there and the font engine is supposed to move it to its correct place in the ligature.

(This comment is a good equivalent of the problem in English. https://news.ycombinator.com/item?id=19575942 )

Wow thanks for spelling that out. That seems like a challenging script!

To add to this, here's a few example complex text layout languages in no particular order: Thai, Khmer, Arabic, Hindi, Hebrew, Burmese, Tibetan, Punjabi, Sanskrit, and many many others.

Many millions (billions?) of people won't be able to use your terminal in their language.

Also ancient language like Egyptian Hieroglyphics.

I use GNU Unifont, with a user-space virtual terminal. Some of GNU Unifont has looked far from "pretty good". I overlay K16-1990, 9x15, and Ubuntu Monospace on top of it for better-appearing characters.

* http://jdebp.uk./Softwares/nosh/user-vt-screenshots.html#Uni...

* http://jdebp.uk./Softwares/nosh/guide/terminal-resources.htm...

Your multilingual console is really far from "perfect". Aside from the aesthetics of GNU Unifont, you do not have good multilingual input, which you have completely left out. You don't have an ISO/IEC 9995 common secondary group. You do not have CJKV input methods.

You still have problems with modifier keys becoming stuck "on" because of VT switching races.

* https://unix.stackexchange.com/a/494685/5132

And you also still have ISO/IEC 2022 8-bit character set switching for messing things up. (-:

* https://unix.stackexchange.com/a/506140/5132

Because English is already well suited to it, is it incredibly ethnocentric of me to believe that instead of trying to shoehorn the complexities and eccentricities of existing written text into the digital medium we should have instead adapted the writing systems to the medium so they'd be simpler to both represent and input? Because every time I look at what it takes to properly support unicode I'm pretty damn thankful ASCII is more than sufficient to represent my language.

The point of our machines is to help humans, not the other way around.

I was saddened when Spain changed Spanish sorting to make life easier for PC programmers in the 90s. Within only a couple of years that effort became pointless, but was now encoded in law.

> The point of our machines is to help humans, not the other way around.

While this is true, significant amounts of friction are caused by being unwilling to adapt to the medium's nature. What would writing look like if we had tried to encode sound waves directly onto paper? Instead we adapted our communication to the medium.

With such a pliable device I consider every adaptation a person must make to be a deficiency. Perhaps a not-yet-addressable deficiency, but a negative nonetheless.

What did they change?

I guess this refers to this:

”Spanish treated (until 1994) "CH" and "LL" as single letters, giving an ordering of cinco, credo, chispa and lomo, luz, llama. This is not true any more since in 1994 the RAE adopted the more conventional usage, and now LL is collated between LK and LM, and CH between CG and CI. The six characters with diacritics Á, É, Í, Ó, Ú, Ü are treated as the original letters A, E, I, O, U, for example: radio, ráfaga, rana, rápido, rastrillo. The only Spanish-specific collating question is Ñ (eñe) as a different letter collated after N.”

(Copied from https://en.wikipedia.org/wiki/Alphabetical_order#Language-sp...)

Indeed, and the justification at the time was because contemporary software got it wrong. I remember it well.

It's not ethnocentric, it's programmer-centric. If you consider supporting Unicode hard, imagine the effort required to get everyone to learn a new writing system and transliterate all existing works into it. At least Unicode support only requires a handful of specialists to create the necessary tooling, rather than some global uprooting of existing traditions.

I think you over estimate the cognitive effort required. Just look at how quickly emoji were adopted into every-day communication. Maybe you can't transliterate Shakespeare into emoji very effectively, but some meaning is lost just reading it in modern English anyway and we could reserve the more complex systems for this kind of preservation.

Emoji work because they're optional. If people were forced to communicate exclusively in emoji to use some particular software, most potential users aren't going to bother with the effort and will choose a competitor instead.

I think I've expressed my point unclearly: emoji are an adaptation of our communication patterns to the medium. Real time textural communication could not express things we were used to expressing in other real time media via tone of voice or body language, so we adapted how we communicate those things to the media. Indeed writing itself is an adaptation of language to a new medium.

I suggested some falsehoods that programmers believe about telephones a while back (https://news.ycombinator.com/item?id=19215636). There is a case for falsehoods that programmers believe about ASCII, starting with:

* It's 8-bit. (It's 7-bit.)

* It is sufficient for English. (Ð ð Þ þ)

* It is sufficient for Modern English. (zoölogy coöperate £ née resumé)

* It at least has everything that one could type on a typewriter. (½ ¼ ¢ , and that's just starting with some contemporary 20th century U.S. typewriters from IBM such as the Selectric)

* It is sufficient for the U.S., at least. (Not for the U.S. Library of Congress in 1969 it wasn't. It needed 174 characters for catalogue cards, per https://link.springer.com/article/10.1007%2FBF02404378 .)

* It was used by Multics. (Multics used a modified version, padded out with leading zero bits and replacing DEL with PAD. See http://web.mit.edu/saltzer/www/publications/multics/bc-2-01.... .)

* It does not have broken vertical bar. (It was in the 1968 standard at 124, a broken bar because the PL/I language people did not want their unbroken vertical bar to be in the then "national variant" range. https://groups.google.com/d/msg/comp.infosystems.www.authori... http://jkorpela.fi/latin1/ascii-hist.html#7C )

* It does have broken vertical bar. (It was replaced in the 1977 standard by an unbroken vertical bar at 124. Then some subsequent 8-bit character sets in the 1980s re-introduced a broken vertical bar in a second position, with much ensuing hilarity.)

* It did not have arrows. (In the original standard ↑ and ← were where ^ and _ now are, for some consequences of which see https://retrocomputing.stackexchange.com/a/9201/1932 .)

* It is the same as ECMA-6. (ECMA-6:1991 allows national-language variants in several code positions. The standard ASCII characters in those positions are merely the "International Reference Version" and one possibility from amongst several in ECMA-6. See https://www.ecma-international.org/publications/standards/Ec... .)

So in actual fact English was adapted to the medium, albeit minimally.

The end result of adapting our language to the medium is to lose in expressibility and gain in portability. Forcing technology to catch up is a more coherent option in the long run

That's the exact same trade off we made for writing in the first place. Yet here we are still conversing with text despite the technology to record and send each other video having been widely available for a decade or more.

That's specious reasoning.

If you mean, body language is the main or majority channel for communication, well, how much body lanuage matters to communication is contested and differs based on context anyway (see: https://www.psychologytoday.com/us/blog/beyond-words/201109/...).

If you mean, there is a loss of expressibility going from the spoken word to the written word. Is there? You would have to prove that that loss is as significant as the symbolic/representational destruction you are advocating. Much of the imperfect synonyms we have developed in english are used to signal tone and intent -- things that are usually signalled in body language. So, just because we are using the written word does not mean we cannot express those things. You have something in a larger space, you can represent it in a smaller space using longer and more complex series of glyphs.

> Yet here we are still conversing with text despite the technology to record and send each other video having been widely available for a decade or more.

Yes, for the very reason that it erases the extra data. you not being able to see me means that you have to go on my words alone, rather than making extra-contextual judgements about me as a whole based on my physical appearance. That seems only to cement my point, though.

Several of these points can be boiled down to "When people say ASCII, they really mean ISO 8859-1 or CP 1252".

>including all the CJK characters.

Is there a version of GNU Unifont for Japanese kanji? I could not find any. A few Japanese kanji share the same code-point as Chinese characters despite not being the same.

https://i.imgur.com/DYqJTvN.png

https://en.wikipedia.org/wiki/Han_unification

It looks like your "perfect multilingual console" here rubbed people the wrong way—the Arabic ligatures are a straightforward counterexample. You got something that's helpful for working in a multilingual environment if you don't intend to read text in other languages but only to refer to it or recognize what it is. So I'd say that in a few cases that don't involve reading and writing or typography, this is potentially more useful, but it's important to be able to draw the distinction!

For example, it's helpful for being able to say "the Arabic name for the Arabic language is spelled ة-ي-ب-ر-ع-ل-ا, alif-lam-ayn-ray-baa-yaa-taa marbuta". But if you want to have an Arabic speaker read it as a word, it ought to be written "العربية" (RTL with ligatures). Your terminal environment is good for the former but not the latter.

> Indeed, GNU Unifont looks awful to many people, and even we only talk about bitmap fonts, Unifont is not the best-looking one.

And this is subjective. I have used Unifont in xterms for years now, and it's very readable and unambiguous to me.