Hacker News Reader

And, for indic languages, there is orthographic syllable pair encoding, which reduces the amount of space taken by indic characters by encoding the most common letter pairs as a single byte. This was presented i think last year, at the URTC in MIT.

(Disclosure, it was published by my friend)

View on HN

Isn't that a specialization of BPE for specific distribution? BPE is essentially a data compression scheme that constructs a straight-line grammar (SLG) that can be directly used as a model encoding, and it is hardly surprising that any other SLG would work well for some other inputs.

It is less byte pair at that point (as the indic languages generally take up 2-3 bytes per codepoint.)

Yeah, of course, but I thought BPE should pick it up anyway if your training data is primarily written in Indic scripts.

> for indic languages

amateur here - but "Indic languages" are quite different. Maybe you mean Hindi and that group ?

Presumably, they’re referring to the Indic scripts (also known as Brahmic scripts): https://en.wikipedia.org/wiki/Brahmic_scripts

I am! I mean to say the languages usually represented in brahmic scripts (With most of the brahmic scripts being phonetic, one can write multiple languages in the same script in a sensible manner)

Reply on news.ycombinator.com