Hacker News Reader

Comment by jll29 | Hacker News Reader

Apart from the learning experience, there is still value in a small-sized, fast TUI text editor for Linux.

However, my two suggestions would be:

- add Unicode support (it's the 21st century, so ∑, ®, etc. should finally work); this is not easy, but the earlier you do it the less dramatic the changes will be (check out https://github.com/unicode-org/icu/tree/main/icu4c);

- don't allocate the lines individually (as you do now with malloc, having looked at the code); at least use an arena allocator, i.e. allocate larger chunks of memory and the provide your own alloc_line function that uses the larger chunks (called arenas) piecemeal. This will speed things up and reduce fragmentation. A more advanced approach would be not to use per-line buffers but to switch to rope data structures e.g. https://github.com/josephg/librope).

View on HN

As an alternative to ICU, there is suckless's libgrapheme (https://libs.suckless.org/libgrapheme/) which is more than a 100x smaller and provides full Unicode compatibility.

Heh, I maintain the MacPorts port for this lib! Like all suckless projects, it’s written in very simple and portable C which makes it a breeze to package.

https://ports.macports.org/port/libgrapheme/

Thanks for your work on packaging my library. Please let me know if I can make the process simpler for you; I take great care to make packaging as simple as possible for the packagers. Likewise I have no sympathy for those writing software that is almost deliberately hard to package.

Thank you for your work on the library! It really couldn’t be easier to package but thank you <3

Are there any similar options that handle collation and normalisation? No small Unicode library seems to implement them, unfortunately.

Author of libgrapheme here: Both collation and normalisation are non-trivial and have many gotchas thanks to the way the Unicode consortium likes to write their specifications. I sometimes get the feeling that they don't even care about implementers and just document what is done in the reference implementation ICU.

The only sensible normalisation one can implement is the full decomposition (NFD), and maybe the full composition (NFC). You rely on the full decomposition if you want to collate correctly, which is a problem because the amount of memory needed to store the decomposition is unbounded in general. I don't want to make the libgrapheme users jump through hoops, and I also don't want to do any memory allocations in libgrapheme either.

There is an idea floating in my head on how to solve this, but I'm currently busy finalising Unicode 15.1 support (Unicode 16.0, released on the 10th, will be trivial to upgrade to) and releasing my already fully-compliant implementation of the Unicode bidirectional algorithm.

I see, thanks for replying. I agree, Unicode specs are hell to work with (I tried doing a auto-codegen thing based on them and just gave up due to the size of tables generated and the seemingly-arbitrary edge cases). libgrapheme looks pretty good otherwise, I'll keep an eye on it for whenever I have to wrangle with Unicode on a low level again (hopefully not for a long time).

Interesting. Cool library, I'll check it out. :) Thanks for sharing

Thanks for recommending libgrapheme. I am honoured, being the author of this library.

> This will speed things up and reduce fragmentation

Does this really make much of a practical difference these last decades? I wrote a text editor back in the late 90s in DOS using DJGPP with per-line allocation and it worked fine on the Pentium MMX i had at the time (and my approach to optimization at that time was be ignorant of the concept :-P), i'm not sure it'd really make much of a difference on any desktop or laptop CPU released past 2000 - at least as far as users editing text files are concerned (might make a difference if you try to open a multiGB file with barely a line break and accidentally press a letter :-P).

No, it won't. You might want an arena for other reasons (better reasoning about lifetimes, preventing nest of pointers, etc.), but an array of line array approach to text editors is totally valid (except in the case of millions of extremely small lines).

I do from time to time open files of millions of small lines. To this day vim handles it better than emacs. These would be log files.

Log files have lines that are long enough, I was thinking more of one/two-character long lines (such that the overhead of malloc bookkeeping would be more than the line size in memory). Regardless, Emacs also stores the data in a single array with a gap in middle (and memcpy is relatively cheap), so what slows it down is probably not the choice of data structure but everything else built around it.

Yeah I have very high limits for paren matching and syntax highlighting so probably those scanners.

I agree. Unicode support is top priority. Thank you for sharing those links.

Well yeah, Seph is right as usual I’ve been updating my MLIR generation code to use Twine lately: https://llvm.org/doxygen/classllvm_1_1Twine.html#details

LLVM has been using rope data structures for some time. There is more discussion about it here:

https://lists.llvm.org/pipermail/llvm-dev/2011-July/041639.h...

In my situation when parsing lines of code Twine stores the file path and concats the line number and column when getting the location within a file. It does this a lot so using Twine there really helps. The rope data structure can be used in a number of ways.

Reply on news.ycombinator.com