Design of a Vim-like text editor | Hacker News Reader

Design of a Vim-like text editor

lists.suckless.org

234 pointsProfDreamer90 comments

Right now, using vim is better than using this new editor (vis), although vis is close to being substantially better than vim.

I use the suckless window manager (dwm) and terminal (st), so gave vis a try for a couple days. Three vim plugins made me return: git-gutters, you-complete-me (ycm), and clang-format.

It is hard to extend vim's functionality. Its plugin interface has issues. Because of that, ycm and git-gutters can not be used at the same time, and something that ought to be straightforward like exposing clang's autocomplete information requires a considerable engineering effort (ycm) spanning three languages and about as much code as vis is total.

Giving vis good extensibility would be easier than fixed vim's plugin interface (sorry neovim). My personal preference for doing this would be a patch to vis that adds vim like gutters and dropdowns, and then patches that require this one and add specific things like ycm and git-gutters. A reasonable programmer could disagree and prefer out-of-process plugins. Either approach being implemented well would be very exciting.

The reason ycm and vim-gitgutter don't work well together isn't the "plugin interface"; it's the limited nature of the sign column which can only display one sign per line. If both ycm and gitgutter want to display a sign on the same line, only one can succeed.

Edit: oops, ycm doesn't use signs...I was thinking of syntastic. As far as I know ycm and vim-gitgutter should work together.

Here is a bug report discussing why they don't work together: https://github.com/Valloric/YouCompleteMe/issues/812

With all sincerity, these plugins are available in Emacs, along with a Vim emulator called Evil. I used to fight with Vim so much, but for several years have been just using it inside Emacs, a decision I've never regretted.

evil-mode is not without problems either. In particular, it doesn't work very well outside of "regular" code/text buffers. Using it in GDB/scratch/other buffers is very inconvenient.

So if migrating from Vim to Emacs + evil, you still have to learn how vanilla Emacs works. At least this was my experience when trying it out.

You could add a hook to use evil only in certain modes, for instance:

    (add-hook 'prog-mode-hook 'evil-local-mode)

This activates evil only for modes that derive from prog-mode.

Yes, that's the only reasonable way to go.

But you still need to learn how vanilla emacs works.

This piqued my interest:

> handle arbitrary files (this includes large ones, think >100M SQL-dumps)

My old hex editor Hex Fiend was a serious attempt to handle arbitrary-sized files correctly. It's hard! In particular, operations which are usually instantaneous (e.g. Find) now may take a long time: they need progress reporting and cancellation, and ideally should not be modal.

A text editor makes that even harder, because now simple operations like "go to beginning of line" may take a long time if you have to find the beginning of the line. There's probably some conditions that you could impose (e.g. handles large files but not long lines); it would be interesting to see what those are.

> Loading a file from disk is as simple as mmap

This is a sketchy decision. For one thing, it means you cannot work with files larger than maybe 3 GB, or even 3 1 GB files, in a 32 bit process. I'd also be uncomfortable relying on mmap over NFS.

Hex Fiend handled this by not mapping files, but reading them (via pread) on demand.

> Since the buffers are append only and the spans/pieces are never destroyed undo/redo functionality is implemented by swapping the required spans/pieces back in.

Saving is what makes this tricky. Say I open a 10 GB file, delete it all, and save it. Can I now undo that delete? (Hex Fiend initially could not, and users were unhappy). If so, where does that 10 GB data live?

For that matter, how DO files get saved? Say I append 1 byte to the end of a file: is the entire file rewritten? Say I delete 1 byte from the front of the file: does it require twice the disk space to save it?

How about copy and paste? Say I open a 1 GB file, copy it, and paste it into another. Does that 1 GB of data get copied into memory, or is it just referenced in the original file? If it's referenced, what happens if I now edit that original file - does the data get copied at that point?

Anyways this is really tricky (but fun) stuff, and I hope the author succeeds since I do want a fast text editor that can operate on arbitrarily sized files.

>For one thing, it means you cannot work with files larger than maybe 3 GB, or even 3 1 GB files, in a 32 bit process.

Serious question : is it bad to assume a 64-bit capable running environment for developer's computers nowadays? I feel like it's better to run with something simple (and way less error prone) if the cost is some 32-bit issues like stated.

Granted, vim is a special case, since we're running this through ssh on servers or whatnot.

Lots of really good cases mentioned in this post, though, bet you make a good debugger

There's definitely still new Linux systems that are 32 bit (Raspberry Pi, etc) though probably not many as a developer computer. Maybe Crouton on a Chromebook?

Anyways you have a point about the simplicity of mmap. Reading up on the suckless philosophy, it does seem to fit.

I think Crouton installs 64-bit Linux.

> Serious question : is it bad to assume a 64-bit capable running environment for developer's computers nowadays?

There's at least one counter-example, since all my machines are 32bit.

Is there anything in particular about software developers which makes you think they're more likely to grind on the hardware upgrade treadmill? I would have thought the opposite.

The software I use (linux + xmonad + bash + text editor) has far lower requirements than that my family and friends use (various versions of Windows, online video players, antivirus, games, etc.)

I tend to give my hardware more respect than my friends and family too, who are much more likely to drop machines, drop/spill things on to machines, rotate them with hard disks spinning, put stress on hinges or weak points (eg. carrying a laptop by the screen), insert/remove peripherals in mechanically stressful ways, etc.

I'm also more able to investigate and fix problems, rather than eg. buying a new computer when the hard drive gets full. My current machines all use AMD K7 chips (Athlon/Duron), so I've built up a collection of compatible CPUs, RAM, IDE cables/drives etc. from dead machines to fix any issues I get with the remaining ones.

I'm perfectly happy with my hardware, so I see no reason to "upgrade"; rather, I've seen news feeds full of compatibility wars, eg. amd64/IA64, IDEx/eIDE, SATA/PATA, HDMI/Thunderbolt/DisplayPort/whatever, BluRay/HDDVD, etc. none of which I've had to care about.

It's been 11 years since the widespread introduction of 64-bit processors in consumer hardware. I think it's a pretty safe assumption to make that a vast majority of people are using 64-bit-capable machines.

There are advantages to the "hardware upgrade" treadmill if you're on a laptop (not many laptops got 9 hours of battery life in 2003), and if you open an IDE from time to time, or do heavy consumer-facing web development, having a computer from the last decade is useful. Also easier to run a high-res multi-monitor setup if you have some good speed behind that.

It's possible to be good and capable on the machines you're citing, but the overwhelming majority of software developers (even the ones using vim/emacs) are not of the linux + xmonad type.

Um, how on earth do you deal with the 10GB delete? You can't store it in RAM, since there may not be enough, especially if you want 32 bit support. Store it on disk? You can't store it in the original file, since you have no flexibility over its contents after a save: a hex editor must write the bytes asked for, unlike a word processor which controls the file format. Ok, maybe we store it in some other file, call it "temp". That almost works, but what if the point was to save disk space? "Please restart the editor"? And if the power goes out one hour later, the user finds out at the next boot that clearing out a 10 gig file did not save any disk space?

I'm out of ideas. What does Hex Fiend do?

The short answer is that it preserves the undo stack, if it can do so without using more than 16 MB of additional memory. If it exceeds that threshold it drops the stack. (Hey, I didn't claim to have solved it well!)

The longer answer: when a file is going to be saved, HF identifies the ranges of the file that will be modified, and then identifies every other "byte array" (what vis calls a "piece chain") that references one of those file ranges. This includes data that may have been copied and pasted from the file, as well as the undo stack.

It then attempts to "break dependencies" on the about-to-be-overwritten ranges in the file, by copying the referenced bytes into memory first. But if that copying would exceed that 16MB threshold it gives up. Giving up in the case of the undo stack means silently dropping it, and in the case of pasting into another document, it prompts the user: http://i.imgur.com/6Vq7VyQ.png

The guiding UI principle is to avoid surprise allocations of large amounts of memory or disk space.

Interesting, I've always wondered how hex editors did it.

You may not have solved it perfectly, but better than a version of Excel I once used. Every time you save, it drops the undo stack. No matter what. I think the latest versions have fixed this by now.

You mentioned Hex Fiend; hed is also a decent hex editor for large files. I've edited 5G files in it.

Edit: Ah, just noticed you're the author. Hi.

"... I do want a fast text editor that can operate on arbitrarily sized files."

Ever try radare (now radare2)?

I'm currently building a similar project in Rust called iota: https://github.com/gchp/iota

I don't intend it to be a vim clone, however I'm currently adding some features which I've borrowed from vim. The main one being modal editing. Its much earlier on than this project, though.

As a side note, building a text editor is great fun, one of the most interesting projects I've worked on!

It looks cool, and I love Rust.

A question, which are your frustrations with other text editors? Can it really be the case that you can't find an existing text editor that you like? :)

Another part of it which I didn't mention is that I've always wanted to build a text editor, they've always intrigued me. So this was me scratching the itch if you like. I also used it as an opportunity to learn rust, which I really like, incidentally.

To answer your question, I actually hadn't found one that suited me, fully. There are many that I like, but non which did it for me. It sounds a bit weird, but it needs to feel right, which none of then did for me. So that's why I started it. It seems others are interested in it too which is cool!

I was curious if you could describe the specific frustrations, and which of these you can fix.

Right, here goes...

- ability to handle large files without freezing or without a noticible drop in performance

- make all aspects of the editor scriptable. Emacs does this well, but I really don't like elisp. It will also be possible to script it in a number of languages, not limited to a single one.

- fast, like really fast. My Emacs config takes ten seconds to load, using IDEs takes much more than that to open. I want to make use of modern processor capabilities and minimise load time. Also to parallelise as much of the editing functionality as possible, so the user is never blocked by a task or operation.

- cross platform, easily installable. Some editors do this well, others don't. With iota you'll be able yo just download a binary and run it. No installation process, no dependencies. Same process to install on all machines.

I guess those are the main ones. Am on mobile right now so its hard to give more detail at the moment. I'll try update the motivations part of the readme with information like this today.

I'm gonna need to read that source code; I've been an Emacs user for a long time now (and vim before that), and although I don't feel the need to switch (yet), I'm always on the lookout for new editors. A recent editor that I found is OpenBSD's mg(1), a super lightweight Emacs clone. If it were not for its lack of support for UTF-8 (I write a lot in French), I would use it a lot more than I do now. Let's hope iota can become that editor!

I'm always interested in the suckless projects. I like emacs and vim, too, but it's interesting to see projects made which can just grab the fork, a sharp knife, and a good pan and make do instead of necessarily including the kitchen sink, if I may get the analogies out.

Also, no tests? Phew!

I've been using Vim for over a decade. I love it. But I feel more and more anachronistic every time I have to hunt down some Vim plug-in for whatever IDE I'm setting up.

I'd love to see a fresh take on hyper-efficient text editing in modern GUI environments.

Well, with Neovim, we'll have a legit fully-featured plugin for any and all IDEs that do plugins. I sincerely can't wait to have Neovim in IntelliJ/RubyMine/CursiveClojure.

IF the project survives.

It will, definitely. I mean, given the amount of work that has been put in and the amount of people that are waiting for the first release, it would be a huge mistake to discontinue it. To make vim a "modern" editor, there are a LOT of things that need to change, and neovim has got most, if not all, right.

It may be still too early to tell but it seems there is enough interest to keep it alive. It's been almost a year since the project started and it has over 120 contributors, over 2500 commits and doesn't seem very top loaded on the project owner (who is responsible for only 18% of the commits).

Which was one of the core goals. Spread the load, avoid single points of failure, create a vibrant community of contributors, allow people in.

I wasn't trying to be negative. There have been a lot of editor projects come and go is all. I like the idea of neovim. I just with Bram was more flexible so that neovim wasn't necessary.

The article mentions the Project Oberon book and the piece list algorithm, which goes back to Xerox PARC's Bravo editor. Coincidentally I came across this paper recently about different data structures for text editors, which covers the piece list algorithm but also several others: "Data Structures for Text Sequences" (1998), Charles Crowley (https://www.cs.unm.edu/~crowley/papers/sds.pdf).

There is also this old book about Emacs that describes data structures for text editing: http://www.finseth.com/craft/#c6

So weird to see this at #1 since I was researching this exact topic last night. My idea is a fully scriptable (Racket or CL) editor with modern package management system. A tiny C core would handle rendering (curses, etc.). Very cool write up though!

As an alternative to ncurses, I encourage you to look at termbox[1]. It's small, minimal, and very robust in my experience. (As a side note, the author also wrote a text editor in Go that uses it.)

[1] https://github.com/nsf/termbox

I'll check it out. Thanks!

A fully-scriptable editor (not in CL but emacs-lisp is pretty close) with a modern package management system is pretty much just emacs+melpa

Instructions to compile the source on Freebsd:

in config.mk, change the line :

CFLAGS += -std=c99 -Os ${INCS} -DVERSION=\"${VERSION}\" -DNDEBUG -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700

to:

CFLAGS += -std=c99 -Os ${INCS} -DVERSION=\"${VERSION}\" -DNDEBUG

The flags (the removed ones), trigger the "__BSD_VISIBLE 0" macro that makes the signal "SIGWINCH" unavailable, as they are used in vis.c

The source code of vis looks great, and the executable takes about 100KiB (x86-64). The editor does not behave exactly like vi/vim, but it is quite acceptable.

I like a lot the suckless manifesto, recommended projects, etc. http://suckless.org/

How about an embedded Lisp interpreter?

Sort of a lisp 1.5 with scheme style syntax. They could opt in within the rc file to allow users not interested to ignore it. Would be closer to the original vi then.

At one point, REPL first editors (ipython notebooks, etc) will meet edition based IDE (them) in an essential cosmic spark and nothing will ever be the same ever again.

Isn't that LightTable? I think they discovered actually everything would remain largely the same.

Good point, how could I forgot LT...

IntelliJ sort of has this for Scala. Scala Worksheets. They are more useful than a REPL, but not quite to iPython in terms of features (mainly because they aren't browser based). But they also have some advantages over iPython (like integrating many of the IDE features).

Yeah everybody is approaching the solution from their angle, I just wondered if there was a center.

PyCharm now offers the ability to open iPython notebooks inside itself - which gives you the nice IDE goodies, plus the notebook goodies.

The downside of course is that Python is very dynamic - so IDEs are of relatively limited use.

The downside of course is that Python is very dynamic - so IDEs are of relatively limited use.

I am surprised to hear you say that with "PyCharm" being the first word in your comment. We have had very good experiences with PyCharm + reStructuredText annotations to help it when the return type or arg type is not inferable. Just like its IJ friend, PyCharm has caught quite a few bugs and helps the team edit Python and not text.

PyCharm's usage is not optional on my team; we tried the "use whatever text editor you want" approach and it turns out that people are not as good at memorizing the type signatures of a complicated codebase as computers are.

Well, there is a vim mode for emacs.

I grew up with the one which didn't have the letter 'm' in it. The lisp/scheme concept could be seen as an alternative to vim/script for embedding/extending where the opt-in was meant for posix ex/regex programmers as an alternative to non vi modality with recursive expression via either McCarthy or Kleene closures and styles of programming with or without /term(cap|info)/ databases.

That's probably not in line with the suckless way.

I curious their philosophy (less is more?). I used dwm for a bit and still use a personally hacked inspired from dwm console derived dvtm, also by Marc iirc. If their going towards UNIX philosophy concepts it will be nice to hack a post config to extend it with personal my schemes and nda|dfa regex engines. This looks like a nice project. Looking forward to watching it.

They actually have a philosophy page which explains it: http://suckless.org/philosophy

It basically boils down to minimalism (in general) with an emphasis on minimalist code instead of minimalist UX/UI (whenever the two conflict).

I have come to favour the Acme[1] approach of having a virtual file system to interact with the editor, so that you can write compatible scripts in any language!

[1] http://research.swtch.com/acme

That sounds like a mighty good idea to me. Not a fan of the Acme interface, but the idea of exposing the editor API through the filesystem means that there is no war like Emacs Lisp vs Scheme in the Emacs world (an no people suggesting that we should switch to JavaScript): people will write plugins in any language they damn please, and they can even follow trends (Go this year, Rust next year, and OCaml from 2017 forth).

IIRC neovim exposes it via a socket which allows similar behavior.

I would like to have a vim-clone, that also can handle embedded images.

Just a wish.

Very interesting. I also found quiet a lot of inspiration from reading about the Oberon system. This author was inspired by the text editor, I was more interested in the language. The author uses 'peice chains' as opposed to a gap buffer which was new to me.

I'm sure many will say, why another editor? I say why not? Looks like a fun project.

Great idea! I wonder why you aren't using github (or bitbucket, etc) for hosting the project. I personally think that early projects can benefit from tools like that, since people can follow the progress (issues, bugfixes, etc) and access the code without cloning the repo.

GitHub etc does not really fit into suckless' minimalist philosophy

It actually is on GitHub[0]. I'm not sure why it wasn't mentioned in the mailing list.

[0]: https://github.com/martanne/vis

That was a very interesting read. I might want to implement some of those things in my text editing component.

I have one question. How would they implement "go to line number" functionality? How fast would it be?

At the moment only the line number corresponding to the start of the viewable area is cached. Further requests are served based on that. Since the underlying pieces/buffers remain unchanged, one could also maintain a more complex data structure such as a balanced binary search tree.

What about uncached? Imagine the user opens a file and immediately issues a command "go to line 12345". Or is go to line not implemented at this time?

Then the underlying file is scanned. The performance of this depends on the number of non contiguous changes made to the document. At load time i.e. with no changes this will be relatively fast since it is a single memory mapped piece/buffer.

Further request are then either served from line 12345 or the start of the file, depending on which is nearer.

How do I submit a patch? Needs a couple of tiny mods to compile on OSX.

I would imagine you can submit a PR at https://github.com/martanne/vis/pulls. He says "As always, comments and patches welcome!" in the README.

Edit: Maybe not, the mailing list mail says "git clone git://repo.or.cz/vis.git" instead. I'm not sure where the canonical source lives.

Author here, both by email or by using the aforementioned github repository is fine.

I think that explaining core of the editor to the people is whole point of how to get people to use/extend/modify it. And this post really hit the nail on the head.

I didn't see plug-in support in that list. Excellent, first class plug-in support is an absolute must have for a good modern text editor.

The piece table/chain description makes me think to an interactive xdelta (which is for instance used in Git).

And neovim ??

You don't have to use posix to suck less you know, or do you?

Most of the "main goals" are supported by ... Eclipse!

yep, at the expense of all your RAM.

And cpu, and battery.

And sanity. ;)

Honestly IDEs in general are the most complex pieces of software that I frequently encounter. Most of that complexity comes in the form of features that I either never intend on using or don't intend on using at the time, which makes that complexity really frustrating. Complexity where it is necessary is fine of course, but I feel like IDEs like Eclipse frequently cross that line.

Eclipse isn't really just an IDE, it's almost an OS in its scope and complexity. I agree it's way too complicated and bloated, and worse: the rewards aren't there like they are in IntelliJ or Visual Studio. It's not as polished and not as easy to set up.

The whole point of integrated software is to be able to do everything in one place: debug interactive in editor, hover to inspect values, edit the code and have the ide recompile and reload the changed program.

The IDE that does that likely does other things, but having unused features should neither add complexity nor incur performance costs!

The complexity of setting up a dev env like the above in emacs or vim is absolutely staggering, but the complexity lies in the configuration and setup, not in the finished env, that's the difference. Personally my tolerance for "configuring" my tools (meaning gluing, telling my editor and profiler where my debugger is) is nearly zero.

The type of complexity that bothers me is complexity that I have to wade through on a day-to-day basis when trying to find features that I do want to use. For example, lots of menus, each with tons of menu entries and submenus. That sort of complexity won't (or at least shouldn't...) impact performance in a meaningful way, but it exacts a psychological toll on me.

As you say, Vim's complexity is different. It has a lot of nonsense built in, but unless you go looking for it you will probably not become aware of it. It also can be complex to set up correctly like you mention, but that sort of complexity bothers me much less. That's more of a "once and done" thing, which I find much more tolerable.

My tolerance for configuring software is actually very nearly pegged at zero, unless I believe that the configuration I am creating will still be useful for at least 10 years. This rules out configuring just about anything for me, with very few exceptions. My current significant configurations are for vim (the emergence of neovim gives me some concern) and zsh (aliases and basic keybindings only. I am confident that any shell I'll be using in the future will allow aliases). My window manager, browsers, etc all use their default configurations (I got burned by Awesome WM changing their configuration scheme a few years ago).

Configuring Eclipse to be tolerable is probably possible, but I have zero confidence that any such configuration could last even two or three years.

Consider the following: multi-tools, mess kits, and sporks exist, yet many people prefer using the specialized equivalents.

The analogy breaks very fast, as people who prefer the specialized equivalents don't go sticking tons of smaller pieces together to try to make them resemble a multi-tool functionality, whereas Vim and Emacs users do add tons of plugins to get IDE like behavior.

FWIW, I think the aversion to IDEs is bogus -- unless it comes from people who don't install plugins on their Vim/Emacs. If they do, then their problem is not IDEs per se, but how popular IDEs are implemented.

The gist of an IDE is "having all the functionality in one place": edit, search, refactor, browse code, scm, REPL, build, etc. People loading Vim/Emacs with plugin try to build the same thing essentially.

The problem with IDEs is not their "IDE-ness" but: 1) they lack a decent, fast, editor component. 2) they are slow to load and memory hogs.

Nothing of the two is inevitable. Just an issue with current implementations. For (1) we can image an ide with a full Vim component, for example, as it's editor part. For (2), the problem is that both 2 popular IDEs (Eclipse, Idea) are built with either Java (which is crappy for GUIs), or have a messed up GUI like Visual Studio.

Why not write a rich-text editor instead? Or a webbrowser?

Seriously. They seem like more interesting endeavours to me. Especially if you document every step of the way. In fact, you could write a book about it.

Writing from scratch is the way of suckless. :)

http://surf.suckless.org/ is already a thing.

Never noticed the logo, elegantly regular and cryptic.

If it seems more interesting to you, then why don't you do it?

Reply on news.ycombinator.com