Xi editor: A modern editor with a backend written in Rust
github.com
github.com
* Separation into front-end and back-end: NeoVIM does this. The UI uses the same interface a plugin does.
* Asynchronous operations: NeoVIM uses libuv to do this (same asynchronous C++ lib as NodeJS).
* Plug-ins over scripting: Yup, NeoVIM does this, as well as dropping vimscript (although it will compile vimscript into Lua for you).
* JSON: NeoVIM uses msgpack, which is significantly less overhead than JSON, but structure compatible.
The two other features listed are Rust and rope-structures, which are implimentation specifics that don't affect users or plugin writers at all.
Edit: The more the merrier! And it's good to see more projects using Rust, but I don't see this as a strong need. See https://github.com/rogual/neovim-dot-app for a nice native Mac UI plugin/client for neovim.
A large part of my motivation for starting this project is to explore how suitable Rust is for this kind of work. End users shouldn't have to care, they should judge it strictly by the quality of what it delivers. But I'm hoping this project will be interesting to Rust programmers for a number of reasons.
Well, when it comes to this new editor, doesn't it remain _even more_ to be seen?
At least NeoVIM has all the basics covered already.
They don't need it, but GUI can provide nice things consoles can't. For example real autocompletion drop-downs. Or see-through file outline/minimap like the one in sublime text. (https://2.bp.blogspot.com/-Co2awzuaA_s/TqQCZ1wwFLI/AAAAAAAAO...) Or better markers for things like region folds, column positions, matching braces, highlights, etc.
That said, I can't let go of modal editing, composeability, and word motions. Here's to hoping sublime implements them and I can relegate Vim/NeoVim to quick fixes in the terminal and Git commit messages.
There's no reason a TUI can't give you "real autocompletion". Vim does pretty well in most cases.
> Or see-through file outline/minimap like the one in sublime text.
I personally don't see the benefit of that, but you're right that you can't see a zoomed-out version of your code in a TUI.
> Or better markers for things like region folds, column positions, matching braces, highlights,
All of those things are supported by Vim and can be done in a TUI. As for "syntax highlighters" that decide to italicise code, I don't really consider that a feature.
As for markers, I know they're supported by Vim. That's why I'm saying better. For example Vim can't do single-pixel column marker (or specifically, create a line between columns). I can do hacks to highlight one column only (100th for example), but then highlight/copy on the terminal is broken, because in reality I just have spaces displayed after my code. Similar thing with region folds - they don't need to take the whole text row and the description of the fold can go off-buffer on a side.
Basically you can do everything with text. You can even print out "this is a placeholder for an image of a sad green frog looking at a computer screen". But actually displaying that picture is better ;)
Fair point, it's my biggest gripe with Vim's autocomplete--which I love. That being said when it works, it's more efficient than any other autocomplete I've seen in a text editor, since it's practically the shell's autocomplete.
> Basically you can do everything with text. You can even print out "this is a placeholder for an image of a sad green frog looking at a computer screen". But actually displaying that picture is better ;)
Once again, valid reasoning, but that's not the point. No one (at least I hope not) is seriously advocating TUI's for graphic design, we're talking about editing text, and that's where Vim excels.
What is the advantage of terminal based editors? As far as I can see, ubiquity, which is made less compelling when you need a host of plugins to do the things you mention.
Terminal editor enthusiasts seem to be under the effects of something like the Blub paradox, where any feature of a GUI editor described to them is dismissed with "why would I ever need that?" while anything in terminal emulator can do is considered essential.
Really? I'd like to see sublime handle text manipulations with the same efficiency and precision that Vim provides.
> What is the advantage of terminal based editors? As far as I can see, ubiquity, which is made less compelling when you need a host of plugins to do the things you mention.
What about integration with the shell? It's not merely a matter of ubiquity: I don't really need a document tree when I have the entire terminal at my disposal a :sh or :wq away.
> Terminal editor enthusiasts seem to be under the effects of something like the Blub paradox, where any feature of a GUI editor described to them is dismissed with "why would I ever need that?" while anything in terminal emulator can do is considered essential.
This would be an apt description if the majority of "terminal editor enthusiasts" were as inexperienced with GUI editors as their GUI counterparts, but in fact the opposite is true. Most Vim/Emacs users tend to have hundreds of hours experience working with GUI editors, but are for some reason convinced that they (the editors) were inferior. Finally, let me give you an irrelevant (but fun) fact: PG, the original proposer of the Blub paradox, uses vi.
I think you've misunderstood the claim here. The claim was not that all existing GUI editors do all the things that all existing terminal editors do. The claim is that a GUI (not a GUI editor, just a GUI) can do everything a console can do, and then do more things besides.
The claim is obviously true, because a GUI can embed a terminal interface in itself, and then augment it. I use emacs that way. It's emacs, but it also can do buttons, menus, proportional fonts, etc. (... not that I use any of those things, but it can), and all sorts of things it can't do in terminal mode even though it's literally the same executable. The point is that the GUI environment is intrinsically more flexible because the GUI has a superset of capabilities, not that a traditionally-GUI-centric design with heavy mouse and menu usage is superior.
The anecdotal suggestion that people who prefer a GUI simply don't know what they are talking about is just pure snobbery. I know both and I still prefer a GUI. You're right that the appeal to the authority of PG on this subject is irrelevant.
Can you give an example of an efficient/precise text manipulation that vim can do that sublime can't?
Because I rarely want to just "edit text" - >50% of the time I want to have a visual representation of the file tree with colored/graphical icons, drag & drop, etc.
A significant part of programming is organizing and exploring trough things, especially in huge code bases - GUI hands down beats any text based solution I've seen. This is also why I still won't use VS code as my go-to editor - no file icons - when I have a file tree with hundreds of nodes icons are the best way to find the stuff I'm looking for.
I keep wanting to try out Spacemacs but I always revert to Vim but it's pretty far from perfect imo.
Vim is more important as a way of text manipulation than as a specific program, and projects like NeoVim and Xi make Vim-the-way more portable and accessible. I would welcome editors that have a high-quality Vim experience and also have nice GUI features like mouse-over popups of function signatures, minimaps, or whatever a GUI unconstrained by the terminal can do -- rather than arguing for workarounds or why people don't need these features.
As a mouse user as well, sometimes selecting text with my mouse/positioning with mouse works fast, and I've always found text editors to not really be too great at letting me click accurately between characters.
Also, if you get into more IDE-y features, not having your "popup"s be overlayed colored text is usually a win for aesthetics. And hey, if I'm staring at something 8 hours a day, having it be pretty is nice right?
So even with GUI apps, I tend not to use the GUI. I don't think I'm alone. Virtually everything I run is either a terminal app to begin with or can run from the terminal. I have tmux set up to work almost exactly like my X window manager, so if I don't run X I barely notice. I'm just as productive without X window as I am with it except for a very few things: browser, gimp, a few games... Honestly I can't think of very much.
It's great that some people like GUIs. I used to use them a lot, but have since found that I'm happier without them. The keyboard is not really a discoverable UI device most of the time, but it can be ridiculously efficient once you learn how to use it. I also have very poor eyesight, so that probably predisposes me to using interfaces where I don't need to use my vision.
The mouse is never as powerful as vim commands. ;)
(Textual user interface)
Also most terms don't properly do utf8.
Maybe they want people to be able to easily distribute binaries of the software via BitTorrent. Apache allows this, the Vim License allows this, GPLv2 does not[0].
Another feature mentioned in the introduction is this:
* Developer friendliness. It should be easy to customize xi editor, whether by adding plug-ins or hacking on the core.
I'm vaguely familiar with NeoVIM, is the core "hackable"? This is really subjective territory but I'm guessing the author is implying that the xi core code is also optimized for human understandability and extensibility and that new code is judged by those criteria as well as the other stuff.
Regardless of my personal preference, I always found it odd: the back-end should be independent of what I perceive as an interface "detail". Even if, in this case, the detail is part of the software's identity.
Which is why I gladly welcome the xi project :)
I notice that you've choosen Rust as the programming language for the editor, a fine choice. Performance appears to be a key factor in your choice of language. However, realize that performance isn't the reason one should use Rust or C++ for the backend of an editor. Let's go though some simple back of the envelope calculations. A realistic goal for editors is to keep ordinary command and keystroke latency under 100ms, this should appear to users as if operations are performed instantaniously. You've picked a more aggressive goal of a maximum latency of 16ms, so let's go with that.
Today, a slow processor runs perhaps 5,000,000,000 instructions per second (speed of Raspberry Pi 2). By the time the editor is ready for general use, most programmer's development systems will probably run something like 20 times faster or 100 Billion instructions per second. For this rough calculation I'll use the speed of a Raspberry Pi 2. In 16ms that's 8 million instructions.
What this means is that in the backend of the editor we have plenty of time to perform the editing operations no matter what language is used as long as it is a straight code path through the editor. The pedestrian operations of the editor, like reading a keystroke or inserting a letter into the text don't require Rust or C++ or C. They can be done with practically any programming language, even a language that runs a hundred times slower than C++ like Python. It is the iterated operations where performance will be a concern: searching big files, complex regular expression searches, syntax highlighting after each keystroke, autocomplete, refactoring, the GUI and interfacing to the GUI.
I couldn't agree more with your requirement that the editor shouldn't hang or crash or lose work. Further, a modern editor is a big project and won't gain traction without additional contributors. From a performance perspective my gut feeling is that any language that performs no worse than 10 times slower than C is a fine choice. For safety, either managed memory (with GC etc.) or an approach like Rust is essential to allow code to be contributed. The JVM languages, while being attractive in many ways carry the burden of a big infrastructure; Jetbrains has figured it out but even after all these years Eclipse feels so klunky--this make me rule out Java, Clojure and Scala. I think this leaves at least Rust, Go, OCaml, Common Lisp, Dart, and Swift. (I'd probably pick Go hoping to get more contributors.)
What about ropes, a clever and much more modern data structure for holding strings than ancient Emac's gap buffer. Ropes do indead have better asymptotic performace for many of the basic string operations: delete an arbitrary character, insert a substring, etc. What about all of the other operations that you'd like your editor to do? Fast searching, backwards or forwards? Handling UTF-8? Doing regular expression matching. Ropes turn the text being edited into a tree of text segments so high speed searching is much more difficult. An abstraction layer over the data structure to make it appear to be indexed like an ordinary array will make some things run much slower, for example, Boyer-Moore string searches, regular expression searching (done for code highlighting), etc.
Ropes or other persistent data structures for sequences will work because they can present an abstract indexed access interface to the rest of the program, but you will always be dealing with it though this abstraction layer just when you'd rather not. Processors can move several bytes per instruction cycle so the gap buffer can close it's gap while editing a 10MB file in less than a millisecond. This leaves the data in one contiguous block. So for simple things the gap buffer is much faster than rope structures because for most simple activity all of the additions and deletions are at the cursor, where the gap is located, and for complex things there is a 1ms overhead before the operation starts (to close the gap) but then things run basically full processor speed without the requirement to go though the abstraction layers. The places where I see this as likely to be important in modern editors is in the design of incremental backups, persistent text attributes, memory mapped access to underlying files, and coordination with asynchronous plugins working on the buffer. I'll be curious to see how this works out for Xi.
Keystrokes and plugins. Emacs can end up running lisp code, through a pretty slow lisp interpreter, on every keystroke. It's completely imperceptable. So performance isn't a reason to treat keybindings differently than other plugins. The overhead of IPC to a plugin is measured in micro seconds.
While supporting Sublime Text keybindings is fine. The programmers most likely to be sensitive to keybindings will probably want either Vim or Emacs keybindings. A good way to support multiple keybindings will make the editor appeal to a much broader set of users. Plugins seem ideal for this. A key comes in, is processed by a plugin and a few elemental backend operations are invoked by the plugin in response to the key. This seems completely reasonable.
The real issue with plugins is direct access to the edit buffer if the plugins run in a separate process and address space. This is complex especially if the buffer is being accessed asynchronously. How is syntax highlighting done in a plugin for example. Where does the syntax highlighting actually get stored? Does the plugin have to be informed each time the user scrolls the screen or types a key (probably yes). What asynchronous commands does the plugin send back to the editor to update the display. Is there a separate plugin working on autocomplete? Does it also have to parse the source code just like the syntax highlighting plugin? Well those are things to figure out, but the keystroke bindings seem much easier to address in a plugin. Plugins provide a convienient place to address the semantic difference between a modal editing model (used extensively by Vim and to a degree by Emacs editors) and the underlying editor backends capabilities.
Using ropes is a tradeoff. A gapped buffer is viable for the reasons you state, but I think ropes come out ahead for what I'm trying to do. It's not so much the cost of filling the gap, as that ropes really facilitate parallel, incremental, and asynchronous computation. For example, I want the "save" operation to write a snapshot of the current buffer to disk, while still being fully editable. The snapshot operation is trivially easy and fast with immutable ropes, but if you tried to hack something like that on top of a gapped buffer, it would increase complexity dramatically.
If you look at the implementation of my Unicode line breaking algorithm, you'll see that the indirection cost of the rope abstraction is minimal. In fact, it _flies_ - in my tests, it's about 3x faster than ICU working on a contiguous string. This is because it's able to work on a leaf at a time. The same can be done for search, etc. Yes, it'll require some reimplementation of string basics, but I personally find that kind of thing fun.
I'm still working out the plugin story. Quick answers to your questions though. A syntax highlighting plugin sees a window of the source buffer (effectively a cache, with invalidation by RPC). I'm thinking a contiguous buffer for the window, not a rope - it'd be bounded by a few megs. For a huge file, computing the syntax highlight of what's off screen can be slow. In no case does it block typing, worst case there's a lag before the colors resolve. The plugin sends the annotations back as rich-text spans, which are stored in the core. Scrolling is instant and doesn't inform the plugin. The autocomplete plugin will almost certainly be separate from the syntax highlighting one (but presumably an author could create a combo plugin if it had advantages).
This doesn't address everything you brought up, but hope it helps illuminate some of my thinking.
One problem with the gap buffer is that as soon as you have multiple cursors/selections or in my case structural regular expression support, then you have to move the gap around all the time. Also undo/redo support is simply not as elegant as with a persistent data structure.
For syntax highlighting copying the relevant text region into a contiguous memory block is simple and works well. For now I'm trying a completely stateless approach to syntax highlighting i.e. the text coloring is always completely recalculated. This trades highlighting accuracy for simplicity, but in practice the results aren't too bad. Efficient syntax highlighting for huge files is a hard problem.
Disagree with the comment "Cross-platform UI toolkits never look and feel quite right." Qt, for one, uses the native toolkit on Mac and Windows and I can't see a difference in look'n'feel.
QT interfaces stick out like a sore thumb on the Mac.
Unless you custom-skin them to the point that they still stick out, but they at least don't fall into the Uncanny Valley.
>Of course there are alternative file managers for other platforms. But none of them make you really go wow.
Btw, there are a couple for Mac too, in case you haven't seen them, e.g.: http://www.binarynights.com/forklift/ , http://mac.eltima.com/file-manager.html and http://www.cocoatech.com/pathfinder/.
I know all the file managers you mentioned (my market research turned up 91 file managers available globally). I believe my point "don't really make you go wow" still applies.
Thanks again! M
And if you've used JavaScript, you may have been using them without realizing it—JS engines will choose rope representations for strings if they observe that the usage patterns would benefit from them.
Gtk3 had one job: make it work with Wayland and HiDPI, but like many open source projects they redid too much and broke more than they fixed.
Case in point: just updated to Firefox 46 and it now requires Gtk3, which introduced many UI regressions and oddities.
So, Qt is the better choice when compared to Gtk3, though those who don't feel the pain of Gtk3 and use GNOME may disagree. I use Qt apps on X, Wayland and Windows, for what it's worth.
https://gcc.gnu.org/onlinedocs/libstdc++/libstdc++-html-USER...
https://github.com/gcc-mirror/gcc/blob/master/libstdc++-v3/i...
It doesn't have support for native widgets, everyone has to emulate them via QML.
https://github.com/google/xi-editor/blob/master/rust/rope/sr... was quite interesting to read (for someone who has never looked into rust)
My current development environment is just Sublime Text for the Rust parts, and Xcode for the parts written in Swift.
That's ridiculous. If I release something into the public domain, then it is forever in the public domain and is perpetually free to use by anyone.
Copyleft is the only defence against that form of attack on user freedom -- attack by distribution.
No it's not. My code is still available to any who want it.
Other modifications made to my code may not be free, but that isn't my code, so your statement is incorrect.
Sounds like you're contorting the perspective.
What's ridiculous is that you're presupposing that I don't care about "perpetual freedom." But I do---that's why I released the code into the public domain.
What you seem to be talking about is something quite a bit different. I guess it's, "perpetual freedom of my code and any other modifications distributed by others."
But that doesn't sound as good, now does it? Your phrasing makes it look like I'm so ogre, which is why it's ridiculous and why it's misleading. Your phrasing is also convenient to certain philosophical leanings among programmers, which potentially makes it disingenuous.
> And talking about how "the original code is free" misses the point
It doesn't miss the point. That the original code is free is a crucial and important point. It just may not be the extent to what you care about, but your phrasing suggests otherwise.
> enslaving users using your software
Enslaving? Oh please.
> arguing that you like permissive licenses while stating that you care about perpetual user freedom is having your cake and eating it.
That's FSF doublethink, plain and simple. Co-opting the word "freedom" was one of the most genius marketing ploys ever done. It's such an overloaded term that nobody will notice, but it's such an enticing term to use to brow beat anyone who disagrees with you into freedom hating slaveowners.
Personally, I don't think that a single source being free captures the point of the word "perpetual" -- it surviving forever (across forks and other changes to the project). When eventually a project you work on is abandoned, someone could fork it and make it proprietary (adding features along the way). When someone then downloads the newer fork of the software, they've had their freedom taken away -- it doesn't matter that the software was free initially.
Why doesn't it happen with SQLite? I think it probably does, it's just that the creators don't care about that (and they're at liberty to do that). But one of the largest operating systems in the world (GNU/Linux) is created using many components under the GPL. So cherry-picking projects doesn't help either of our cases (and there are many more high-profile projects under copyleft licenses than in the public domain).
This is the kind of subversive language that I take issue with. Once I release a piece of code, it exists now and forever, until its bits are deleted from every last storage medium. A fork has nothing to do with that code's existence. It may be the case that a fork gains favor and is more widely used than the original piece of software, but the original piece of software still exists and is perpetually free.
> Personally, I don't think that a single source being free captures the point of the word "perpetual"
I'm not taking issue with your ideas (in this thread, anyway). I'm taking issue with your phrasing and the way you've expressed your ideas. I thought I was pretty clear on that. You're using words like "enslaving" and "they don't care about freedom," which is just a way to discredit, shame and condescend those that don't share your beliefs. It's a classic debate tactic.
> Why doesn't it happen with SQLite? I think it probably does, it's just that the creators don't care about that (and they're at liberty to do that). But one of the largest operating systems in the world (GNU/Linux) is created using many components under the GPL. So cherry-picking projects doesn't help either of our cases (and there are many more high-profile projects under copyleft licenses than in the public domain).
It is unbelievable how much you're twisting my words. I called out SQLite as an example that public domain is in wide use beyond a "few western countries", which was a response to your snubbing your nose at my terminology. Like public domain is some obscure thing? Well, no, actually, it's used by one of the most popular software projects in the world. How could it still be obscure? That was my point.
The idea that I cited SQLite as some kind of evidence that public domain is "better" than copyleft is ridiculous. It's even more ridiculous that you implied that I argued that "X doesn't happen to SQLite." I didn't.
Rather nice font in the screenshots though. Then
> The main author is Raph Levien.
Oh. Now I see why this is on hacker news. Now I'm interested too. If nothing else, this will be gorgeous.
I'm asking "why isn't this handled by freaking CSS? We have a goddam tag for this!"
It may be a good example of programming that's worth reviewing as I learn or it may just help provide a sense in the kind of community that's growing (or not) around something that's relatively new (in some stable status at least) like Rust.
Had it not said "Rust" in the title... I may well have not bothered to look at this at all.
The choice to start a new editor in Rust signals to me that this editor won't have as much trouble optimizing for speed as other recent editors, and it also signals that it may be able to hook on to recent Rust energy to get collaborators. To the end user, that potentially means speed and longevity.
Whether rust is beneficial to a text editor project is up for debate (my guess is no), but I agree that language is a feature of an open source project in the types and quantity of contributers it attracts.
The implementation of Xi, has some novel things that might interest Rust programmers, like a practical Rope implementation, which are quite rare in Rust and often abandoned.
I was interested in Rust before, but it now seems like Rust programmers are like vegans (I have to hear about it every 10 minutes.)
I'm not saying anything specifically about this case, because clearly the information here is very limited, but there is a case to be made for rewriting projects that work to increase maintainability in the future. Most newer languages do have an edge in this department, as they are more conscious of those issues.
There is sometimes the benefit of a rewrite regardless of language, as well; cleaning stuff up, organizing stuff better because you now know better, etc..
Any idea what language and APIs the Windows version will use? All of the Microsoft-sanctioned choices for desktop apps (as opposed to the wanna-be mobile/tablet platform introduced with Windows 8) are kind of sucky.
But I'm very much open to ideas from more experienced Windows devs who understand the tradeoffs better.
The best solution would probably be one that uses both DirectWrite and Direct2D. WPF 4 uses DirectWrite, but AFAIK, it's still on top of the older Direct3D-based milcore graphics system. But to use both DWrite and D2D, I guess you'd have to get down and dirty with C++, or at least use a lot of P/Invoke and COM interop in your C# code.
To be honest, I don't understand why GDI isn't good enough. Can you explain what you find lacking in GDI's text rendering, or point me at an existing explanation?
If my experience is anything to go by, then it also introduces a whole new class of pain around gluing things together.
You can alleviate a lot of it by using a real and backward/forward compatibly extensible(!) schema language for the interchange language. Your interchange language should also support some form of 'algebraic data types' just for maintaining general sanity.
I personally like Thrift as it can use JSON or plain text (perfect for debugging) or whatever, not just a binary encoding of communication messages and its type system is IMO slightly easier to work with.
http://blog.neil.brown.name/category/edlib/ https://github.com/neilbrown/edlib
In particular, it should use little more memory
than the buffers being edited.
Do any of the decent OS X -compatible editors have things like
- find usage of a method/class/constant in a project
- change method signature and have calls update
- rename a class/method/constant/variable and have usages updated
This isn't a troll, I'm honestly asking. I'm using IDEA Ultimate at the moment and it works fine, it just has the usual downsides for a Java app.
If you write java and value those features, though IDEA will be a better experience.
Also for reference I don't really write Java. I write php, sql, shell scripts, ldif files, html, occasionally JavaScript and/or css, lua...
It's called a "compiler" ;)
Jokes aside, here's a few tools that might give you an idea what's out there:
- Racer for rust
- Go fmt
- Youcompleteme
- ghc-modOh and a plugin specifically for Vim, so not an external tool at all..
So, basically the answer is the same as before: an IDE is still where it's at.
Are there some quick and simple examples of this?
For one, the build system/dependency system is much more easier to handle. Rust uses the cargo package manager, and depending on an external library is as simple as adding a single line to the Cargo.toml config file. Likewise, building, running, documenting and testing is all a build-in matter of calling `cargo [build|run|doc|test]`, which makes prototyping and getting started way easier than my experiences with C++ so far.
As far as the actual language goes it basically behaves like C++ and has a similar feature set, so you get the same sense of how given source code will end up in the compiled binary, and can write fast efficient code with, for example, features like pointers. The big distinction there is that Rust is memory safe per default, (including pointers in form of safe references) and upholds this invariant with compiletime checks and explicit language barriers (needing to encapsulate "unsafe" operations in unsafe {} blocks), so per default you are protected from whole classes of bugs that are possible in C++, like dangling pointers, memory races, iterator invalidation, etc.
Interestingly, rustc also seems to generate programs that leak memory. I recall an issue where the simple and venerable "hello world" program in Rust (i.e., println!) had memory leaks.
While the human readability of JSON is nice, it has some serious flaws when the goal is to handle arbitrary data.
I have yet to decide what kind of protocol to use in my own vis editor[1], hence why I'm asking.
Just a heads up, you misspelled "sparse" and accidentally a word after "completely".