Hacker News Reader

Comment by vincnetas | Hacker News Reader

Running subjective benchmark now. Generated 1G text file. Vim and nano after some delay are showing things on screen, ox is still thinking. 5 minutes and counting...

  base64 /dev/urandom | head -c 1000000000 > file.txt

View on HN

One thing about this particular benchmark is that the file it produces is a little bit unusual even for large text files, because it is only contains one (massive) line. Most huge text files you open in text editors are things like multi-gigabyte logs and database dumps, which are composed of many relatively short lines. Text editors often bake that assumption into their data structures. A more representative test would probably be to use hexdump instead of base64 there.

Though I take the point: if a text editor advertises itself as being "fast", this is the kind of stuff you expect it to handle.

EDIT: ok, so i just tried the hexdump version. Vim and Emacs took a couple of seconds each (Emacs being slightly slower and chunkier when you navigate, Vim handles it without issue and is perfectly repsonsive). less opened it instantly (obviously, it's less!). ox has been going for a few minutes now, nothing happening.

It's not the benchmark's fault, ox is just slow for large files.

> the file it produces is a little bit unusual even for large text files, because it is only contains one (massive) line.

XML and JSON without pretty print. Minimized HTML with embedded resources. While not often gigabyte sized those seem to be enough to get most editors to hang and make editing the files a miserable experience.

Yes, true enough, it does happen occasionally, but it's pretty rare (and usually there are other ways to deal with it: using less or similar tools, or just running your massive single-line JSON thing through jq first). And my point was exactly what you're saying: many text editors choke on files like this, so it's not such a huge indictment of ox that it chokes on it as well.

However, ox clearly performs poorly even when your text files aren't massive single lines, so it doesn't really matter: ox basically can't handle files like this regardless of line distribution.

> and usually there are other ways to deal with it: using less or similar tools, or just running your massive single-line JSON thing through jq first

Older Unixen at least (SVR3-based) had a tool called bfs - big file scanner. Used it some, e.g. when, as a system engineer, I helped IT staff of one of our customers, a university processing exam results of tens of thousands of students from dozens of colleges affiliated to that univ. IIRC, its UI was something like a read-only ed (google "unix bfs command"). You used it to scan really large (for the time) files, e.g. data files (input/output) in large data processing environments, for purposes like checking if the input or output files anecdotally looked okay, no major noticeable garbage in them. Haven't checked if it is present in modern Linuxes. Also don't remember if it could handle large files without newlines. Likely not, if based on ed. Didn't have such files to work on then.

Incorrect. The base64 command wraps at 76 columns by default.

Whether there's a wrapping feature depends on the implementation. coreutils base64, yes. Other implementations, maybe not. macOS's /usr/bin/base64 for instance doesn't have the concept of wrapping. FreeBSD base64, judging from the manpage, doesn't do that either. https://www.freebsd.org/cgi/man.cgi?query=base64

Edit: Oops, posted some misinformation. The mistaken part, reproduced below:

> First, wrapping by default only occurs when writing to a tty. When stdout is redirected to a file, wrapping base64 output usually doesn't make sense, so wrapping doesn't happen.

Correction: coreutils base64 wraps regardless of whether stdout is a tty.

> macOS's /usr/bin/base64 for instance doesn't have the concept of wrapping

Yes it does. "-b N" or "--break=N" inserts a line break every N characters.

Yeah. I know -w doesn't work, and missed -b when glancing at the manpage. (I use coreutils day to day.)

I ran it on macOS, and there I got:

    $ base64 /dev/urandom | head -c 1000000000 | wc
       0       1 1000000000

I haven't tried anywhere else, so it might be different on Linux or GNU or whatever.

As an emacs user, one of the things I really wish it had was robust support for arbitrary file sizes and shapes.

(It'd be fine if not all features worked on these files.)

Sure, most files are reasonably sized and shaped, but sometimes you do find yourself needing to e.g. dig into that single-line 1GB json file to try to debug something, and it's inefficient to have to think, before opening any file, "is more normal tool sufficient for this, or should I use something else?"

There's no fundamental reason we can't have one tool that does it all!

emacs -q or -Q is often quite fast. It's things like syntax highlighting, word wrapping, etc. which modes add which usually slow it down. Anything trying to run regexes over the buffer isn't going to scale well.

I think long lines are (were? [1] seems to be a new feature) a well known problem in emacs even without a custom config. There have been ways to work around this for awhile I think, but as far as I know emacs -Q isn't one of them.

[1] https://www.gnu.org/software/emacs/manual/html_node/emacs/Lo...

oh that's neat, i didn't know about that! could help.

my ideal editor would never block, and use file-sized based approximations to allow scrolling to arbitrary locations w/o having to actually read the contents of the entire. things like syntax highlighting &c would be either strictly time-limited to not drop frames, or done in the background.

Try M-x find-file-literally to avoid any mode-specific hooks, colorization, etc.

Could this be summarized as "algorithm" (or data structure more specifically) beats language implementation performance?

Certainly could!

Summary of replies: This benchmark doesn't count because No true Scotsman.

Most would happily trade an editor that is 100x slower for gigabyte single-line text for one that is 10% faster on gigabyte multi line text.

99% of files I work with are <1000 lines, so I don't mind. For larger files I will fall back to Vim.

The important thing to realize is that text editor performance is less about the technology used and more about the data structures and i/o.

This is not an excuse for an editor being claimed "fast". Textadept is written in C and is also very fast but has the exact same issue -- choking on huge files when Vim and others are able to handle it.

I rarely open gigabyte single-line files, but when I do, I don't want to switch editor.

Most of the time I only realize that everything is on a single line after opening the file.

A good editor doesn't require me to think about these things.

If it's over 1 GB and single line you're almost always going to exit the editor, run some scripts on it, and open it again when it's correctly formatted.

I would rather open it in editor, strike a "reformat all" shortcut, and have the file right there.

I don't want to chase separate tools that can format different text files, that's what editors are for.

It's still annoying if your editor hangs and you have to find some way to kill it before your computer becomes usable again.

Sure. I'd not be able to actually do anything useful with a gigabyte single-line file so what I'd want my editor to do is say "Longest line is over 100k characters, truncating display to the first 100k".

issue is that the editor can't tell if it's a multiline file with a first big line or an editor with one big line if it's cut at the first line. But I'd rather have the editor bail at the first 100k line than not opening it.

Sure, but this one isn't 10% faster on gigabyte multiline text. Others in this thread are testing wrapped text and Ox still chokes.

Emacs opened it in about 4 secs on my 2018 MBP.

I wonder: Couldn't we have some kind of simulated line breaks, which an editor places virtually, so that it can manage very long lines easily and efficiently internally, while writing out the file without these virtual line breaks? Basically only so make the conditions appropriate for the editor's string data structure?

In fact, you basically need to do that anyway to wrap long lines effectively.

Software developers find lazy/partial file loading unnecessary but, it is not and it won't...

Vim is amazing. Try searching for patterns in this enormous file ;) Definitely a hard nut to crack.

Personally i see no reason why this is needed (just use vim), but I'm sure a lot of people will use it just because its written in rust^^

I highly doubt you have to manually edit a GiB file by hand. For read-only, less opens it instantly.

I have had to do this in the past. Admittedly, that was only a few times over the course of 15 years.

I tweak 1.5GB files by hand on a weekly basis on servers across the globe. Fortunately vim works for my use case.

In that case, you would be better served by some automation tools.

You'd have to know my specific use case to say that.

Unfortunately it's not automatable.

Yeah but that's not a realistic usecase for most users/developers. I haven't measured if Ox is faster or slower, but for me startup time and ability to quickly open files under 10MB are more important.

I open massive files in a text editor all the time.

I also am regularly limited in opening large text files working with genetic data.

Probably, but then we don’t use our normal editors when we want to open huge files, we use the fast ones.

But for 10MB range do you even feel the delay for any of text editors? For me it feels instant whatever (terminal based) editor i'm using. And what does it even mean "fast" in this case. They all are definitely faster than me typing.

It is very realistic when I copy a json dump into an editor and then do a key stroke to format it.

My beloved vim doesn't fair so we'll. VScode does very well on this.

I noticed some years ago that vim can be much slower due to highlighting of particular syntax of huge files. For example it was fine with a 10MiB C file, but super slow with a 10MiB XML file, but fast again with ":syntax off"

Are you doing the formatting via a Vim plugin?

I often find it's much faster to pipe out to a command line program to do formatting on huge files.

e.g.

   :%! jq

   :%! xmllint --format -

Huh, thanks, I didn't know you could put the contents on stdin like that (%!) - just in case you (or anyone) was similarly unaware in reverse, you can also put `%` wherever you want to use the filename in the command.

i.e. I would have done:

    :! jq %

and have probably been guilty of (not knowing `%!` per above):

    :! cat % | ...

I frequently open minified CSS/JS by accident. IntelliJ's project search window includes a small preview of each result when you hover it, so when you navigate through the results your might pass through a minified file and get your IDE to hang for a couple of minutes :)

I open massive text files semi-regularly. If a text editor can't deal with that it's a big con for me.

> Yeah but that's not a realistic usecase for most users/developers.

It also looks like vim and emacs don't decide what a realistic use case is, or if you're "most" users/developers. They just open text files for editing.

Is something like Emacs, Vim, or Nano genuinely slow for you?

Even for large files, I've never noticed anything beyond microsecond-type delays with those. Definitely does not feel laggy to me, usually.

Reply on news.ycombinator.com