Atom now opens files larger than 2MB | Hacker News Reader

Atom now opens files larger than 2MB

github.com

39 pointsvruiz59 comments

This is 2015, 2MB files shouldn't be news worthy for a text editor. We seem to be bloating software faster than hardware improvements can offset it.

This has nothing to do with bloat. While unimpressive not to be able to open files larger than 2MB, it's easy to see how and why such a limit came about, and as Atom seems to be styled as a text editor for programming, not really unreasonable, as very few human-created text documents get anywhere near 2MB.

Sorry, it's not obvious at all how the limit came about. That is pretty terrible.

Atom is a desktop/native wrapper around browser-like internals. The stuff inside is all just HTML/javascript/css running on node. There are some neat advantages, as well as just seeing how far a web interface for a desktop app can go.

The limit came about because trying to open a 100MB file as an html page (even without all the extra styling and interactive options) will break any browser. It was just a hard problem to solve, having an HTML-based editor deal with so much text at once.

> trying to open a 100MB file as an html page (even without all the extra styling and interactive options) will break any browser.

I'd love to test that. Anyone willing to privide an example file? HTML with css perhaps, no js. Should be pretty small when zipped.

Assuming you don't need anything complex, it's pretty easy to generate it.

Here's a 100MB HTML file (compressed to 300KB): https://curl.io/get/yt7zbmfq/0eb1564f570236acf1d9f304855a7c7...

As a side note, VIM opens it in an instant /troll

The limit came about because trying to open a 100MB file as an html page (even without all the extra styling and interactive options) will break any browser.

I remember opening HTML files in that size range (they were logs) in IE6 around a decade or more ago, on a machine with <1GB of RAM, and it handled them just fine - there was a noticeable delay, but it wasn't unusable. Even searching for strings worked.

It is obvious to me: I think they chose to load, parse, prettify, colour etc. and keep the entire file into an in-memory data structure, which is easy to reason about, and totally reasonable for "human sized" data. However, for arbitrary sized files, you need a mechanism for leaving most of the file in a raw state - that's harder (but not impossible, obviously).

So, it makes sense to me that they would assume only "human sized" data (and expect users to use another tool for looking at large files, "do one thing well" and all that) and opt for a slightly simpler programming model. Apparently, they decided that the assumption was wrong, and so changed the model.

very few human-created text documents get anywhere near 2MB.

True, but as a programmer i very often generate and need to view files greater than 2MB.

This.

Not to mention log files which commonly consume megabytes.

Nitpick inbound.

You shouldn't open log files with an editor unless your intent is to edit the file. There are a whole host of tools (grep, less, awk, sed, etc) useful for parsing and exploring log files.

You also very often do X, but that doesn't mean your programming text editor is the correct tool for doing X.

Insisting on all tools having an X because X is a useful thing is how you get the very bloat the GP is talking about.

On the other hand, virtually every other editor is the correct tool to do that task, yet this one is not. Are we supposed to accept that?

2MB is 0.025 percent of my ram, I shudder to think how much ram is actually being used by atom... At least the html looks pretty.

The biggest problem I see is that this is an artificial, arbitrary hard-coded limit:

https://github.com/atom/atom/commit/ec65a71d6dc8c3c44cb1b77a...

If I have 64GB of RAM, I expect my applications to be able to use all of it when asked to, and not just give up completely. It's fine to slow down, and giving a warning "Opening large files may be slow. Do you want to continue?" would be nice, but working with a 2097151-byte file and giving up if it's 2097152 bytes is really unpleasant.

Interestingly, Visual Studio disables all editor services (cold folding, intellisense, etc.) except syntax highlighting on XML files bigger than 10MB.

Code editors are surprisingly complex, it's not about simply putting lines on the screen. A good exercise in understanding the complexity is writing a simple syntax highlighting code viewer with native controls (not HTML) and see how much memory/CPU cycles your implementation uses for a 2MB file.

Tons of optimization needs to happen - for example, as far as I know very little information about lines that aren't visible is kept in memory in Visual Studio - so called "virtual lines." I'm sure other more mature code editors have this type of optimization as well. From a native perspective this type of optimization isn't a complete nightmare to implement (although it's hard if you consider code folding), however, imagine trying to do it in HTML. The browser is still going too do all the reflow etc. for the lines that aren't even visible, eating up CPU cycles and memory.

Even merely syntax highlighting is a problem. If you make one edit the entire file might be re-lexed: once again you need a unique solution for editors ("progressive lexing").

imagine trying to do it in HTML

The question on my mind is, why do it in HTML? Especially if you're going to wrap that browser in something that approximates a desktop app anyway.

Imagine if designers tried to do everything in PowerPoint for some unknown reason. "3D animation? Yeah, we can do it in PowerPoint with this incredible hack that makes slides sort of like faces on a mesh..."

That's how it is with HTML today.

"Imagine if designers tried to do everything in PowerPoint for some unknown reason."

A fair percentage of the employees where I work seem to have reached that point already - reports, training material, data for general discussion, GANTT charts with individual symbols that I guess must be manually moved every week. Gah!

Because of extensibility concerns and the sheer number of JavaScript developers? I'd agree if you say it's not a good compromise, but it is one nevertheless.

HTML is not the problem.

Light Table doesn't have any issues with large files, and is built on essentially the same stack (Electron + native HTML/JS/CSS for Atom vs Electron + ClojureScript compiled HTML/JS/CSS for Light Table).

Visual Studio Code doesn't have any issues with large files, and is also built on web technologies (no source available but I assume some variant of Electron + TypeScript?).

The problem is somewhere in Atom itself.

> Visual Studio Code

Virtual lines in action.

1. Open VS Code with a file that extends below the bottom of your screen.

2. Press F12 to open the dev tools.

3. Use the magnifying glass to inspect the final visible line of code (it should be the predecessor sibling of <div class="contentWidgets">).

4. Scroll the text editor up and down and watch as the lines are added and removed from the HTML.

Edit: Looked at Atom and, yep, all the lines are kept in the HTML all the time. They seem to be chunked into "tiles" but those tiles always exist in the document.

Edit2: Actually it seems as though there only ever exist 3 tiles (partially obscured by top, fully visible, partially obscured by bottom) containing a dynamic list of lines. So this isn't why Atom is slow, arguably this is a better approach than appending/removing single lines at a time (which is what VSCode does).

> Visual Studio Code

It is based on Visual Studio Online. Its text editor component is non-trivial and Microsoft spent a lot of money on it. The editor (codename "Monaco") is from Erich Gamma (ex-IBM/Rational), the "Design Patterns" book author from Switzerland. Microsoft bought the Monaco editor with him and its the single reason why TypeScript exists and why JavaScript ES6 looks like it looks ("class" syntax) and why he leads the Visual Studio team: http://en.wikipedia.org/wiki/Erich_Gamma

The Monaco editor (coded in TypeScript) will never be open source, its their competitive advantage. The same reason why HTML5's contentEditable API will never be fixed in IE/Chrome/Safari as a web office suite is their competitive advantage of Microsoft Office365 Web Word , Google Docs and Apple iCloud Pages. With a bug-free contentEditable API would render also Monaco editor and many other web based rich text editors obsolete: https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement...

I've actually done not 3D, but 2D animation in Powerpoint (as a kid, with access to an office suite, but not the internet and thereby more appropriate software). Each slide contains a frame of animation (drawn in Paint, of course), and slides advance automatically in milliseconds. You can attach an mp3 to play in the background for the soundtrack. Save it as a .pps and you have a cartoon that you can share with your friends.

That made me wonder whether the demoscene had ever done something with Powerpoint... and indeed it has:

http://www.pouet.net/prod.php?which=54670

http://www.pouet.net/prod.php?which=30319

Excel too:

http://www.pouet.net/prod.php?which=53021

Completely impractical and very silly, but that's what the demoscene is for...

Atom changelog could be a good parody media (except that it’s not)

There was this little gem a while back:

“Fixed an issue where Atom would to fail to launch from directories that contained a file named undefined or null.”

That's pretty golden, but I like my jokes sourced. Do you have a link for that particular change-log entry?

https://github.com/atom/atom/releases/tag/v0.150.0

How does that even...

If they're not treating filenames as strings - and nothing but strings - then it could have some pretty surprising security implications, possibly to running arbitrary code just by opening files or directories with certain contents?

This is so sad...

I've been trying out Light Table recently, and it seems to have no problems with large files (and generally feels much more snappy in use compared to Atom).

I just tried a bundled 3MB+ .js file and it loaded practically instantaneously. Even opening a 10MB+ .mp3 file (for science!) didn't result in any delay longer than a second. Whereas opening the same files on the latest Atom took a good 5 seconds for the .js and 10 seconds for the .mp3 (and it looks like Atom had to give up on doing syntax highlighting on the .js file, whereas Light Table did not). These tests were all performed on a rather modest Core M processor, so YMMV.

AFAIK, both Atom and Light Table are built on Electron and web technologies (Light Table uses ClojureScript but at the end of the day they both compile down to HTML/JS/CSS).

So my takeaway from this is: Atom's performance woes are most likely not intrinsic to the web-based platform it's built on, but are rather results of implementation flaws of Atom itself.

http://lighttable.com/

I once made the mistake of trying to edit a 50MB SQL file in Sublime Text. It took about 10 minutes to open. Once it had, every edit I made, every line I scrolled froze Sublime Text for another 2 to 3 minutes. I can only presume Sublime Text reads and renders the entire file at once, re-rendering on each edit, even if your window only shows 100 of the 10,000 lines. I love Sublime Text, however it is just impossible to use for large files. I suppose this is why we have stream editors like sed.

Or textpad. I regularly edit 400Mb files in that no problems at all, running macros and all sorts. Shame it is unicode challenged though :(

Sublime Text's performance highly depends on the line length of the loaded files. A 5 MB JSON file as one long line can kill ST, but neatly formatted in 50k lines is no problem.

For this reason, I generally use Notepad2 to handle larger text files (5 to 300 MB).

> stream editors > sed

Oh. Now I know where the name comes from!

It's also explained on the man page:

    NAME
          sed - stream editor for filtering and transforming text

FWIW the latest dev builds have changed syntax highlighting and the speed improvements can be massive.

My suggestion though is to turn of SQL highlighting and disable word wrapping when opening a large file (I do the exact same with SQL and CSV files).

There's a reason I stick to vim, which treats multi-gigabyte files as a supported use case. (My daily use case: McKeithing through bloody huge log files.)

May I ask the meaning and origin of the verb McKeithing?

Alternative medicine fake doctor. https://en.wikipedia.org/wiki/Gillian_McKeith#Diagnostic_tec... "the awful poo lady". A phrase for grovelling through log files. Anyone in the UK will understand the term immediately.

How long ago was this? I just opened a 40MB SQL file in Sublime Text2 and it took less than 40 seconds to open. Once open, editing and scrolling takes no time at all and searching takes maybe a second or 2. Doing a replace on a sting that appeared in every line took about 90 seconds.

I agree the Sublime Text isn't as good as some other editors when it comes to large files, but 50 MB is generally not a problem

This was about 1 year ago, Sublime Text 2.0.2, same version I run now. I'm waiting on Sublime Text 3 to be released before I upgrade.

I'm also running 2.02 and, as I mentioned, have none of the problems you described.

Funny how "untrendy" editors like Emacs (from the 70s!) handles 50MB files and larger without even raising an eyebrow, while all the "modern" editors seem crippled when asked to to do the same things.

I'll stick to what works, thank you.

I edited (not once) an assembly for AI that was almost 100MB in size in notepad. Took a few minutes to load, but then worked just fine. That was almost ten years back. I can't comprehend how something today can't open a 2MB file.

I just don't get the appeal for these not-really-native editors.

I bite the bullet and use a Java based IDE because frankly there are very few good ones that aren't Java based and support the languages I use, but if you're just using it as an editor there are plenty of native apps for Mac, Windows and Linux.

The attraction is having one editor on all the platforms which developers use (this is one of the few market niches where the users are seriously split between Windows, Mac and Linux). That means it can always be installed no matter what platform you're on, and justifies more effort than would be possible on any one platform, which may cover only a third or a quarter of the market. Also, I wouldn't say there's a clear distinction between an IDE and an editor. People want a modular program, which starts simple, and can be upgraded when the upgrade becomes useful or necessary.

Does this joke of an editor still spy on you with Google Analytics? Things like this (and MongoDB) really make me want to get out of this industry. The amount of pointless wheel-reinvention that goes on simply due to fashion and short attention spans is (or ought to be) scandalous.

Well. The comments here so far are about what one would expect. We're well on our way to making a 2MB+ file of "LOL USE VIM/EMACS" comments.

I've been using Atom for a bit now. I like it. I started out with vim but it never really stuck. I still use vim bindings in Atom for basic movement/yanking/pasting et cetera, but I admit, I am a GUI using creature and I like to grab the mouse every once in a while.

I rather enjoy the fact that I can poke at my editor just using a basic understanding of web development technologies. I enjoy that Atom was easy to set up and install plugins, unlike the dozen different ways you can/should customize vim. I already have a much more tweaked environment with Atom simply because slogging through .vimrc and vim bundles (even with vundle) was a major pain for me. I've never run into performance issues or the 2MB file limit (yes, I do have to deal with log files, but opening them in a text editor is not part of my workflow).

Clearly, others have different skills and priorities and Atom doesn't work for them. That's great. I'm glad there are options out there. Vim isn't a giant pile of shit just because it doesn't work for me.

I find it crazy that we still struggle to have simple things like a decent text editor. Speaks volume on the average quality of software in general.

A +1 for Notepad++ on this note.

Cute emoji in commit messages, along with this sort of achievement... OK, seriously, what is the Atom's raison d'être?

So you can edit your files on GitHub?

Which text editor do you use to open and edit 100+MB files (on Windows, Linux, OSX)? Which text editor can handle a 100GB text file and has a fast search and replace function?

On Windows notepad.exe, wordpad and notepad++ aren't really prepared for that task. Helpful would be an editor that streams in only a small part of a big text file.

On Windows I always used HiEdit http://www.winasm.net/free-small-fast-text-editor.html Written in asm and can open files including several gigabytes.

Vim does a quite good job too.

an editor that streams in only a small part of a big text file

vi does that.

ed, by Ken Thompson. It's actually extremely useful for this one specific task of editing particular lines in a large file. There's also sed if you want to perform arbitrary search and replace.

CRiSP was pretty good at that.

Visual slick edit claims to support files up to 2TB http://www.slickedit.com/products/slickedit/new-features

UltraEdit http://www.ultraedit.com/support/tutorials_power_tips/ultrae...

Reply on news.ycombinator.com