Hacker News Reader

Comment by pkolaczk | Hacker News Reader

They debugged the problem to be excessive allocations in the tight loop of the TSV parser. If you allocate a separate object per each field, I can imagine how it can result in throughput counted in kB/s instead of GB/s.

We have had very similar issues with Java - excessive heap allocations can bring the system to a crawl and there is often no easy way out.

View on HN

They didn't debug anything. They just stuck a job queue on it and then rewrote part of it in Rust. That's the point of my comment. Using their code in the post (gist here[1]):

   $ pv lol.tsv | node lol.js
   1.71GiB 0:00:07 [ 225MiB/s] [================================>] 100%
   Finished reading the file.

Do we live in a world where engineers using a mature language like NodeJS with a pretty simple line-by-line file reading process and a string split think they hit a hard wall at 700kb/s, and the problem is not their bad code?

1. https://gist.github.com/orf/92bc26381e3dc00c8e96af262768591b

But is the code shitty? It looks like boring normal code to me. I'm not sure what point you're making here.

The point of the article is that you can write really boring normal code in NodeJS and it's really slow, or you can write really boring normal code in Rust and it's a lot faster, and it's not that hard to use the latter to replace hot paths in the former.

It's probably all the memory allocations. I know that when I do work like this, creating a huge array up front and assigning to that typically results in massive speedups.

I know basically nothing about Node though, so I could be completely wrong here.

Yeah that's one way to do it. Another is to reuse structures where possible to not force reallocation. There's a bunch of ways.

Node.js has a decent JIT, or so I've heard. I have experience with the HotSpot JVM. For the JVM this code would absolutely stop allocating on the heap past 10k iterations of the loop. "Split a string then grab the parts in a hot loop" is ground stakes for JIT compilers.

If they are seeing any GC activity once the loop gets hot, something else is going very wrong.

Yeah but if you're appending in a loop you'll need to allocate memory every time, right? Can a JIT avoid or rewrite this?

Yep the Hotspot JIT will absolutely allocate on the stack and reuse memory allocations inside loops. It also has escape analysis and it will detect you don't need the array of strings after the iteration. In that case it usually just allocates the strings directly at their final locations on the record object.

Wow, that's super cool. Almost makes me wish i did stuff on the JVM (but until someone rewrites LAPACK for it, that seems unlikely).

> But is the code shitty? It looks like boring normal code to me. I'm not sure what point you're making here.

1. It should never be that stupidly slow

2. That should be obvious to anyone with a napkin and some math

3. Excessive allocations are highly unlikely to be the root cause of the slowness

4. Instead of debugging, they seemingly spent a lot of time putting it on Kubernetes, scaling it out to 25 instances and then finally rewriting a portion of it in Rust.

The whole thing smells of confusion and multiple issues being conflated together, with the rewrite in rust fixing these by virtue of deletion.

And if the root is was the GC, then some `--trace-gc` output in the post would be nice at the absolute minimum. Showing a for loop with a string split doesn't cut it.

Exactly. I'm not a super-expert in Node.js but worked a lot with it and also know a lot of internals.

And I can definitely say: IO speed is pretty good. To be honest, I've never come across a language with bad IO speed and I've seen quite a lot of languages.

I also did some benchmarks (like orf did) and speed is pretty good. Maybe Rust is a little bit better because it's nearer to syscalls than Node.js is but we're not talking about 760kb/s vs. 250mb/s.

200GB of logs and 25 instances means 8GB per instance. Not sure what they're doing, but processing of 8GB taking 3h is like doing it with my 80286 processor ;-) Even with inserting stuff in MongoDB this seems way to slow. And they only replaced the file IO part with Rust...

I guess the records array is getting pretty huge which might result in swapping to disk. But then: don't let it happen, just flush it to the database or whatever. I believe the problem could have been solved with Node.js without issues.

The point the parent is making is that the boring normal code in NodeJS is not slow, at least not as presented in the article. The bottleneck is probably somewhere else. Allocation is usually not slow in a garbage collected language. Deallocation can be, but that is usually easy to check (for example with the --trace-gc flag, or running with profiling enabled and checking the results with chrome devtools). It would have been nice if they shared such analysis.

With the limited information provided in the article, there are other points that I would check before going as far as writing a module in another language. For example

> records.push({

They are collecting all the parsed rows in a single array. This is going to consume memory proportional to the size of the file, potentially in the same order of magnitude (depends on how much information is being collected and how much discarded).

The low throughput is definitely a code smell.

In my experience, a common mistake in handling files too large for memory is to read the data faster than you process it, consequently running out of memory.

For example, maybe they have a loop reading lines and doing a bunch of async DB writes: your process will read at disk speed and write at database speed (relatively slower), and you'll run out of memory due to async tasks/context queuing up in memory.

Perhaps this is why it worked to split the file into 8GB units X 25 servers: it didn't blow their memory budget per server.

Try it with something a little bigger than what you used. Then you would have seen that the GC kicks in and performance goes down after about 2GB of processed data.

With my first try with bigger data it blew up with an out of heap memory error. The article mentions them having memory problems as well. They probably operated at the upper levels of available memory and even got into swapping and then 700kb/s is not surprising.

So yes, this absolutely looks like a memory allocation issue.

Yes, of course the unbounded array will cause you to run out of memory. That’s… not news, and it absolutely isn’t a “memory allocation issue” in the sense they are using it in the post.

Try modifying it to clear the array (a DB flush, if you will) every 100k lines.

That's what you get if you use the (simplified) example code that is in the article. It's also what you can derive from their description of the problem.

Now we don't have any Rust code to compare so we can't say how it's different. But everything points towards a memory allocation issue that is just not their in simple Rust code.

Following you in the abstract, but still a bit confused.

* "unbounded array" - are you referring to `records`?

* If so, is the memory of `records` too large? Or is it that `records[i].pathname` has a reference to `fields[7]`, AND v8 will retain all copies of `fields` until the loop is done?

Do you think this code would be more GC-friendly?

```

let count = 0;

const MAX = 1000;

const flushedRecords = [];

for await (const line of readlineStream) {

  const fields = line.split('\t');

  const httpStatus = Number(fields[5]);

  if (httpStatus < 200 || httpStatus > 299) continue;

  records.push({

    pathname: fields[7],

    referrer: fields[8],

    // ...

  });

  if(count == MAX || isLastLine(line)){

   flushedRecords.push([...records]);

   records = [];
   
    count = 0;

  } else {
    count++;
  }

}

```

Or were you thinking something totally different?

We know a few concrete things:

1. It parses 200gb at an average rate of 700kb/s

2. The job splits strings into a format that is loaded into mongodb

3. There where OOM errors that went away with a smaller workload

None of this points to an inherent issue with NodeJS and the way it references memory.

The speed issue alone points to a different problem root cause, and the post is completely lacking in concrete analysis.

If v8 is being dumb and retains references to huge strings, then understanding this is the case and forcing a copy (and thus enabling the gc to reclaim data) is quicker and easier than any of their attempts at fixing the issue.

Splitting a small TSV file and loading it into mongo can be done with even the most memory inefficient language without issue.

"and the post is completely lacking in concrete analysis."

I do agree here, but I think it's telling that the rust solution is, without analysis, just better.

Is this the best developer or best case study? Perhaps not, but it is interesting in that little analysis was needed.

In the case where you have some constraint placed on you (running in the browser, company policy forbids rust e.g.), the analysis is worthwhile. Otherwise, assuming the fix didn't take long (relative to the requisite analysis and dev training), the decision to switch to a less shoot-yourself-in-the-foot language seems reasonable to me.

I think we can all agree that the blog article, while interesting and tells a story about the way the author solved a problem, does not have high enough rigor to serve as a benchmark for "don't do this, do this instead", or "this language is bad at this, and this one is better". Those types of statements would require much more rigor and A vs B comparisons where code A and code B are the same, just in different languages.

BUT, what I am interested in - if I'm using javascript to parse a large file line by line, plucking off specific fields like the example does, what can I try to see if it's more GC-friendly? Obviously the answer is always some flavor of "profile it" and "it depends". If the blog author could have gotten a big boost in performance by adding 10 lines of js code to their loop, that is something all readers of the article could benefit from.

I will play around with it and see what I can learn about the V8 GC. For reference: here is a talk about profiling the ruby gc, which helped me fix a memory leak at work (https://www.youtube.com/watch?v=kZcqyuPeDao).

There's a lot of other considerations when adding a new language. Questions come up such as:

* How long does it take to ramp up on the new code base (author notwithstanding as a user of Rust outside of the company)?

* If the author of the code is sick and there's a bug, or logic needs to be added, how hard would it be for someone else to do this work?

* What is the build story for the Rust code like? How does it integrate with existing CI?

* How do you store the Rust library? Some artifact store that Node can interact with?

Without more details, unless the company was already writing Rust, this doesn't seem like a reasonable choice at all. Doing some profiling and experimenting with allocation, along with comments as to why you're doing something that's not naive (e.g. appending to a large array) seems much cheaper than doing all this Rust work.

I've been in situations at $WORK where we were locked into using Python for certain code. Rewriting portions of it in Rust (if it needed running in the runtime) or Java (if it was across a service boundary) would have been enjoyable. But answering all those questions made us stick to Python. And surely, in most of those instances, we never really came back to those hot loops again so it never needed subsequent work. There were a few cases where we did find ourselves constantly trying to eke out more performance and in most of those cases a rewrite did end up going through, but those are rare and part of the craft of software engineering is to identify these areas and get ahead of rewrites early enough to not waste cycles.

> They just stuck a job queue on it and then rewrote part of it in Rust

IMO that's not an inappropriate response here. You can make this code in Node, but it's super-awkward trying to reduce string allocations in JavaScript. Much easier to rewrite in Rust where such things will be fast even with the naive code. For code such as this you can likely do a 1:1 port of the Node code and it will be 10x faster. Which is much better than getting yourself in twists trying to make Node fast at tasks it's not good at.

I took the code from their post, the one they highlighted as the issue, string allocations and all, and ran it. It was 1000x as fast as their self-reported runtime.

Does this really, really not scream “there is another issue” to you?

Do you really think, even with string allocations in a hot loop, NodeJS with its very optimised runtime would struggle to hit 1mb/s?

Naive Rust code is super-awkward compared to naive JS code. I doubt one can do 1:1 port of anything more complex than Hello World (even FizzBuzz may get you tripped up).

Not sure what you mean by naive rust, but rust has good webassembly integration. To give you some context, here's a post I made about rewriting 10k lines of js into rust: https://www.reddit.com/r/rust/comments/k3jy5g/i_rewrote_10k_... (I went on to rewrite the server from nodejs to rust too)

YMMV, but my Rust code ends up looking pretty much identical to my TypeScript code most of the time. And especially for code like this where there are no complex lifetimes.

I disagree. At my job I rewrote an environment variable injector module written in JavaScript into pure Rust. Not only was it straightforward, but I'm fairly certain I finished the Rust port faster than it would have taken to port it another language like Python or Go.

A common technique (at least in Java, never seen it in nodejs) is to have a pool of pre-allocated objects and just reuse them as they are discarded. Essentially manual GC

Happens JS world aswel SimonDev did a video about it, this one I think: https://www.youtube.com/watch?v=easvMCCBFkQ

This channel is a real gem, show what talent really is: https://www.youtube.com/watch?v=a1L7k35EHIc

Thanks for the vid links, always looking for new channels with informative content.

It is easier said than done, when the whole ecosystem is based on libraries and frameworks that allocate like crazy. Preallocating and reusing introduces complexity and increases likelihood of bugs. And at least in Java you can run into surprising GC troubles with pooling as well. Object pools tend to be promoted to a tenured generation and if you do it too much, you essentially break the generational hypothesis many GCs are optimized for. Generational GCs don't like dealing with objects with long lifetimes.

I recall hearing there's something of a parallel ecosystem of near-zero-allocation Java libraries, used in HFT among other things. I'm unsure why people use a GC'd language in the first place if they're going to go to those lengths to avoid it, but presumably there are good reasons (perhaps memory safety, before Rust came along)

You use it only for the one class you are allocating a million times per second and let the other ones be GC, often this technique is used within a single heavy function with a tight loop

Most of your application probably doesn't need it and the tight loop might not be big enough to be worth using JNI to write in a non-GC language (as well as maintaining another compiler stack for)

Ideally the runtime should allow you to manually manage memory when you need to, like some keyword that lets you manually keep track of an object lifetime. But as far as I know most GC languages don't have that

Object pooling is typically not a common pattern in Java these days outside of HFT stuff, and even in those cases it’s often not performance, but determinism that’s the issue. (Connection pooling is still common as with other expensive objects, just not plain object pooling.) Java’s garbage collectors are excellent if you cut with the grain.

Yeah, you would only use it in very tight loops in special cases, ideally only within a single heavy function. It would be madness to do object pooling across a large scale codebase

That reminds me of how iOS could achieve fast scrolling of lists, which Facebook and Twitter struggled with for years because they tried to stick with HTML. Basically if you scroll down, a list element from above is moved to the bottom and has its contents replaced.

Anyway, I'm not sure if this is in Java already or is upcoming, but Java will finally add value objects, that is, stack variables instead of just making everything a heap object. For tight loops that will be much more efficient, since it can just reuse the same memory, stay within the cpu registers, etc. (I'm talking out of my ass btw from a superficial knowledge of Go and Java and half-remembered theory from college).

Virtual scrolling is a very common technique for rendering large tables or lists in HTML (and pretty much any other environment), and is by no means unique to iOS...

And also common in Android from the very beginning of their ListView now called RecyclerView.

The Windows 3.1 list box supported it.

By the way the very same code in Java would absolutely rip through the files with no extra allocations of arrays once the JIT gets seriously involved. A big loop with temporary arrays in it that are used to populate the fields of a record every iteration is HotSpot C2's favourite thing.

Does node not similarly allocate on the stack in hot loops?

Did you actually test it? Some time ago someone at another forum made a similar claim that returning an Optional from a method in Java will be surely optimized as we well, but it turned out that it wasn't (https://pkolaczk.github.io/overhead-of-optional/).

Good point, I should probably check.

Reply on news.ycombinator.com