Comment by curveship | Hacker News Reader

Comment by curveship | Hacker News Reader

I just spent an hour looking at the Imba benchmark. Yep, it's cheating. Which is a shame, because I really like the framework as a whole.

The vast majority of the speedup comes from a single sneaky line of code. The majority of their "Everything" benchmark's time is spent in the reorder step. They've implemented this as "remove a random todo, render, push the removed todo back onto the end, render." The Imba implementation, and it alone, caches the rendered todo view, so that they can re-use it once the todo is reinserted.

This single optimization is responsible for the vast bulk of their claimed speed. Removing it puts Imba only 2x faster than React, not 60x.

If you want to try it yourself, look at line 55 of app.js. Change:

    res.push((this['_' + todo.id] = this['_' + todo.id] || t$('todo')).setObject(todo).end());

... to:

    res.push(t$('todo').setObject(todo).end());

Furthermore, this isn't a caching strategy you'd want to use in a real app. It holds onto all DOM nodes ever created, thereby leaking quite a lot of memory.

Again, I think Imba is cool, and fast, just not otherworldly fast. I hope this was just an "oops!" and not an intentional misrepresentation.

Hi there.

This is utterly wrong, and if you had cared to read about what the benchmark is trying to achieve, you would understand (https://github.com/somebee/todomvc-render-benchmark).

You cannot simply remove caching and reusing nodes from the benchmark (which you do with that change). This is the way Imba does diffing, and it would be akin to removing the React virtual dom!

As we mention in the readme: "Even though it looks incredibly boring, the "Unchanged Render" is possibly the most interesting benchmark. All of these frameworks are plenty fast if they only render whenever something has changed. But if the frameworks are fast enough to render the whole app in 0.01ms, you could skip thinking about all sorts of tracking to keep the view in sync."

In a real world app you do not create 1000000 todos. The actual data rarely change that much. As for purging the cache, see comment: https://news.ycombinator.com/item?id=10092454. Quote:

One thing to be aware of is that Imba doesn't automatically remove the cache for you, because we've found that it's tricky to determine when you actually want to purge it. For instance: if mouseOver <something> else <something-else> Just because `mouseOver` becomes true for one frame doesn't mean you want the `<something-else>`-tag to be purged from the cached and removed. Keeping it around is nice for performance and convenience (e.g. state, jQuery plugins). In practice it turns out you want to purge whole pages/sections at the same time, which is way friendlier for the garbage collector as well.

If you _really_ want to not cache things this way, you can change the line to:

    res.push((this['_' + i] = this['_' + i] || t$('todo')).se ...

Which would only ever cache as many dom nodes as there are tasks, but change which nodes are used for which tasks.

I wonder how React with Immutable.js would fare in "Unchanged Render" benchmark.

I now ran the benchmarks with the this['_' + i] change (which disables per-task caching, but does not really remove any/all caching alltogether. Imba is still 35x faster than react on the 'everything' benchmark (and even faster on the others). I still would write my apps exactly like they are in the original benchmark, but do you agree that the ['_' + i] change removes what you call 'sneaky code'?

UPDATE: Since the performance was just as good with this dumber type of caching I have changed the actual benchmark to work this way. Would you still consider that caching sneaky? If so, I'm not sure what to say. Yes, Imba caches dom nodes for reuse. That is the whole philosophy behind its 'virtual dom'. Now it does not 'leak memory' anymore either, even though this 'leak' is a feature (ref comment about manual pruning) and not a bug.

Reply on news.ycombinator.com