Hacker News Reader

Comment by PinkPigeon | Hacker News Reader

> A Highly Optimized Build Process: that will span as many CPU cores as you can throw at it to make building your site as fast as possible. For reference, Elder.js easily generates a data intensive 18,000 page site in 8 minutes using a budget 4 core VM.

How does this compare to something like Hugo?

View on HN

Eight minutes!? That’s over 100ms of CPU time per page. I’m baffled and more than a little horrified that you would call that highly optimised. I would honestly expect anything calling itself highly optimised to be doing that in closer to 8 seconds. 1.7̅ms is ample for something carefully put together, probably even in JavaScript and with a database (fetch everything up front in one or so queries), though it’ll certainly be easier in a language like Rust. I admit I’d be decidedly impressed by a reasonably-featured JavaScript SSG doing 18,000 pages in 8 seconds; but more practically, in a JavaScript ecosystem, I’d be targeting definitely under a minute.

I will admit I’ve grown used to Rust code and code that’s been designed to be fast. (Zola’s not that, by the way; it makes many decisions in favour of subjective ergonomics and flexibility at enormous runtime cost, as well as doing a lot of theoretically-unnecessary cloning and such all over the place, which could speed things up a lot if it were sorted out somehow (I haven’t contemplated the matter). For my own website with some admittedly very complex templates, and that’s probably Zola’s slowest part, it’s scaling at about 2ms per page with a warm disk cache. Actually, it does some multithreading, but much of the most expensive stuff in the process isn’t parallelised, so repeated runs average using almost a quarter of my 16-thread laptop. I feel I should multiply that 2ms by something, but I’m not sure quite what!)

Edit: Later on the page: “As of October 2020, ElderGuide.com has ~20k pages and builds in 1 minute 22 seconds.” 4ms wall time (16ms if still operating on four cores) is much more encouraging. I guess the page could do with revision. Later still, “Build times on our ~20k page site are routinely less than 10 minutes on a modest VM.” which seems to be the first set of figures again.

I get what you are saying.

In that example most of it was waiting on the remote database. (I’m ans SQL newb).

That said, Elder.js makes it easy to see what is causing performance hits with full perf tracking.

These days that same site is 1 min 22sec.

General rule of thumb with databases is to reduce the number of queries. Fetch large batches of information. This can be difficult to arrange with how such systems are commonly built and/or coded; some sort of intermediary between the consumer and the database can be warranted, which you can tell, “oh btw, these are the items that I’m going to want, d’you mind just fetching them all now so you can hand them directly to the consumers when they ask for them rather than issuing a new query for each?” In a well-done system like this, the database will be a minor component in the performance—templating and other JS-side transformations are much more expensive than fetching easily-identified rows from a database that’s been designed to handle zillions of requests per surprisingly-small-unit-of-time.

I’m curious what fraction of the 1m22s it now takes is waiting for the database.

Building performance tracking into the product? Good job! That shows me that you’re genuine in caring about it, which is still more encouraging. I’m going to investigate Elder.js.

> much of the most expensive stuff in the process isn’t parallelised

Are you sure? Markdown and template rendering of all pages/sections and image processing is done in parallel. I expect those to be the most expensive stuff. The things that are currently done sequentially are taxonomies (an oversight, the pages in them are rendered in parallel) and the search index generation (oversight as well).

No, I’m not sure; I apologise for being wrong (since you know what you’re doing). I just went searching for signs of threading and rayon and thought I was only finding it on the earlier parts of the process and image stuff. But I wonder why repeated runs of `zola build --drafts` on my site with no images for it to process average less than 25% of total CPU usage? I would have thought >170ms was long enough for it to hit its stride inside each process.

> Eight minutes!? That’s over 100ms of CPU time per page.

Your calculation seems wrong.

(8×60×1000)/18000 = 26.666667 milliseconds per page

Four CPUs were involved, so multiply that figure by four.

It beats it. Here are the benchmarks one of our users ran: https://github.com/Elderjs/elderjs/discussions/166

That's pretty impressive. Hugo appears to have a steady growth in overhead right up until the point where elde.js overtakes it when the number of pages hits 10,000.

I was wondering the same. How does this end up differing from SvelteKit using Vite?

(Elder.js Author)

SvelteKit's offering is great. Much better than Sapper. For Vite the HMR on SvelteKit is better than Elder.js but SvelteKit doesn't offer Partial Hydration.

The main difference is Elder.js is designed for static sites and offers tools to help make building large static sites easier.

For instance, when it comes to building non-trivial static sites, there is a lot of data massaging that needs to happen and be in sync across the entire project. A good example is when reading from a headless CMS or generating a sitemap.

With Elder.js, you can massage this data once and add it where you need to via a hook and it will be available on all pages.

SvelteKit is less opinionated. It is apples and oranges.

edit: a word.

Can't you do partial hydration by setting certain components to not hydrate inside the <script context="module'>? Thats what it says in the docs but it might not be the same thing in my understanding (which is limited). Also aren't app-wide stores basically the least trivial way of keeping data in sync across an app? Certainly not difficult in SvelteKit as it is - in fact I can't imagine it being any easier.

No, SvelteKit does not have support for partial hydration.

What you're describing is the no js option in SK. That doesn't allow you to selectively add some JS, it's either all or nothing. You set this option on a per page basis, not per component.

The massaging of data that Nick talks about here has to do with data that is needed to build the actual pages, not runtime data (which you could use Stores for).

https://github.com/sveltejs/kit/issues/1390

Reply on news.ycombinator.com