Hacker News Reader

Blades currently doesn't break the sitemap, but it can easily be implemented. I see that 50k pages is a maximum for one sitemap file. Do you think it should break into separate pages only after reaching 50k, or sooner?

IMO the easiest implementation would always break into an index and sitemap(s) as needed. 49,999 URLs? 1 index and 1 sitemap. 50K+ URLs? 1 index and sitemaps as needed. This gives you the most consistent output and users who cross the 50K threshold over time will not need to update their sitemap in search tools to be compliant since pointing to the index will already deal with that.

You are not wrong, although I would argue that use cases where you have 50k blog posts (or pages) are not that common...

I should have been more clear on the terminology - 50K URLs. Which doesn't always mean 50K posts or pages, but those plus all your taxonomy pages plus all your pagination pages. A 10K post or page site can give you 50K URLs pretty easily if it uses taxonomies and pagination.

My case might be unique where the site is being built at the end of a pipeline that sorts through millions of rows of data and gives you the content you want based on your filters, but it's honestly not difficult to get to 50K URLs for many sites that have some age on them.

I would say a 10k page or post site is also very rare.

Nah, you'd be wrong. I use a password protected static site as my knowledge dump. I accumulate 1000s of posts in months.

You doing it is not the definition of ‘not rare’ :P

I think the conversation here is getting a little sidetracked :)

(not indicating Aeolun specifically, just replied to the lowest level)

Seems to me like every newspaper would accumulate 10k pages within a year or two?

I think static generators are mostly used for personal blogs or pages, but I'd love to hear about newspapers that use them!

Just run my company's internal Hugo build on commodity hardware (4 cores), and I get about 1000 pages per second, i.e. a 3966 page site took "4841ms" - so 20,000 pages assuming linear scaling, would still only 20-30 seconds. Not minutes. Perhaps our theme isn't complex enough?

Interesting. Are you using Related [0] content? I always suspected the long build was due to related content but the build profiler isn't specific enough to point that out.

[0] https://gohugo.io/content-management/related/

I've faced this issue as well.

I run a Python script after the build has completed that reads the entire sitemap and chunks it into parts of 50k.

+1 for mentioning incremental builds as well. These two features are often overlooked and people just ask "why do you even have so many pages?" and it ends there because nobody wants to entertain the power user.

A third disadvantage I've seen is search. I use fuse to generate a JSON for client side search but it takes a lot of memory to render and is a cause for concern.

What kind of sites are you building with static site generators that has 10K+ pages of content?

I always got the impression that they were more for small blogs/sites.

You can take your slow dynamic site and throw some caching in front of it or you can just skip all the moving parts and make the whole thing static from the start and let nginx do what it does best and serve static files super fast. I've done a lot of work for media and news sites that previously ran on WordPress and it is not uncommon to see 10K posts or more. I would argue static site generators are considerably underused for many types of sites that genuinely don't need an active datastore connection.

This is actually very interesting. Can you tell more on the recommended stack for that kind of websites? You said media but I guess also directories

There are so many options but it genuinely depends on how your content is being generated. If you have everything in a giant CSV or JSON file then there's not much work to do at all. Many shops I worked with were used to WordPress and after WP released the API functionality it became possible to completely decouple the editing & publish function from the front end display. You can still use a securely locked-down WP install for content creation and editing, then just use the (also locked-down) WP API endpoint as your data source that gets fed into your site generator. Hugo has an awesome feature where if you feed it a URL it will go fetch your content for you, so it doesn't even need to exist in the /content dir prior to build. Thanks to the free tiers for Gitlab CI/CD or Github Actions, your repo only contains theme code and with a 10-line CI/CD config file and a container image that has Hugo installed, you just need a trigger to run the build pipeline - then you're left with the static site that you can publish anywhere you like.

If you don't need the user roles and editor experience you get from WP then you can feed your generator the content from any source you like as long as the files have the correct frontmatter to turn them into pages/posts. You might need a little glue script here and there to chain systems together, but it's not super difficult to adapt to whatever pipeline you need after you understand where your content comes from.

Shout-out to https://wp2static.com/

Why are you assuming a dynamic site is slow?

Everything after the first request is cached by a CDN anyway so the only difference is when that original page is generated. It only takes a few milliseconds to generate the page from a database vs reading a file from disk.

The bigger advantage to static sites is portability (even though you can have a webapp that reads markdown files just as easily as database rows).