Lessons from Building a Static Site Generator (2020)
nicholasreese.com
nicholasreese.com
Out of laziness, most sites I run have their own 100-200 line Python static site generator that takes Markdown (if I'm really feeling it) or HTML files with Jinja templates and generates pages around them. The core generator code hardly ever changes year by year. Here's an example [0].
This isn't to say that everyone should always write their own. I am just a bit surprised by all the debate around each generator because they all produce the same thing and the only (or major) variables are the template language and what themes are built in (though of course you can always bring your own CSS).
But _using_ a static site generator is a very good idea. If you have no other inclinations, I think the stack that makes sense for anyone with multiple contributors is to use WordPress for editing and then have a plugin that will generate static pages from it so not every request to your site hits the database.
[0] https://github.com/eatonphil/notes.eatonphil.com/blob/master...
In my experience playing with several generators before building Elder.js it isn't so much about the output that matters it is about how the static site generator lets you do non-trivial customization. Things that would be hard without a larger framework.
More importantly, when building a major project on a static site generator, it is important to have an upgrade path beyond a static site generator should you require it. Elder.js was built with that use case in mind though I didn't cover that in the article.
Being able to move to SSR should the project require it is a huge plus in my book.
I think that's a bit reductive (especially since template libraries themselves are already complicated, not in a bad way). To me a static site generator is: 1) a string template library, plus 2) a file system walker/trigger to generate from a template, plus 3) additional data to feed the templates, plus 4) the actual content.
Then there's of course the additional features you may or may not need: a tagging system, a comment system, a subscription system, etc.
Thankfully despite all these components you don't need to write much of the actual code since they all exist as builtin (file system walking) or major OSS (Jinja, Mustache.js, etc) libraries. The SSG is primarily glue.
Indeed! For example, https://github.com/sunainapai/makesite is a simple and lightweight static site generator written in Python. It can be customized easily by modifying the Python source code and adapting it to one's needs. I like that when I need a new feature, I can add it quite easily by writing a few Python functions. It is meant to be programmer-friendly.
Disclosure: My wife wrote this project. I am just a happy user of the project.
The animating idea ("use this as a template but don't be afraid to customize or reinvent certain parts") liberated me from feature-by-feature evaluation of a bunch of complex config/templating systems.
For a small site like mine, getting the content right is the main thing and the site generator should mostly get out of the way. I update the site sporadically and don't want to re-learn a complex templating and config-file system every time I go back to it.
I maintain my site exactly the same way (minus the Jinja) and it is a workflow that works for me. Simplicity is best, even beating flexibility) when it comes to tools that help you express yourself. Otherwise you spend all your time wrestling with your tools rather than creating.
Code is available at github: https://github.com/john-bokma/tumblelog
Currently the input file is 1.4MB (828 blog entries) which Emacs can handle without any problems. Searching works very fast. I keep the file open and writing a new entry is just adding stuff to the top of the file.
Most of the static site builders I tried were either way too complex, or else just straight-up didn't work at all (looking at you, coleslaw).
I tried to go full emacs and use org export (org being my favorite text format) but the default export is horrendous and the documentation for org to html export is so bad it might as well not exist.
Software is simultaneously awesome and infuriating.
So after three days I just gave up on org-export and now I have pandoc shit out an html snippet that I concatenate with a hand-rolled html preamble and postamble via a Makefile.
Found a few rough edges, probably because org is underspecified. It's not elegant but it works for my use case.
It uses Scrolldown instead of markdown, which is simpler, cleaner, and incredibly extensible.
The command line app has just a few commands—-and they all take zero params.
A site is just a single folder, and because content is written in Scrolldown, it works great with git and is great for content sites or collaborative strongly typed databases.
It fast, I get about 300 pages per second, and not a lot of code (sub 1k excluding dependencies), and the code is tested.
It’s in nodejs now but no reason scroll can’t be language agnostic.
I’ve been around SSGs for over a decade and designing this one to be simple, reliable, and to stay out of the creators’ way. I think it could be the last SSG you’ll ever need.
pandoc \
--standalone \
--css=/style.css \
--highlight-style=code-highlight.theme \
--variable=lang:en \
--include-before-body=navbar.html \
--include-after-body=footer.html \
--template=template.html \
$MD -o $HTML
0: https://pandoc.org/MANUAL.html#option--standalone
1: https://github.com/jstrieb/personal-site/blob/master/compile...
Happy to answer any questions. I'll be going through and adding context to the questions I see.
> the ElderGuide.com team expects to maintain this project at least until 2023-2024
So nice to see the absence of a hype train, and also so nice to get an insight into your view of the design space
I would never be your market for elder.js. But I appreciate learning from you, and appreciate building software alongside you
It's infuriating to see blog posts or even news that don't clearly display the publish date near the title.
I agree they should probably make it visible on the article page itself...
Infuriating is a bit much
Ok maybe I was exaggerating :) but it really annoys me.
Nobody visits homepages anymore. :)
How to win friends and influence people was written close to 100 years ago, but it is still recommended in plenty of places.
The dragon book is older than me, but it is still one of the recommended books to write a compiler in.
Why does a blog post have to have a date displayed on it? If it is about a specific version of some software, I can understand, and agree with you, why it should be mentioned in the post.
Because human nature hasn't changed. Front end stuff changes every day.
1. Template system
There are tons of different template systems out there for things like the "shortcodes" in the article.
{{youtube id="123asdf4" /}}
<embed-youtube id="123asdf4"></embed-youtube>
While on the surface it looks verbose because of the closing tag, in most editors, you only have to press a key or two to close the tag. HTML5, strictly speaking, does not support self-closing tag syntax for custom tags. That syntax is only supported for void elements. There are only 16 void elements in HTML5.
I use "prefix-suffix" syntax to avoid ambiguity... any tag with a hyphen is obviously a custom tag.
2. Routing
Something you can use to tackle the routing complexity is to place your source files in the same path as the canonical URL. You only need routes for generated content, like index pages and such.
3. Index data
You'll naturally want to generate indexes and create previews for links. I suggest that you start by looking at the schema.org schema for web pages and work with a useful subset of that. This way, you can generate indexes on your web page using the same exact data, same exact schema, that you use for the JSON-LD data you provide for search engines like Google.
This is a minor point, but it reduces duplicated effort between the code for generating content for your website and the code for generating JSON-LD metadata.
Don't dive too deep into the schema.org schema, just take a couple bits and pieces that you need, and refer to the feature guides in Google's documentation:
https://developers.google.com/search/docs/guides/intro-struc...
> <embed-youtube id="123asdf4"></embed-youtube>
Thank you.
Just one comment: found implementing a Svelte Leaflet Map component a bit of a struggle - an addition to the docs on this would be very useful.
Hugo has shown me that my ideal generator will likely be built with Go, although I could see a C, C++, or possibly Rust equivalent being fine for my needs. However, what I desire more than anything is to have a really performant default template using pagination and hierarchy based on the file system.
I don't want to have to define titles or empty _index.md files to define sections, just want to point it at a nestable directory on my file system and use filenames for pages and directories for sections. This significantly improves the usefulness for anyone, because at that point it can be used as a knowledgebase/note-taking system from the get-go without needing any custom templating to make it work out of the box.
The other important factor is building in search capabilities. While I think that client-side JavaScript and a JSON output works for this goal and can be optimized, Xapian would be a great alternative and extend the usefulness outside of the JavaScript browserbase. So far Hugo has left serving production sites up to third-party web servers, but I see no reason why they couldn't finish implementing the `serve` components to be production-ready and implement a search component and possibly other dynamic components.
I really hope to see someone with a strong understanding of efficiency come in and own this arena building off Hugo's single binary approach, but with an open mind to make it robust to handle all kinds of self-hosted content. Even implementing a bookmark manager using this approach would be awesome, having it auto-sort YAML/TOML/HCL or whatever and be able to serve up searchable bookmarks without even relying on JavaScript. The possibilities are endless with the types of apps that can be built with this design using readable files as data for the content.
Traditional frameworks like Next.js, Gatsby, Nuxt.js all "fully hydrate" the client.
This means that every bit of HTML that is sent to the is browser is taken over by JS on the client.
This has it's costs but it is done to give interactivity.
Partial hydration is where you are only adding interactivity to the parts of the site that need it... think of it like the good old days of jquery but with a modern front end framework... Svelte.
In my experience, writing a static site generator is very much a continuous trade between flexibility and ease of use.
I also agree with most comments that it is often easier to write your own static site generator rather than finding the one to use amongst all the available choices.
That is why I made my own, and obviously I ended up writing a whole bunch of documentation because you can't escape the need to clarify how things work.
If you're interested you can check it out at https://yassb-foss.github.io/
I'm seeing "shortcodes" and I'm like -- as in variables and/or configuration files?
Or, more broadly -- why Javascript for the backend? This looks like a silly level of complexity. I'd start thinking about it in Bash and then probably head over to e.g. Python once databases et al start getting involved. What am I missing here?
People usually start their projects with the language they feel most comfortable in, not the one that is best suited for the project.
While Elder.js does allow for full server side rendering making "Javascript the backend" the goal of the static site generator is generate static HTML/CSS/JS that can be hosted from a CDN, S3, or other static host.
That said, one of the biggest pitfalls with building a major site on a static site generator is there is often no upgrade path to server rendering. Elder.js does offer that out of the box.
I could ask the same thing: why Python for the back-end?
Both languages have similar feature-sets and are roughly equally suitable to this task. Given that, a person should use whichever they're most comfortable with. Lots of people are comfortable with JavaScript these days, ergo there's lots of activity in the JavaScript SSG space.
That being said, I think what I mean by "e.g. Python" is specifically - "older slash more text oriented slash proven"
If you like Javascript, fine. But I think there's a case to be made that there is a completely unnecessary "bloat" to Javascript -- especially even as the author himself has suggested that at the end of the day it's all string concatenation.
If that's the case, (as it frequently is) it ends up boiling down to "what handles text well and in an established and smooth way," and Javascript does not score high there, I'd suggest.
> I think there's a case to be made that there is a completely unnecessary "bloat" to Javascript
"Bloat in JS" is something of a trope, and I'm not really even sure what you mean by it here. Typically people mean "too many JS dependencies are often drawn in for web pages", but that's not really relevant. Maybe you mean syntax bloat (method calls instead of list comprehensions)? If so, you're not totally wrong, but it's also not really a big deal in my experience. If you're talking about runtime/performance, well... V8 tends to be faster than CPython in raw compute (excluding native modules) because Google has put so much work into optimizing it (not that that matters much for a SSG either)
> it ends up boiling down to "what handles text well and in an established and smooth way," and Javascript does not score high there
Whenever I need to process some nontrivial text, the first thing I do is open up a Chrome tab for the JS repl. I think the following it pretty smooth:
const csv = `
Name,Email,Phone Number,Address
Bob Smith,bob@example.com,123-456-7890,123 Fake Street
Mike Jones,mike@example.com,098-765-4321,321 Fake Avenue`
const lines = csv.trim().split('\n').map(line => line.split(','))
const [headings, ...data] = lines
const objs = data.map(datum => Object.fromEntries(
headings.map((heading, index) => [heading, datum[index]])))
console.log(objs)
// output:
[
{
"Name": "Bob Smith",
"Email": "bob@example.com",
"Phone Number": "123-456-7890",
"Address": "123 Fake Street"
},
{
"Name": "Mike Jones",
"Email": "mike@example.com",
"Phone Number": "098-765-4321",
"Address": "321 Fake Avenue"
}
]
const header = `
<h1>Welcome to ${pageTitle}</h1>
`
Consider someone who starts off text-mangling in Javascript, and then later has a go at a more "classic" approach, e.g. command-line with Bash or Python or even Perl. Then consider the reverse; someone who starts off classic and learns/tries this Javascript approach.
I'd be willing to bet huge sums of money that a decent number of the Javascript people will end up adopting the classic approach, and that very few, possibly zero, of the classic people will adopt the Javascript approach.