Hacker News Reader

Comment by jez | Hacker News Reader

A problem with this approach is that deleting a file from source/ does not delete it from build/.

In my own projects, simply rebuilding the whole site is fast enough, so I opt to remove the whole build folder before a rebuild:

https://github.com/jez/jez.github.io/blob/source/Makefile#L1...

This defeats a big part of why you’d want a build system in the first place (incremental builds), but at least if you know the page you want to regenerate you can still `make` that file directly.

If there’s a common workaround for this pattern in makefiles I’d love to learn it.

View on HN

Not sure if it’s a common pattern, but my solution to this was to always run a command that deletes all “unexpected” files, using GNU Make’s “shell” function to enumerate files and the “filter-out” function to filter out “expected” outputs. Edit: I ensure this runs every time using an ugly hack: running the command as part of variable expansion via the “shell” function.

Edit to link my Makefile: https://github.com/jaredkrinke/make-blog/blob/main/Makefile

File deletions and renames are common problems with many revision control / build systems.

Other than the nuclear option ("make clean"), another is to have a specific rename / remove make target, so:

  make rm sourcefile

  make mv sourcefile newsourcefile

... which will handle the deletion and renaming of both the original and generated targets.

In practice for even fairly large blog and online projects, a make clean / make all cycle is reasonably quick (seconds, perhaps minutes), and is often called for when revising templates or other elements of the site design. If you're operating at a scale where rebuild time is a concern, you probably want to be using an actual CMS (content management system) in which source is managed in a database and generated dynamically on client access.

> If there’s a common workaround for this pattern in makefiles I’d love to learn it.

"make clean"?

How does that solve the problem? That forces a total rebuild, which is exactly what he said he didn't want.

Yes, I didn't read properly.

I guess you could do some magic to delete "unexpected" files, but are there tools which do solve this problem?

The cleanest way to do it is essentially "make install". You do all the heavy build steps into a build directory, and then the final stage is to delete the "output" directory and copy all the files you need there. Incremental builds should still be pretty fast since the only repeated action is copying files (and you could link them if you want instead).

Conventionally, install puts the outputs in their where they will live for use. It does so in a way that 'make uninstall' will leave things as they were before install. The install target should also run any pre- and post-install commands. There's also a 'make dist' convention, to build a release tarball.

This is the way, because intermediate build artefacts also end up in `build/`. You don't want those in your `output/` directory, but you also don't want to delete them because they help speed up the incremental builds.

Edit: `make install` also protects you against broken builds breaking your live site.

Something like this should do the trick:

  rm/%.html:
    @rm -f source/%.html build/%.html

Run with:

  $ make rm/page.html

The shake build system (a general-purpose build system similar-to/better-than make) has a "prune" feature for exactly this purpose:

http://neilmitchell.blogspot.com/2015/04/cleaning-stale-file...

But I think the best solution (that also works with make) is to have a "make dist" target that creates a final .tar.gz archive of the result. If the rule is written properly then it won't contain any stale files. The disadvantage is for large project it may be slow, but you are not supposed to use this rule during development (where it is useless anyway), only for releases (which still can be built incrementally -- only the final .tar.gz needs to be created from scratch)

Not sure if anyone actually uses it, but I would approach the problem with find, comm, and a sprinkle of sed:

    comm -23 <(find build -type f -iname "*.html" -printf "%P\n" | sed 's/\.html$//' | sort) \
             <(find source -type f -iname "*.md" -printf "%P\n" | sed 's/\.md$//' | sort)

The find commands get you a list of all the files (and only files - directories will have to be removed in a separate step) in each of the build and source folder, sed chops off the extension, while comm -23 compares them, printing only the files unique to the build folder, which you can then deal with as you see fit (e.g., by feeding them to xargs rm).

Using comm was exactly what I did use in my little experiment of a barebones SSG.

I did save a list of generated files and compared them. This one liner is the meat of the whole solution:

  comm -23 <(awk 'NR>1' "$DSTDIR/build-info") <(find "$SRCDIR" -name "*$EXT" -type f -printf "%P\n" | tee >(gen_index) | xargs -n1 "$0" "$SRCDIR" "$DSTDIR" | sort | tee -a "$DSTDIR/build-info.new") | (cd "$DSTDIR" && xargs rm)

Full source here: https://gist.github.com/hadrianw/060944011acfcadd889d937b960...

I think the best solution is to use something like webpack or vite or whatever. These usually have their own dev server and can watch directories for changes.

My personal site is also using a custom make-like ssg, but after spending a disproportionate amount of time writing the bundling/packaging code, I decided to just switch over to one of these tools. It’s a solved problem, and it greatly reduced the complexity of my site.

I use Nix, so I get incremental builds and your problem goes away.

You’re using Nix to drive your static site generation? If so, please share more details because that sounds intriguing!

Here you go: https://juuso.dev/blogPosts/nix-as-a-static-site-generator/n...

Wow, I ask and a blog post just appears!

My initial reaction is: I should probably get around to learning about Flakes. I’m not sure I’d want each blog post to pin its templates, but it’s nice to have that choice.

A flake is a function, right? Invoking `nix build` resolves its inputs and computes its outputs. So if the static site is the output, wouldn't the content have to be one of the inputs?

Personally, I think I'd use two flakes, one that builds the content into something that's ready to hand-off to code, and a second one that turns it into a usable site. That way you wouldn't end up with a new input for each post, but instead would have a versioned something which represents your content all bundled together, and then the site builder consumes it as a single dependency--but conceptually it's the same.

I guess I'm just saying that it's not conventional, but it's a pretty logical conclusion to reach.

I was just reading this and thinking that `nix build` would do the same trick even more nicely.

I posted a link to the other comment, but here it is for you as well: https://juuso.dev/blogPosts/nix-as-a-static-site-generator/n...

I admire how nimble you are. I aspire to write blog posts at the drop of a hat like this, but I rarely do.

I also like the use of flake inputs for content.

It reminds me of a world that I've been imagining where the conclusions in scientific papers are generated as a flake outputs (an adjacent output would be the human readable thing, a PDF or whatever).

In this world, you can just run `nix flake update && nix build`, and if a paper that you cite published an update which invalidates your conclusion, you know right away because their output is your input, so your build fails.

We think about repeatable builds being for executable binaries, but they could equally well be for conclusions and assumptions.

Perhaps nix is too big of a hammer for the job, but it seems like the best shot we have at achieving this without also constraining the scientist re: tooling.

I realize that you don't want to be storing mountains of data in the nix store, but it would work just as well if the output in question is an IPFS CID, to be resolved during the build instead. The publisher can then be in charge of keeping that CID pinned and of notifying scientists when they're "build" starts failing.

> I admire how nimble you are. I aspire to write blog posts at the drop of a hat like this, but I rarely do.

Thanks! I took up blogging more often as of recent, and for me, having a manageable system is a large part of that. The last thing I want to happen on a Sunday evening is breaking some page of my website. That being said, I hope to one day make the workflow easier.

> It reminds me of a world that I've been imagining where the conclusions in scientific papers are generated as a flake outputs (and adjacent output would be the human readable thing, a PDF or whatever).

I happen to be a reviewer for software artifacts in a scientific journal, and I often use Nix here. Not that many projects do use it, but if I'm able to reproduce it with Nix, then I know the author has not missed any implicit dependencies. I like to imagine it's also useful for the authors as a feedback, whether they use Nix or not.

> I realize that you don't want to be storing mountains of data in the nix store, but it would work just as well if the output in question is an IPFS CID, to be resolved during the build instead.

I maintain separate build serves of my own using Nix integrations and the Nix cache is quite large already (so called remote builders) sitting at around 500GB. I host these at Hetzner.

I have also thought adding IPFS integration for my website, but haven't got around to it.

> I happen to be a reviewer for software artifacts in a scientific journal

That's very cool. I have a question for you.

I'm taking a bioinformatics class, despite not having the chemistry prerequisites. I'm getting a crash course in biochem, and the rest of the class benefits from having an expert in what-kind-of-quotes-to-use.

I've been thinking: would it be helpful if the care and maintenance of these compute environments wasn't left to each scientist but was instead aggregated (perhaps per-class or per-university)?

We're setting these chemists up with conda in Ubuntu in WSL in a terminal whose startup command activates the conda environment. Not exactly a recipe for reproducibility after they get a new laptop.

What if certain compute-heavy classes published flakes which the students could...

a) use while taking the class so we stop wasting time on troubleshooting ssl deps via conda

b) reference in publications after the fact. They could say:

> Here's a Jupyter notebook, download it and run it in the UCCS biochem environment like so: `nix run github:UCCS/CHEM4573?rev=16afd67`, its output lets us make the following conclusions...

I know it would be helpful for the students in the class. Do you think it would be helpful to them later on when they were publishing things?

I'm thinking about packaging the dependencies for this class, giving it to the teacher, and pitching it to the university:

> Set up a technical fellowes program. Waive tuition for us nerds and in exchange we'll support your students and faculty through the maintenance of these environments.

I don't mind paying tuition so much, but I'd like to do something to get a bit more cross pollination going between scientists in need of tech support and techies in need of something meaningful to work on.

Am I dreaming here, or would it solve some problems? Do you think I have a shot at convincing anybody?

Couple of thoughts:

> I've been thinking: would it be helpful if the care and maintenance of these compute environments wasn't left to each scientist but was instead aggregated (perhaps per-class or per-university)?

This is definitely something that Nix can abstract quite well. In my company we have [an infrastructure of computers](https://github.com/ponkila/homestaking-infra) that we manage with NixOS. We have gone over the system such that `cd` into a directory "conjures" the environment using devenv or direnv. We don't do anything too fancy yet, but we have a project commencing next month in which we start to also manage routers this way. We speculate that this will help us to do things such as follows: register new node, and it gets automatically imported by the router which establishes DNS entries and SSH keys for each user. The idea is that we could have different "views" of the infrastructure depending on the user which the router could control. For administrators, we have a separate UI created with React that pulls NixOS configuration declarations from a git repository (note: these don't have to be public) and shows how the nodes connect with each other. The UI is still under construction, but imagine this but now with more nodes: https://imgur.com/a/obBfRk0. We have this set up at https://homestakeros.com.

Depending on a project you are working on, you could then have a subset of the infrastructure be shown to the user and have things such as SSH aliases and other niceties set up on `cd` in. When you `cd` out, then your view is destroyed.

We have quite overengineered this approach -- we run the nodes from RAM. NixOS has the idea of "delete your darlings" which is having a temporary rootfs. We have gone the extra mile that we don't even store the OS on the computer, the computers boot via PXE and load the latest image from the router (though any HTTP server will do, I boot some from CloudFlare). We do this because it also forces the administrators to document changes that they do -- there is nothing worse than starting to call up people when theres is downtime and try to figure your way back up from what the mutations are. PXE booting establishes a working initial state for each node -- you just reboot the computer, and you are guaranteed to get into a working initial state. I'm personally big on this -- all my servers and even my laptop works like this. We upgrade servers by using kexec -- the NixOS configurations produce self-contained kexec scripts and ISO images for hypervisors (some stakeholders insist on running on Proxmox). I've suggested some kernel changes to NixOS which would allow boostrapping arbitrary size initial ramdisks, because otherwise you are limited to 2GB file size.

> We're setting these chemists up with conda in Ubuntu in WSL in a terminal whose startup command activates the conda environment. Not exactly setting them up for reproducibility if they ever move to a different laptop.

Python in specific is a PITA to setup with Nix, dream2nix etc., might help but it's definitely the hardest environment to set up of all languages I've tried -- even GPGPU environments are easier. Oftentimes, the only problem is not the packaging, but also the infrastructure used. For that, you could also publish the NixOS configurations and maybe distribute the kexec or ISO images.

A notable thing is that devenv also allows creation of containers from the devShell environment, which may further help your case. Researchers could reference docker images instead of insisting on everyone to use Nix.

In any case, I put some emails on my HN profile so we can also take the discussion off platform -- we are looking for test users for the holistic approach using PXE, and we are currently funded until Q3 next year.

Not sure on your issue with reproducibility using conda is. We (team of RSE working with many researchers) have had good success with storing conda environment files in git along side the code, only a few commands to get a working environment. We provide class room training to researchers and provide the training material and environments this way.

I don't have the link to the github issue handy, but I remember that the key was to ask for a lower version of python and then supply the `--force-reinstall` param. The fact that it only happened to some students was evidence that conda wasn't as hermetically sealed as nix.

My real gripe is that in that issue, the app developer couldn't really help--since it was a packaging problem--and the conda folks were unaware because the users had gone straight to the app developer. If there must be a third party doing curation, it seems to me that they should be more narrowly focused on whatever particular suite of tools enables whatever particular group of people--not on individual packages.

I know that conda lets users do this too, but I don't think that the environments compose as well as they do with nix. If you want Jim's envioronment, but with Susie's custom build of foo-tool, you can just take both as inputs, overwrite foo-tool as desired, and output composition. Your maintenance burden remains small. If conda handles environments with this kind of compositional attitude, I'm unaware of it.

> I don't have the link to the github issue handy, but I remember that the key was to ask for a lower version of python and then supply the `--force-reinstall` param. The fact that it only happened to some students was evidence that conda wasn't as hermetically sealed as nix.

Conda can be a bit fiddly, it makes life much easier if you specify the required version of python when you create a new environment.

>I know that conda lets users do this too, but I don't think that the environments compose as well as they do with nix.

Yes, though we find Conda great for most of our use cases, we still have to resort to creating containers.

Reading through your link I caught myself thinking if I would put up with all those boilerplate nix steps just to add a new page to the site.

Don't get me wrong, I get that you gain big amounts of flexibility out of it the way you do it but if we think about the tasksat hand, adding a page to a predefined blog, it seems a bit involved.

A fair comment, I do not disagree. I do plan to one day do an `ls` command on the root Nix file so that the manual update to the root flake for both the inputs and the RSS feed would be redundant.

Reply on news.ycombinator.com