Why I Prefer Makefiles over Package.json Scripts | Hacker News Reader

Why I Prefer Makefiles over Package.json Scripts

spin.atomicobject.com

235 pointsducaale177 comments

This seems to be missing the obvious point: Make isn't anchored to any single language. If you will only ever use npm for your entire life, then you go ahead an live happily in package.json script land.

But I've lost track of the number of languages and environments that I've worked in. Make ties it together by being the Good Enough anchor point that documents and launches all the other tools and compilers.

- Compile this Rust app with Cargo then flash it to the esp32? Make.

- Build this page with JS using Tool Of The Week and push it to a test container? Make.

- Compile ancient C app and build a .deb? Make.

And so on.

Why not use shell scripts you may ask? Because shell scripts are way too free form and your tastes will change over time. Make forces just enough structure that you won't get carried away with yourself. Not for the basic task running stuff anyway.

Bazel would be a better solution in terms of reproducibility and user-friendliness, though.

I agree, but make is usually more abundant on much more systems (which is useful when there is no root privileges available). Also people are used to run make when they encounter a Makefile. I personally use Makefiles even to create Docker images, I find it simpler to run make images or make run, than remembering (or putting in a script and remember what parameters to use, as pointed in the comment above scripts are too free-style) how to do it manually.

Though I also agree that Makefile is not a silver bullet and some more complex/niche methods may be required in particular cases.

It's just a nightmare to manage unless you're a Google-sized company

I've been working with Bazel by myself. My experience is that it is not a nightmare... it's just the documentation is missing some pretty crucial "how-to" guides, and there are a couple features that changed a lot prior to 1.0 (so using them is a bit difficult).

Some stuff that is very easy with makefiles is a bit harder with Bazel, so I can't give it an unqualified recommendation. The payoff is that some things that are very, very hard with makefiles are much easier with Bazel (using multiple languages, cross-compiling, downloading dependencies automatically, distributed builds, etc).

My experience is mostly through consuming packages that other people have written, and the downloading existing packages is a big part of why it's problematic. If I already have PrettyCommonStandardLibraryX, I don't really want Bazel to download another copy of it for me just to incorporate your project. Other build systems are simple enough to change, or support this straightforwardly. Doing it with Bazel is quite painful IMO.

I like nothing more than dropping into a new project, and, rather than figuring out the commands to make it run, see that it has a Makefile with a handful of targets for common tasks.

With some collegues we wrote an article about the benefits of this approach a few years ago: https://blog.capitaines.fr/2016/09/30/standardizing-interfac...

I could not agree more. However most people are on windows for which obtaining gnu make is painful, or maybe even impossible - at least the path handling has many sharp edges.

Indeed – although I guess WSL made unix tools slightly easier to use nowadays.

But even without executing the Makefile, simply reading it can tell new developers which language-specific command needs to be run to build the project (and then the command can be copy-pasted and run manually).

WSL made running a linux VM less hassle.

You can put a copy of make in the repo.

For added stability, make sure it’s statically compiled (which it might be already...)

I've been using GNU Make 4.2.1 from here: https://github.com/mbuilov/gnumake-windows - depends only on kernel32.dll. I assume the latest 4.3 build there is the same.

I’m wondering if you could compile make to wasm and include it as a cross platform dev dependency?

I don't comprehend how people can develop on windows (without linux subsystem).

I have to do that every once and then (to ship the occasional C++ or Rust build on Windows) and it's the stuff of nightmare. Stuff breaking randomly from one day to another, env variables to be set in weird ways, GUI installers, 8 different versions of mingw or similar. Recently I've seen that there are a few package managers now (I used chocolatey and at least 2 others just trying to get something to compile) but still, compiling something trivial is always an adventure.

Mac OS X is kind of okay. Brew is barely decent and things mostly work (unless you discover you need to install 12GB of XCode for some dependency or your script is expecting coreutils instead of bsd).

Every linux distro comes with a package manager and compiling is trivial

> I don't comprehend how people can develop on windows

They want to distribute their app on Windows?

That’s why, not how. And you can cross-compile from Linux for some stacks.

It's easier to target Windows from Windows than from Linux.

It's usually not by choice. Medium to large enterprises often demand this so they can manage the employee hardware.

It depends a huge amount on the language and toolchain.

MSVS is really quite nice.

Homebrew is a subpar choice on macOS; MacPorts is faster, has more packages, and is implemented more correctly than Brew.

Additionally, MacPorts was co-created by an engineer who also created the original FreeBSD Ports system, and thus hews much more closely to standard UNIX/BSD practices.

I’m not sure how and when Homebrew became the standard, but it is definitively worse.

I came to realise this too.

Using Homebrew and multiple users is excruciating and an eye opener on how system-level software should really be installed.

Homebrew insists on avoiding root privileges whilst also installing packages system-wide. That works fine and is invisible with one user but falls down hard otherwise.

Their documentation is incorrect too, saying that this is all fine because “we install in /usr/local/bin”. It’s not easy to change this.

The solution was to embrace MacPorts which correctly requires root privileges to install system-wide packages.

I haven’t looked back since. I haven’t missed brew or any software that’s available on brew alone.

Too bad when I want to install newer fancy tools on Linux, it's easier to unify the environment with Homebrew instead of using Homebrew only on Linux (which has a very weird quirk that it wants to install in /home/linuxbrew/.linuxbrew (or else it'll start compiling so many packages instead rolling binary packages) instead of /opt/homebrew like any sane decision would've been.)

I also don't understand Nix when it wants to make 30 users for build process and a few unintuitive decisions. Otherwise it's good that it works same on macOS and Linux.

For me compiling usually is something like `cargo build`, `ng build`.

I remember having problems with libs that require installing & registering a library somewhere such that CMake can find it. However, I distanced myself a bit from C(++), so that doesn't really happen anymore :)

I avoid mingw, don't use any package manager besides Windows Store (if you want to call that a package manager).

Can't complain. Sometimes, there is stuff that simply doesn't support Windows -> WSL. When there is docker, it doesn't matter anyways...

My strategy is don't fight Windows and you'll be happy

From your description it sounds like you might be going off the beaten track and hitting problems.

When I was doing C++ on Windows getting a dev environment setup just meant installing Visual Studio with an appropriate Windows SDK version (or the Windows SDK + build tools for a build system).

You can have multiple VS versions installed side-by-side. To get a terminal with environment variables set correctly you just need to use the shortcuts from your VS installation.

For third party dependencies we checked the headers and (pre-built) binaries into the repository. I don’t remember ever having more than a dozen or so in total. It was usually things like boost and zlib.

Having done that you can just point CMake directly at the packages rather than worrying about FindPackage.

Working in tools like Python and Node, personally I often miss the simplicity and stability of this approach.

Are most people who would have a use for a tool like Make on Windows?

Well, most developers I work with benefit from a task runner. Now, we use different runners for different repos. I’d prefer being able to go into a repo and do «make test» and have it work, regardless of language, framework, purpose of the repo.

If you show up to the job with a windows box I expect you to know how to do the job in windows

It's the same problem with docker-compose files; how do you expect the developers not running in windows to fix your windows problems?

Thank god WSL is around. Otherwise I would've given up on supporting Windows environment for those who still stick to Windows.

If you're already using the Chocolatey package manager for Windows (I do, and am mostly happy with it), it offers GNU make:

https://community.chocolatey.org/packages/make

I have never bumped into that article before, thanks for that.

I did implement though that uniform interface in https://github.com/ysoftwareab/yplatform

And it still holds true for development today! Really any organization that has multiple languages should consider making their devs lives easier with Make.

And of course, it plays great with Docker & Compose, here's a write-up we did on using Make with container tooling: https://shipyard.build/blog/makefiles-for-modern-development...

The only bad part about Make is that it doesn't provide even the most basic utilities for doing cross platform work like removing a directory. It mostly works with Unix, but writing a Makefile that works on Windows and Unix is really hard.

If Make had just a few more batteries included, it would be perfect. However, maybe adding those batteries would take away from its other qualities.

Nowadays there's Windows Subsystem for Linux. There's no excuse not to successfully run "Linux" scripts on Windows.

I've been running very complex build systems via https://github.com/ysoftwareab/yplatform (disclaimer: author here) since 2016 on Linux, Mac and Windows without a problem.

That’s why you always provide an nmake for the project :)

Make is a file-based dependency tree that only defaults to a specific shell. If you really want to go crazy, you could actually change it to anything that can be invoked with "<executable> <file>", such as python, and use that language instead.

As a really crude example:

  SHELL := python
  .ONESHELL:

  out.csv:
      import csv
      with open("$@", "w") as f:
          writer = csv.writer(f)
          writer.writerow(["header1", "header2"])

and so:

  $ make
  import csv
  with open("out.csv", "w") as f:
      writer = csv.writer(f)
      writer.writerow(["header1", "header2"])
  $ cat out.csv 
  header1,header2

I get that this doesn't exactly solve your "cross-platform work" gripe, but my point is that a lot of what people perceive as make providing isn't actually make in the first place.

Thank you for reminding me it! Awesome finding.

Worth noting, each recipe can have its very own SHELL[0] (e.g., ruby recipe, that uses ruby command with -e flag):

  ruby: .SHELLFLAGS := -e
  ruby: SHELL := ruby
  ruby:
   greeting = "labas"
   puts "#{greeting}, ruby!"

[0]: http://agdr.org/2020/05/14/Polyglot-Makefiles.html

Nice, being able to mix them is new to me. I thought SHELL could only be set once.

Do we not have modern alternatives for make?

One thing I appreciate about the javascript ecosystem and "scripts" is the focus on keeping everything contained in the project. The only assumption about the system is that it has node, from there it's npm i, npm start. Scripts are a way to make sure people always run the project-local version of various development tools and not whatever they happen to have installed.

When I see a makefile, I expect they will be making more assumptions about my system and expect a little more work.

I do appreciate the "has worked fine for decades" aspect of makefiles, and I guess docker solves some of this, but I still prefer a fully self contained javascript project.

Yeah this seems to miss some of the biggest features of npm scripts. Access to the project's local packages is probably the biggest feature not mentioned IMO.

npx to the rescue

To add one tiny thing to this, it’s absolutely imperative that your project be setup with safeguards for node version dependencies. I can’t tell you how many projects I’ve joined that don’t even have an nvmrc file, much less some guard to ensure a clean install and build.

When you say everything contained in the project, you forget that there's more to life than just node/javascript. Just two silly examples: you cannot run/deploy your API backend without docker/kubernetes/aws-cli/etc, nor your mobile app even if it's in JavaScript (react-native).

The only way to keep to node/javascript is if you only develop libraries (a website that doesn't get deployed -> still on library level).

API backends/middleware can be Node.js too

That's true, kind of. If you have those kinds of tasks, I see the appeal of the makefile over various commands in package.json that won't run after a simple npm install anyway. In practice, you may be building your react native app with EAS Build and deploy your website/api to Vercel or to AWS with pulumi. That's all still npm dependencies.

I would like to call attention to `just` (https://github.com/casey/just) which is a modern take on "make if we didn't have to keep it backwards compatible", and more focussed on just task running rather than tracking dependencies.

Yes! This is what a modern task runner should be. Includes ability to list recipes, and a consistent format between multiple platforms. Just two things Make doesn't have.

I was just going to say that. Just is awesome and I use it it most of my projects.

There's no -j option nor mention of parallelism on the docs. Does it run independent targets in parallel? This is easily the main feature of make.

I don't understand how it can work in parallel if targets are not somehow associated to files. What's the advantage to a shell script with separate sub-commands, then?

‘just’ apparently isn’t a build system, just a ‘task runner’.

So I assume it doesn’t need/want to implement parallelism.

It sounds like `just` is not a replacement for make, then. It seems like it could be a nice wrapper on top, i.e. `just build` could invoke the make build system.

Parallelism is pretty useful for just a 'task runner' too.

I've (ab)used Make for server pushes, and sometimes you want those to run one at a time, and sometimes you want -j 100 (sometimes -j 100 is even effective).

The key aspect here is that since Make knows what the dependencies are it has enough information to automatically parallelize the task. A simple task runner would need to be specifically configured with the dependency graph, and if you have that you've just re-created Make.

If I'm using the task runner for deployment, it doesn't need to have a sophisticated dependency graph, it "just" needs to be able to have a list of tasks that are declared as independent/parallelizable, and a way to specify how many to run at once.

Sophisticated dependencies are nice to have of course, maybe make push-all transitively includes building the thing locally, but it's still useful to have just the push step (or the reload step, etc), have optional parallelism. If you start to build a tool that does do these things first, in order; then these things in any order; then these other things, that does look a lot like Make. but if you just handle ordered steps and independent steps, and let a person do the mixing, then it's not nearly as much work as Make.

> Sophisticated dependencies are nice to have

What are you talking about? Make does not deal with "sophisticated" dependencies. Make is insultingly simple, not sophisticated at all. It's just a damn directed tree. The simplest data structure in computer science after the array.

There are also some fancier shortcuts for expressing rules in makefiles, but you can ignore them and use only the simple stuff. That's already a much more powerful tool than "just".

Tracking dependencies is what makes make make. Love it or hate it, Ant got this right (imho), and modern build systems almost universally get this wrong. Every language can run and compose tasks. Dependency tracking, parallelization, and language agnosticism are the killer features of make (again, imho).

I agree with you on all grounds (I personally use make daily) however note that `make` checks dependencies via last updated time flag on files. I think it would have been a better default to cache based on hash of file. This way if document was modified, by hash wasn't changed, make shouldn't rebuild tasks that depend on this file.

But computing the hashes of all the files would take the same time as rebuilding everything, or maybe even slower. And you'd need to re-compute all these hashes even for a no-op. It's absurd.

The nice thing about dates is that they can be checked instantaneously, and independently of file size. If a file has changed, its date has, so it's all the information you need.

Computing the SHA sum of a file is plenty fast on modern hardware, nowhere close to the same time as a rebuild.

Yep, I can take the sha256sum over my all my project's (vendored) dependencies in 10ms using the command:

time find ./vendor -type f -print0 | sort -z | xargs -0 sha256sum | sort -k2 | sha256sum

Note if using this code for practical purposes, you should be sure your collation is set appropriately. That's taken with 1280 files that are in total 19MB.

For raw throughput, you can use BLAKE2 for even more speed https://www.blake2.net/

What % of that time is required for just computing the timestamps of these files? I bet it's a few orders of magnitude faster.

Sure, but who cares?

While 1280 files isn’t exactly small, it’s not really big either. Big as in where incremental builds really show their shine.

This might be approaching niche but I’ve seen a few mega-projects with 100 times more than that (also mostly due to vendored deps). You really don’t want to read 10G from disk for every incremental build. The actual sha computation isn’t the bottleneck, disk is. Ninja on the other hand, using mtime, handles this without a sweat.

The best solution would probably be to integrate the change detection with git, it is already content addressable and has lots of other smart things built in to make it fast.

Rebuilding potentially has side-effects. You can have a build system that, if a file is changed, it makes HTTP calls, generate docker containers, store artifacts in S3 etc. Given this, computing the hash would be negligable amount of work. Besides make does iterate over all dependencies every time to check last modified date filesystem keeps. So, at that point one can compute hash.

You shouldn’t have such a build system. Idempotence is a feature, not a bug.

idempotent does not mean side effects free; a compile-to-the-cloud system for distributed caching can be idempotent (ocaml is building support in this direction)

Side effects that target on non local dir seems shouldn't be included in the build step. It should be in a standalone script/ target/whatever. Or the build process will not be reproducible at all.

builds should not use non-local inputs, but they can have non-local outputs. the simplest case would be automatically publishing a package or image to a private registry

> If a file has changed, its date has, so it's all the information you need.

Unless the date hasn't changed -- there is no hard guarantee that mtime will be accurate.

If the file's contents have changed but the timestamp didn't, then something is broken and _that_ is the thing to fix, not rewriting the fundamental mechanism underneath the oldest relevant build tool.

There are always different preferences on deciding when something needs to be run again. File modification time is a reasonable default that serves many (most?) cases pretty well.

Yes, file hashes might be another way. On the other hand, if I modify a comment, the file hash will change, but re-compilation is not necessary. In this case, you might need yet another way...

You can always write your Makefiles to compute and depend on hashes or whatever else you might consider appropriate for your exact use case.

I've been using make for a long time but make really really sucks at parallelization and language agnosticism.

I mean those things work, but they're by no means nice. Parallelization works, but output of different tasks (and even the log that certain commands have been run) are interleaved, making it impossible to view progress. Language agnosticism works by operating on the level of shell commands, but that's really about it. If shell commands need to be customized on a per-system or per-OS level... well it's possible to hack around that with enough external scripts and even esoteric if statements inside the Makefile, but it's not nice.

Documentation for make feels like it's from the 80s. Which it is.

All these things (with the exception of having to rely on arcane sets of external scripts and if-statements-in-Makefile) are perfectly fixable. But they don't get fixed.

I haven't understood what the point is for a "task runner" over having a bin/ directory in the project with some shell scripts. If it's not going to do anything extra for you (like run commands according to a dependency tree like make), I don't see the point.

`just` seems like a useful tool, but it only covers a secondary use case of `make`. `make` without a dependency graph is like a bicycle without wheels.

All I can say is that I’ve only ever used make as a task runner myself, so just matches my need way better.

IME make’s model is often too simplistic to be of use, so it simply gets in the way of tools already doing things internally with proper insight, or requires force-cleaning everything because it's too dumb and just breaks the build when you e.g. switch branch.

Just’s author is also looking at integrating `redo` for building / dependency tracking, though the issue has no activity since it’s been created.

If you find yourself running "make clean" because it is "too dumb" you've built your Makefile wrong. Usually it means you've tried to manually re-create the inherent Make rules and have gotten it wrong somewhere. This is incredibly common, since the inherent rules are not always the same across various versions of Make and people don't always trust them.

IMHO, this is the primary weakness of Make. The standardization efforts never really happened so you have various forks of 1980s code with annoying incompatibilities. BSD Make is missing some of the implicit build rules from GNU Make (only able to compile simple single source file applications without an explicit rule), and Windows Make tends to be extremely primitive. Worse, Make still in 2022 can't handle spaces in file names, like c:\Documents and Users\.

I would be so much happier if everybody could standardize on one flavor of Make. Probably GNU Make.

> I would be so much happier if everybody could standardize on one flavor of Make. Probably GNU Make.

With included Guile scripting support!

https://www.gnu.org/software/make/manual/html_node/Guile-Exa...

Only if your version of Make is compiled with it. "GNU make may be built with support for GNU Guile..." https://www.gnu.org/software/make/manual/html_node/Guile-Int...

I really wish it was more common because having it would be great.

On Debian and Debian-based system you can install the “make-guile” package.

> If you find yourself running "make clean" because it is "too dumb" you've built your Makefile wrong.

What's built wrong is make itself. Because the signal it uses for "this dependency has changed" is the mtime, any change which does not move the mtime forwards will be ignored. Switching between concurrent git branches, for instance.

> Usually it means you've tried to manually re-create the inherent Make rules and have gotten it wrong somewhere.

And pray tell how do you "manually re-create the implicit make rules" when such rules don't exist? GNU Make has implicit rules for C, C++, Pascal, Fortran, Raftor, Modula-2, and assembly. That is, frankly, not much. And mostly outdated.

The python library/CLI tool «invoke» is also nice, although requires python.

I'm pleased to see this being espoused. One thing I always tell junior people is ""don't be tempted to think of make(1) as out-of-date [1], and to view packages available via npm etc as better replacements, as many tasks fit the model of make(1), if not the title "a program for directing recompilation" "" I am often viewed with suspicion until I show them a (much more compact) replacement for their custom python/js program expressed as a makefile without any of the process management requiring debugging. :-)

An alternative approach for the "batches of files to process" situation is to generate the commands and feed them to GNU parallel for execution.

[1] no pun intended

Agreed if you only have to support linux/unix, but the one good thing about the Node ecosystem is that they tend to work on windows too.

While i agree, I’ve spent literal days fighting with node modules in recent months. Sure make isn’t perfect either but it’s a lot less, confusing? Ie json isn’t a language for humans, but often you really need to read it to correct some issue or bad merge from another contributor.

> Sure make isn’t perfect either but it’s a lot less, confusing?

I think that's a matter of what you're used to. I don't use make all that often, so I find make files beyond a few lines quite a lot to get my head around, whereas I spend all day writing JavaScript so I can read/write JSON in my sleep.

Maybe. I actually use node much more often, i don't even recall the last time i used a make file in production, i just find parsing json as a human to be rather painful.

not the point, also "WSL"

Make has worked on Windows longer than Node.js has existed.

Make works fine, it's the commands you call from make that don't tend to as compatible. For example, try running rm -rf on windows. It might work if you have GNU tools installed, but it certainly won't work out of the box. On the other hand, the node.js `rimraf` package will do the same thing with no cross-platform compatibility issues.

When using package.json, does npm intercept a script containing "rm -rf" and translate it to something different?

No, that's why there's a bunch of packages such as rimraf[0] that implements that sort of functionality in a cross-platform way that most people use in their scripts

[0]: https://www.npmjs.com/package/rimraf

No, but it does put binaries from any installed node packages in your PATH (locally when running scripts).

The history of Make on Windows would Make one think twice about relying on it. What shell do you get when running Make on Windows?

http://gnuwin32.sourceforge.net/packages/make.htm

https://community.chocolatey.org/packages/make

Might it not be a disservice to juniors to point them away from deeply understanding the platform they're working with?

Showing someone an alternative approach does not prevent someone "deeply understanding" their current method; it's not zero-sum.

Might it not be a disservice to juniors to point them away from deeply understanding the platform they're working with?

I cannot overstate how annoyed I was by the patronising tone of this comment.

Biggest pain of package.json to me is having to be crammed on one line. Slap in a conditional and now we have a messy thing to read, compared to its multiline counterpart.

Speaking of debugging, I found this handy:

  print-%: ; @echo '$(subst ','\'',$*=$($*))'
  include Makefile

Save somewhere, eg as ~/Makefile.debug, then when you want to know the evaluated value of some variable or task in some Makefile, you prefix it with print- — for example, LOCALES:

  make -f ~/Makefile.debug print-LOCALES

It'll print the evaluated LOCALES of the Makefile in the directory you run this in.

This is amazing. Thank you for the tip!

Very useful, will give it a shot. Thank you. :-)

May I also direct your attention to self-documenting Makefiles:

https://www.thapaliya.com/en/writings/well-documented-makefi...

Here is the one I use and recommend: https://github.com/awinecki/magicfile

A Makefile inside a JavaScript codebase is a good indicator that you will encounter other less idiomatic ways of doing things in that codebase.

That's a bonus.

Node and NPM parted ways with idiomatic JS a long time ago. Most packages in the Node ecosystem are pretty much, "What if a bunch of people who clearly hate JS nonetheless decided to spend most of their time and energy inside files ending in .js (or .ts)?" So to the extent that "idiomatic ways of doing things" really means "idiomatic wrt the Node ecosystem" and not "idiomatic to JS as it was meant to be written", that sounds like a win.

<https://news.ycombinator.com/item?id=24500593>

Make has been around for decades, and with good reason. I've enjoyed watching the grunts and gulps and whatevers appear as the New Shiny Thing, become complicated and collapse under their own weight as people struggle with overly complex configs, as I quietly do the thing that has always worked and has not changed in forever.

Over here in crazytown I'm using Makefiles to automate system deployments.

I thought it was insane at first but actually I've come around to it; Make hits that sweet spot of being ubiquitous, simple, and flexible. There's a little bit of a learning curve but once you're over that initial hurdle it's pretty straightforward, especially for one-way operations, e.g. install without uninstall, build without clean, etc.

I've only started using a Makefile ~two years ago when I started a big Go project at work, and I quite like it. There's a few caveats here and there - Makefile specific syntax, shell-script specific syntax, and whether I should put .PHONY in front of every command (I guess 'no unless you have a file with the exact same name'?), but it's quite compact and straightforward.

The most complicated command I have is something that removes a folder or set of files, invokes the swagger generator with a list of options, copies it to a target folder, and passes it through a formatter/processor. But it's very straightforward, and it's "just" shell script, not shell script wrapped in a JSON document, or some Java tool invoked indirectly through an XML configuration file like back in the day with Maven.

If using Make as a task runner and not build cache, I just put `.PHONY: %` at the top which means everything is marked `.PHONY`

Thanks!

Reminds me of something I wrote years ago: https://blog.jgc.org/2010/11/things-make-got-right-and-how-t...

Has anyone Fixed Make in a way you find good and right since you wrote that?

(My last few forays into the build systems domain have mostly turned up “Bazel but for X audience” so I can’t recall a satisfying solution)

Why not both?

Use make for dependency graph, but also use "npm run" or "yarn run" to make sure you're running with the correct Node and correct paths to all your tools.

I know you can specify those in your Makefile too, but the plus of running via npm/yarn is that they know where your binaries live, where your mode_nodules are, and that sort of thing.

replace that with `pnpm` and you've got a winner. npm and yarn are old hat (unless you're working with react native, which is another discussion)

Yes, I usually treat Make as the user-friendly facade to the underlying framework tooling (which might be e.g. Rails, Docker, or npm). `make bootstrap` gets the project setup. `make up` starts the project.

For npm projects these usually just wrap package.json stuff. But I don't need to remember what tech a project is in, or where to look for the commands, if Make is the entry point.

For a lot of the same reasons in this article I like to use tup[0] when I can. It doesn't integrate into anything which is both good and bad. I wish my IDE could check with tup to see what dependencies get pulled in by what files. However, it's nice that it doesn't care what language or ecosystem I'm using.

Also, it's very strict about declaring dependencies properly, and will actually fail the build if you've set things up in a way that depends on something not tracked (by watching filesystem access as the compiler runs). That gives me warm fuzzies that my builds are reproducible.

Also I can get a neat dependency graph as a PNG if I want.

[0] https://gittup.org/tup/

The problem I have with Make is that it has a syntax that most web developers I have worked with would find archaic and unfamiliar.

On some projects I've worked on, it's been used as basically a command alias system that only a few people can maintain. None of its caching or dependency chain capabilities were utilized.

In these cases, shell scripts would have been a better option, and in some cases were later introduced with success.

I use makefile for all my builds.

cmake can help portability across OSes but it has a layer of its own complexity, I mostly work on linux alone, makefile seems more than enough.

google etc produces new tools to build its huge code base but I don't have that large scale source code, makefile so far worked well for my scale.

unless you have specific needs I feel makefile will do just fine. it's simple, readable, manageable, and get the job done fast.

I also use Makefiles, been using them for 40 years. I find that Makefile targets for misc. things like grabbing remote training data, running tests, building documentation, etc., etc. augments information in READ files, that is in addition to functionally saving time, serves as documentation when I haven’t looked at a project in 6 months.

Why I prefer package.json over makefiles:

- turborepo (https://turborepo.org/) can describe dependency pipelines and provide automatic caching. Makefiles aren't designed for multi-input, multi-output scenarios - its really awkward: https://www.gnu.org/software/automake/manual/html_node/Multi... and https://stackoverflow.com/questions/39237306/makefile-compil...

- turborepo can also run never-ending targets in parallel (e.g. API server and static file server in development mode). Not sure how well make supports that

- turborepo can depend on env var values. Makefiles can to, but like everything with makefiles, its an awkward workaround: https://stackoverflow.com/questions/14840474/make-target-whi...

- makefiles do not work on Windows (They are portable, just not to platforms most node devs care about)

- Its unclear whether `make` will ever add features to remove the awkwardness for supporting the various scenarios above. It doesn't seem likely to happen.

Also, it says something about the design of `scripts` that its so easy to build something like turborepo's dependency pipeline on top of it :)

For the clojure crowd, there's bb tasks [0].

* parallelism

* hooks (before/after each task, etc.)

* command-line arg parsing

* --help equivalent

* bash/zsh/fish autocompletion

* you can also just spawn a shell with bb/shell

* yes, dependencies

* regular programming language (Clojure), with commonly used shell functions like glob()

* import code from locations (don't need to copy/paste between Makefiles)

* built-in JSON and CSV support, so you can use your app's JSON files right in your build file

[0]: https://book.babashka.org/#tasks

Stick around in any programming ecosystem long enough, and you live long enough to see strategy, taste, and "discovery" come full circle. We had make, then package.json "scripts," then grunt, then gulp, then the trend shifted back towards package.json "scripts." (Note: Using "scripts" to differentiate the package.json property versus the generic word use)

I like that the author isn't being authoritative, but there could be some additional due diligence. I ran the make-is-faster loose benchmark via PNPM and runtime was 0m0.022s for make and 0m0.012s for PNPM. If I care about those 10ms, PNPM is my horse. Yarn is a glacier compared to PNPM.

Another thing I would've liked to see comment on is the automatic pre and post paradigm that package.json "scripts" affords. The big three package managers all support pre and post, and it makes arranging "scripts" a breeze, it breaks down dependent steps into separate console output, and is generally easy to organize imho.

All in all a nice write up for folks who might not really like package.json "scripts" to begin with, or for those who'd rather not gain more granular insight into how "scripts" works, but I don't see this being the nail in the coffin case against them.

What in god’s name is pnpm?

> What in god’s name is pnpm?

Take a look: https://pnpm.io

https://pnpm.io/ it ships with Node.js along side npm and yarn these days, has for about a year I believe.

What's your source on pnpm/yarn shipping with node? It doesn't in my experience.

`corepack enable`

https://nodejs.org/api/corepack.html

The caveat here is that the binaries aren't "shipped" with node.

`corepack enable` adds shim scripts to node's $PATH to intercept calls to `npm` `yarn` or `pnpm` and downloads the binaries from a URL without much checking along the way. This functionality has had some vocal opposition.

It's a replacement for NPM and YARN. Never used it myself.

https://pnpm.io/

I’ve used pnpm religiously for over a year and have seen it being adopted as a golden standard replacement for npm in my circles. Highly recommend it to anyone who works with JavaScript or Typescript!

But why? Ok it is faster and save storage. Is that really a huge problem?

(GNU) Make's advantages include ease of use, a modicum of portability, and support for a variety of different software development command line stacks.

Unfortunately, the shell commands that are typically setup in a Makefile, are rather unportable. They waste a touch of time and space. They can break in surprising ways.

For this reason, I try to write my build tasks in terms of the same programming language as the application language. For example, Grunt for JavaScript projects, Shake for Haskell projects, Gradle for Groovy projects, Rez for C/C++ projects, Dale for D projects, TinyRick for Rust projects, and Mage for Go projects. Leiningen for Clojure projects, SBT for Scala projects. Vast for shell projects.

Make, like Python, suffers from the appearance of simplicity, at the cost of long term maintainability. Ideally both the build system and the main application are compiled. That doesn't have to be a cumbersome process, but rather a guide to identify potential bugs sooner during development.

Make is my first choice for random projects, but only when better build systems are unavailable.

I use make to run daily/hourly/etc batch processes in the ERP system at work. There's a Python shim to actually run processes in the ERP, plus a shell script to get things ready to run and send success/failure emails, and cron to kick batches off, but other than that it's pure make. In theory I should also be able to pass '-j 4' to make and have it run multiple processes at once, in the correct order with their dependencies properly satisfied, but I haven't actually tried that yet.

It's clearly an abuse of a build tool, but it's also a testament to make's incredible flexibility.

I've also heard of someone who replaced their Linux startup scripts (in the pre-systemd days) with make to speed up boot time. That's actually what inspired my batch schedule work.

I like Make, and use it in personal JavaScript ecosystem projects that do actually need to build things with interdependency (eg https://github.com/justjake/quickjs-emscripten/blob/master/M...). I’ve also seen Makefiles grown to be horrendous monstrosities masquerading as command-line tools; for about a year at Airbnb we used a 1000+ line Makefile as the main tool for fiddling with Kubernetes cuz one of our senior engineers didn’t like Ruby, and I’ve seen another one get close to that level of cravenness.

What I learned from supporting Make and shell among a few different audiences is that most developers have no interest in how to write or maintain shell-like tooling. They forget or mess up quoting rules constantly, and eschew learning things and good design in these tools to a much greater degree than in their “normal” work in Java/Python/Ruby/Typescript/Golang.

For every POSIXly Correct HN Commenter (of which I count myself a member), there’s 100 regular software engineers who won’t read a `man` page on what $@ or $< mean in Make. I know that if I start writing a Makefile for $JOB, it’s gonna be me and that one guy who uses tcsh who are gonna maintain it and answer questions.

(Although for what it’s worth we don’t use package.json scripts either thank goodness. All our complicated build steps are typescript commands, and our glue is CI system YAML files.)

I would try to use JavaScript tooling in JavaScript projects, especially if I share it with other developers.

I would love to see better handling of filename transformation, file watching, and parallelization.

A few times over the years I had given up and just used Make instead of Gulp or Grunt. Most recently, I wanted to watch if files had been deleted (if directory entries had been modified). It ended up being a few lines of Makefile.

And once you understand the syntax, it is expressive and simple, and way more portable. There was some good discussion awhile ago on this[1]:

[1]: https://news.ycombinator.com/item?id=7622296

I'm also a fan of make but whenever another frontend dev starts working with one of my projects the first thing they do is rip out the perfectly good Makefile and replace it with npm scripts (often introducing errors). I guess it's a generational thing.

I don't have a problem with make files, but I do prefer to work directly with `package.json` when reasonable. Most notably, most of the IDE's work very well with `package.json`, but don't always handle make as cleanly.

I was amazed to discover Taskfile. Didn't realise it was a copy of Make.

https://github.com/adriancooney/Taskfile

https://github.com/ysoftwareab/yplatform that this a notch higher (disclaimer: author here)

As others have mentioned on this thread, the problem is not just "use Makefiles instead of package.json scripts", but quickly ending up with duplicate Makefile content and then supporting more than just one language. It's inevitable.

for those that don’t know, npm’s “scripts” functionality as we know it today was not “designed”, but it was simply a byproduct.

First introduced as an internal functionality for lifecycle support https://github.com/npm/npm/commit/c8e17cef720875c1f7cea1b49b... on 1 Mar 2010

and later extend to run arbitrary targets https://github.com/npm/npm/commit/9b8c0cf87a9d4ef560e17e060f... on 13 Dec 2010

This is the entire design backstory https://github.com/npm/npm/issues/298 .

Needless to say that GNU Make’s equivalent (or any other build system) is not to be found in npm’s scripts.

We are using make targets to automate ci/cd, local dev and a lot of other things by adding a set of easy to extend, customise template makefiles as submodules.

The main advantage for us is it's same everywhere and it's agnostic to underlying technologies. (we are only supporting unix based environments and have guidelines to setup gnumake in macos). By using this way, we ensure that the general SDLC is same across different repositories and underlying tooling can change anytime without inducing a lot of refactoring

We are also using make targets to document themselves. How to use, what are other targets, variables you need to define etc. This makes it a powerful CLI app for us that can run and support many things.

For anyone interested: https://gitlab.com/ska-telescope/sdi/ska-cicd-makefile and the similar parsing method for self documenting makefiles: https://gitlab.com/ska-telescope/sdi/ska-cicd-makefile/-/blo.... We didn't know about magicmake that's linked here that does the similar thing

Just wanted to share nps as a nice way to manage scripts for your package.json

https://www.npmjs.com/package/nps

You can create a dedicated JS file (with comments and more!) that will house your scripts, which you can run by using "nps" instead of "npm" as a command.

Out in Grand Rapids, Mi where AO is located there seems to be a bit of a tradition of abusing make in a good way. It's where I picked up the habit as well. I think its just institutional knowledge/tradition that has creeped through the various firms here.

I kind of wonder what other "traditions" get passed around on a regional level?

But you don't write scripts in package.json and it also failed to show an example how the javascript dependencies could be loaded with make.

I think make works well in combination with package.json, but not as a replacement. You can also do the same stuff in an NPM command (which just loads node code) that you could do in a makefile script.

Not to mention that the top three package managers will load binary files from node_modules automagically by name only, without having to specify paths. That gets hairy in a monorepo, and package managers running package.json scripts manage them just fine.

`npx` solves this elegantly as long as you `cd` to the project directory first.

I have very similar use-cases, and I also I found makefiles a bit limiting. I wrote lk[1] to make this sort of thing easier, and you can just write plain bash functions instead of makefiles.

[1] https://github.com/jamescoleuk/lk

The second I see a node project built in make, I remove it

That's a person who doesn't understand portability, tooling, or the need to stick to community norms. Everything about their library is going to be pure pain.

Back when I was doing network engineering I used make to localize configs from boilerplate code templates. Not a common use for it or maybe even the best tool for the job but because I already knew Makefiles it was an easy solution for me to implement. Saved me endless hours of tedious error prone text editing. Just an example of why it's worthwhile to learn how to use these fundamental tools that have survived the test of time. Even if you don't have a specific use for it today it's a very useful tool to have in your toolbox.

At one point I tried to do the same thing but then I realized there isn't any make feature I need to use. The key thing about make is to avoid building already built artifacts, if you don't need that you end up with things like .PHONY. There is'nt anything in make that helps with composing or showing a list of scripts etc either. At the end I ended up with plain javascript scripts for js project and plain bash scripts for everything else.

package.json scripts are definitely a bit out of place. I think their prevalence stems from popular npm projects using the pattern for easy cross-platform developer experience. I've seen some shell scripts in opensource node projects, but not much makefile, and I actually had an interviewer once joke that makefiles were old and out of style.

But as I have worked with less Windows developers over time, my projects have relied more on shell scripts and makefiles. I think the containerization movement has started to shift software development towards Linux-based tooling as well. As a cross-platform effort this is a bit ironic, but having overlapping developer experience and CI/CD tooling is pretty convenient when you're on a Unix kernel or WSL2.

However, much of the npm ecosystem and community are focused on frontend work that often does not involve infrastructure-related tooling. Without the absolute need for any OS-specific tools, and with the many composable crossplatform scripts in vogue already, package.json scripts make sense (until they get out of hand).

Here is Makefile "starter" I use: https://github.com/awinecki/magicfile

People call this "self-documenting makefile".

It migrated with me from company to company, from project to projects. Through node, php, aws, docker, server management, cert updates, file processing and many many more.

Task is nice:

https://taskfile.dev/#/

Earthly is an interesting open source project in this space that offers many of the benefits of a make file plus containerization plus some of the performance benefits of a Bazel/Pants.

https://earthly.dev/

Believe it or not, it isn't an open source project. https://github.com/earthly/earthly/blob/main/LICENSE

It's BSL which is source available with the one restriction being you can't build a commercial competitor. And the source becomes MPL2 after 3 years.

https://earthly.dev/bslfaq

What you said does not contradict what the parent comment said. You're both in agreement that it is not open source.

It’s helpful for mitigating an NPM arbitrary code execution vulnerability along with ‘ignore-scripts=true’

https://blog.uidrafter.com/getting-rid-of-npm-scripts

The argument about variables in makefiles was convincing to me, but looking it up I learned for the first time you can actually use variables in package.json files as well. After all these years mucking about with package.json...

Pretty convincing writeup. I've only used makefiles when compiling C / C++ but lately I'm finding even those are moving to CMAKE with makefiles becoming less and less common.

I too have been using Makefiles in both my C++ and Deno projects and have been very happy with the unity it provides.

one of the few annoyances is spaces in paths. Just avoid them anyway, right?

This is one of the few fixes I'd really like to see. I'm from a Unix background so I don't tend to have spaces anywhere, but it's still a splinter that needs removing.

What are the benefits of Makefiles compared to a bunch of shell scripts?

That you get parallelism and partial re-running for free.

parallelism is common in package managers these days https://pnpm.io/cli/run#--parallel

No, make understands the directed acyclic graphs of dependencies, regardless of language, and will walk that graph in parallel. No forcing or ignoring sorting, and no enforced package level blocking on scripts.

It's a shame this doesn't work for Java in particular (not JS) as Java's coding conventions/requirements do not lend themselves at all to manually updating a list of file dependencies.

A single, system-and-language-agnostic entry point to the actions you want to run with very clear dependency rules built in.

Besides Windows, shell scripts are system-and-language agnostic (not using bash/zsh-specific syntax of course).

Also I'm not sure Windows can run Makefiles out-of-the-box.

The Makefile is the facade to the project. So you can have `make run` do `make run-windows` or `make run-nix` under the hood. And when you change hosts, or you redo the `run-X` targets into a single cross-platform binary or you put it in a Docker container, nothing changes. You don't need to remember that you need to invoke X or Y script.

I'm pretty sure git-bash provides `Make` on Windows. I believe thats how I tested it for this project: https://github.com/J-Swift/cod-stats

I’ve successfully used both together before :)

The makefile lets me be very expressive about task dependencies and encode some developer conveniences. Ie make can know `make bootstrap` should invoke bootstap_windows.sh on Windows and bootstrap_osx.sh on Mac. Then the work happens inside the appropriate script.

They’re two tools that solve different problems but they work together quite well.

Resumability is the main one, in my mind

Imagine a simple two-step process. The first takes a long time. The second one fails on occasion. You don't want to run the first one again. With Make, if the first one succeeds, it won't run again unless you `clean`

Easy, but now imagine many, many steps

Makefile just checks whether a file exists, nothing more.

if [[ -f myfile ]]; then

It also compares the last-modified timestamps of the source and target files so it only runs if the target is out of date. Plus because the dependencies are explicit, it's trivial to run in parallel the parts that can be.

Thanks, didn't know about timestamps.

But it's trivial to run shell scripts in parallel as well, just add ` &` at the end.

Reply on news.ycombinator.com