Hk, a new Git hook manager | Hacker News Reader

Hk, a new Git hook manager

hk.jdx.dev

96 pointsDrBenCarson82 comments

The most compelling thing about this for me is that it uses pkl instead of yaml or json.

pkl is so good I don't think I need plugins for hk

> lefthook is written in go, hk is written in rust. This will make hk faster but the advanced parallelism logic in hk should make hk much faster. Because git hook managers are often limited not by their own logic but the speed of the commands they run, this should make hk significantly faster in real-world usage.

What a strange sentence. First Hk will be faster than Lefthook because it will be written in Rust[1], but then later on it says that it doesn't actually matter? But either way it will be faster because it has "advanced parallelism"?

Looking at the comparison to Lefthook it's not clear to me why this isn't a PR to Lefthook. It's already a well-established solution which is present in many package managers. Surely the distinction between "checks" and "fixes" would be possible to introduce there as well?

Do we yet another tool instead of working together on improving the existing ones?

[1]: Which also is not a given. Programs which allocate a lot can be faster in Go since the garbage collector only works on the live set of objects whereas Rust have to explicitly deallocate all memory. CLI tooling is actually kinda a sweet-spot of GCs: You don't want to spend time reclaiming memory until you reach a certain threshold of used memory. In scenarios where you use less than the threshold you end up spending zero cycles on deallocation. (And completely leaking all memory is bound to cause problem on bigger commands or smaller machines.)

It makes sense. If you have 4 hooks, and you run them serially, it will be slower than if you ran them in parallel.

I say this as a huge fan of Rust, Rust adds almost nothing to this kind of parallelism.

It doesn't claim that.

It claims:

1) Rust is fast, so that helps some.

2) They run hooks in parallel, and that helps a lot.

I was replying to OP not the article. The article really muddles through and hand waves away the Rust vs Go - I’m seriously skeptical the overhead of the language would show up at all with the work being done here.

You are moving the goalposts? You first claimed that Rust doesn't help with parallelism. I pointed out they never claimed it did. Now you are talking about something else?

> lefthook is written in go, hk is written in rust. This will make hk faster

Direct claim from the article.

That assumes the hooks themselves can be run in parallel. Without a way to describe dependencies between them there's big scope for race conditions.

Hooks are generally not meant to modify the source code. They should ideally just analyze the code and either succeed (exit code 0) or fail. Race conditions won't happen in such situations.

In practice though, hooks sometimes run tools that are were not designed like that and instead modify the file (eg: formatters). However, this is usually done on staged files (in case of the pre-commit hook) and the modification is applied to the working copy. This can cause race condition in that one tool may overwrite the changes made by another tool. But since the source is not modified, it won't end up in a deadlock. It will also fail as desired. So the race is not serious. It will resolve itself after a few runs, unless something is done to prevent it in the first place.

Which is why it is best if you plan for parallelism from the beginning, so that your configuration has a straightforward way to indicate dependencies and making sure certain hooks aren't run in parallel with conflicting hooks

The subject program doesn't seem to provide a way in which to specify the dependencies between hooks, hence they will just race each other assuming they are all fired off in parallel.

in fact I do believe I have a solution to this problem. I should have this fully implemented soon but right now it is not.

First, I intend to be able to define simple "dependencies", but IMO that's not the interesting part.

I am going to use rw mutexes on every file with staged changes. If 2 steps are running against disjoint files there is no problem. If 2 steps are running against "check" (not "fix" which edit files by convention), they can also run at the same time. The only scenario where blocking is necessary is when you have 2 "fix" or 1 "check" and 1 "fix" on an intersection of their files.

For that scenario there is going to be a setting that you can enable to run a steps "check" command first (only if this scenario is about to happen, if no files will be in contention it will simply run "fix"), and if "check" fails, then it will switch to "fix".

This is my current idea and in my head I think it would work. There are caveats, notably running "check" and then "fix" might be slower than just waiting and running "fix" once which is why it needs to be optional behavior. You also may not care about things stomping on each other (maybe the linters themselves have their own locking logic).

Sure, but Go can do things in parallel too. How is Rust's faster than Go's, especially if they're limited by the speed of the commands they run?

Imo this problem would be fine in either Go or Rust. I've done both. For extreme cases you can, i imagine, make Rust parallelism faster than Gos, but by default if you just design them similarly -- eg: throwing Tokio at the problem and not hyper optimizing Rust -- i imagine they'd perform quite similarly.

As much as i'm a proponent of Rust, Go is very capable and is generally a great language. Its warts (as i see them) are not going to be apparent in the parallelism needed for a Git hook manager lol.

It mentions "lefthook" by name. I'm guessing that specific go-based hook manager doesn't do this parallelism. So it's saying "rust instead of go might give a small performance boost, but that's not where most of the performance boost comes from".

Why mention Rust at all? How is that possibly relevant to the problem? There seems to be a lot of projects that are rewrites of existing project in Rust where the fact that they are in Rust seems to be their distinguishing feature. Languages are less important than the solutions that a piece of software provides.

> Why mention Rust at all? How is that possibly relevant to the problem?

There are a lot of command line tools that are written in JS and other scripting languages.

Having the tools that are involved in interactive use be written in a compiled languages gives hope that they might be fast enough to not be annoying.

Me personally I do not install nodejs on my machines. So knowing that this tool is not written in JS is relevant for me.

> There are a lot of command line tools that are written in JS and other scripting languages.

???

Also, this tool replaces another which was written in Go, which I would put in a similar performance category as Rust. It shouldn't make a difference in this scenario

I prefer knowing what language things are written in. Not only is it interesting, but i like knowing what sort of installation i have to look forward to, if i'm interested in contributing, etc.

If they didn't mention the language, it would be an immediate question. The main focus is parallelism.

Could the Go solution add parallelism, sure. Did they? Not yet. Does that mean no other improvement in any other language can ever be written? No.

"Rust is faster" as an off the cuff comment that should have been left out seeing it has triggered some folks to hyper focus on that point.

> Could the Go solution add parallelism, sure. Did they? Not yet.

They did: Lefthook lets you define a "group" where you can specify that every command should be done in parallel: https://lefthook.dev/configuration/group.html. In addition, it's possible to configure the whole hook to run in parallel through another property: https://lefthook.dev/configuration/parallel.html.

I'm assuming that Hk's innovation here is that it's a bit smarter with what it runs in parallel. Maybe it uses the globs to automatically run commands in parallel which targets different files?

> "Rust is faster" as an off the cuff comment that should have been left out seeing it has triggered some folks to hyper focus on that point.

It's not a a hyper focus: This was the first reason (out of only three) that Hk itself presented as a reason to use it over Lefthook. So yes, I agree: It should have been left out if the intention wasn't for people to focus on it. Put it somewhere in a footnote if it's not so relevant.

> Why mention Rust at all?

I like knowing what language a tool is written in. If it's written in Python or JavaScript and it isn't something that's absolutely essential I can just immediately move on. It also lets me know if it's something I'd be willing to contribute to. It's odd that the authors mentioning the language is so triggering for you.

Overall, the whole logic "A is written in X, but B is written in Y, hence it is faster" is so deeply flawed, that one has to consider a possibility that the author doesn't understand the mere basics of computer program engineering. There are so much nuances that I don't even want to start untangling it.

Agreed. And for tools like this (which are light wrappers around much heavier tools), the difference in performance is insignificant. It’s all the other factors that start to matter.

mise (née rtx), by the same author, could confidently claim this because the tool it replaced – asdf – is written in bash. Bash is perfectly capable for a great many things, but unless you’re getting fairly deep into the weeds of what it can do (e.g. eschewing external calls to sed et al. whenever possible in favor of parameter substitution), you’ll tank performance from sub-shell spawn times. mise was objectively and obviously faster out of the box, though nearly any language that included basic text processing functionality would’ve been acceptably fast.

In general though, I agree that the blanket statement of “X is good because it’s language Y” is absurd, though I stubbornly cling to the opposite case for NodeJS, because I despise the idea of a frontend language running anything but a browser window. I have no objective defense.

`asdf` has now been rewritten in Go for anyone interested, so choosing mise over it due to performance is less of a concern now.

Oh, good to know! Kinda sad that they gave up on bash; I wonder how they’ll differentiate themselves now.

I had submitted a PoC patch for the bash version that greatly sped it up, but there were a couple of tests I was struggling to get to pass, and I ran out of free time.

It says should be much faster, which makes me think they’re talking about performance but haven’t actually compared it.

I find hk's choice of programming language for hooks pretty exotic TBH.

This all made me look at lefthook, and it’s pretty darn interesting. I’d been missing out!

I haven't even built it, let alone benchmark it. I don't think people are realizing this thing is like a 3 day old project (in terms of actual work). The about page is a bunch of scratch I wrote today (and didn't even finish that doc before I moved onto something else).

This project is brand new. I've only spent 3 days on it and this doc I half finished today. That said, I just did some benchmarks and it certainly is much faster than its peers. 4.6x faster than lefthook and 6.1x faster than pre-commit (in the best case).

I'll put these benchmarks on that page in different scenarios.

Update: https://github.com/jdx/hk/blob/main/docs/public/benchmark.pn...

As I said in the doc I think real-world performance in a large codebase will show that hk is even faster still—though that will depend on the project in question. It's really just a matter of providing the right levers in the right places and having good defaults.

Despite what everyone here says: yeah, just doing CLIs in Rust will be faster than Go and for CLIs like this milliseconds matter.

I'm sorry, but it's very hard to take this project seriously: You originally claimed that it was 1.5x faster than Lefthook, but then you removed the parallel support from the Lefthook (https://github.com/jdx/hk/commit/ad473331db2866f6574555ac4b0...) and changed your claim to "4.6x faster". These numbers are also just for some a specific workload (Prettier, ActionLint, pkl eval, ripgrep, cargo fmt) on this specific repo. For now the "4.6x number" is heavily based on the fact you have five jobs and Hk runs commands in parallel by default while you let Lefthook run them sequentially. I bet you can turn that number into 10x if you just add a few more jobs!

> Despite what everyone here says: yeah, just doing CLIs in Rust will be faster than Go and for CLIs like this milliseconds matter.

You've shown nothing to suggest that this is true. On my computer "lefthook --help" is exactly as fast as "hk --help". There's also many other differences between these tools. YAML vs Pkl is one difference. Another one is that Lefthook shells out to "git" to determine the list of staged files, while Hk uses libgit2. Which of these are faster? I'm not sure! It might even depend on repository size and/or other details. And as we all agree: In practice the parallelism strategy will matter the most.

> As I said in the doc I think real-world performance in a large codebase will show that hk is even faster still

I'm absolutely sure that you will be able to tweak hk to become the "fastest" pre-commit runner in your designated category. I'm also pretty sure that similar optimizations will apply quite easily to Lefthook. They're after all doing pretty much the same thing.

of course I selected the best benchmark. Once I ran a benchmark that had higher numbers I edited my response—I was up front about that. So far 0 benchmarks show that lefthook is faster and I doubt there could be one knowing how both of them work. I'm not just creating scenarios where I know hk will outperform but I'll certainly highlight the best one.

> then you removed the parallel support

that was not intentional. I fixed that, but it didn't change the results that much:

https://github.com/jdx/hk/commit/dfe1fc1724b8f6c43b184dc98ac...

In any case, I don't know why anyone would take such a new project "seriously". I certainly don't.

Thanks for creating and sharing this :) Commenters here can be pretty awful so I wanted to say that I really enjoy using the tools you have created.

My question is how do I get this stuff into the repo itself so that it is replicated on a clone/fetch/pull?

I don't want to have to have 3rd party tools (including things like PRs/issue management) that don't record their info in the repo itself.

Should be possible with the content addressable store underneath and new blob types? Or will that break too much of the existing git?

nix develop

Sorry how is this relevant to wanting to have issues and commit hooks as part of a repo that is cloneable and shareable in the same way as the existing

commit->treeish->blob

and refs?

I appreciate the comparison to the leading existing tools, but it also assume a familiarity with them. This doc could also use a basic introduction for people who are new to this area and don't already use a hook manager. Examples of hooks, what the benefit is, etc.

While your concern is valid, I recommend learning about git hooks from their canonical source rather than from the documentation of any hook manager. Hooks are features of Git itself. Hook managers are an after-thought. Meanwhile, this tool and its docs also appear to be works in progress. I use pre-commit (a python program, not the hook itself) as a hook manager. But nothing beats the git book and docs. Here are some links:

1. https://git-scm.com/docs/githooks

2. https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks

Here is another site dedicated to git-hooks. You'll find some good examples and resources there:

https://githooks.com/

I wrote a tool in this space as well, precious (https://github.com/houseabsolute/precious). It's odd to me that these tools are often talked about with such a focus on Git hooks. Yes, that is definitely one use case, but I also want to run this sort of thing in CI to check PRs, and I _also_ want to run it locally to apply _pretty-printing_ to new code, rather than just having it check the code that I wrote.

I think hk does do all those things, but it's a bit obscured by the focus on Git hooks in the docs. But the docs are also still in a super early state, so maybe that will be fleshed out more in the future.

I use https://github.com/cachix/git-hooks.nix which provides all of it:

- Pre-commit hook setup

- Run locally anytime (`pre-commit run -a`)

- Check in CI (as Nix flake check)

Example repo: https://github.com/srid/haskell-template

The pre-commit configuration: https://github.com/srid/haskell-template/blob/master/nix/mod...

For me the issue to be solved is how to easily get the team to run the tool. Some focus should be on integrating build tools.

When I or a team member do gradle build, npm build or cargo build or similar on a fresh checkout the tool should ask to be installed.

That is what mise from the same author is for. It lets you define per-project sets of tools with specific versions (including languages). I find it quite useful and I'm planning to get my team at work to adopt it once I get some round tuits.

I would use a wrapper sh script that checks for the tool and prompts an install, basically how Java projects include maven or gradle into a repo

One of the goal of hk is to be used with mise (https://mise.jdx.dev/dev-tools/)

mise supports an experimental bootstrapping feature: it can download itself and install the tools required for the project.

See https://mise.jdx.dev/cli/generate/bootstrap.html and https://mise.jdx.dev/continuous-integration.html#bootstrappi...

I don't know if this is well-known or not, but nix flakes are _amazing_ for managing and configuring cross-platform hooks.

In less than 40 lines of nix, most of which is boilerplate, you have fully cross-platform automatically installed and declaratively managed hooks, as part of your repo.

Need to run it in CI too? Well, no problem! Nix runs equally well in CI as it does locally.

Check this flake for example:

https://github.com/Azeirah/remarks/blob/main/flake.nix

See line 62 in the shell script.

This flake:

1. Manages ALL my dev env dependencies. Including even the specific version of bash that the script is running on, also the specific python dependencies down to the bit.

2. It's (posix-compliant) cross-platform, so it runs on MacOS, WSL, Linux, NixOS, ARM, x86 etc. Also docker.

3. It uses no specialized tools other than Nix, which is now a 20 year old Linux project which is quickly gaining even more traction. Nix is a programming language for reproducible, reliable and declarative dependency management (think docker but with a pure and functional programming language, rather than a recipe with installation instructions)

4. It also creates binaries as well as docker images from the binary. The docker image is like 2 lines of extra code.

Highly highly recommended. It's a bit difficult to wrap your head around initially, but holy damn if all dependency management problems don't just magically disappear forever once you learn Nix.

And I mean ALL of them. Whether it's Linux packages, docker containers, development environments, CI, virtual machines, containers, compilers, C headers from a 1970 Bell Labs project. No matter the architecture or OS (except Windows! :D).

Oh, did I mention it has a lockfile? So you can rollback or upgrade piecewise? Whenever you want?

It's like a programmable version of the superset of npm, cargo, pip, apt, brew, composer, gem, make, cmake, docker, git, Jenkins and even Linux itself (if you dare go the way of NixOS)

Nix is slow enough that having it in your pre-commit gets annoying

Haven't run into that yet, but can imagine that it gets worse if your devenv is large.

I'm sure it's something that can be engineered away and optimized in the Nix core though. Because yeah it isn't a hyperspecialized tool for specifically git hooks alone written in Rust. No.

It's not a saw nor a hammer. It's an entire workshop.

Flakes are less of a workshop, but more like a bus with tools or a heavy-duty toolbox.

When combined with direnv, the nix portion is almost entirely cached, so there's nothing to make it slow. I use pre-commit hooks via devShell in many projects, and they're nearly instant. The only exception is when the hook itself is slow.

Putting aside that blanket statement, that just makes no sense whatsoever. You use Nix to generate pre-commit configuration. That configuration contains unique paths to linters. Those paths will always point to an identical build of a Nix package, ensuring consistency. There's no room for Nix to slow down the pre-commit hook, because it doesn't run during the execution of the hook at all.

And since this thread is talking about git hooks, there's also this project for managing git hooks inside Nix https://github.com/cachix/git-hooks.nix

Git hooks are like make files for me. Copy/paste that `git staged files` command and then add your linter commands (in .githooks/pre-commit file). Add `set -e`. Set package manager post-install hook to config git to the `.githooks` dir.

Sadly I think git hook managers remove the simplicity of the whole design. Understandably since no one reads manuals anymore and projects don’t mind tacking on yet another module/plugin, however easy.

I can get behind a hook manager that's a single binary. Where I have an issue is with (for instance) python hook managers where you now need to have python interpreter management that's aware of both the python app you're working on, and the tooling which might have a non-intersecting version requirement.

While I understand your valid criticism with the overhead of using an interpreted language here, I must point out that using multiple versions in the same repository is quite an antipattern that comes with various complications. Maybe you should consider switching to a version manager of[0] your[1] choice[2].

[0]: https://github.com/jdx/mise

[1]: https://github.com/asdf-vm/asdf

[2]: https://github.com/version-fox/vfox

You have misunderstood my criticism. I don't care in the slightest about the interpreter overhead. I care that I am forced into needing multiple versions in the same repository by having one version required by (for instance) a hook manager, and a different version required by the project itself. I am already using a version manager; it doesn't solve that problem.

The big difference with Makefiles is that git hooks can’t be committed to the repo. Hook managers allow hooks to be shared and updated across a team.

Committing git hooks to the repo is possible. These are the 2 ways in which they're commonly handled:

1. Link the scripts from the worktree to the .git/hooks directory - perhaps using a bootstrap script.

Ref: https://codeinthehole.com/tips/tips-for-using-a-git-pre-comm...

2. Declare the directory in the worktree to be the local git hooks directory.

Ref: https://knpw.rs/blog/direnv-git-hooks

So do dev-environment start-up scripts, which in my experience are always simpler, clearer, and more maintainable than these managers.

Just put your hook logic in a script and copy it to the git hooks folder on startup as necessary -- and then, voila, you've avoided the nonsense of some well-intentioned package whose author thinks Rust in git hooks is a selling point rather than a head scratcher.

Hooks shouldn’t be shared and updated across a team. Don’t force your workflow on others. Which is exactly why you can’t or shouldn’t commit them.

What is the technical justification behind such as statement? Sharing hooks across the team doesn't equate to forcing a workflow on them. Each developer must manually activate the hooks in some manner in all the cases I've seen. Otherwise, they work as usual without the hooks and without interfering in the workflow in any manner. As a corollary, the hook conditions are enforced in the CI (since it can't be enforced in the developer's system). Git hooks are instead offered as a convenience function to help the developers catch mistakes early and often, before it hits the CI.

100%, I was using husky because we were using it at work. But it turns out for my use-case all I needed was this in my package.json:

    "scripts": {
        "postinstall": "git config --local core.hooksPath etc/hooks"
    },

Ooh, core.hooksPath is quite nifty. I usually use something like

         ln -sf ../../scripts/git-pre-commit-hook .git/hooks/pre-commit

which simply adds a pre-commit symlink to a script in the repo's scripts/ dir. But hooksPath seems better.

Sometimes I wonder if the death of blogging - which in turn killed a lot of showing people how to use a tool where they’d otherwise skip the manual - means we entered a world where people jump immediately to building a tool for an otherwise simple concept.

My git pre-commits are usually calling other executables or other tools, so I'm not quite sure how much it matters how fast is the git pre-commit tool. Can we have a benchmark of a normal use case of how much time are we saving with this new tool?

or is this "Yet Another Rust Rewrite" (YARR)?

Being written in rust instead of python or go probably doesn't matter that much. But being able to run multiple tasks in parallel, which hk claims to do, could make a big difference.

this is written by the author of mise

Interesting. I've been working on a pre-commit replacement too, written in Rust but using WASI for all plugins (no exceptions!). I haven't got very far but I think this will have huge advantages over pre-commit, mostly in reliability.

Me and my colleagues have had numerous issues setting up pre-commit because it inherits Python's atrocious infrastructure.

I'm curious how this is going to deal with actually running plugins? Will it take the same approach as pre-commit and add dedicated not-very-good support for a load of different languages?

Not gonna say Python isn't a mess but I'm surprised a self-contained application is so bad. All the pieces are there to have it work— give it its own venv, use wrappers or the shebang so transparently uses it, and plug-ins get installed into that venv. All should be happy.

Is it that difficult to use `pipx install` or `uv tool install`?

Yes. Those are both more effort and less reliable than what I am planning (a static binary with WASI plugins).

I agree to both. It just sounded to me like it is an unbearable experience, which I wanted to correct. Certainly, on a clean slate, I prefer a static binary

You will be able to use mise to set up any tools required for the linters. See https://mise.jdx.dev/dev-tools/

I also looked into WASI as well as lua until I ultimately decided that I think I can get rid of the concept of plugins altogether in favor of pkl. We'll see how well that works out but right now I don't have any use-cases that would require plugins and if I can stick to that it will definitely help performance.

I don't really understand. Pkl seems to be a configuration language? How are you going to run clang-format, rustfmt, go fmt, pyright, etc. using Pkl?

it just needs to define shell script

So you have to already have all the linters installed? Ok I am planning to solve that problem too, but we'll have to see whether it's actually possible to compile e.g. rustfmt to WASI. In my experience so far WASI is pretty alpha quality.

mise is my solution to that problem. I don't think it should be the job of a hook manager to install things developers should have installed for other reasons anyways.

Pre-commit has like the worst DX to me, so I'm really looking forward to a replacement.

My biggest gripes are:

1. Pre-commit hooks are designed to modify files in-place, which turns `git commit` into something that can alter your work tree. IMO those tools should never ever modify files (unless explicitly asked by the user) and only output a diff and exit code that signifies if the check failed.

2. It manages the tools with its own environment but never exposes it. I always have to dig around its cache directory to find out which executable it is running if I have to reproduce the problem after something goes wrong.

I like Rust as much as the next guy, but statements like this makes me roll my eyes:

> hk is written in rust, pre-commit is written in python. hk will be much faster.

I have no idea how fast hk or pre-commit is, i have never used them. What matters to speed is the algorithms used and their complexities. If you implement a shitty algorithm with exponential complexity in Rust it's going to be slower than a linear complexity algorithm in python.

This isn't true when it comes to CLI tools, where fixed costs and warm-up time can easily dominate over algorithmic complexity, especially comparing a scripting language to an ahead-of-time language. Without careful construction, a tool written in python with whatever O(log(n)) time complexity may still be booting up by the time a Rust tool finishes the job in O(n^2) time: frequently n is actually pretty small outside whole-repo builds, and the cost of interpreter is high especially with accompanying tooling -- python3 itself will run `print("hello")` on my machine in 40ms, but when wrapped in pyenv (which I think is typical?) it takes 400ms. The same goes for node/npx and ruby/bundle. Compare to an ahead-of-time compiled binary from go, which can boot and print hello in 3ms on my machine -- it has between 10x to 100x the wallclock budget the python program just spends booting up.

These days I'd say tools pipx (and now also uv) are far more common than pyenv for tool management

Reply on news.ycombinator.com