6,499 karma · joined November 17, 2015
That sounds pretty cool! I will definitely take a closer look.
- very fast? run it all the time (shell prompt drawing, if you want, like direnv)
- fast? run it in a pre-commit hook
- a bit slow? run it in a pre-push book
- really slow? run it in CI, during code review, etc.
Fwiw: I also rewrite history often-ish but it's never that fast for me because I have commit signing turned on and that requires verifying my presence to a USB smartcard on each commit. For me, it's okay if a commit takes a second or two. As it creeps up beyond 3 or 4 seconds, I become increasingly annoyed. If a commit took a minute I would consider that broken, and if I were expected to tolerate that or it were forced on me, I'd be furious.I generally expect an individual pre-commit hook to under ~200ms (hopefully less), which seems reasonable to me. Some of the ones we have are kinda slow (more than 1s) and maybe should be moved to pre-push.
Since you seem especially sensitive to that latency, here's what I'd propose if we worked together:
If you own a repo, let's make all the hooks pre-push instead of pre-commit. On my repos, I like many hooks to run pre-commit. But since the hooks we use are managed by a system that honors local overrides via devenv.local.nix, let's make sure that's in .gitignore everywhere. When I'm iterating in your codebases and I want more automated feedback, I'll move more hooks to pre-commit, and when you're working in mine you can move all my hooks to pre-push (or just disable them while tidying up a branch).
But if you don't want to have hooks modify code, in a case like this you can also just use `tofu validate`. Our setup does `tflint` and `tofu validate` for this purpose, neither of which modifies the code.
This is also, of course, a reasonable place to have people use `tofu plan`. It you want bad code to fail as quickly as possible, you can do:
tflint -> tfsec -> tofu validate -> tofu plan
That'll catch everything Terraform will let you catch before deploy time— most of it very quickly— without modifying any code.
> you can't force folks to run them
I think it's useful to be able to disable things like this when debugging or reconfiguring them. I sometimes disable the ones I've set up for myself, then reenable them later.
> Why not just call a shell script directly?
Because it's manual; I have to remember to do it each time. But there's no reason not to have a script you can invoke in other ways, if that's something you want.
> How would you use these with a CI/CD platform?
The thing that sets up your environment also installs the git hooks configuration, and/or you can have it invoke specific hooks via CLI.
If you use a tool like this via Devenv instead of using its built-in mechanisms for installing tools:
- you can add a linter without putting it on your path
- you can put a linter on your path without enabling any git hooks for it
- if you are already using a linter in a git hook, adding it to your environment will get you the exact same version as you're already using for your git hook at no additional storage cost
- if you are already using a linter at the CLI, and you add a git hook for it, your hook will run with the exact same version that you are already using at the CLI at no additional storage cost
- your configuration interface is isomorphic to the upstream one, so
- any custom hooks you're already using can be added without modification beyond converting from one format to another;
- any online documentation you find about adding a custom hook not distributed with the upstream framework still applies;
- and you can configure new, custom, or modified hooks with a familiar interface.
- any hook you write as a script or with substantial logic can also be plugged into a built-in task runner for use outside git hook contexts, where
- you can express dependency relationships between tasks, so
- every task runs concurrently unless dependency relations mandate otherwise.
Which imo solves that problem pretty well. My team uses that kind of setup in all of our projects.> The interface isn't built with parallelism in mind, it's sort of bolted on but not really something I think could work well in practice.
I'm curious about what this means. Could you expand on it?
You might be interested in something like `treefmt`, which is designed around a standard interface for configuring formatters to run in parallel, but doesn't do any package management or installation at all:
https://github.com/numtide/treefmt
(That might address both of the issues I've replied to you about so far, to some extent.)
> It also uses a bunch of rando open source repos which is a supply chain nightmare even with pinning.
If the linters you're running are open source, isn't this what you're ultimately doing anyway? Nix gives a bit more control here, but I'm not sure whether it directly addresses your concern.
> I am working on a competing tool
Oh, dammit. Only after writing all of this out did I realize you're the author of mise. I'm sure you're well aware of Nix and Devenv. :)
Because I think your critiques make sense and might be shared by others, I'll post this comment anyway. I'm still interested in your replies, and your opinion of having some environment management tool plug into prek and supplant its software installation mechanisms.
And because I think mise likely gets many of these things right in the same way Devenv does, and reasonable people could prefer either, I'll include links to both Devenv and mise below:
When Android was new, I very frequently used Termux and ConnectBot with my first few Motorola Droid phones. For a brief moment, I had a working phone with a great physical design only held back by an outdated chipset and being locked to Planet Computers' abandonware. I could touch-type at 80 WPM on an easily pocketable device! Termux shone there.
So many things about Android were not just more exciting in terms of potential when it was new, but actively better: wider variety of hardware, widely unlocked bootloaders, no remote attestation, etc. Termux sadly feels like a painful reminder of that to me.
And for more modern software distribution mechanisms (e.g., Nix, Guix, Flatpak), centralized package updates may not actually run any vendor code with high privileges at all.
The norm for proprietary software updates on Windows is indeed a free-for-all of every publisher downloading and running code with admin rights, and it is indeed a terrible way to operate. Avoiding that kind of madness doesn't necessarily mean running lots of old, vulnerable software.
That's essentially being done with Flatpak.
Linux is largely still built on the old (and indeed, outdated) Unix trust model. The system itself is assumed to be trusted, and the primary security boundaries on the system are drawn between users. Since Linux package managers actually install and manage the base system as well as end-user software, anything the package manager installs is treated as part of "the distribution", and thus trusted. It's not a good idea to use such a thing to install proprietary, third-party software. The curation and vetting of the distro maintainers is actually vital here, and when you add a third party repo, you're giving it a lot of trust. At the same time, why would distro maintainers give free labor to integrate proprietary software? Most are not super interested in that, and even if they are, they don't generally have the rights necessary to redistribute, let alone modify, proprietary software. On the other hand, those third-party developers and publishers don't want to master and manage a half-dozen different packaging formats, and various other packaging ecosystem differences that vary across distros.
Flatpak is positioned to solve all of these problems, and it's no secret that enabling (relatively) responsible use of proprietary software is one of the goals. It enabled distributing a small number of large, common runtimes of which different versions can safely coexist on the same system, addressing fragmentation. To reduce the amount of trust given to installed apps, it separates what it installs from the base system, and offers sandboxing to help limit the permissions granted to an app that still runs under the OS user of the person using it. And it supports third-party repos that publishers can run themselves.
I'm not currently a daily Flatpak user, so idk how much the current reality lines up with that goal, but that's where the movement towards this is on the Linux desktop today.
If you can be respectful and act like a guest, it's worth reading a little there. You'll see the worrisome aspects in more detail but also a level of savvy that sometimes seems quite strange given the level of attachment. It's definitely interesting.
- a large number of incredibly fragile users
- extremely "protective" mods
- a regular stream of drive-by posts that regulars there see as derogatory or insulting
- a fair amount of internal diversity and disagreement
I think discussion on forums larger than it, like HN or popular subreddits, is likely to drive traffic that will ultimately fuel a backfiring effect for the members. It's inevitable, and it's already happening, but I'm not sure it needs to increase.I do think the phenomenon is a matter of legitimate public concern, but idk how that can best be addressed. Maybe high-quality, long form journalism? But probably not just cross-posting the sub in larger fora.
For example?
I recently picked up a used New 3DS XL to give this a try for the first time, and I actually really like it. With the eye tracking, it's pretty good imo. Did you also try the later iterations of the hardware?
For both scripts, everything interesting is installed via Nix, so there's little reliance on special casing various distros', built-in package managers.
In both cases, all scripts have to pass ShellCheck to "build". They can't be deployed or committed with obvious parse errors or ambiguities around quoting or typos in variable names.
In the case of the scripts that are tools for developers, the Bash interpreter, coreutils, and all external commands are provided by Nix, which hardcodws their full path into the scripts. The scripts don't care if you're on Linux or macOS— they don't even care what's on your PATH (or if it's empty). They embrace "modern" Bash features and use whatever CLI tools provide the most readable interface.
Is it my favorite language? No. But it often has the best ROI, and portability and most gotchas are solved pretty well if you know what tools to use, especially if your scripts are simple.
Imo there's an identifiable core common to all of these kinds of package managers, and it's not terribly hard to work out a reasonably good hierarchical ontology. I think OP's greater insight in this section is that internally, every package manager has its own ontology with its own semantics and lexicon:
> Even within a single ecosystem, the naming is contested: is the unit a package, a module, a crate, a distribution? These aren’t synonyms. They encode different assumptions about what gets versioned, what gets published, and what gets installed.
The uneven terrain also makes package managers more interesting to compare to each other than many other kinds of software, imo.
Codeberg at least has some integrity around such things.
This exact frustration (in his case, with a printer driver) is responsible for provoking RMS to kick off the free software movement.
There are a ton of other open-source process supervisors you can use to manage long-lived processes in a portable way, too, notably Foreman[8] and various clones written in languages other than JS, and GNU Shepherd[9]. In the course of writing this post, I discovered one called dinit[10] which looks sort of similar to s6 and the GNU Shepherd in that it supports both pure usermode operation as well as functioning as an OS's init system. Anyway, all of 'em are in Nixpkgs, so you can pick them up and use them without any packaging work, too.
Service orchestration and containers are basically orthogonal concerns. Before Docker was born, there were already plenty of portable tools for standalone "process supervision", "service management", whatever you wanna call it. So it is after Docker, as well.
If I needed this for one of my dev environments I would take a look at process-compose to decide if it's acceptable to me. If it isn't, then after surveying the contemporary landscape of usermode service managers, I'd then write a devenv module that generates configs for it, and use that.
> Should I run Nix in Docker, Docker from Nix, or is there a nix-only solution for this?
I'd do this in a "Nix-only" way if possible, but if it's convenient for you to run a service via Docker (or Podman or any other container runtime), you can still do that.
If you can safely assume that all of your devs have it available, you can ship the client (`docker` or `podman` CLI or whatever) as part of your Nix environment, then have your process manager launch it via that command line interface. I'd avoid running Nix from within Docker for the purposes of development environments.
--
3: https://devenv.sh/supported-process-managers/process-compose...
4: https://f1bonacc1.github.io/process-compose/
5: https://sandervanderburg.blogspot.com/2020/02/a-declarative-...
6: https://github.com/svanderburg/nix-processmgmt
7: https://skarnet.org/software/s6/
8: https://github.com/ddollar/foreman