HNHacker News
TopNewBestAskShowJobs

tylerjl

201 karma · joined February 19, 2015

https://tjll.net

[ my public key: https://keybase.io/tjl; my proof: https://keybase.io/tjl/sigs/jmqcf9anf6zPbl4QuF-2s2cfI7FVEV42LvPTqD2zr08 ]

submissionscomments
tylerjl··on Self-host your mail server
I've hosted and actively used my own email server for > 10 years and there are two things that are true at the same time:

It is absolutely possible to host, use, and achieve decent deliverability with your own server. It takes work and is a different beast than running a webserver, but that frontier is not dead (yet).

At the same time: it is an ABSOLUTELY TERRIBLE IDEA to suggest that folks operate an email server out of residential IP through your ISP. The spam prevention ecosystem is so draconian now that email traffic flowing out of address ranges belonging to residential owners is yeeted into the sun. If you've operated email infrastructure at decently large scales, you know that SPF/DKIM/DMARC are just one part of how spam is identified, and mixing in the source of email traffic into the detection heuristics is a significant part of that identification process. Achieving decent delivery can often involve "warming up" an IP over time to establish its reputation, and consumer address ranges are a hodge-podge of activity that incur very little trust about what they're doing.

I agree with some commenters that exhort the exercise of self-hosting email to prevent the venerable protocol becoming a homogeneous GMail and Outlook-only world, but there are reliable ways to do this: using a VPS, relying on a company for the sending/SMTP part, or something similar. But footguns abound, and disgorging SMTP packets from your LAN edge is one of them.

tylerjl··on Elastic lays off 7% of employees
It's interesting to contrast this announcement with a similar post from the CEO in 2022 [1]: those past layoffs had much more of a victim-of-circumstances tone as ZIRP was beginning to dry up, but apparently those "bad times" versus "good times" during AI mania just accounts for a delta of +6% additional layoffs.

Another commenter questioned what size bucket Elastic falls into these days; in April 2025 their SEC filing [2] cited about 3,500 employees. So not a startup any more but definitely not fully-fledged FAANG-sized.

(not sure whether it even applies here; but full disclosure, I left Elastic in 2022.)

[1]: https://www.elastic.co/blog/ceo-ash-kulkarni-email-to-elasti... [2]: https://www.sec.gov/Archives/edgar/data/1707753/000170775325...

tylerjl··on Love systemd timers
Thanks for the kind words! Especially given how ubiquitous systemd is now, skilling up on the toolbox with commands like `systemd-analyze` and `systemctl list-timers` feels super valuable.
tylerjl··on Love systemd timers
Hey everyone, author here. I spotted the hn traffic a little late but I'm happy for any feedback or comments and will try and address the top-level comments as I can.

(Aside: I wrote this article early last month but it caught on only just recently. For better or worse, touching a third rail topic like systemd seems like a sure-fire way to elicit strong and numerous reactions both positive and negative.)

tylerjl··on Blog Post Is Your Sign to Start Self-Hosting
Oh hey, I'm the author. Happy to chat about the post if anyone wants to. (Also the title here probably needs the leading "This" to make sense)
tylerjl··on Ask HN: Share your personal website
https://blog.tjll.net/
tylerjl··on Why Self-Host?
Another sort-of-recent development in the space has made self-hosting dramatically more accessible: even though hardware costs were reasonable before, they're now _very_ reasonable and also resource-efficient.

Repurposing an old tower would offer you enough compute to self-host services back in the day, but now an Intel NUC has plenty of resources in a very small footprint and branching out into the Raspberry Pi-adjacent family of hardware also offers even smaller power draw on aarch64 SBCs.

One experiment in my own lab has been to deploy glusterfs across a fleet of ODroid HC4 devices to operate a distributed storage network. The devices sip small amounts of power, are easy to expand, and last week a disk died completely but I swapped the hardware out while never losing access to the data thanks to separate networked peers staying online while the downed host got new hardware.

Relying on container deployments rather than fat VMs also helps to compress resource requirements when operating lots of self-hosted services. I've got about ~20 nomad-operated services spread across various small, cheap aarch64 hosts that can go down without worrying about it because nomad will just pick a new one.

tylerjl··on Systemd has been a complete, utter, unmitigated success
One convenience of journald is that it exposes a single place to plug in log collection for observability tooling

opentelemetry-collector, promtail, and so on have native plugins for it, which makes aggregation easier to setup

Most tools have "tail this plaintext file" as well, but if it's all flowing to journald, setting up log collection ends up being that much simpler

tylerjl··on Systemd has been a complete, utter, unmitigated success
Hi, author here. The title is intentionally a little attention-grabbing hyperbole but I've appreciated the discussion and feedback about the post. Thanks for reading!
tylerjl··on Jazz² Resurrection: Open-source Jazz Jackrabbit 2 reimplementation
I loved this game growing up, I'm definitely going to give this a try.

As a minor observation, I'm pretty (pleasantly) surprised that the project provides a NixOS package for the application. As somebody who tends to flit between Arch, Fedora, and NixOS, seeing packages for NixOS before Fedora availability is a very surprising signal in terms of Linux distribution popularity.

tylerjl··on Ask HN: Could you share your personal blog here?
https://blog.tjll.net/

The posts that made it to the top of HN (that I can recall at this moment)

35 Million Hot Dogs: Benchmarking Caddy vs. Nginx (https://blog.tjll.net/reverse-proxy-hot-dog-eating-contest-c...)

SSH Kung Fu (https://blog.tjll.net/ssh-kung-fu/)

Building My Ideal Router for $50 (https://blog.tjll.net/building-my-perfect-router/)

I've been using nix/nixos a lot lately and will probably end up publishing more in that general area of interest. That and my excessively-overengineered homelab.

tylerjl··on MRSK: Deploy web apps anywhere
Slightly off-topic, but my ChatGPT instincts always tend to fire whenever I read copy that concludes with the formula, “Ultimately…” or “Overall…”.

I wonder how much (if any) of this landing page was GPT generated.

tylerjl··on Polymode: Multiple Major Modes and How to Use SQL and Python in One Buffer
It's kind of hard to fully appreciate how powerful polymode is without actually _using_ it. When you use an editor that can mostly-understand the syntax behind markup like fenced markdown code blocks like this:

```ruby

puts "Hi"

```

You may be used to native syntax highlighting take over, but while you're in polymode, moving the cursor into the Ruby portion of the code actually activates Ruby-mode - any code checking (with something like flymake) works as expected, any major-mode specific actions that reformat code like M-q or == (while evil is active) will do what you expect, and polymode isn't just limited to fenced markdown.

Like the linked article demonstrates, polymode is super flexible. I myself wrote my own, small polymode that watches for comments like /* python */ that precede literal string blocks (which look like ''print("Hi")'' and are usually multi-line) in nix code to transform regions into their native modes (it doesn't need to just be python) and it's really great.

You can see an animated description of this here: https://twitter.com/leothrix/status/1597327756102336512

edit: my fenced block example was weird

tylerjl··on Ask HN: Self-hosting in 2023: Nextcloud on Linode, or...?
That's a great plan! I'll say that self-hosting may be _the_ number one thing I'm most passionate about due to concerns similar to yours (privacy, ownership, and so on). I've self-hosted many of my own services for a very long time and so I have my own experiences to share as well.

I'll say right off the bat that I don't see any red flags with your proposed plan. The following bullet points are primarily meant to offer some additional options or mental nudges to help you brainstorm - like I said, there's nothing abjectly wrong with your architecture, so this list may just offer more ideas:

- I've self-hosted a few email servers (and still do) and I think punting on that (or just doing the backup plan) is probably the right approach - you can DIY it today, but it's a part-time job. If you ever do decide to take ownership of your email, bringing your own domain to Fastmail or Proton Mail has also worked well for me. Today I host one domain on Linode and one on Ramnode. As with most things email, there are tons of nuances with doing it yourself - I had to get both my email servers' public addresses placed on an allowlist with their respective providers.

- I self-host most of my services on my own hardware in my homelab. I eschew the big, expensive, loud, power-hungry hardware in favor of smaller, cheaper, and swappable hardware, and the strategy has worked out really well. I primarily use ODroid hardware (they offer both ARM and x86-64 hardware). You mentioned a floating/non-public address as a constraint, so you could still do this with tailscale/headscale/something similar and gain the benefit of cloaking your services inside a private network (and using some public/free cloud instance as a low-power VPN endpoint). I don't think DigitalOcean/Linode are bad choices, but I very much like owning the hardware layer as well.

- I've been self-hosting before Nextcloud existed and used its progenitor (ownCloud) and developed a harsh distate for the huge, sprawling complexity of the system (it was hungry for resources, broke on upgrades constantly, etc.). That story may be better now, but I've sinced move on to hosting very targeted, smaller services. For example, instead of Nextcloud's file syncing, I run syncthing everywhere, and instead of Nextcloud's calendaring, I run radicale. Nextcloud will probably be fine, but I've been happier with running a smaller collection of services that do one thing well (syncthing in particular is an exceptional piece of software)

I could really ramble on but I'll just include a list of the stuff I host if you have any questions about it. I blog[1] about some of these, too: Transmission, Radarr, Sonarr, Jackett, Vaultwarden, espial, glusterfs, kodi, photoprism, atuin, Jellyfin, Vault, tiny tiny rss, calibre, homeassistant, mpd, apache zeppelin, and minio. Outside my lab hardware I run a few instances of nixos-simple-mailserver, mastodon, and goatcounter (used to run plausible). I also run a remove ZFS server that I mirror snapshots to as my remote backup solution.

[1]: https://blog.tjll.net/posts/

tylerjl··on Zero to Nix, an unofficial, opinionated, gentle introduction to Nix
> [...] am I understanding correctly that nix could be used instead of packer to generate a vm image/template for proxmox(or whatever hypervisor)?

That's correct, although as you mention later, it's limited to NixOS. It's an unfortunate constraint given how tremendously powerful it is, but the limitation makes sense. nixos-generators is literally a nix function that accepts a NixOS configuration and operates on the nix value in order to construct the VM image, so it wouldn't work quite the same without nix at the core.

Because nix "knows" about the total definition of the system, nixos-generators can do wonderful things like construct an AMI image or VMWare image without talking to any hypervisor APIs or cloud APIs at all - it knows how to construct the system from the "ground up" and you just end up with a big disk image file after the appropriate `nix build ...` command, no AWS keys or running Proxmox API required.

It's a tradeoff, to be sure. But if the use case fits - which for us, it does quite well - you can be outrageously productive. Adding an entirely new image format takes as long as typing a new line in `flake.nix`, and I leverage a common NixOS module between all the image types to invoke a quick and easy local qemu VM when I'd like to experiment locally.

It's become hard to imagine what managing a "normal" system would be like with the features I get by baking it all into a NixOS configuration.

tylerjl··on Zero to Nix, an unofficial, opinionated, gentle introduction to Nix
It's a good question, and a very mature/well-engineered Docker dev environment probably gets you near-parity with an equivalent nix setup. That said, my reasons are:

- Although not _all_ of our projects need nix builds in the end, at least a few do, and acquiring their devshells is essentially zero-effort (you just ask nix for the devshell for the package instead of the package output itself)

- As some other commenters have noted, dealing with large container contexts can get hairy/slow. A devshell just tweaks a few environment variables, which is less of a tangle when working on the various projects (I use direnv, so my emacs session hooks into `flake.nix` and finds everything in-path automatically)

- While you could get a bit-for-bit identical dev environments by pushing a built container image to a registry that all devs pull from, I think most people would write a `Dockerfile` once and let folks build it locally before hopping in, which leaves a small (but extant) possibility that some environments may be subtly different (shifting container tags, ad-hoc apt-get commands, etc). A flake.nix coupled with a flake.lock means all devshells are lock-step identical.

tylerjl··on Zero to Nix, an unofficial, opinionated, gentle introduction to Nix
> I would love to see a discussion from somebody who really likes Nix on why it isn't ready for prime time yet/just play devil's advocate aloud on why it isn't the greatest thing since sliced bread.

Top reasons in my mind:

1. Error messages. Even with my >1 year of experience using NixOS full-time, I've encountered errors that I simply _cannot_ fix. This is getting better (recent nix releases let you introspect problems more easily).

2. Documentation gaps. Much of the nix docs are actually pretty good now! But sometimes you run into "missing links" that make it really hard.

> What is a real world use case where Nix isn't overkill? I've read toolchains but... nvm (node version manager), rustup. I still Rust on a machine once and I never think about it again.

For me, nix is unbelievably powerful to construct system images for various virtual machine formats. I'm using the nixos-generators[1] project to construct 8 or 9 image formats from one NixOS configuration. Packer and similar tools are the non-nix analog, but nixos-generators requires essentially adding a singular line to support something like Proxmox as opposed to much more work in some other tool.

I'm also using nix to build all our team's software - which varies from Vue.js to Rust to Django - and fold _all_ those development dependencies into a singular nix `devShell`. This means you can clone our repository, use `nix develop .`, and all engineers use the identical verisons of all software, dependencies, and libraries across several toolchains. It's incredibly convenient. (Imagine that you're a .js developer who needs to make a quick edit to a Rust backend route but doesn't sling Rust all day - you don't need to know how to setup rust at all, the `devShell` provides it).

[1]: https://github.com/nix-community/nixos-generators

tylerjl··on Zero to Nix, an unofficial, opinionated, gentle introduction to Nix
I think the nix community is doing an increasingly-better job at addressing several outstanding issues, and although the OP link resolves one of them - the getting-started experience - the "why" that you highlight here is still a hard one to answer without getting into the technical weeds. I ran an informal Twitter poll a few months ago and this question (the "why" instead of "how") was the most-requested kind of content.

One potential answer to this is, "imagine building and running software with lockfiles for _literally everything_". I'm not just talking about _versions_ of dependencies or shared libraries - which nix does - but also things like:

- Locking the current point in time (nix resets the build sandbox to the unix epoch)

- Locking out network conditions (all build dependencies need to be fetched and therefore expressed as part of the instructions and not left to "at some point during the build")

- Locking out access to any system state (again, builds occur in a sandbox populated only with what you indicate within the build instructions)

That's what's behind the marketing for reproducability and repeatability. If you lift those principles into new and interesting applications, the various other uses for nix fall out of it:

- NixOS takes the principle of those nix builds and applies to it building not just packages, but the system entirely, like the files it places in /etc or the running kernel.

- Projects like devenv[1] or flake devShells in general re-use the portability of a fully-defined nix package to ship hard-to-break executables into share-able devshells so your peers can work with the bit-for-bit same version of Terraform or python (without worrying about what version of /lib/libssl.so they may have)

- Since nix "owns" the _entirety_ of the inputs and outputs to a piece of built software, shaping it into different artifacts becomes trivial. For example, as long as you're able to build a rust project with nix, nix easily lets you kick out a minimal OCI container (without needing to write a `Dockerfile`), or produce a tiny qemu clone of your system (or any system) by feeding your NixOS configuration into a function that produces images instead of configuring your running system.

Hopefully that's helpful, sorry if it isn't - but nix is sort of alien software, and nix people are still learning how to best share its potential with others!

edit: list formatting

[1]: https://devenv.sh/

tylerjl··on Zero to Nix, an unofficial, opinionated, gentle introduction to Nix
In your specific case - a _channel_ versus a flake _input_ - consider how you're tracking your system configuration. If you have an /etc/nixos/configuration.nix, then your system can be reconstituted _only_ if you have that configuration.nix in addition to the revision that your channel is currently on. Compare this with a system defined in a flake's `nixosConfiguration`, which accepts its version of nixpkgs from the flake's input, so you can rebuild/recreate the system from the flake entirely without needing to piece together other bits of state like the current channel revision/nixpkgs checkout.
tylerjl··on Where are my Git UI features from the future?
Apologies if this is only tangentially related, but it's a thought I've often had:

When I went through my university degree (which, admittedly, was over a decade ago), the skills and/or training for the day-to-day tools of software engineering - like git - were nearly never taught in a dedicated way in the same approach as concepts like data structures or algorithms. With degree in-hand the average graduate would probably be more comfortable reversing a linked list or estimating the O(n) of an algorithm than performing a git rebase. Yes, the latter is technology-specific rather than a generalized CS principle, but I would have to imagine that some capstone-type work would really spend time properly training a white-collar worker deeply on a standard few industry tools.

I'm not saying that the author is wrong, because git is indeed deviously inscrutable at times. However, after my twelfth or thirteenth failed merge or rebase, I really _dedicated_ some time to figure out what the hell git was doing from the ground up - which wasn't easy or quick - but I haven't felt truly mystified and incandescently angry at git since. It probably isn't fair to expect every Joe Software Engineer to spend that kind of time on each tool they're expected to use (I'm a masochist and enjoy doing it) but it bums me out to run into the occasional software engineer who never had the opportunity to engage in deep learning and/or practice about their closest tools like a nurse with an ultrasound machine or a court reporter with a stenotype. In a more perfect world a well-rounded software engineering education program would give people space for that, but we're left with "squeezing it in-between sprints" or "on weekends" because it's hard to imagine a company granting people ample time for professional development that doesn't transparently inflate OKRs.

Maybe my anecdote is out-of-date now and the situation is better, but I do still feel like most of us have had that "coworker had to blow away their local repo because git became too tangled" experience in recent memory. I also sort of assume that, in a paradoxical kind of way, code bootcamps may do this type of thing better, but I don't have that experience to draw from.

tylerjl··on 35M Hot Dogs: Benchmarking Caddy vs. Nginx
Thank you, that's very kind! There's a reason I included Corey's name in my hastily cobbled-together skeleton meme [1]. Hopefully my writing achieves that level of technical approachability.

[1]: https://blog.tjll.net/reverse-proxy-hot-dog-eating-contest-c...

tylerjl··on 35M Hot Dogs: Benchmarking Caddy vs. Nginx
Damn. This is probably worth swapping out k6 for if I manage to pull off a second set of benchmarks. Thanks for the heads-up.
tylerjl··on 35M Hot Dogs: Benchmarking Caddy vs. Nginx
This is hard because while, yes, some platform with less potential for jitter and/or noisy neighbors would help eliminate outside influence on the metrics, I think it's also valuable to benchmark these in a scenario that I would assume _most_ operators would run them in, which is a VPS situation and not bare-metal. FWIW, I did try really hard to eliminate some of the contamination in the results that would arise from running in a VPS by doing things like using the _same_ host reconfigured to avoid potential shifts in the underlying hypervisor, etc.

But I would certainly agree that, for the utmost accurate results, a bare-metal situation would probably be more accurate than what I have written.

tylerjl··on 35M Hot Dogs: Benchmarking Caddy vs. Nginx
Correct. Maybe it blows my credibility out of the water and I'll be shamed for life, who knows
tylerjl··on 35M Hot Dogs: Benchmarking Caddy vs. Nginx
That's very kind of you to say, thank you!
tylerjl··on 35M Hot Dogs: Benchmarking Caddy vs. Nginx
This is - along with some reverse proxy settings tweaks - one of the variables I'd be keen to test in the future, since it's probably _the_ most common delta against my tests versus real-world applications.
tylerjl··on 35M Hot Dogs: Benchmarking Caddy vs. Nginx
FWIW I'm a big fan of HAProxy as well, but I was just constrained by the sheer volume of testing and how rigorous I intended to be. Maybe once my testing is a little more generalized I can fan out to additional proxies like HAProxy without too much hassle, as I'd love to know as well.
tylerjl··on 35M Hot Dogs: Benchmarking Caddy vs. Nginx
Hey y'all, author here. Traffic/feedback/questions are coming in hot, as HN publicity tends to engender, but I'm happy to answer questions or discuss the findings generally here if you'd like (I'm looking through the comments, too, but I'm more likely to see it here).
tylerjl··on 35M Hot Dogs: Benchmarking Caddy vs. Nginx
Thanks, Matt! I've pushed the revised section measuring sendfile and metrics changes, so those should be accurate now.

Phew. Caches are purged, my errors are fixed. I can rest finally. If folks have questions about anything, I'm happy to answer.

tylerjl··on Our Roadmap for Nix
Scoped down to the problems that nix solves, I get the same vibes using nix as I did in the early days of docker. It solves painful problems, with a community and suite of tools large enough to be useful, and people who have paid the initial mental investment are happily using nix in lots of ways today.

But when you widen that scope to incorporate the rest of nix's warts, I'm not sure whether the project carries the requisite momentum to become a ubiquitous tool. I hope it does! The project has lofty goals and it really deserves a layer of porcelain to soften its usability and bring its benefits to more developers, because the actual, pragmatic results it delivers border on magic sometimes, and I wish more people were able to experience it. I don't blame people for avoiding that ramp-up time investement. I tried, gave up for many years, and only in the past six months or so devoted enough time to use it proficiently without training wheels. Not everyone has that luxury, nor should we in the nix community expect people to pay down enormous initial learning debt.

I began freelance consulting at the start of the year, and interesingly, more than one of my engagements turned into "let's solve this with nix", which is both surprising and encouraging. If the nix community can trim down the harsh thorns that encircle the language and tools (and provide rock solid learning materials), there's an actual chance that nix could really bloom in the near future.

Page 1 of 2Next →