Healthchecks.io now uses self-hosted object storage | Hacker News Reader

Healthchecks.io now uses self-hosted object storage

blog.healthchecks.io

195 pointszdw79 comments

> In March 2026, I migrated to self-hosted object storage powered by Versity S3 Gateway.

Thanks for sharing this, I wasn't even aware of Versity S3 from my searches and discussions here. I recently migrated my projects from MinIO to Garage, but this seems like another viable option to consider.

First time hearing about Versity for me too. I thought "S3 Gateways" were an Amazon-only service rather than something mere mortals could set up.

I've been trying to give some containers (LXC/D and OCI) unprivileged access to a network-accessible ZFS filesystem and this might be what I need. Managing UID/GID through bind-mounts from the host to the container (ie NFS on host) has been trickier than I was expecting.

It sounds like you're not using Incus, but for those who are, it looks like there is a S3 compatible object storage API built in. I haven't used it for anything yet but it seems relevant:

https://linuxcontainers.org/incus/docs/main/howto/storage_bu...

Not really s3, and I haven't touched LXC in a long but this may help on the OCI side. I apologize if this is redundant to you.

Remember that UID mapping on namespaces is just a facade with an offset and a range typically based on subuid[0] and subgid[1] today.

In the container `cat /proc/self/uid_map` or by looking at the pid from the host `cat /proc/$PID/uid_map` you can tell what those offsets are.

     $ cat /proc/self/uid_map
         0       1000          1
         1     100000      65536

Here you know that PID =0 in the container maps to PID 1000 in the host, with a length of 1

The Container PID offset of 1 maps to the host offset of 100000 for a length of 65536

With subuid/subgid you can assign ranges to the user that is instantiating the container, in the flowing I have two users that launch containers.

     $ cat /etc/subuid
     debian:100000:65536
     runner:165536:65536

     $ cat /etc/subgid
     debian:100000:65536
     runner:165536:65536

Assuming you pass in the host UID/GID that is how you can configure a compatible user at the entrypoint.

But note that only highly trusted containers should ever really use host bind mounts, it is often much safer to use mount NFS internally.

Host bind mounts of network filesystems, if that is what you are doing, is also fragile as far as dataloss goes. I am an object store fan, but just wanted to give you the above info as it seems hard for people to find.

I would highly encourage you to look into the history of security problems with host bind mounts to see the wack-a-mole that is required with them to see if it fits in with your risk appitite. But if you choose to use them, setting up dedicated uid/gid mappings and setting the external host to the expected effective ID of the container users is a better way than using Idmapped mounts etc...

[0] https://man7.org/linux/man-pages/man5/subuid.5.html [1] https://man7.org/linux/man-pages/man5/subgid.5.html

I appreciate the detailed explanation of ID/GID mapping.

> it is often much safer to use mount NFS internally

This is the config I'm trying to move away from! I don't see how an unprivileged LXC with a bind mount is worse than a privileged container with NFS, FUSE, and nesting enabled (I need all of that if I can't aggregate on the host).

NFS and CIFS within the container requires kernel-level access and therefore the LXC must be privileged. I'd rather have a single defined path.

I tried to get around this using FUSE but it creates its own issues with snapshots/backups (fsfreeze).

If my solutiom works for a regular LXC it will probably work for Podman.

I can't comment directly on LXC but LXC is very different from runc/crun/your-CRI here, not better or worse, just different.

With podman, unfortunately we don't k8s Container Storage Interface (CSI), so you have to work with what you have.

When I said:

> it is often much safer to use mount NFS internally

What is more correct, is having the container runtime or container manager mount them, not the user inside the container.

But as you are trying to run unprivileged or at least with minimal privileges, which is all we can do with namespaces, you are cutting across the grain.

I do use podman pods and containers, mostly for the ease of development, but on more traditional long lived hosts.

I have a very real need to separate UIDs between co-hosted products, but don't need to actually run a VM for these specific use cases.

So I have particular rootful tasks that have to be done as the user root root in ansible:

1) Install OS packages 2) Create service admin and daemon user 3) Assign subuid/subgids ranges to those user security domains as needed 4) For specific services add NFS data directories to /etc/fstab with the 'user' and 'noauto' flags

In Podman I would then create

     podman volume create --driver local --opt type=nfs --opt device=192.168.1.84:/path/to/share --opt o=addr=192.168.1.84,....

     podman run -d  --name nfs_test -v nfs-shared:/opt docker.io/library/debian:latest

Which if you don't have the fstab entry will give you:

     Error: mounting volume nfs-shared for container ...: mount.nfs: Operation not permitted for 192.168.1.84:/path/to/share on /home/user/.local/share/containers/storage/volumes/nfs-shared/_data

That `_data` is one of the hints of the risk of host bind mounts, the risk is either having an inode that the host cares about or issues across containers etc...

While imperfect, this is following the named volume pattern, which really just uses tells about it being in a container and doesn't expose the mount inode to the container.

What does happen inside the container entry point is validating that the expected UID is reachable, adding a user with the right UID offset and switching to that user.

A misconfigured host bind mount or leaking because you can't view who has access are the most common problems, and as containers run with elevated privileges until you drop them they can get around those protections, even if they aren't elevating to root in a rootless situation, they can still access the data of any running container with just a few trivial mistakes or new discovered vulnerabilities.

    $ capsh --decode=00000000800405fb 0x00000000800405fb=cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap

While NFS is absolutely a whole new ball of wax with other issues, one nice thing is that (at least the servers I know of) don't even support the concept of user namespaces and UID mapping, which makes it fragile and dangerous if you start mapping uid/gid's in, but can be an advantage if you can simply isolate uid/gid ranges.

IMHO it will be horses for courses and depend on your risk appetite as all options are least worst and there simply will be no best option, especially with OCI.

Wow, I really appreciate you coming back for the follow-up! It's too late for me to read through it in detail at this moment but:

In the end I discovered that I can combine a "mapall/squash" on the NFS server, a regular NFS mount on Proxmox, and then an `lxc.mount.entry` for the LXC config and the combined effect is an unprivileged container with read-write permissions for the UID/GID specified on the NFS server. If I need more UID/GID combinations I can just create bind mounts and then export those with the appropriate mapall/squash settings.

Thanks again :)

As a user for over a decade, just here to submit my appreciation. healthchecks.io is fantastic.

Self Hosted object storage looks neat!

For this project, where you have 120GB of customer data, and thirty requests a second for ~8k objects (0.25MB/s object reads), you’d seem to be able to 100x the throughput vertically scaling on one machine with a file system and an SSD and never thinking about object storage. Would love to see why the complexity

(Author here) that's more or less what I have right now – one machine with a file system and an SSD. S3 API on top is there to give multiple web servers shared access to the same storage. I could have used something else instead of S3 – say, NFS – but there was a feature request for S3 [1] and S3 has a big ecosystem around it already.

[1] https://github.com/healthchecks/healthchecks/issues/609

The complexity for that is almost always for redundancy and for ease of deploys.

For S3 self-hosters check out Garage, who are developing with NLnet funding: https://garagehq.deuxfleurs.fr

Covered by the article:

>I ran local experiments with Minio, SeaweedFS, and Garage. My primary objection to all of them was the operational complexity

Out of the listed options, Garage is actually really simple for something that runs on multiple servers. It was also the only easy option I found that can replicate over WAN and isn't super latency sensitive.

I'm sure it's a lot better now but everytime I see btrfs I get PTSD.

Same here. Had a production node running btrfs under heavy write load (lots of small files, frequent creates) and spent two days debugging what turned out to be filesystem-level corruption. Switched to ext4 and never looked back. The article doesn't mention what filesystem sits under Versitygw here, which seems like a pretty relevant omission for anyone thinking of replicating the setup.

I hit a panic in btrfs using an ubuntu 24 LTS kernel. The trauma is still well and alive.

I'm a little surprised it's not ZFS. Too difficult to add to their Linux environment? That's still a problem here in 2026.

Same, and reasoning around inodes feels easy fixed by just upping inode per KB from 16k to 4k which is likely block size anyways.

I'd worry about file create, write, then fsync performance with btrfs, but not about reliability or data-loss.

But a quick grep across versitygw tells me they don't use Sync()/fsync, so not a problem... Any data loss occurring from that is obviously not btrfs fault.

Care to elaborate? I've heard good things about it, but am personally a ZFS user.

Years of serious corruption bugs.

Gluster was that for me

Yup, still get nightmares about glusterfs.... still have one customer running on it.

I heard it got better, but we ran into the BOTF (billions of tiny files) issue around 2016. (For a genealogy startup this was a serious issue)

Ah, another one! Yep, also same, before ceph days at least (although I've had my own, albeit self-inflicted, nightmare there too).

I don't get it, if it's running on the same (mentioning "local") machine, why does it even need the S3 API? Could just be plain IO on the local drive(s)

The app was already built against the S3 API when it used cloud storage. Keeping that interface means the code doesn't change - you just point it at a local S3-compatible gateway instead of AWS/DO. Makes it trivial to switch back or move providers if needed.

(Author here) There are multiple web servers for redundancy (3 currently), and each needs access to all objects.

with average object size of 8.5kB I'd honestly consider storing it as blobs in cloud DB, with maybe some small per-server cache in front

If the app was written using the S3 API, it would be much faster/cheaper to migrate to a local system the provides the same API. Switching to local IO would mean (probably) rewriting a lot of code.

Surely "read object" and "write object" are not hard to migrate to local file system. You can also use Apache OpenDAL which provide the same interface to both.

Yeah, unless you have the raw S3 API throughout your codebase you should be able to write a couple dozen lines of code (maximum) to introduce a shim that's trivial to replace with local file access. In fact, I've done this in most projects that work with S3 or similar APIs so I can test them locally without needing real S3!

Compatibility with existing code, middleware and so on.

seperate machine I think given the quoted point at the end:

> The costs have increased: renting an additional dedicated server costs more than storing ~100GB at a managed object storage service. But the improved performance and reliability are worth it.

The S3 API doesn't work like normal filesystem APIs.

Part of it is that it follows the object storage model, and part of it is just to lock people into AWS once they start working with it.

What kind of vendor lock-in do you even talk about. Their API is public knowledge, AWS publishes the spec, there are multiple open source reference client implementations available on GitHub, there are multiple alternatives supporting the protocol, you can find writings from AWS people as high in hierarchy as Werner Vogels about internals. Maybe you could say that some s3 features with no alternative implementation in alternative products are a lock-in. I would consider it a „competitive advantage”. YMMV.

Apart from all these other products that implement s3? MinIO, Ceph (RGW), Garage, SeaweedFS, Zenko CloudServer, OpenIO, LakeFS, Versity, Storj, Riak CS, JuiceFS, Rustfs, s3proxy.

Add Tigris to the list as well please.

We maintain a page that shows our compatibility with S3 API. It's at https://www.tigrisdata.com/docs/api/s3/. The test runner is open source at https://github.com/tigrisdata-community/s3-api-compat-tests

Riak CS been dead for over a decade which makes me question the rest. Some of these also do not have the same behaviors when it comes to paths (MinIO is one of those IIRC).

Also, none of them implement full S3 API and features.

There's a difference between S3 API spec and what Amazon does with S3 - for isntance, the new CAS capabilities with Amazon are not part of the spec.

Ceph certainly implements the full API spec, though it may lag behind some changes. It's mostly a question of engineering time available to the projects to keep up with changes.

> There's a difference between S3 API spec and what Amazon does with S3 - for isntance, the new CAS capabilities with Amazon are not part of the spec.

Sure, but those are S3 APIs and features that provided by AWS. We not talking about S3 spec, we're talking about s3 product.

You’re wrong. “Implementing the S3 API” means the spec, not an Amazon product.

We're talking about using AWS S3 and using something that implements S3 spec. Really not that hard to understand.

When customer wants to switch away from AWS S3, customer cares about AWS S3 feature coverage and not the spec.

Spec and features are intertwined. Customers who switch away from AWS S3 still want to use the same SDKs, libraries, etc that support S3 API. They don't want to rewrite their applications to use a new API. So then is it feature coverage or spec coverage?

It's both? Customer doesn't care if spec is 100% covered if feature that they are using in AWS S3 isn't supported.

Also, who is rewriting their application to change interaction with an object storage? People that directly use some S3 sdk all around the app should read a book on software engineering or at least a blog post.

Wrong again. Not only have I worked on DigitalOcean’s S3 implementation but I currently work on an open source product that targets S3 spec and can be used with any cloud provider and any other spec-compliant drop-in, like Garage.

All that means: you only used AWS S3 features that are in the S3 spec.

What does RadosGW miss?

I'm 100% aware of how S3 works. I was questioning why the S3 API is needed when the service is using local storage.

Sometimes API compatibility is an important detail.

I've worked at a few places where single-node K8s "clusters" were frequently used just because they wanted the same API everywhere.

The API has sort of become a standard. There are many providers providing S3 API-compatible storage.

> part of it is just to lock people into AWS once they start working with it.

This is some next-level conspiracy theory stuff. What exactly would the alternative have been in 2006? S3 is one of the most commonly implemented object storage APIs around, so if the goal is lock-in, they're really bad at it.

> What exactly would the alternative have been in 2006?

Well, WebDAV (Document Authoring and Versioning) had been around for 8 years when AWS decided they needed a custom API. And what service provider wasn't trying to lock you into a service by providing a custom API (especially pre-GPT) when one existed already? Assuming they made the choice for a business benefit doesn't require anything close to a conspiracy theory.

And it worked as a moat until other companies and open source projects started cloning the API. See also: Microsoft.

WebDAV is ass tho. I don't remember a single positive experience with anything using it.

And still need redundant backend giving it as API

When I was in school, we had a SkunkDAV setup that department secretaries were supposed to use to update websites... supporting that was no fun at all. I'm not sure why it was so painful (was 25 years ago) but it left a bad taste in my mouth.

WebDAV is kinda bad, and back then it was a big deal that corporate proxies wouldn't forward custom HTTP methods. You could barely trust PUT to work, let alone PROPFIND.

When S3 launched the core API could be described with 4 requests. It was (and still mostly is) super simple.

Saying they should have used WebDAV instead shows a lack of knowledge on your end rather than theirs.

So you don't need to refactor your code?

And when/if you decide to head back to a 3rd party it requires no refactoring again.

yeah, sure, those 5-10 different API calls would surely be a huge toll to refactor... I'd rather run an additional service to reimplement the S3 API mapping to my local drive /s

Or a simple SAN

love this. one person, 119GB, two drives, rsync. no kubernetes, no distributed cluster, no nonsense. just works.

this is the kind of setup that lets you actually go to bed without checking your phone every 20 minutes.

As someone who has dealt with wacky storage issues/designs, a lot of this "felt" strange to me. Btrfs? Rsync? Then I got to the bottom and saw that they were only handling about 100 GB of data! At that scale, nearly anything will work great and TFA was right to just pick the thing with the fewest knobs.

At a previous job years ago, we had a service that was essentially a file server for something like 50TB of tiny files. We only backed it up once a week because just _walking_ the whole filesystem with something like `du` took more than a day. Yes, we should have simply thrown money at the problem and just bought the right solution from an enterprise storage vendor or dumped them all into S3. Unfortunately, these were not options. Blame management.

A close second would have been to rearchitect dependent services to speak S3 instead of a bespoke REST-ish API, deploy something like SeaweedFS, and call it a day. SeaweedFS handles lots of small files gracefully because it doesn't just naively store one object per file on the filesystem like most locally-hosted S3 solutions (including Versity) do. And we'd get replication/redundancy on top of it. Unfortunately, I didn't get buy-in from the other teams maintaining the dependent services ("sorry, we don't have time to refactor our code, guess that makes it a 'you' problem").

What I did instead was descend into madness. Instead of writing each file to disk, all new files were written to a "cache" directory which matched the original filesystem layout of the server. And then every hour, that directory was tarred up and archived. When a read was required, the code would check the cache first. If the file wasn't there, it would figure out which tarball was needed and extract the file from there instead. This only worked because all files had a timestamp embedded in the path. Read performance sucked, but that didn't matter because reads were very rare. But the data absolutely had to be there when needed.

Most importantly, backups took less than an hour for the first time in years.

what happens if you suffer catastrophic failure, you'd lose up to one hour of data?

The system had RAID, so disks could be replaced without taking anything down.

We never had a catastrophic failure, but yeah, and hour of lost data wouldn't be the end of the world. For regular maintenance we would just coordinate with the QA team to pause testing temporarily. (Testing was only around-the-clock near a product release.)

> Our S3 API is now served by Versity S3 Gateway and backed by a plain simple Btrfs filesystem.

With apologies to the SRE Book ("hope is not a strategy")... Btrfs is not a strategy.

> The costs have increased: renting an additional dedicated server costs more than storing ~100GB at a managed object storage service. But the improved performance and reliability are worth it.

Were your users complaining about reliability and performance? If it cost more, adds more work (backup/restore management), and the users aren't happier then why make the change in the first place?

Not the OP but I have some… similar experience. When you run a high availability service without a full ops team, reliable infrastructure is non-negotiable. Burn out has to be managed.

It's interesting that cloud providers are unable to provide stable S3 as a service. Hetzner is unable to deliver stable object storage, but given the article neither are OVHCloud and UpCloud.

Given the individual file size and total volume, I'd argue it make sense to use move to local only storage.

On a separate note, what tool is the final benchmark screenshot form?

The graphs are from Netdata. I'm using it to monitor the servers, and also feeding it some application metrics via its statsd interface.

Do you like Netdata? I'm looking into it. I'm curious if you use all the features or just a few.

Moved object storage from AWS to CloudFlare and have been pretty happy. No problems with performance so far. Bills were 90% cheaper too (free bandwidth)

You can wire lsyncd for near real-time syncs and btrfs snapshots for offline storage

great writeup. Is s3 a customer or internal requirement? Why not write to your disks? Easily 1/3 - 1/5 the price and better performance.

how do you provide highly available fault tolerant storage?

versity does not include any erasure coding or replication...

"Our current (April 2026) object usage is: 14 million objects, 119GB"

I mean, I appreciate the openness about the scale, but for context, my home's personal backup managed via restic to S3 is 370GB. Fewer objects, but still, we're not talking a big install here.

This is pretty much like that story of, if it fits on your laptop, it's not big data.

My phone has 327.1G of storage in use right now :-) (years ago a friend of mine who was also old enough to complain about "640k" suggested that we should just denominate storage in dollars, so "reasonable" amounts scaled with Moore's law...)

Reply on news.ycombinator.com