adamcharnock

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between:

1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based).

2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50% cheaper than AWS et al). We rent or colocate bare metal, manage it for you, handle software deployments, deploy only open-source, etc. Only really makes sense above €$5k/month spend.

3 - Rented Bare Metal – Let someone else handle the hardware financing for you. Still minimal cap-ex, but with greater hiring/skilling and risk. Around 90% cheaper than AWS et al (plus time).

4 - Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills, scale, cap-ex, and if you plan to run the servers for at least 3-5 years.

A good provider for option 3 is someone like Hetzner. Their internal ROI on server hardware seems to be around the 3 year mark. After which I assume it is either still running with a client, or goes into their server auction system.

Options 3 & 4 generally become more appealing either at scale, or when infrastructure is part of the core business. Option 1 is great for startups who want to spend very little initially, but then grow very quickly. Option 2 is pretty good for SMEs with baseline load, regular-sized business growth, and maybe an overworked DevOps team!

[0] https://lithus.eu, adam@

Agreed. The one bastion of sanity in all this is (/was) the UK. I formed my first company there, 18 years ago, online, in 30 minutes, for around £20.

I then moved to Portugal and started not one but two companies there. The whole process is so clearly setup to discourage people from actually forming companies. Everything from the attitude of all involved ("are you sure?!"), to the practical bureaucracy and costs involved.

I thought perhaps Portugal was just an outlier. But then I moved to Germany. And just wow. Definitely worse. Rounds of paperwork, notary offices, and fees. A process taking weeks. And for a GmbH a minimum investment of €25k. Sure you can form a UG. company for €1, but that effectively just announces "don't trust us, we're tiny" (IMHO).

It is something that really saddens and frustrates me with the EU.

Edit: And sure, you can form an Estonian company. But then you have to try and fly under the radar with regards to the 'permanent establishment' rules.

Well yes and no.

Yes you can form a company in Ireland while living in France. But you cannot get an Irish VAT number without a physical presence in Ireland.

And if – for example – France learns that you are running an Irish company from France (i.e. you have a 'permanent establishment' in France), they'll want you to file and pay French corporation tax. Which is likely sufficiently annoying that you may as well have formed a French company in the first place.

In Portugal (for example) is is mandatory for an Lda (limited company) to have an employee, and they must be paid minimum monthly amount. In practice this means a few hundred euros a month go to the government in taxes. Then on top of that is another hundred or so in accountancy fees.

We're looking at alternatives, I've made some previous comments on that front. Sadly MinIO was the only option with sufficient performance for this particular situation. Thankfully we're not using any MinIO-specific features, so at least the migration path away is clear.

It was definitely MinIO related, I probably should have made that clearer. We noticed that with zero fault tolerance, MinIO objects would randomly become corrupted, which MinIO would present as "you're making too many requests, please slow down". We were certainly not making too many requests.

We've been using JuiceFS in production for a few months now and I'm a big fan. I've felt for a while that block-level filesystems do not adapt at all well to being implemented across a network (with my personal experience being of AWS EBS and OpenEBS Mayastor). So the fact that JuiceFS is interfaces at the POSIX layer felt intuitively better to me.

I also like that it can keep a local read-cache, rather than having to hit object storage for every read. This is because it can perform a freshness check with with the (relatively fast) metadata store to determine if its cached data is valid, prior to serving the request from cache.

We back it with a 3-node (Redis-compatible) HA Valkey cluster, and in-cluster MinIO object storage, all in bare-metal Kubernetes. We can saturate a 25g NIC with (IIRC) 16+ concurrent users.

It is also one of the few Kubernetes storage providers that provides read-write-many (RWX) access. Which can also be rather helpful in some situations.

In an early test we were running it against MinIO with zero redundancy. Which is not recommended in any case. There we did see some file corruption creep in. In which case some files in JuiceFS become unreadable, but the system as a whole kept working.

Another reason I think JuiceFS works well, is indeed because of its custom block-based storage format. It is disconcerting because you cannot see your files in object storage, but instead just a lot of chunks. But this does buy some real performance benefits, especially when doing partial file reads or updates.

Another test we're doing is running a small-to-medium sized Prometheus persisted to JuiceFS. It hasn't shown any issues so far.

And, if you've made it this, far: check us out if you want a hand installing and operating this kind of infra: https://lithus.eu . We deploy to bare-metal Hetzner.

Well, they didn’t like it! :) I considered just providing views onto the tables that would present the data in the way that made the accountants happy. But in the end we just ended up changing to using separate debit/credit columns. They managed to talk me around somehow, but I don’t remember the details any more. I think this was the relevant issue:

https://github.com/adamcharnock/django-hordak/issues/59

I wrote this a long time ago. It does tend to upset some people, but it did work as an accurate and testable mental model:

https://django-hordak.readthedocs.io/en/latest/accounting-fo...

In case any SME-sized companies here are wanting to do something similar but are looking askance at the risk/investment/hiring required, then we'd [0] love to talk to you.

We specialise in doing this but on a smaller scale. Eg. 10-100 person companies that have 0-to-a-few DevOps engineers. Included is DevOps time each month to use as you wish, we're on call for SLAs, around 50% reduced cost vs AWS/Google/Azure, etc.

Somewhat differently to most, we deploy onto bare metal. In addition to dropping costs we typically see at least a 2x speed-up overall. Once client just reported a 80% reduction in processing time.

CTOs like us because we're always on-hand via Slack (plus we're the ones getting woken up in the night), and CFOs like us because billing becomes consistent.

Anyway, blatant pitch complete.

[0]: https://lithus.eu/

adam@ above domain

Not used it yet, but RustFS sounds like it has self healing

https://docs.rustfs.com/troubleshooting/healing.html

Agreed. Also Lefthook: https://github.com/evilmartians/lefthook

Copy/paste from a previous thread [0]:

We’ve done some fairly extensive testing internally recently and found that Garage is somewhat easier to deploy in comparison to our existing use of MinIO, but is not as performant at high speeds. IIRC we could push about 5 gigabits of (not small) GET requests out of it, but something blocked it from reaching the 20-25 gigabits (on a 25g NIC) that MinIO could reach (also 50k STAT requests/s, over 10 nodes)

I don’t begrudge it that. I get the impression that Garage isn’t necessarily focussed on this kind of use case.

---

In addition:

Next time we come to this we are going to look at RustFS [1], as well as Ceph/Rook [2].

We can see we're going to have to move away from MinIO in the foreseeable future. My hope is that the alternatives get a boost of interest given the direction MinIO is now taking.

[0]: https://news.ycombinator.com/item?id=46140342

[1]: https://rustfs.com/

[2]: https://rook.io/

The min bend radius isn’t that large in my experience. On the order of 10cm IIRC, possibly even less.

We’ve done some fairly extensive testing internally recently and found that Garage is somewhat easier to deploy, but is not as performant at high speeds. IIRC we could push about 5 gigabits of (not small) GET requests out of it, but something blocked it from reaching the 20-25 gigabits (on a 25g NIC) that MinIO could reach (also 50k STAT requests/s)

I don’t begrudge it that. I get the impression that Garage isn’t necessarily focussed on this kind of use case.

We've been looking at Forgejo too. Do you have any experience with Forgejo Actions you can share? That is one thing we are looking at with a little trepidation.

Something about queuing systems that often gets me is that they can start to seem like the wrong abstraction as soon as one has tasks that enqueue additional tasks. Particularly when features start growing, and double particularly when modelling business processes.

This is because the code enqueuing the task needs to be aware of what happens next, which breaks separation of concerns. Why should the user sign-up code have to know that a report generation job now needs queuing?

Really what starts to make more sense to me is to fire off events. Code can say, "this thing just happened", and let other code decide if it wants to listen. When then makes it an event stream rather than a queue, with consumer groups at al.

I made the (now unmaintained) project https://lightbus.org around this, and it did work really well for our use case. Hopefully someone has now created something better.

So I'd say this: before grabbing for a task queue, take a moment to think about what you're actually modelling. But be careful of the event streaming rabbit-hole!

In this particular case the only need was for Postgres migration, no other services needed.

The problems we've seen or heard about with Supabase are:

* Cost (in either magnitude or variability). Either from usage, or having to go onto their Enterprise-tier pricing for one reason or another * The usual intractable cloud-oddities – dropped connections, performance speed-bumps * Increased network latency (just the way it goes when data has to cross a network fabric. Its fast, but not as fast as your own private network) * Scaling events tend not to be as smooth as one would hope

None of these are unique to Supabase though, they can simply all arise naturally from building infrastructure on a cloud platform.

Regarding self-hosted Supabase - we're certainly open to deploying this for our clients, we've been experimenting with it internally. Happy to chat with you or anyone who's interested. Email is adam@ company domain.

This is exactly true, and is something we have built our business around. In fact, I just kicked-off a multi-TiB Postgres migration for one of our clients this morning. We're moving them out of Supabase and onto a bare-metal multi-AZ Postgresql cluster in Hetzner.

I'm going to say what I always say here - for so many SME's the hyperscaler cloud provider has been the safe default choice. But as time goes on a few things can begin to happen. Firstly, the bills grow in both size and variability, so CFOs start to look increasingly askance at the situation. Secondly, so many technical issues start to arise that would simply vanish on fixed-size bare-metal (and the new issues that arise are well addressed by existing tooling). So the DevOps team can find themselves firefighting while the backlog keeps growing.

The problem really is one of skills and staffing. The people who have both the skills and desire actually implement and maintain the above tend to be the greying-beards who were installing RedHat 6 in their bedrooms as teenagers (myself included). And there are increasingly few of us who are not either in management and/or employed by the cloud providers.

So if companies can find the staff and the risk appetite, they can go right ahead and realise something like a 90% saving on their current spend. But that is unusual for an SME.

So we started Lithus[0] to do this for SMEs. We _only_ offer a 50% saving, not 90%. But take on all the risk and staffing issues. We don't charge for the migration, and the billing cycle only starts once migration is complete. And we provide a fixed number of engineering days per month included. So you get a complete Kubernetes cluster with open source tooling, and a bunch of RedHat-6-installing greying-beards to use however you need. /pitch

[0] https://lithus.eu

I don't quite go this far, but I do use Claude/Codex to write Ansible playbooks/roles/collections that then do this kind of thing.

It is very easy to see what actions are being taken from the code produced, and then one gets a tool that can be used over and over again.

You can then also put these into mise tasks, because mise is great too.

No problem!

And it is very true. Although the binary does also need building, which is also handled in the above actions workflow.

Done: https://github.com/golithus/minio-builds

We [0] use MinIO with for our clients so we've just thrown together a nightly build process. Use/fork as you wish:

https://github.com/golithus/minio-builds

Example use:

    docker run -p 9000:9000 -p 9001:9001 ghcr.io/golithus/minio:latest

[0]: https://lithus.eu

We're working on a binary build process now. We hope to have something up at https://github.com/golithus soon.

We use MinIO (community edition) a fair amount. And while we like it, it is also becoming increasingly clear that our days of deploying are numbered.

We want to start experimenting with Garage for smaller deployments, and would be interesting to hear of any production experiences there. (Anyone done multi-PiB deployments?)

Other than that we're going to start looking at Ceph/Rook for larger deployments.

I very much understand this, and that is why we do what we do. Lots of companies feel exactly as you say. I.e. Sure it is cheaper and 'better', but we'll pay for it in salaries and additional incurred risk (what happens if we invest all this time and fail to successfully migrate?)

This is why we decided to bundle engineering time with the infrastructure. We'll maintain the cluster as you say, and with the time left over (the majority) we'll help you with all your other DevOps needs too (CI/CD pipelines, containerising software, deploying HA Valkey, etc). And even after all that, it still costs less than AWS.

Edit: We also take on risk with the migration – our billing cycle doesn't start until we complete the migration. This keeps our incentives aligned.

We're hoping to write a case study down the road that will give more detail. But the short version is that not all parts of the client's organisation have aligned skills/incentives. So sometimes code is deployed that makes, shall we say, 'atypical use' of the resources available.

In those cases, it is great to a) not get a shocking bill, and b) be able to somewhat support this atypical use until it can be remedied.

I understand the concern for sure. But we don't bill hourly in that way, as one thing our clients really appreciate is predictable costs. The fixed monthly price already includes engineering time to support your team.

I cannot overstate the performance improvement of deploying onto bare metal. We typically see a doubling of performance, as well as extremely predictable baseline performance.

This is down to several things:

- Latency - having your own local network, rather than sharing some larger datacenter network fabric, gives around of order of magnitude reduced latency

- Caches – right-sizing a deployment for the underlying hardware, and so actually allowing a modern CPU to do its job, makes a huge difference

- Disk IO – Dedicated NVMe access is _fast_.

And with it comes a whole bunch of other benefits:

- Auto-scalers becomes less important, partly because you have 10x the hardware for the same price, partly because everything runs 2x the speed anyway, and partly because you have a fixed pool of hardware. This makes the whole system more stable and easier to reason about.

- No more sweating the S3 costs. Put a 15TB NVMe drive in each server and run your own MinIO/Garage cluster (alongside your other workloads). We're doing about 20GiB/s sustained on a 10 node cluster, 50k API calls per second (on S3 that is $20-$250 _per second_ on API calls!).

- You get the same bill every month.

- UPDATE: more benefits - cheap fast storage, run huge Postgresql instances at minimal cost, less engineering time spend working around hardware limitations and cloud vagaries.

And, if chose to invest in the above, it all costs 10x less than AWS.

Pitch: If you don't want to do this yourself, then we'll do it for you for half the price of AWS (and we'll be your DevOps team too):

https://lithus.eu

Email: adam@ above domain

I totally agree. So much complexity for generally no good reason [0]. I saw so much of this that I ended up starting a company doing the exact opposite. I figured I could do it better and cheaper, so that that's now what we do!

If anyone wants to bail out of AWS et al and onto a few beefy servers, save some money, and gain a DevOps team in the process, then drop us an email (adam at domain in bio).

[0] My pet theory about the real reason: the hyper-scalers hire all the engineers who have the skills to deploy-to-a-few-beefy-servers, and then charge a 10x multiplier for compute. Companies can then choose between impossible hiring, or paying more. Paying more is easier to stomach, and plenty of rationalisations are available.

An example from one of our monorepo's ansible directories:

    [hooks]
    postinstall = [
        "uv sync",
        "ansible-galaxy role install -r ansible/requirements.yml",
        "ansible-galaxy collection install -r ansible/requirements.yml",
    ]

So following a `mise install`, the user also gets all the needed python packages installed via uv, and also all the galaxy roles/collections installed