3,683 karma · joined May 3, 2010
https://lithus.eu - We run your infrastructure so you don't have to. Managed Kubernetes on bare metal in Germany. ~50% cheaper than AWS.
They have a special place in my heart. Vinge didn't write much, but what he did write I think is very special indeed.
I've also taken to buying hardbacks of my favourite books from ebay.
https://www.goodreads.com/en/book/show/77711.A_Fire_Upon_the...
https://www.goodreads.com/book/show/226004.A_Deepness_in_the...
> [This standard is] for anybody who creates or helps create documents. The widest use of plain language is for documents that are intended for the general public. However, it is also applicable, for example, to technical writing, legislative drafting or using controlled languages.
You don't actually have the buy the standard, but this is it: https://www.iso.org/standard/78907.html
And you can read it for free here: https://www.iso.org/obp/ui#iso:std:iso:24495:-1:ed-1:v1:en
https://github.com/adamcharnock/python-srtm
I originally used it for doing line-of-sight calculations for a wireless ISP I started.
- Router OS Diff - Can diff two configs and give you the commands needed to bring the existing config up to date with the desired config. It's certainly not perfect, but a starting point of anyone needs something like this. [1]
- Netbox Routeros – A netbox plugin for updating the config of RouterOS devices directly from the Netbox interface. [2]
It has been many years since I touched these, but perhaps they will be of interest to someone.
Aside from that, I have had excellent experience with Mikroik. Everywhere from in-datacenter to it running in an off-grid hut on a mountainside. I've even heard reports of people finding rain streaming through a CRS and it just happily ticking along.
And yes, adding a Redis cluster is fine, it is just another moving part to manage. But given that the alternative is made out of unobtainium, I guess that is just the way of it :-)
I've found myself wanting this on several occasions too. I.e. wanting all my rust backend processes (k8s pods) to have some minimal shared state, without having to spin up a Redis cluster. I've talked to Claude about it a couple of times, and it descends into something like, "you gotta use Raft or CRDTs, and pick 2 out of 3 from CAP". Which honestly seems pretty fair, and indicates to me that I'm dreaming for something magical.
Nonetheless, it is nice to hear someone else asking for this. If this is indeed feasible (even if simple/limited), then I'd be interested to try it.
I hope this isn’t the case. It is the route I took, but it also doesn’t seem to be a likely route going forward. Strong CS grounding is feasible for sure, but I have a hard time believing that a meaningful number of people will be spending the requisite years coding manually.
But this doesn't seem to make sense when someone comes to a topic with an LLM in-hand. They need to know high-level techniques, architecture, best practice, etc. As they pursue the topic they start to get down into the details, although probably never learn to do it fully independently.
I quite like this view because it paints a somewhat optimistic way forward from where we are now.
Shameless plug: We started our company[0] on this basis, i.e. managed Kubernetes on bare metal in EU DCs. We run everything on open source tooling, provide DevOps engineering time to our customers' engineering teams, take on the migration risk ourselves, and offer response-time SLAs. So yes I'm biased, but I did this because I do actually really believe in this approach.
So I'd flip it around. Perhaps building a sovereign hyperscaler is the wrong approach. I'd say we need workloads that aren't welded to any one provider in the first place. And open source tools have come a very long way since EC2 first came online.
[0] - https://lithus.eu/
I'd teach someone to drive it and say, "now push down on the clutch". They they would heave and struggle, then eventually succeed and look victorious. I'd say, "well done, it is now half way down! But that's all you need for now!"
EDIT: To fully explain: It has a two-stage clutch. You half-press it and it disconnects the wheels from the engine. If you fully depress it all the way to the floor, it additionally disconnects the power-take-off shaft (PTO) from the engine. The PTO shaft is a spindle on the back of the tractor which drives things like your flail mower, wood chipper, etc.
EDIT 2: Edit 1 was for the general audience, not the parent commenter ;-)
And there was no fancy technology in it at all. If I was in the forest and had forgotten the key, I'd just reach behind the dashboard and hot-wire it. The air filter was basically a shisha-pipe that bubbled the incoming air through wire wool and engine oil.
Its fuel gauge didn't work either. You just had to take a look in the tank, or quickly react as soon as the revs started dropping. I ran it dry a few times and had to sit there with a spanner in one hand and YouTube into the other, while trying to bleed all the fuel lines. But they were all on the outside of the vehicle, which made it comparatively easy I imagine.
I've never actually driven a modern tractor, so don't know how it compares. I imagine the clutch is easier on the knees these days!
Anyway, this just felt like the place to share this.
For backups we use both Velero and application-level backup for critical workloads (i.e. Postgres WAL backups for PITR). We also ensure all state is on at least two nodes for HA.
We also find bare metal to be a lot more performant in general. Compared to AWS we typically see service response times halve. It is not that virtualisation inherently has that much overhead, rather it is everything else. Eg, bare metal offers:
- Reduced disk latency (NVMe vs network block storage)
- Reduced network latency (we run dedicated fibre, so inter-az is about 1/10th the latency)
- Less cache contention, etc [1]
Anyway, if you want to chat about this sometime just ping me an email: adam@ company domain.
[1] I wrote more on this 6 months ago: https://news.ycombinator.com/item?id=45615867
If you're interested I'll see about getting the PR created sooner rather than later.
The down side is that the starting price is kinda high, so the math probably only works out if you also have a number of other workloads to run on the same cluster. Or if you need to run a really huge Forgejo server!
I suspect my comment history will provide the best details and overview of what we do. We'll be offering the Firecracker runner back to the Forgejo community very soon in any case.
It has been a pretty smooth process. Although we have done a couple of pieces of custom development:
1) We've created a Firecracker-based runner, which will run CI jobs in Firecracker VMs. This brings the Foregjo Actions running experience much more closely into line with GitHub's environment (VM, rather than container). We hope to contribute this back shortly, but also drop me a message if this is of interest.
2) We're working up a proposal[1] to add environments and variable groups to Forgejo Actions. This is something we expect to need for some upcoming compliance requirements.
I really like Forgejo as a project, and I've found the community to be very welcoming. I'm really hoping to see it grow and flourish :D
[0]: https://lithus.eu, adam@
[1]: https://codeberg.org/forgejo/discussions/issues/440
PS. We are also looking at offering this as a managed service to our clients.
We're not really that kind of product company; we're more of a services company. What we do is deploy Kubernetes clusters onto bare metal servers. That's the core technical offering. However, everything beyond that is somewhat per-client. Some clients need a lot of compute. Some clients need a custom object storage cluster. Some clients need a lot of high-speed internal networking. Which is why we prefer to have a call to figure out specifically what your needs are. But I can also see how this isn't necessarily satisfying if you're used to just grabbing the API docs and having a look around.
What we will do is take your company's software stack and migrate it off AWS/Azure/Google and deploy it onto our new infrastructure. We will then become (or work with) your DevOps team to supporting you. This can be anything from containerising workloads to diagnosing performance issues to deploying a new multi-region Postgres cluster. Whatever you need done on your hardware that we feel we can reasonably support. We are the ones on-call should NATS fall over at 4am.
Your team also has full access to the Kubernetes cluster to deploy to as you wish.
I think the pricing page is the most concrete thing on our website, and it is entirely accurate. If you were to phone us and say, "I want that exact hardware," we would do it for you. But the real value we also offer is in the DevOps support we provide, actually doing the migration up-front (at our own cost), and being there working with your team every week.
https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-i...
But. Don't leave it until the last minute to talk to them about this. They don't make it easy, and require some warning (think months, IIRC)
It works because bare metal is about 10% the cost of cloud, and our value-add is in 1) creating a resilient platform on top of that, 2) supporting it, 3) being on-call, and 4) being or supporting your DevOps team.
This starts with us providing a Kubernetes cluster which we manage, but we also take responsibility for the services run on it. If you want Postgres, Redis, Clickhouse, NATS, etc, we'll deploy it and be SLA-on-call for any issues.
If you don't want to deal with Kubernetes then you don't have to. Just have your software engineers hand us the software and we'll handle deployment.
Everything is deployed on open source tooling, you have access to all the configuration for the services we deploy. You have server root access. If you want to leave you can do.
Our customers have full root access, and our engineers (myself included) are in a Slack channel with you engineers.
And, FWIW, it doesn't have to be Hetzner. We can colocate or use other providers, but Hetzner offer excellent bang-per-buck.
Edit: And all this is included in the cluster price, which comes out cheaper than the same hardware on the major cloud providers
It sounds like they probably have revenue in the €500mm range today. And given that the bare metal cost of AWS-equivalent bills tends to be a 90% reduction, we'll say a €10mm+ bare metal cost.
So I would say a cautious and qualified "yes". But I know even for smaller deployments of tens or hundreds of servers, they'll ask you what the purpose is. If you say something like "blockchain," they're going to say, "Actually, we prefer not to have your business."
I get the strong impression that while they naturally do want business, they also aren't going to take a huge amount of risk on board themselves. Their specialism is optimising on cost, which naturally has to involve avoiding or mitigating risk. I'm sure there'd be business terms to discuss, put it that way.
We figured, "Okay, if we can do this well, reliably, and de-risk it; then we can offer that as a service and just split the difference on the cost savings"
(plus we include engineering time proportional to cluster size, and also do the migration on our own dime as part of the de-risking)
5 - Datacenter (DC) - Like 4, except also take control of the space/power/HVAC/transit/security side of the equation. Makes sense either at scale, or if you have specific needs. Specific needs could be: specific location, reliability (higher or lower than a DC), resilience (conflict planning).
There are actually some really interesting use cases here. For example, reliability: If your company is in a physical office, how strong is the need to run your internal systems in a data centre? If you run your servers in your office, then there's no connectivity reliability concerns. If the power goes out, then the power is out to your staff's computers anyway (still get a UPS though).
Or perhaps you don't need as high reliability if you're doing only batch workloads? Do you need to pay the premium for redundant network connections and power supplies?
If you want your company to still function in the event of some kind of military conflict, do you really want to rely on fibre optic lines between your office and the data center? Do you want to keep all your infrastructure in such a high-value target?
I think this is one of the more interesting areas to think about, at least for me!