adamcharnock

That is true now, but 4-5 years ago I was starting a small ISP and prices were almost triple that. $80-$100/address seemed somewhat common IIRC, and prices only seemed to be going up. So I can certainly see an ISP looking at that and making a plan for a future where IPs cost $200+ each.

Since then prices have come down. I can only speculate that high prices caused a bunch of organisations to decide to sell their unused IP blocks. I suppose the question is: Does the market now have enough spare capacity to see us through to IPv6-only adoption at reasonable IP costs, or will we be seeing high prices again in 5+ years?

This just spurred me to check recent sales prices: https://auctions.ipv4.global/prior-sales

Check the charts for price since 2020. It looks like IPv4-block prices almost tripled 2020-2021. Then seem to have been in gradual decline since then. I'm rather surprised.

I’ve just been reading some of Vernor Vinge’s books and they’ve certainly hit different the second time around (A Deepness in the Sky, A Fire Upon the Deep).

I think it really is mostly that I’ve changed. It’s been a slow change that has been hard to notice from the inside, but one that becomes quite stark when re-experiencing a fixed point such as a book.

They’re like cast iron pans. The flavour builds over time as long as you don’t wash them too thoroughly.

I’m in the market for new monitors (or maybe only one in this case!)

A question if you don’t mind - Do you find 4K resolution to be sufficient on a 40” screen?

Also just eager to hear any others reasons why you like it

In addition to the other replies, I also recall hearing some time ago that the AC EM field interacted with wildlife in surprising ways (causing sharks to attack the cable, IIRC). It could be an urban legend at this point though.

Our experience with Hetzner /Cloud/ is similarly shaky. But Hetzner dedicated servers all altogether a different story.

We (lithus.eu) deal with their custom solutions team regularly and often provision private networks in the 10-100G range. On top of that we deploy MinIO and OpenEBS/Mayastor, and the whole thing just hums along very nicely indeed.

Interesting. This is something I've yet to encounter, and we use Hetzner a lot for our clients (dedicated servers though, not cloud, so maybe that makes a difference).

> If you look closely, you'll also see the coffee/tea cups actors sip from are usually empty. Can't afford the risk of accidentally spilling liquid on the costume and delaying the shoot.

If I were a prop-master (is that what it is called?) I've always thought that I'd just have a bag of plaster of paris handy. Then 30 minutes before going on set just dump some in the prop-cup with some water.

Sets quickly, density is about the same, physics of the cup should look convincing. Probably best for disposable cups though.

Oh wow, this is exactly what we're doing. We've been doing this for a while, but since last year we've gone all-in on it. See: https://lithus.eu

We saw that companies were hesitant to deploy on bare metal because of: risk of project failure, need to retrain, need to hire, and general fear of unmanaged hardware.

So we did the math and realised that for half the price of AWS we could provide the infra /and/ provide the DevOps engineering time to support their software developers. We'd already built the IP to do this over the proceeding years anyway.

So we're definitely a services company rather than a product company, but we are actually doing this in a really meaningful way. We migrate people out of AWS (et al), then operate their infra and become their DevOps team. And now we have some very happy customers!

Always happy to chat if anyone is interested – adam@ domain.

> UDP message delivery order is not guaranteed

My thinking was that ordering would be pretty unaffected when there is only a single hop. But yeah, we would have needed to test that under load.

I had a developer colleague a while back who was toying with an idea that would require emitting and consuming a _lot_ of messages. I think it was somewhere on the order of 10k-100k/second. He was looking at some pretty expensive solutions IIRC.

I asked if the messages were all under 1.5kb, he said yes. I asked if at-most-one delivery was ok, he said yes. So I proposed he just grab a router and fire messages through it as UDP packets, then use BGP/ECMP to balance the packets between receivers. Add some queues on the router, then just let the receivers pick up the packets as fast as they could. You'd need some kind of feedback to manage back pressure, but ¯\_(ツ)_/¯

A fairly cheap way to achieve 1M+ messages per second.

I never got the chance to flesh-out the idea fully, but the simplicity of it tickled me. Maybe it would have worked, maybe not.

You're welcome! Always happy to give advice. Let's have a call next time. I'm adam@ (company domain in bio)

> If we manage to bundle our whole app in one server and have only 1-2 clients on -premise, do you still suggest Kubernetes or a simpler rsync on all servers is enough?

rsync is nice and simple. Personally I'd say at least use Ansible, with its built-in rsync support. Then you can do more than just copy files.

> Also, should we have a separate database instance for each client, or a Postgres cluster sharded? The latter seams more manageable.

Depends on the size. Run postgres separately for your on-prem clients. For your cloud clients, I'd say keep them on the same server until you start to get over 100GB-1TB of data, then start to think about sharding. RDS gets super expensive, so sharing too early may be uneconomical.

> I am afraid Kubernetes is overkill for our lambda needs.

For just Lambda, I agree. But if running everything outside of AWS (i.e. racking servers) then it shines, because then you have your app, postgres, valkey, etc, all balanced between servers.

So this is something I/we have a bit of experience in [1].

Firstly, I'd test what level of on-prem they are really asking for, because there are various shades of this:

- servers actually in their office

- their own servers in a datacenter (DC) near their office

- someone else's servers of which they are the only users, in a DC anywhere

- deployment into their own AWS account

Which one they actually need has an effect the kind of options available to you.

> From experience, is it really necessary or does a bigger server do the trick?

In our experience big servers do indeed to the tick. And if your costs are minor on Lambda/SQS/etc, then I imagine any new server is going to have 10x-100x the capacity you actually need.

> 3. Worse service: can SQS/SNS/lambda easily be replaced without feature loss?

Broadly speaking: yes. SQS can go to Redis/Valkey queues (ideally), or something heavier if needed. SNS also has good options, depending on the features you are using (FanOut -> Redis Streams, or Kafka; Email -> pick your provider). Lambda I would just replace with Kubernetes pods, but I'm a Kubernetes guy so hey.

> Also I am talking about on-premise but is it the best solution to mitigate the risk of service interruption?

I think it depends on what you are deploying. It will be easier for you to centralise more most likely. What we (business hat on now) would suggest would be a Kubernetes-managed cluster of physical servers, with the ability to isolate clients to their own server(s). This runs your enterprise clients separately from your public deployment, makes it easy to scale to new enterprise customers, easy to scale when individual customers grow, and provides easy migration of clients between servers in cases of hardware failure. You could also offer your customers a HA option which runs your software over three servers, with HA failover of services. Our company would then run it all for you, but you'd still have full access. (Removes business hat)

[1]: https://lithus.eu

This is certainly interesting, and is something I often wonder about. In our case we mostly run Kubernetes clusters on EX130 servers (Xeon, 24c, 256GB). In this situation there are a lot of processes running, for which increased cores-count and memory availability seems worth it. Particularly when we have the cost of 10-25g private networking for each server, lower node counts seems to come out being more economical.

But, but with fewer processes I can totally believe this works out to be the better option. Thank you for the write-up!

There are some excellent comments here, so I'll just add my particular flavour.

I think using Kubernetes effectively in 2025 is more about what you _don't_ use than what you _do_ use. As an early stage startup you can get a long way with no RBAC, no network policies, no auto-scalers, and even no stateful workloads. You can use in-cluster metrics metrics and logging before you need to turn to Prometheus, Loki, etc. Use something managed like AWS EKS.

Try to solve your problems first by taking away, and only if that isn't feasible then start adding. Plain old Deployments will get you a long way.

Now this next bit is going to sound like a pitch, and that's because it is – but when those free credits start running out, your bill starts reaching mid-four-figures, and you start thinking about your first DevOps hire, _call us_. Just for 30 minutes. We can migrate you out your cloud infra and onto a nice spacious bare metal k8s cluster, and we'll become your 24/7 on-call DevOps team. We'll get woken up in the night when things break, not you. And core-for-core it will cost a lot less than AWS.

The fact that we can do all that is a testament to how expensive AWS really is. K8s is a good choice now if you keep it simple, positions you well for growth in the future, and for a cluster under a couple of hundred cores it is going to be pretty economical to run it in the public cloud.

PS. Link in bio

Very cool and fun!

The first thing did when I opened it was to point my phone at the floor though, trying to find Australia. Took me a moment to realise it wasn’t that kind of pointer!

I love the sound of this, but the readme could really benefit from the screenshots. That way I can get an idea of what it looks like without having to actually start it running.

To answer from a Kubernetes perspective: Both OpenEBS Mayastor and LocalZFS now support disk encryption.

This is exactly what we [1] do! We migrate clients out of AWS and into Hetzner bare-metal k8s clusters, and then we also become the client's DevOps team (typically for a lot less than Amazon charges)

I will say that there is a fair bit of lifting required to spin up a k8s cluster on bare metal, particularly for things such as monitoring and distributed block storage (we use OpenEBS). I would ballpark it as a small number of months.

It is likely easier on their cloud offering, but we've found that to be a little less reliable than we would hope.

Happy to chat more: adam@...

[1] https://lithus.eu

We're setting up a data-heavy client at the moment who has a similar need. We're working with Hetzner's custom solutions team to provision a multi-AZ setup, with 25G networking and 100G AZ interconnects. Link in bio if you want to chat, email is adam@...

This certainly looks like a pleasingly straight-forward way to spin up k8s.

I do notice that this deploys onto their cloud offering, which we've (https://lithus.eu) found to be a little shaky in a few places. We deploy clients onto their bare metal line-up which we find to be pretty rock solid. The worst that typically happens is the scheduled restart of an upstream router, which we mitigate via multi-AZ deployments.

That being said, there is a base cluster size under which a custom bare-metal deployment isn't really viable in terms of economics/effort. So I'll definitely keep an eye on this.

I left a comment fairly related to this a while back:

https://news.ycombinator.com/item?id=43465403

But there _is_ also an attitude difference. In terms of willingness to take risks and innovate, the USA does do very well for itself, and I think the UK does ok too. But that cannot be said for the EU countries I’ve lived in. Stable reliable long term safe jobs seem to be more the name of the game, and starting a company is seen as a big and risky commitment. Whereas in the US and UK you can start a company in your lunch break.

It is a generalisation, and I’m part of a wonderful entrepreneurial community here in Munich. But even there everyone says how risk averse European businesses are. I really really wish it wasn’t true.

This may be blindingly obvious, but I’m going to say it anyway: If Amazon was willing to actually give up control of AWS EU, then this kind of announcement would be entirely surplus to requirements. But they will (obviously and rationally) not be giving up control of AWS EU because that would essentially have to be an act of charity, so they need to dress it up a bit.

(Before hitting ‘add comment’ I’m taking a moment to consider if I’m being overly cynical. But no, I really don’t think I am. But my company does compete with AWS, so that is a bias.)

I think this is a really interesting point. I have a few thoughts as a read it (as a bit of a grey-beard).

Things are moving fast at the moment, but I think it feels even faster because of how slowly things have been moving for the last decade. I was getting into web development in the mid-to-late-90s, and I think the landscape felt similar then. Plugged-in people kinda knew the web was going to be huge, but on some level we also know that things were going to change fast. Whatever we learnt would soon fall by the wayside and become compost for the next new thing we had to learn.

It certainly feels to me like things have really been much more stable for the last 10-15 years (YMMV).

So I guess what I'm saying is: yeah, this is actually kinda getting back to normal. At least that is how I see it, if I'm in an excitable optimistic mood.

I'd say pick something and do it. It may become brain-compost, but I think a good deep layer of compost is what will turn you into a senior developer. Hopefully that metaphor isn't too stretched!

I think this is more akin to shutting down an AZ, rather than a region, which certainly does happen. Except AZ lettering is randomised per AWS account, so your 'us-east1-a' isn't my 'us-east1-a'. Which means AWS can migrate people away, over time. I believe older accounts which still use the old AZ are given notice that it is closing.

Plus, there was the whole closure of AWS EC2 Classic, replaced with AWS VPC.

This another reason why deploying over multiple AZs has its benefit. Not just for technical failure, but it means you can still move should one region close down.

I suppose an interesting questions is: would I prefer to move a single-AZ deployment such as this in the cloud, or in the real world? And honestly I can see the pros and cons of each.

In the cloud it involves a bunch of engineering time (possibly minimal, likely a lot more given reality). In the real-world it involves a temporary fibre connection to the next DC over, and a gradual or rapid move of hardware with the help of some specialist contractors (for example). But at least the state and implementation quirks move with the compute. I can see it either way, but I can feel myself wanting to believe in the latter. There is something about trucking servers across town that appeals me.

It's always either DNS or MTU.

(Or, as I recently encountered, it can also be a McAfee corporate firewall trying to be helpful by showing a download progress bar in place of an HTTP SSE stream. I was sure that was being caused by MTU, but alas no.)

> 2-5x pipeline performance at 1/2 cost just by using self-hosted runners on bare metal rented machines like Hetzner

This is absolutely the case. Its a combination of having dedicated CPU cores, dedicated memory bandwidth, and (perhaps most of all) dedicated local NVMe drives. We see a 2x speed up running _within VMs_ on bare metal.

> And knowing how to deal with bare metal/utilize this kind of compute sounds generally useful skill - but I rarely encounter people enthusiastic about making this kind of move

We started our current company for this reason [0]. A lot of people know this makes sense on some level, but not many people want to do it. So we say we'll do it for you, give you the engineering time needed to support it, and you'll still save money.

> I just don't see why going bare metal is always such a taboo topic even for simple stuff like builds.

It is decreasingly so from what I see. Enough people have been variously burned by public cloud providers to know they are not a panacea. But they just need a little assistance in making the jump.

[0] - https://lithus.eu

An interesting idea! I suspect a major speed up would come from the fact that the column is staying the same size. So (I assume) far fewer bytes would need to be moved around.