HNHacker News
TopNewBestAskShowJobs

adamcharnock

3,685 karma · joined May 3, 2010

Senior SRE and founder

https://lithus.eu - We run your infrastructure so you don't have to. Managed Kubernetes on bare metal in Germany. ~50% cheaper than AWS.

submissionscomments
adamcharnock··on Millions are visiting the European Alternatives site. What trends are we seeing?
In my experience it is a lot of things, in varying amounts depending on the situation (and each of these is both good in some ways and bad in others):

- The USA is a huge and fairly homogenous fairly wealthy fairly risk-positive market.

- The USA has a bigger = better mindset. "Your restaurant is fantastic, you should open another one!".

- Abysmal work-life-balance. Great for making stuff.

- Minimal language and cultural barriers.

- Because of the USA market, you can grow huge before having to expand to other countries.

- Because you can grow huge in the USA market, that is where the funding goes.

- All the the USA's allies didn't mind too much, because they are(/were?) allies

- All the non-USA-allies developed their own markets (Russia, East Asia, Africa, India, etc).

- And so now, given the messaging coming out of the USA, EU governments/companies/people are giving their USA-entanglement a second look.

Which, being in the EU, is pretty exciting. I feel like there is a lot of opportunity here right now.

Disclaimer: I'm sure there are 1001 reasons, and the above is just one potential view.

adamcharnock··on Millions are visiting the European Alternatives site. What trends are we seeing?
We're[1] seeing a huge influx of new clients right now, and not just any clients, but excited and enthusiastic ones. These are from EU, UK, Canada, SE Asia, Australia. Typically they are growing startups or SMEs who are no longer see investing further in AWS/GCloud/Azure as a good idea. It isn't solely because of politics, but politics is generally the catalyst.

Honestly, it's pretty exciting.

[1] EU Infrastructure with DevOps included - https://lithus.eu

adamcharnock··on Zero-Downtime Kubernetes Deployments on AWS with EKS
> You can run Postgres in any managed K8s environment (say AWS EKS) just fine and enable any extensions you want as well.

You absolutely can do this, and we do ineed run Postgres in-cluster.

We generally see that people prefer a managed solution when it comes to operating their databases. Which means that when it comes to their (eg) AWS EKS clusters, they often use RDS rather than running the DB in-cluster.

Our service is also a managed service, and that comes with in-cluster databases. So clients still get a managed service, but without the limitations of (eg) RDS.

adamcharnock··on Zero-Downtime Kubernetes Deployments on AWS with EKS
I've come at this from a slightly different angle. I've seen many clients running k8s on expensive cloud instances, but to me that is solving the same problems twice. Both k8s and cloud instances solve a highly related and overlapping set of problems.

Instead you can take k8s, deploy it to bare metal, and have a much much more power for a much lower cost. Of course this requires some technical knowledge, but the benefits are significant (lower costs, stable costs, no vendor lock-in, all the postgres extensions you want, response times halved, etc).

k8s smoothes over the vagaries of bare-metal very nicely.

If you'll excuse a quick plug for my work: We [1] offer a middle ground for this, whereby we do and manage all this for you. We take over all DevOps and infrastructure responsibility while also cutting spend by around 50%. (cloud hardware really is that expensive in comparison).

[1]: https://lithus.eu

adamcharnock··on [dead]
Coverage of 'day 5':

https://news.ycombinator.com/item?id=42933219

adamcharnock··on Turning the database inside-out (2015)
This all sounds to me very close to the event-sourcing/CQRS/DDD area of thinking. In which case you look at it in two parts:

- Event firing: Here is where you fire an event saying that the thing has happened (i.e. item_added_to_cart, not add_item_to_cart). Crucially, this event states the thing has happened. This isn't a request, it is a past-tense statement of fact, which is oddly important. It is therefore at this point where you must do the validation.

- Event handing: Here you receive information about an event that has already happened. You don't get to argue about it, it has happened. So you either have to handle it, or accept you have an incomplete view of reality. So perhaps you have to accept that the cart can have more than 10 items in some circumstances, in which case you prompt the use to correct the problem before checking out.

In fact, this is typically how it goes with this kind of eventual-consistency. First fire the event that is as valid as possible. Then when handing an 'invalid' event accept that its just got to happen (cart has 11 items now), then prompt the user fix it (if there is one).

Not sure how helpful this is here, but thought it a useful perspective.

adamcharnock··on Is the world becoming uninsurable?
I definitely understand what you are saying here, and it makes sense. But concrete is quite common in Europe these days, which I suspect would also be a good option for earthquake zones.
adamcharnock··on Take the pedals off the bike
I would love to see this approach in language learning, which I am fairly bad at. I'm very much driven by results rather than accuracy, and so often a teacher will correct me on the finer points of the language I'm using and I'll either no idea what they are talking about, or know that I am certainly not going to remember that detail. In either case I find it very demoralising.

If there was a "Learn to speak German like a 5 year old" course then I would love that. Give me something usable to motivate me further, then I can come back for more complexity when I actually want and need it.

But isn't this the case for all language courses? They start you slow and build up? I feel like it isn't, although perhaps it is just the courses I have seen. It seems to me that the people who teach languages generally really like languages, so they understandably revel in the details. I, however, do not (although I wish I did).

I realise now this is a bit of a rant. Apologies!

adamcharnock··on Postgres for everything (e/Postgres)
If anyone has seen a steams implementation in Postgres please let me know. Not just pub/sub, but full implementation of streams with consumer groups would be very interesting to me.

Second best, how would you implement this in Postgres? I’m tempted to give it a go but I haven’t a fully-baked plan yet.

adamcharnock··on Kubernetes on Hetzner: cutting my infra bill by 75%
How about we have a chat? I think it is hard for startups to justify implementing this infrastructure from scratch because that is a lot of time & skills that are really best focussed elsewhere.

Ping me an email (see bio), always happy to chat.

adamcharnock··on Kubernetes on Hetzner: cutting my infra bill by 75%
I don't have one I can share publicly, but if you send me an email I'll see what I can do :-) Email is in my profile.

You'll need a bit of baseline networking knowledge.

adamcharnock··on Kubernetes on Hetzner: cutting my infra bill by 75%
It is under their costing for 'additional hardware'[1]. You need to factor in the switch, uplink for each server, and the NIC for each server.

[1]: https://docs.hetzner.com/robot/general/pricing/price-list-fo...

adamcharnock··on Kubernetes on Hetzner: cutting my infra bill by 75%
I wholeheartedly agree, I'm trying to be generous as I know I have a bias here.

I think the AWS way made clear sense in the days before the current generation of tooling existed, when we were SSH-ing into our snowflake servers (for example). But now we have tools like Kubernetes/Nomad/OpenShift/etc/etc, the logic just doesn't seem to add up any more.

The main argument against it is generally of the form, "Yes, but we don't want to hire for non-cloud/bare-metal". Which is why I think a consultancy provides a good middle ground here – trading off cost savings against business factors.

adamcharnock··on Kubernetes on Hetzner: cutting my infra bill by 75%
> As a consultancy company, you want to sell that. As a customer, I don't see how that's worth it at all.

Well I do rather agree, but as a consultancy I'm biased.

But let's do some math. Say it's 4 months (because who has uninterrupted time), a senior rate of $1000/day. 20 days a month, so 80 days, is an $80k outlay. That's assuming you can get the skills (because AWS et al like to hire these kinds of engineers).

Say one wants a 3 year payback, that is $2,200/month savings you need. Which seems highly achievable given some of the cloud spends I've seen, and that I think an 80-90% reduction in cloud spend is a good ballpark.

The appeal of a consultancy is that we'll remove the up-front investment, provide the skills, de-risk the whole endeavour, even put engineers within your team, but you'll _only_ save 50%.

The latter option is much more appealing in terms of hiring, risk, and cash-flow. But if your company has the skills, the cash, and the risk tolerance then maybe the former approach is best.

EDIT: I actually think the(/our) consultancy option is a really good idea for startups. Their infrastructure ends up being slightly over-built to start with, but very quickly they end up saving a lot of money, and they also get DevOps staffing without having to hire for it. Moreover, the DevOps resource available to them scales with their compute needs. (also we offer 2x the amount of DevOps days for startups for the first year to help them get up and running).

adamcharnock··on Kubernetes on Hetzner: cutting my infra bill by 75%
We've toyed around with this idea for clients that do some data-heavy data-science work. Certainly I could see that running an on-premise Minio cluster could be very useful for providing fast access to data within the office.

Of course you could always move the data-science compute workloads to the cluster, but my gut says that bringing the data closer to the people that need it would be the ideal.

adamcharnock··on Kubernetes on Hetzner: cutting my infra bill by 75%
We've [1] been using Hetzner's dedicated servers to provide Kubernetes clusters to our clients for a few years now. The performance is certainly excellent, we typically see request times half. And because the hardware is cheaper we can provide dedicated DevOps engineering time to each client. There are some caveats though:

1) A staging cluster for testing updates is really a must. YOLO-ing prod updates on a Sunday is no one's idea of fun.

2) Application level replication is king, followed by block-level replication (we use OpenEBS/Mayastor). After going through all the Postgres operators we found StackGres to (currently) be the best.

3) The Ansible playbooks are your assets. Once you have them down and well-commented for a given service then re-deploying that service in other cases (or again in the future) becomes straightforward.

4) If you can I'd recommend a dedicated 10G network to connect your servers. 1G just isn't quite enough when it comes to the combined load of prod traffic, plus image pulls, plus inter-service traffic. This also gives a 10x latency improvement over AWS intra-az.

5) If you want network redundancy you can create a 1G vSwitch (VLAN) on the 1G ports for internal use. Give each server a loopback IP, then use BGP to distribute routes (bird).

6) MinIO clusters (via the operator) are not that tricky to operate as long as you follow the well trodden path. This provides you with local high-bandwidth, low-latency object storage.

7) The initial investment to do this does take time. I'd put it at 2-4 months of undistracted skilled engineering time.

8) You can still push ancillary/annoying tasks off onto cloud providers (personally I'm a fan of CloudFlare for HTTP load balancing).

[1]: https://lithus.eu

adamcharnock··on Engineers do not get to make startup mistakes when they build ledgers
If you're looking for a pre-built double entry accounting Postgresql schema that provides database-level integrity checks:

https://github.com/adamcharnock/django-hordak

I created and maintain this, along with a couple of others. It is built for Django (so great if you're using Django), but extracting the schema wouldn't be too hard. It also has MySQL support, but the integrity checks are more limited.

(Side note: I'm a freelancer and available!)

adamcharnock··on WireGuard: Beyond the most basic configuration
I agree that would be useful. I'm fairly sure it is because all the entries in `AllowedIPs` are just written as-is to the routing table, and the routing logic in the kernel (and most/all routers?) has no facility for 'does not match'.

Instead the solution would be to add a explicit route to state where the excluded CIDR should be sent to. That would would be more specific and would therefore be used for matching packets rather than the 0.0.0.0/0 (or whatever) routed pointed at the wireguard tunnel.

adamcharnock··on Phased Array Microphone (2023)
I would love to see this come to our various mobile devices in a nicely packaged form. I think part of what is holding back assistants, universal-translators, etc, is poor audio. Both reducing noise and being able to detect direction has a huge potential to help (I want to live-translate a group conversation around a dining table, for example).

Firstly it would be great if my phone + headphones could combine the microphones to this end. But what if all phones in the immediate vicinity could cooperate to provide high quality directional audio? (Assuming privacy issues could be addressed).

adamcharnock··on Blood Sugar Control Is a Key Factor in Slowing Brain Aging, Mediterranean Diet++
Seems like the site is getting hugged to death right now:

https://web.archive.org/web/20241105075317/https://www.bgu.a...

adamcharnock··on AWS data center latencies, visualized
It is possible I was measuring latency in a single direction, rather than round-trip-time. My memory is a little hazy now.
adamcharnock··on AWS data center latencies, visualized
This isn’t just an issue for cloud providers. It’s also not easy to find collocation space either.

My best guess is that it is a combination lower demand (vs rest of world), and infrastructure availability (connectivity + power).

I can imagine a bunch of secondary factors too, but this to me sounds like the key broad reasons.

adamcharnock··on AWS data center latencies, visualized
Random fact: I did some planning around this for a client a while ago. While measuring the AWS latencies I found I could get approximate latencies (within 10%) by measuring the rough undersea cable length (km) and dividing by 150.

While not overly surprising, it was very consistent.

Edit: I think it was actually 155

adamcharnock··on AWS and Azure Are at Least 4x–10x More Expensive Than Hetzner
This is what we're trying to address at Lithus[1]. We're offering both the raw compute resources, and also the DevOps time needed to setup and manage the services your engineering team needs.

[1] https://lithus.eu

adamcharnock··on The Copenhagen Book: general guideline on implementing auth in web applications
My practice on this is to store the user-provided case, but do case insensitive lookups.

This means that you send emails to the case-sensitive address originally entered, but the user is free to login case insensitively.

The downside is that you cannot have two distinct users with emails that only differ in their case. But I feel rather OK about that.

adamcharnock··on A Brief Intro to Clean Architecture, Clean DDD, and CQRS
I’ve found that CQRS & DDD can be useful when modelling actual business processes, although it certainly isn’t without its overheard.

The benefit is that it can map nicely onto how existing processes work. You can sit with people in a meeting, they can describe their processes (by something as simple as passing post-it notes around the table), and those post it notes then directly become the basis of your system design.

Not that this is the only way to achieve that, but it is _a_ way.

adamcharnock··on Brazilian Electric "Suicide" Shower Heads [video]
> Same experience in a Costa Rica hostel! Except in that case my skin just gently tingled where the water touched it. It was a case of, “I know that sensation, and I certainly shouldn’t be feeling it in the shower”

~ A comment I left on the topic of these shower heads a little while ago. https://news.ycombinator.com/item?id=41250231

adamcharnock··on Teacher caught students using ChatGPT on their first assignment. Debate ensues
My takeaway from the thread is that LLM-use seems to be the manifestation of student's disinterest, dissatisfaction, and potentially poor attention spans. Comment from one student (who, from what I glean from the thread, is probably more capable than average).

There is one comment from a student in the thread. Sure they seem angry, but I also see their point:

> I have used chatgpt in literally all of my essays this semester, and got an average grade of 8-9.

> ...

> But you know why, at least in my case, I always get away with it? Because all of the work I have to do requires ZERO critical thinking from my side. In fact, it is even worse to use it sometimes. Don't you even dare to come up with your own opinions!!! Just quote someone that is already quoting someone and so and so on.

> I know most students don't give a single flick about your class and that unmotivates but damn it is the 5th year you ask for the same essay and we know you don't even read half of them. Why would I lose so much time doing something literally no one cares about.

> TL;DR: If a machine can do most college essays it's because they were mechanical jobs already.

adamcharnock··on Teacher caught students using ChatGPT on their first assignment. Debate ensues
That... sounds like a fantastic idea.

Having read the reddit thread linked in the article, I think the main problems would be one of (depending on the teacher):

1. Not wanting to

2. Needing funding/support to AI-transcribe all the video for each of their numerous students, but their institution not providing it.

But those seem [eventually] surmountable. I really like the idea.

adamcharnock··on Teacher caught students using ChatGPT on their first assignment. Debate ensues
I found the linked Reddit thread to be particularly interesting (educators discussing use of ChatGPT by students, 1y ago):

https://www.reddit.com/r/Professors/comments/17v6478/chatgpt...

← PreviousPage 5 of 16Next →