916 karma · joined January 24, 2016
I've wondered the same - though never been able to put it into such concise words. I formerly worked as a Mechanical/Controls engineer and did not feel the same frustration with interruptions at the time despite being an equally challenging industry (as far as my experience went). My conclusion is that the hard constraints of a physical system keep the problem space (and thus the required working memory) relatively small in the physical world. When you're writing software that has no physical manifestation, the problem space is typically massive since the constraints are fewer.
As a former maintainer of packages in an enterprise Linux ecosystem, I agree. But...the patches are often meant to solve a dependency mismatch and the maintainers can't easily pull related dependencies forward without breaking other applications. It's great that other distros like Arch just simply upstream the patches - it's the right thing to do after all - but that takes time that the enterprise distributions don't always have.
Both companies had a huge moat until Amazon came along blew their doors off. And Amazon has been investing heavily in electric delivery vehicles: https://www.cnbc.com/2021/03/18/amazon-begins-testing-rivian...
-------------------------------------------------------
The problem I see with kafka is that it was built before cloud architectures were commonly adopted (with distributed systems everywhere). Confluent has put a lot of effort dragging kafka's architecture to the present, but some major features are missing:
- Auto-scaling: Confluent finally introduced "elastic scaling" a few months ago but it only allows you to scale up and must be triggered by the admin (no threshold-based auto-scaling).
- Multi-tenancy: Planning for a multi-tenant kafka cluster is not for the faint of heart. Achieving isolation tends toward liberal usage of topics of which starts to become unmanageable in the low thousands. This isn't crazy when you've got a few hundred microservices and several tenants to keep isolated.
- Decoupled brokers and storage: Any broker scaling or failure can lead to downtime while event storage is redistributed.
Confluent's Cloud service reduces operational overhead but isn't always feasible due to cost, resource limits (like service accounts or schemas for instance), data controls, etc.-------------------------------------------------------
The biggest story in recent years was the Elwha dam removal. It's stunning to see how much the landscape has changed since: https://therevelator.org/elwha-dam-removal/
And a recent story about a Skagit river dam: https://www.king5.com/article/news/investigations/seattles-s...
More details: https://www.howtogeek.com/337719/how-to-tame-your-chromecast...
But I hope most people here don't feel a morale hit when their code gets tossed. Inherently, I understand the desire for one's creative work to live on, but that's essentially incompatible with this industry. And I also have found much joy in turning-down/deleting existing services I've written. My team celebrates these events and it's always fun to give that high-maintenance service a viking funeral.
- Auto-scaling: Confluent finally introduced "elastic scaling" a few months ago but it only allows you to scale up and must be triggered by the admin (no threshold-based auto-scaling).
- Multi-tenancy: Planning for a multi-tenant kafka cluster is not for the faint of heart. Achieving isolation tends toward liberal usage of topics of which starts to become unmanageable in the low thousands. This isn't crazy when you've got a few hundred microservices and several tenants to keep isolated.
- Decoupled brokers and storage: Any broker scaling or failure can lead to downtime while event storage is redistributed.
Confluent's Cloud service reduces operational overhead but isn't always feasible due to cost, resource limits (like service accounts or schemas for instance), data controls, etc.I generally agree strongly (e.g. car vs air safety). Though this group is highly cognizant of KNOWN risks. It's the UNKNOWN risks (usually found at those interfaces I mentioned above) that get overlooked. Unknown risks definitely will be more problematic in a compressed-schedule environment.
But I think it's worth noting to the (mostly software oriented) HN crowd - aerospace projects have massive manufacturing cycle times. Some things literally take years to manufacture for the proof-of-concept stage, let alone production. You can't NOT be incredibly schedule-oriented in this environment. This can cause some perverse incentives for management (which must be mitigated), but there are always going to be somewhat risky last-minute changes that could seem ill-advised in hindsight if you want to deliver something.