How to take down production with a single Helm command | Hacker News Reader

How to take down production with a single Helm command

ounapuu.ee

44 pointshddherman16 comments

Tangentially related, cannot help but be reminded of the "Ohnosecond": https://en.wikipedia.org/wiki/List_of_humorous_units_of_meas...

I've seen enough of these over the last two decades, between Ubuntu/Debian and RedHat, that I use and inherently trust other distros more. Just compile the source packages. If there is some issue, file a bug. Monkey patching everything silently leads to things like this, which I've never experienced with distros like Arch, Void, or Solus.

For helm and most other golang apps, you just need to download and put binary into your $PATH. They usually put them onto github releases, so it's really low friction way to install a genuine application.

Now to keep them updated is another story.

Last sentence. I love Go's install and run story, but keeping it up to date is a pain.

We trust, or used to trust, distros for this. Why are they messing with the source?

My understanding is that they've more or less always done this for various reasons: security patches, compatibility, dependency versioning. I understand the historical reasons that led to this structure for package management, especially with how brittle C dependencies seem to be, but I truly hate this practice. It seems to make it exceptionally difficult for authors of major software to establish any sort of invariants or security boundaries.

Debian is very very diligent about patching out phone-home and expiration timers and other such “developer knows best” misfeatures. Some apps also include autoupdate (aka solarwinds style RCE) which they also remove.

Overall they do way more good than harm. Trashing them because they sometimes make mistakes is probably incorrect.

But when they get it wrong, they can get it catastrophically wrong - like breaking ssh-keygen so that for given set of parameters there existed only 2^32 keys.

That's a good argument for not running IaC tools on your own workstation but on a dedicated machine where the environment is also somehow vetted.

On the other hand I am not sure how to actually vet all used tools. For instance if you take Helm there are many pointers to repos for various Linux distros and other OSs: https://helm.sh/docs/intro/install/

Who to trust? While the Helm page says "members of the community" are in charge of the Ubuntu repo the Fedora repo is called an "official repository". Presumably judging from the OP that means "official Fedora repo" opposed to "official Helm repo".

So essentially there are no repos of the Helm maintainers which makes installing and auto-updating it on any Linux distro virtually impossible.

Explanation here: https://github.com/helm/helm/issues/12681#issuecomment-19593...

Looks like it's a bug in Helm, but actually isn't Helm's fault, the issue was introduced by Fedora Linux.

(This issue is linked from the article, but it took me some time to find it.)

why would someone use Fedora in production, tho ?

helm runs on the operator machine, not server. While fedora server might not be very popular, fedora workstation seems to be pretty popular.

> Apparently the version of Helm packaged in Fedora Linux included a patch that introduced this issue

Let's translate: how to take down whatever by living on the bleeding edge.

Fedora is hardly bleeding edge

I always thought of Fedora as QA for red hat.

Apparently bleeding enough in this case.

And increase the bloodshed by, apparently, configuring your staging environment significantly different than your production environment. An instigating the bloodshed by doing something first in production and then testing it in staging (sure, the first time staging wouldn't have caught it, but maybe then you could look to see where you misconfigured staging...)

Reply on news.ycombinator.com