chronid

You arent' forced to use service mesh and complex secrets management schemes. If you add them to the cluster is because you value what they offer you. It's the same thing as kubernetes itself - I'm not sure what people are complaining about, if you don't need what kubernetes offers, just don't use it.

Go back to good ol' corsync/pacemaker clusters with XML and custom scripts to migrate IPs and set up firewall rules (and if you have someone writing them for you, why don't you have people managing your k8s clusters?).

Or buy something from a cloud provider that "just works" and eventually go down in flames with their indian call centers doing their best but with limited access to engineering to understand why service X is misbehaving for you and trashing your customer's data. It's trade-offs all the way.

The k/v store offers primitives to make that happen, but for non-critical controllers you don't want to deal with things like that they can go down and will be restarted (locally by kubelet/containerd) or rescheduled. Whatever resource they monitor will just not be touched until they get restarted.

I am fairly simplifying, but you are expected to know your direct dependencies (and normally wil), pagers have embedded escalation rules with prinaries and secondaries, etc. The tooling once you know what to do is better than anything outside of FAANGs I've seen in terms of integration and reliability.

Escalation teams are usually reserved for the "oh fuck" situations, like "I don't work on this site but I found it broken" or "hey I think we are going to lose soon this availability zone" or "I am panicking and have no idea how to manage this incident, please help me".

They're a glue mechanism to prevent silos and paralysis during an event, usually pretty good engineers too.

Some FAANGs at least (though they may not cover everything) have a "help something is broken but I don't know what to do" team and/or rotation for incident response, staffed on multiple continents to "follow the sun".

But you need to know they exist. :)

This has been done forever. Ops team had cronjobs to restart misbehaving applications out of business hours since before I started working. In a previous job, the solution for disks being full on a VM on-prem (no, not databases) was an automatic reimage. I've seen scheduled index rebuilds on Oracle. The list goes on.

You should look where the economy is growing and where the salaries are growing. It's not uniform at all.

The entire situation (as an EU country citizen who moved to another EU country) and the narratives around it are funny to me because they're the same as the ones going around for years in my birth country.

"Side X should learn they should get better candidates, otherwise people are not going to show up" way of thinking included, which has only led to further decline as the "conservatives" win and make the situation worse taking more and more seats and control in state controlled companies while at the same time pushing their own companies to absorb more and more of the budget. Yeah, not showing up because you did not like the candidate was a great success - if you wanted the decline to accelerate, that is.

Well, good luck US friends, to you and us all.

Even software (at least outside academia) eventually has to fight physics and the thing with most gravity of it all, money.

Doesn't a lot of optimization target making your code and data more cache friendly because memory latency (not bandwidth?) kills performance absolutely (between other things like port usage I guess)?

If something is in L3 it is better for CPU "utilization" than stalling and reaching out to RAM. I guess there are eventually diminishing returns with too much cache, but...

The badging system will notice someone has not left. This should have made security panic a fair bit on Friday evening...

The "hate" was not about the philosophy of linux.

It was about the udev/systemd maintainers writing "fuck you Gentoo developers" in a mailing list when they wanted udev to support only systemd, actually that udev should be part of systemd (for no real reason) and reimplementing things (sometimes badly for years) instead of reusing what was in place. It was about pretty big bugs (I remember a bug around coredumps being cut at ~700MB in size, which the systemd maintainers responded with the equivalend "well that's your problem not ours, journald is not fast enough and so we cut the max size" and was only closed eventually - ~3 years - probably because some RH customer screamed loudly enough). And the original discussions were even before systemd decided to hardcode what essentially is random IP addresses into resolved ("maintainers should change those values if they don't like them!") and other network services (NTP? I don't remember now anymore). It was about the billion libraries (later on there have been some security fun issues) it brought in and made your stuff link to (I vaguely remember something to parse qr codes?). It was about not-documented DBUS interfaces and making DBUS a core part of the system...

Most complex services still had heavy, long prestart script written in bash anyway at that point in time even when using systemd, so the script complexity was not really gone (and I think postgresql for example still has something like that).

Now, Redhat pushed hard for it and you cannot just ignore what they do and go your own way since the number of maintainers they employ is massive along the entire stack, from kernel to GNOME - Systemd was convenient to use and people went for it even with all the issues it had/has for good reasons (same as pulseaudio), but thinking the only discussion at the time was about the "linux philosophy" is a strawman and changing history a fair bit. :)

I'm not trying to defend CrowdStrike, but pointing to the fact Delta is the one maintaining and owning critical infrastructure and the executives trying to shift this responsibility onto someone else is the reason this happened in the first place. :)

Did crowdstrike force delta to accept running what essentially is a permanent RCE in their production fleet? You do not buy a software that is capable to do that and you put the fact it's not capable of doing that in the contract.

The update policy may work for the client version updates, but not for the "policy definition", otherwise delta won't get the sweet "all vulns mitigated with a 4h SLA" they crave.

AFAIK crowdstrike can push updates at any time at any host. There are staging areas they may use, but don't have to (particularly for definitions updates).

Crowdstrike should have done a better job, but Delta chose them (to offload the responsibility and work) and now they're claiming foul. They knew the risk. This is a classic executive play of claiming the fault lies in the consultants/vendor and taking no responsibility.

Crowdstrike is not handling critical infrastructure. Delta is.

The reality is the industry wants its cake and eat it too. No one forced Delta to buy a software which could force upgrades in their production fleet. They're a billion dollars company, and should put their big boys pants on.

This is not really correct, and assumes the state of a cloud service (let's say load balancing) is binary.

In my experience it's not. The cloud will glitch. The load balancing algo will break subtly for your workload. Your traffic will get blackholed for no apparent reason. I spent a week trying to convince a cloud provider they fucked up (the time it took for them to give us someone who could run the appropriate tcpdump) once. There was no global outage.

It's on you to determine if this is important for you or not in your case, but you will need to mitigate it above a certain SLA threshold requirement. It's far for consigning CAP to history or a curiosity, which is like saying you will never have network issues if you use "serverless" stuff.

Not talking about anyone in particular, but sometimes I feel people building and using the cloud reach hubris level of surety in their systems - I worked both sides of the fence, and I know the long tails of fuckups that impact customers though...

Sorry, this is untrue. Enterprises have SOCs and oncalls, if there is a high risk they can do at least minimal testing (which would have found this issue as it has a 100% bsod rate) and then fleet rollout. It would have been rolled out by Friday evening in this case without crashing hundred of thousands of servers.

The CS customer has decided to offload the responsibility of its fleet to CS. In my opinion that's bullshit and negligence (it doesn't mean I don't understand why they did it), particularly at the scale of some of the customers :)

> I am sympathetic to that, but its only possible if both policy and staffing allow.

We are not talking about small companies here. We're talking about massive billion revenue enterprises with enormous IT teams and in some cases multiple NOCs and SOCs and probably thousands consultants all around at minimum.

I find it hard to be sympathetic to this complete disregard of ownership just to ship responsibility somewhere else (because this is the need at the of the day let's not joke around). I can understand it, sure, and I can believe - to a point - someone did a risk calculation (possibility of crowdstrike upgrade killing all systems vs hack if we don't patch a CVE in <4h), but it's still madness from a reliability standpoint.

> for policy, there are lots of places that demand CVEs be patched within x hours depending on severity.

I'm pretty sure leadership when they need to choose between production being down for an unspecified amount of time and taking the risk of delaying (of hours in this case) the patching will choose the delay. Partners and payment integration providers can be reasoned with, contracts are not code. A BSOD you cannot talk away.

Sure, leadership is also now saying "but we were doing the same thing as everyone else, the consultants told us to and how could have we have known this random software with root on every machine we own could kill us?!" to cover their asses. The problem is solved already, since it impacted everyone, and they're not the ones spending their weekend hammering systems back to life.

> However you are also dependent on programs you install not autoupdating. Now, most have an option to flip that off, but its not always 100% effective.

You choose what to install on your systems, and you have the option to refuse to engage with companies that don't provide such options. If you don't, you accept the risk.

Windows updates are also cybersecurity, but the customer has (had?) a choice to how to roll those out (with Intune nowadays?). The customer should decide when to update, they own the fleet not the vendor!

You do not know if a content update will screw you over and mark all the files of your company as malware. The "It should never happen" situations are the thing you need to prepare for, the reason we talk about security as an onion, the reason we still do staggered production releases with baking times even after tests and QA have passed...

"But it's cybersecurity" is not a justification. I know that security departments and IT departments and companies in general love dropping the "responsibility" part on someone else, but in the end of the day the thing getting screwed over is the company fleet. You should retain control and make sure things work properly, the fact those billion dollar revenue companies are unable to do so is a joke. A terrible one, since IT underpins everything nowadays.

I wrote this in another thread already, but the fuck up was both at crowdstrike (they borked a release) but also and more importantly their customers. Shit happens even with the best testing in the world.

You do not deploy anything, ever on your entire production fleet at the same time and you do not buy software that does that. It's madness and we're not talking about small companies with tiny IT departments here.

I also blame the customers here to be completely honest.

The fact the software does not allow for progressive rollout of a version in your own fleet should be an instantaneous "pass". It's unacceptable for a vendor to decide when updates are applied to my systems.

Centralized management is very useful, just a random delay is not enough. One of the (big) companies I worked with had jury rigged something with chef I believe to show different machines different "repositories" and roll things out progressively (1% of the fleet, 5%...).

What I find definitely depressing is the fact we used to roll out progressively even OS upgrades (I guess now that is done through intune?) and was one point in favor of windows (on Linux you had to do things yourself at the time AFAIK, I don't think the situation has improved much).

Nowadays we get mandated random software upgrading at once on the entire company fleet and no one bats an eye - I counted more than a dozen agents installed for "security" and "monitoring" purposes in my previous company servers, many of those with hooks in the kernel obviously, and many of those installed with random policies to tick yet another compliance box...

This is not my experience, working in small shops/enterprise companies (some regulated). What I've seen is a constant, hard resistance from security "departments" to do anything that is not making policies (one company I worked with for a while had a security policy denying usage of managed identities in Azure...) and buying yet another magic solution from a vendor that will fix all our security problems (offloading its maintenance on... operations teams!), sometime with configurations that resemble the proverbial "very expensive firewall with ACCEPT ALL policies in all directions".

The companies with working security in my - limited, sure - experience had security teams owning the tools and making the life easier for developers and ops, from something "simple" like certificate rotation automation, to mTLS that is "transparent" for apps, to authn/authz, to secret management, all owned and managed by the security org.

Increased efficiency makes systems more fragile though, so minmaxing isn't actually a good outcome. Long term at least.

Next quarter we may all get a bonus, unless a rival AI exploits our fragility...

If they manage it in my company at least of course, the real question is if they want to manage it (and know what that means). Usually they don't, in my experience.

Those people will actually be grateful they don't have to deal and spend weeks to debug yet another cloud provider hidden gotcha or bug and convince support they are right. They will now very happy as they can now deliver value (at relative scale) and really speed up processes. I know I am.

Good ol' bare metal is real nice, but it won't save you from application complexity, security requirements, and so on - you still need to manage it somewhat. If you're not a startup looking for market fit at least.

Salt is unfortunately (in my two year old experience) full of features and strange bugs related to them - I do remember encountering a bunch of memory leaks master side for example. Deployment had the classic issues of python applications.

The core was (mostly) solid, but if you need anything done on it do buy the time of some consultant working on it already.

It's not as accessible as ansible, but definitely more accessible than cfengine. At least coming from Ansible.

I have worked on and around systems with an order of magnitude more data and a single node failing did not matter. We weren't using btrfs anyway (for data drives) and it definitely was not cheap. But storage never is.

But again, most systems are not like that. Kubernetes cluster nodes? Reimage at will. Compute nodes for vms backed by SAN? Reimage at will. Btrfs can actually make that reimage faster and it's pretty reliable on a single flash drive so why not?

Machines die. Hardware has bugs, or is broken. Things just bork. It's a fact of life.

Would I build a file storage system around btrfs? No - without proper redundancy at least. But I'm told at least Synology does.

I'm pretty sure there's plenty of cases where it's perfectly usable - the feature set it has today is plenty useful and the worst case scenario is an host reimage.

I can live with that. applications will generally break production ten billion times before btrfs does.

Yes, that would be great, but unfortunately there are application teams (particularly in the enterprise) lacking such tact when blaming infrastructure for issues.

Good old silos are alive and well, and ownership is not always part of the culture.