How does Kubernetes, microservices and front end frameworks fix that?
They don't. In fact, as someone who works at the more modern end of these things, I'd suspect that "no meaningful monitoring" probably correlates quite well with cloud era microservices thingies.
But we can't blame the tools, that space is just younger and by nature contains more immature stuff.
I don't think the argument here is for keeping things out of version control and monitoring. It's that you can actually leave stuff alone for several years, come back and find it working just as before, when the layers underneath doesn't constantly change. That can be liberating.
I think the article's point was that the industry is perpetually young and immature because it keeps (unnecessarily) chasing fads.
Doesn't being in a container mean its much easier to rebuild it from zero and thus allowing security updates to be more regular?
There's no benefit to using containers here, only more pain.
It’s really difficult to pin everything. Even if you manage that, upstream resources change/move.
Yeah observability is great, but no one I've seen in cloud manages to launch with it. So in place of SRE org with full observability stack, they launch with effectively nothing.
Strong attitudes of "we don't measure hardware metrics like cpu/memory/disk, we measure customer facing stuff like throughput/error rates etc". Sounds good, but what do you think happens to those if you run out of disk bro? Pretty sure the running out of disk happens first. Imagine if you could like.. catch it?
However theres lots of cloud people who don't know linux.
Therein lies the challenge. Too many of these cloud devs forget that whether its "server less" or not, theres a .. server, somewhere.
All the layers of abstraction work when they work, and when they don't .. best of luck figuring out what's going on in a timely fashion. Hope you aren't running any money on top of it.
Disk: essentially limitless. If disk of VM runs out, node will crash, new node will start. Service should keep running on other nodes meanwhile. If restarts happen too often, throughput/error rate will suffer or cost will rise.
Modern mentality is "let it crash" [0]. Software has bugs. Design so it can crash and scale depending on need.
[0] https://medium.com/@vamsimokari/erlang-let-it-crash-philosop...
However there are industries applying these lessons in HPC / data analytics / things that touch money live .. operating on scales of users in the 10s to maybe 100s. So stuff where downtime is far more costly both in dollars and reputation.
I'm also intrigued by the constant cloud refrain of "stuff crashes all the time so just expect it to" coming from a background where I have apps that run without crash for 6 months at a time, or essentially until the next release.
I'm all for scaling, recovery, etc.. I just fail to understand why it is desirable for this to be an OR rather than an AND.
What if stuff was highly recoverable and scalable but also.. we just didn't run out of disk needlessly?
IMHO, those aren't mutually exclusive. Your app code should be robust enough to run 6+ months at at time, and the "stuff crashes all the time so just expect it to" attitude should be reserved for stuff outside your control, like hardware failures.
And "let it crash" is hardly "My service is invincible!" I'm sure any serious Erlang deployment has seen what I've seen as well, which is a sufficiently buggy process that crashes often enough that the system never gets into a stable state of handling requests. "Let it crash" is hardly license to write crap code and let the supervisor pick up the pieces. It's a great backstop, it's a shitty foundation.
I haven't used Erlang, but my impression is that it's probably the same experience there?
In my case, IIRC, it was something to the effect of, a lot of our old clients out in the field connected with a field that crashed the handler. Enough of these were connecting that the supervisor was flipping out and restarting things (even working things) because it thought there was a systemic issue. (I mean, it was right, though I had it configured to be a bit to aggressive in its response.) The fact that I could let things crash didn't rescue the fact my system, even if I fixed that config, would strain its resources constantly doing TLS negotiations and then immediately crashing out what were supposed to be long term connections.
Obviously, the problem was in the test suite, we were able to roll back, address the problem, ultimately this was a blip not a catastrophe. I just cite it as a case where "let it crash" didn't save me, because the thing that was crashing was just too broken. You still have to write good code in Erlang. It may survive not having "great" code but it's not a license to not care.
Many software development processes considered as a whole are anti-fragile... mine certainly is, which is a great deal of why I love automated testing so much (I often phrase it as "it gives me monotonic forward progress" but "it gives me antifragility" is a reasonably close spin on it too)... but that's not unique to Erlang nor does Erlang have anything special to help with it as everything Erlang has for robustness is focused at run time. You can have anti-fragile development processes with any language. (The fact that I successfully left Erlang for another language entirely is also testament to that. I had to replace Erlang's robustness but I didn't have to replace its anti-fragility, since it didn't have it particularly.)
It's not the tool, it's how you use it.
It pertains to e.g. single database connection, single HTTP request etc. If something crashes there your APM reports it and Erlang's unique runtime continues unfazed. It's a property of the BEAM VM that no other runtime possesses.
"Let it crash" is in fact "break your app's runtime logic to many small pieces each of which is independent and an error in any single one does not impact the others".
Scaling an Erlang node is very rarely the solution unless you literally run out of system resources.
RE: error handling, Elixir apps I coded and maintained never had a problem. Everything was 100% transparent which is another huge plus.
In general CGI and PHP had the right idea from the start but the actual implementations were (maybe still are? no idea) subpar.
Erlang's runtime is of course nowhere near what you will get with Rust and very careful programming with the tokio async runtime, but it's the best out there in the land of the dynamic languages. I've tried 10+ languages which is of course not technically exhaustive but I was finally satisfied with making super parallel _and_ resilient workloads when I tried Elixir.
For a lot of tasks I can just code in the Elixir REPL and crank out a solution in an hour, including copy-pasting from the REPL and into an actual program with tests. Same task in Rust took me a week, just saying (though I am still not a senior in Rust; that's absolutely a factor as well) and about 3/4 of a day in Golang.
The only other small-friction no-BS language and ecosystem that came kinda close for me is Golang. I like that one a lot but you have to be very careful when you do true parallelism or you have to reach for a good number of libraries to avoid several sadly very common footguns.
Please read the last three paragraphs in [0]: "a well-designed application is a hierarchy of supervisor and worker processes" and "It handles what makes sense, and allows itself to terminate with an error ("crash") in other cases."
I've personally designed and co-implemented mission critical real-time logistics systems which dealt with tens of thousands of events per second, with hundreds of different microservices deployed on a cluster of 14 heavy nodes. Highly complex logic. At first we were baby-sitting the system, until it became resilient by itself. Stuff crashed all the time. Functional logic was still intact. Then we had true silence for months on our pager alerts.
I call it anti-fragile and Taleb is right. You can't make a system resilient if you don't allow it to fail.
* microservices help security because for each service you write, you create a minimal surface area (as opposed to monolithic applications where each method has access to maximum dependencies).
* microservices also help monitoring, because failures can be contained and one application does (should) not impact another. This means that your focus of monitoring is on application performance, and less on the underlying system architecture.
* monitoring can be a cross-cutting concern. For example, you can monitor all HTTP requests by instrumenting your ingress layer. Similarly, you can support a unified authorisation scheme by bringing this to the ingress layer. Even more, using microservices, you can independently promote services to production (sort of like feature toggles) or do green/blue promotion.
* security of your microservices is improved, because you can independently upgrade your dependencies per service. Many older monoliths are 'stuck' in a certain dependency configuration and cannot be upgraded without a 'big bang' of expensive dependency resolution and testing.
I could give many more examples, but I hope this captures the spirit a little.
Setting up monitoring for your docker containers is also a whole thing. :)
I think you’re taking my example a little too literally. My point is not that docker/k8s/whatever is bad; just that the ‘new’ adds features at the cost of simplicity.
Which is also the point of monitoring a container, which is fairly reliable nowadays, compared to managing your own health check service to ping your box to check if it's alive.
All in all, I think k8s is way too complicated for what it does for most people, but administrating servers was also a complex task to begin with, and the things sys admins were dealing with looked nightmarish to me. Heck, there was a time companies would have their own SMTP in house...
Those sort of attacks have tapered off though, right? Unless you're engaged in a specific feud, or have caught the ire of a social justice warrior, these days we're mostly looking at data exfiltration as the primary goal. That being said, there are a lot of ongoing feuds and active social justice warriors.
In short: the more moving parts you have, the less time you have between major malfunctions.
(Manufacturing and hardware world understands it well, which is a big part in why they like integrating things so much.)
So e.g. over in the backend-land where I live, it used to be that I had to occasionally update the compiler or one of the few third-party dependencies that I used. Today, I have many more libraries (to the point there's something to update for security reasons roughly once a month, on average), and on top of that, I have CI/CD introducing its own mess, Conan updates which occasionally get messed up, or make some existing recipes incompatible, CMake updates which are done unexpectedly and break stuff, now also Docker is adding more of its own problems, etc. So I have to deal with some kind of tooling breakage every other week now.
And always, always, when I think it's all finally OK and I can get on with my actual job, some forgotten or hidden component craps itself out of the blue. Like that time our git precommit hooks broke for me, because someone changed them in a way that doesn't work with my setup. And then me wasting a day on trying to fix it, eventually giving up and degrading my setup to unblock myself. Or another day where, for no apparent reason, some automation that made automated commits to some git repos started losing Change-ID headers in commit messages, making Gerrit very sad, leading to several people wasting a total of several person-days trying to fix it. Etc.
There's always something breaking, the frequency of such breakages seems to be increasing, and it's a major source of frustration for me in this job. Which is why I too am often thinking back to "good old days", and am increasingly in favor of keeping the amount of dependencies - both libraries and tooling - to a minimum.
I can say with great certainty: Almost no one rebuilds their damn container anywhere near as often as the the gray-beard in the basement updates the Debian packages on the the server that runs the container.
I had the debate with a client, we did monthly security update, unless something horrible happened. The client was rather upset that we didn't patch more frequently, like weekly or daily. My argumentation is that it doesn't really matter if the Linux kernel or bash is patched, when the only thing running is a container with a beta version of Tomcat that hasn't had security updates applied in three years.
Even worse are the people who just pull things from Docker Hub, with no plan as to how and when they'll pull newer versions. But fine, let's just keep running KeyCloak from 2017, and that old Postgresql image which the developer never configured to do backups, I'm sure it's fine.
Most probably the gray-beard simply enabled "unattended-upgrades".
You can do something similar with a container (track the security fixes for the packages used, force rebuild and deploy when needed), but it is a bit of work and I don't know of any ready solution.
I just did a "npx create-react-app text" and I got
"74 vulnerabilities (69 moderate, 5 high)"
40 major security flaws in 10 years sounds almost like a bargain.
Out of curiosity, why is it discouraged?
I’m not really in the React world, but have been fixing up a Create React App for a client recently, so learned a bit more about it.