karmarepellent

This is why I've become a fan of StrictYAML [0]. Of course it is not supported by many projects, but at least you are given the option to dispense with all the unnecessary features and their associated pitfalls in the context of your own projects.

Most notably it only offers three base types (scalar string, array, object) and moves the work of parsing values to stronger types (such as int8 or boolean) to your codebase where you tend to wrap values parsed from YAML into other types anyway.

Less surprises and headaches, but very niche, unfortunately.

[0] https://hitchdev.com/strictyaml/

The sarcasm is duly noted. But I simply answered the question. I don't have any strong opinion regarding passkeys.

A service that lets you sign up by uploading a SSH public key could just as well let you upload multiple public keys in your profile to be able to connect from other devices.

This is incorrect. SSH certificates work just like x509 certificates in that regard. Also, with PubkeyAuthentication, there exist all kinds of ways to collect host keys before connecting to them for the first time and thus avoiding the trust-on-first-use problem. Especially in private networks where you control all the nodes.

> Signing in is cryptographically signing a commitment to the current ephemeral tunnel.

I can see how SSH could be used for authentication on the web. And I have no doubt that it would be sound out-of-the-box. But I am not sure what you mean by your last sentence. Do you mean that authentication targets are gated and only reachable by establishing a tunnel via some kind of forwarding?

Aside from the wonderful possibilities that are offered by using port forwarding of some kind, you could also simply use OpenSSH's ForceCommand to let users authenticate via SSH and then return a short-lived token that can then be used to log into an application (or even a SSO service).

I guess no one uses SSH for authentication in this way because it is non-standard and kind of shuts out non-technical people.

I'm curious to know if people see this as a viable alternative to a PXE installation, especially when it comes to the deployment of large-ish (possibly air-gapped) clusters.

Actually I think the trick is to change ones own perspective on these things. Regardless of how many redundancies and how many 9's of availability your system theoretically achieves, there is always stuff that can go wrong for a variety of reasons. If things go wrong, I am faster at fixing a not-so-complex system than the more complex system that should, in theory, be more robust.

Also I have yet to experience that an outage of any kind had any negative consequences for me personally. As long as you stand by the decisions you made in the past and show a path forward, people (even the higher-ups) are going to respect that.

Anticipating every possible issue that might or might not occur during the lifetime of an application just leads to over-engineering.

I think rationalizing it a little bit may also help with the paranoia.

Its a matter of evaluating what kind of infrastructure your application needs to run on. There are certainly mission critical systems where even a sliver of downtime causes real damage, like lost revenue. If you come to the conclusion that this application and everything it involves better run on k8s for availability reasons, you should probably focus on that and code your application in a k8s-friendly manner.

But there are tons of applications that run on over-engineered cloud environments that may or may not involve k8s and probably cost more to operate than they must. I use some tools every day where a daily 15 min downtime would not affect my or my work in the slightest. I am not saying this would be desirable per se. Its just that a lot of people (myself included) are happy to spend an hour of their work day talking to colleagues and drinking coffee, but a 15 min downtime of some tool is seen as an absolute catastrophe.

I think it depends on the definition of "bricking the cluster". When you start to upgrade your control plane, your control plane pods restart one after one, and not only those on the specific control plane node. So at this point your control plane might not respond anymore if you happen to run into a bug or some other issue. You might call it "bricking the cluster", since it is not possible to interact with the control plane for some time. Personally I would not call it "bricked", since your production workloads on worker nodes continue to function.

Edit: And even when you "brick" it and cannot roll back, there is still a way to bring your control plane back by using an etcd backup, right?

Do you mind sharing what these operations were? I can think of a few things that may very well brick your control plane. But at the very least existing workloads continue to function in this case as far as I know. Same with e.g. misconfigured network policies. Those might cause downtimes, but at least you can roll them back easily. This was some time ago though. There may be more footguns now. Curious to know how you bricked your cluster, if you don't mind.

I agree that k8s offers many features that most users probably don't need and may not even know of. I found that I liked k8s best when we used only a few, stable features (only daemonsets and deployments for workloads, no statefulsets) and simple helm charts. Although we could have probably ditched helm altogether.

In theory: absolutely. This is just anecdata and you are welcome to challenge me on it, but I have never had a problem upgrading Kubernetes itself. As long as you trail one version behind the latest to ensure critical bugs are fixed before you risk to run into them yourself, I think you are good.

Edit: To expand on it a little bit. I think there is always a real, theoretical risk that must be taken into account when you design your infrastructure. But when experience tells you that accounting for this potential risk may not be worth it in practice, you might get away with discarding it and keeping your infra lean. (Yes, I am starting to sweat just writing this).

Agreed. The best thing we did back when we ran k8s clusters, was moving a few stateful services to dedicated VMs and keep the clusters for stateless services (the bulk) only. Running k8s for stateless services was an absolute bliss.

At that time stateful services were somewhat harder to operate on k8s because statefulness (and all that it encapsulates) was kinda full of bugs. That may certainly have changed over the last few years. Maybe we just did it wrong. In any case if you focused on the core parts of k8s that were mature back then, k8s was (and is) a good thing.

I think the value proposition holds when you are just getting started with your company and you happen to employ people that know their way around the hyperscaler cloud ecosystems.

But I agree that moving your own infra or outsourcing operations when you have managed to do it on your own for a while is most likely misguided. Speaking from experience it introduces costs that cannot possibly be calculdated before the fact and thus always end up more complicated and costlier than the suits imagined.

In the past, when similar decicions were made, I always thought to myself: You could have just hired one more person bringing their own, fresh perspective on what we are doing in order to improve our ops game.

We ran only two (very small) clusters for some time in the past and even then it introduced some unnecessary overhead on the ops side and some headaches on the dev side. Maybe they were just growing pains, but if I have to run Kubernetes again I will definitely opt for a single large cluster.

After all Kubernetes provides all the primitives you need to enforce separation. You wouldn't create separate VMWare production and test clusters either unless you have a good reason.

I have use cases for both approaches (letting a reverse proxy handle TLS, letting the application listen on an external socket and handling TLS in the application).

I find is is easier to configure an application with a reverse proxy in front when different paths require e.g. different cache-control response headers. At the end of the day I do not want to replicate all the logic that nginx (and others) already provide when it integrates well with the application at the back.

Other commenters suggest that both ways (with or without additional reverse proxy) add "tons of complexity". I don't see why. Using a reverse proxy is what we have done for a while now. Installation and configuration (with a reasonable amount of hardening) is not complex and there exist a lot of resources to make it easier. And leaving the reverse proxy out and handling TLS in the application itself should not be "complex" either. Just parse a certificate and private key and supply them to whatever web framework you happen to use.

Makes me wonder what part of the pattern you think is "too clever"? I think it is fairly easy to reason about when the lock is restricted to the encompassing block and automatically dropped when you leave the block.

I agree. Sometimes the best you can do to cope with the sprawling ecosystems around programming languages and having to deal with convoluted codebases and inefficient programs, is sit down at home and building something simple (if you can spare the time and energy). That is what I do every now and then and it is incredibly satisfying to be able to explore how therapeutic programming can be when you are not boxed in.

I think this is not how companies work though, because they do not operate based on your views alone. And getting all people to agree on some of the topics you mentioned (e.g. importing dependencies vs. rolling your own implementations) is an incredibly complex task.

I can confirm that upkeep of Gitlab is rather easy and does not take much time. Just the occasional update every few weeks. For its size it is also quite robust and the integrations it offers work well.

The only thing I can really complain about are the constant UI changes that do not actually improve anything. I fell they are just changes for change's sake to keep some frontend devs busy.

Not sure why JS is the problem though. You can use JS in small doses to drastically improve the user experience on websites. The fact that heavy frameworks are sometimes used in contexts where they are overkill is not strictly JS' fault. By that logic you would also need to go down the rabbit hole of what compilers produce efficient programs on the server side and ban everything else.

I think the tools we have nowadays are perfectly fine. It's a matter of how they are used. And I am pretty sure efficiency is not what companies think of when they launch a product.

I have no data to back it up, but this is why a lot of people living in the Alps supposedly still drive cars with a manual gearbox, although generally sales of new, automatic cars are now ahead of manual ones, even in Germany.

It makes me question what we are doing too. I'm using HTMX extensively at work, but I never use it to only update local state. A few lines of Javscript on the client will do that.

However I think it's a powerful solution to updating the DOM when state changes on the backend and a roundtrip is required anyway. For that matter the examples on the official project site are better to understand proper use cases for HTMX [0].

Personally I use Active Search and Edit/Delete Row a lot. While Active Search does not change state in the backend, I found it's still a valid use case for HTMX when searching large data sets, since the backend is (in my case) just way faster to search through a large data set.

It all comes down to your ability as a developer to find the proper solution to the problem at hand. A counter does not need HTMX.

[0] https://htmx.org/examples/

I have never seen a Debian system break due to auto-updates when they only subscribe to the common main, contrib and even non-free Debian repositories. They are incredibly stable and robust.

However one might want to use apt preferences/version pinning if they also use external non-debian repositories to e.g. install a newer version of PHP. Since some repositories keep different major versions in the same repository, an auto-update might otherwise install a version that causes an application to break.

That being said, most systems can probably get away with auto-updates and a little apt preference configuration and call it a day.

Unfortunately in air-gapped systems you cannot simply pay pennies for a managed k8s platform. In these cases you have to bootstrap and manage k8s on your own in your data centers. While I do not think bootstrapping and managing a cluster is difficult at all (especially if you only handle stateless workloads) it may still not fit or integrate well with a companies overall management infrastructure.

While I am a happy cloud infrastructure user in private, I have to go through some extra hoops to deploy applications at work, regardless of if k8s is used or not.

To be fair in other comments OP made an effort not to get involved into those endless Kubernetes vs VM discussions. However either side eventually posts a snarky comment and there goes.

I think everyone just has to acknowledge that there are use cases for both. Also Kubernetes and "classic" configuration management via Ansible (or others) are orthogonal to each other. So these discussions are somewhat misguided in the first place.

For example: you might want to deploy a VM or auto-install and configure a physical machine with custom tooling and something like Ansible or Puppet and _then_ configure said machine as a Kubernetes node that handles the actual workloads. In other cases some Dev might want to install and run an application without the k8s layer using Nginx as a webserver. In this case, too, Puppet/Ansible might or might not be involved in configure the application but only handle the "OS layer" if there is such a thing. And in yet other cases you get away with a simple cloud-init script that makes your machine a k8s node and leave out other configuration management tools altogether.

Guess what: All of this is fine. Evaluate solutions based on what you need, not what other people working in giant corporations urge you to use. And then go and build it, ideally having fun doing it.

Representing either tool as a one-size-fits-all is misguiding at best and seems to be overly simplistic to the complex problem of deploying your applications.

Thank you for your thorough explanation. Interesting to see that you basically use your monitoring system as a scheduler to run Puppet and it sounds beneficial to closely integrate it with your monitoring to have it all in one place.

At my place of work we went the "traditional" way of running Puppet locally. It has been our experience that Puppet failures due to user misconfiguration or some such do not require our immediate attention (e.g. after hours), so we just check Puppetboard a few times per day to identify failing nodes.

Another reason why we use Puppetboard to monitor Puppet nodes is that every alert that our Icinga monitoring system produces is automatically interpreted as an incident which needs immediate attention. We are currently in the process of changing that so we are able to process non-critical alerts in a saner way.

Anyway, interesting to see how a fellow Puppet user manages their setup. Keep it up!

Now I am curious: how do you detect despair regarding cloud tech in job descriptions?

> while monitoring configuration compliance with a custom Naemon plugin.

While I absolutely agree with you and your approach, would you mind elaborating what kind of configuration compliance you are referring to in this statement? I suppose you do not mean any kind of configuration that your Puppet code produces as that configuration is "monitored", or rather managed, by Puppet.

Thorough auditing can still happen on the target host. If every single one of your hosts is properly configured to produce audit logs, maybe you can get away without auditing or even session recording on the SSH bastion host.

Yes, that's a fair point, it's part of their job to spend time in calls.

Ironically this gives me even less confidence that I would be interested in what they offer. Maybe the mere fact that they conveyed the offer and spent an hour in a phone call is all they wanted actually?

Maybe I'm just being overly pessimistic on the matter though.

Well there is marketing and then there is marketing. Given the fact most of us would be out of a job without advertising of some kind, I cannot condemn it the way you do.

However I despise deceitful marketing as well. I am also not a big fan of the way my company markets itself. It is neither deceitful nor misleading, but just ... irrelevant to people working in the space and thus hard to align with.

But I also know companies (that we have worked with) who spend their marketing budget on hosting small-ish conferences and choose to post content from their (technical) blog in their LinkedIn. For me personally, that way of marketing themselves just inspires more confidence than ... overconfident salespeople.