I get the feeling you think I am okay with this bug. I am not. I am not okay with any system getting hosed, but I am very not okay with production servers being destroyed.
Your work flow is like a good set of armor. You have different stages where things will fail -- and they will fail. The goal of your armor is to prevent failure on the most important thing -- and that is your production servers. The thing that brings in money, customers, users, whatever, the reason you are here.
So yes, if I had to chose between a developers workstation getting destroyed or a production server I would 10 times out of 10 pick the developers server.
> It's all very well saying "know your trade", but the reality is that most organisations aren't running state-of-the-art orchestration tools. Heck, not so many years ago, many of these modern tools didn't even exist yet, and they've had plenty of problems of their own that make keeping up with the bleeding edge dangerous in itself.
I am not suggesting any such thing. Nobody needs state-of-the-art orchestration tools. If you ever happen to bump into any of my other post you will see I argue against most things like kuberneties. The problem at hand is a very well known problem, and the solutions for preventing production server failures -- or at least minimizing them have been around for at least as long, if not longer than the web its self. Maybe part of the problem is we have wrapped our self in these tools to make things seam easy, and have lost basic system administration skills - because the things you are describing would make it out like I am asking you to be Elon Musk and land rockets on floating barges. I am not. I am asking for simple and free tools to be used to automate the building of artifacts that can then be deployed to simple vms or servers and verified to not cause adverse effects. Then for the same artifacts to be deployed to your production servers. All of the tools needed to do this are free. All of these notions are things that should have been taught in school, or on the job training. This is apparently not the case, this is why post like mine are being made to point out how it should be done, so maybe somebody reading this will learn something new.
> Hopefully even in the early stages you would still have some sort of staging set up, and I think you misread my comment there; I was in no way advocating not having staging servers. I was only observing that even if you take out staging catastrophically rather than production, it can still be a pain to set everything back up, just less of a pain than losing production while you're doing it.
Please my comments about armor above. Its okay and will happen something will fail. The goal is to make sure it is not your production server.
> Right, but how many organisations have 100 production servers? If you've reached that scale, you're already probably in some sort of 1% group, and obviously you might have far more resources available to deploy management infrastructure around those servers.
You introduced the 100 servers number, and that is why I used it. If you have a 100 severs than your first argument of not having "tools" even though If find it faulty in its own right is blasted away by anybody with any real number of servers.
> Right, but how many organisations have 100 production servers? If you've reached that scale, you're already probably in some sort of 1% group, and obviously you might have far more resources available to deploy management infrastructure around those servers.
100 servers is really not that many. But I feel this highlights my point even more. If you are a small organization and have a few servers that means each server represents a larger % of the workload and business. This intern really means you can't afford to not have good practices in place to avoid downtime on servers represent a much larger % of the work load should one go down.
> That philosophy might be something you can afford once you're no longer operating in small/early mode, if you get that far. But while you're still worrying about say getting from MVP to ramen profitability in your startup, everything is too costly, and you never have the luxury of doing the ideal thing everywhere right now. Hoping for basic staging isn't out of the question. Hoping for a full-time ops person to deploy the best-in-class orchestration tools that came out last week because you can't trust running apt to install security updates on your production Debian servers without destroying them is probably beyond your wildest dreams.
You can't not afford to do these things. So you get MVP and your service crashes and now you are a big zero because you lost all your initial clients. Please don't gamble with both investors money and developers you hire to work for you. Writing software is not pulling a lever on a slot machine. It takes real skill and attention to detail to pull off. No point in putting on your best tuxedo top only to enter the ballroom without pants on. You will look good from the car, but be the laughing stock of the event.
> Many, many organisations will not have the luxury of doing everything ideally, because they lack the time, people, budget, knowledge or omnipotence to do it all at once,
These are not luxuries. They are must. If you can't do these things then you don't have a product, budget, or the people that are suitable for the job at hand. You must build your foundation on rock, and if you can't afford that rock then you are not ready to start building anything other than a hobby.