They can't be doing one-off undocumented config, package, and network/firewall changes which make it impossible to setup another server reliably. At $company I moved us to Terraform+Packer (to get them used to immutable deploys, but still just an EC2 instance) then Pulumi+Docker+Fargate so we could fix our deployment velocity. The CTO was constantly afraid everything would break; mostly cause it actually would break all the time. Now basically anyone can deploy even if they're not a SysAdmin.
That's not to say you can't automate a Pet Server, but it's a lot more likely for someone to "just once" make some changes and now you don't trust your automation. In our case we had SaltStack and we were blocked by the CTO from running it unless it was off-hours/weekend.
I chose to work with the CTO/Team to figure out a solution everyone could live with. I even chose a more annoying solution (Packer) initially just to make sure people felt comfortable and avoid changing things anymore than I had to.
It was so trivial to terminate and restart dozens of servers at any given time since unless there was a mistake in the cloud-init, we could bootstrap our entire infrastructure from scratch within an hour.
It was amazing, never had to deal with something missing on a server or a config being wrong in a special case. Dozens of hosts just purring along with 0 downtime since the moment anything became unhealthy, hosts would start auto-booting and terminate the old instance.
Got any tips in this regard?
Ansible, Puppet, Python, Terraform, Openstack Pulumi springs to my mind. Still need to cover a PXE server. (Maybe with Ansible)
I'm completely missing. I have searched arpund and I have some solutions, but back in my head, some people have something else.
The part before Ansible or puppet to kick in.
When I wanted to set up NixOS, I wanted to use it to manage a bunch of NixOS virtual machines (so, nixops), and then run a couple of services. Here are a few points from my anecdotal experience:
1. Learn an entirely new language that has absolutely horrendous documentation. It used to be (I've been told it got better but I abandoned Nix before this) extremely difficult to debug as well, since oftentimes you would get an error in some library file when the error was obviously in your own file.
2. Try packaging your own stuff. Again, barely any documentation. One time, I wrote a simple Rails application in a day. It took me more than three days to figure out how to deploy it, and that involved just figuring it out. Rails adds a PID file to the directory with your Ruby code in it, but that directory in a Nix derivation (fancy [which makes it harder] name for something like package from my understanding) is immutable. Good luck packaging complicated projects that someone else hasn't packaged yet.
3. Nixops was so incredibly outdated that it caused my server to say it was using an "insecure" library from Python 2.7 and my auto updates started failing. On paper, nixops looked nice (manage NixOS virtual machines/VPSes with Nix). But it broke my updates. I eventually wrote my own replacement for Nixops, which worked, but I still don't trust the rest of NixOS to not break. The tool might have some merit on a different Linux distribution, though. (I'd also note that I believe that nixops fixed their insecurity bug, but there were a myriad of other issues I don't remember that I had with nixops)
4. Setting up pre-packaged services sounds easy. They were, somewhat. But permission errors were pervasive. SystemD tempfiles were finnicky at best and gave path traversal errors that were extremely hard to debug.
Eventually, I got a new server, installed Arch on it, and use docker-compose for everything. It takes me maybe 20 minutes to set up a new service on a good day, instead of four hours of Googling for some obscure error from NixOS. While not perfectly "reproducible," neither was NixOS, because I could simply not trust a virtual machine on nixops to boot successfully—there would always be more required even if it worked once.
I might be open to NixOS on a desktop, though.
I run NixOS, but I recognize that user experiences might be very different depending on the packages you use. NixPkgs is huge, enormous. Some stuff is really well maintained, whereas other stuff isn't.
Furthermore, declarative systems make a difficulty tradeoff. Lots of stuff is much easier, but problems are sometimes harder to debug as there's an extra abstraction layer.
IMHO, these days the best distributions out there are either very simple imperative ones with binaries (Arch, Alpine, etc.) or Nix-like (NixOS and GuixSD). Getting experience in those two extremes is really valuable.
2) Debian
Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also.
My hunch is that the 5000 onprem pet servers are not going away any day soon, because a massive amount of it is legacy systems that take a long time to migrate to cloud, if ever. Also the work stress is just ridiculous. So much stuff to do, even with automation. Only reason I still do this is that I like the "old school" tech stack vs. cloud IaaS/PaaS alternatives.
I am not so sure... I am a well seasoned sysadmin, been doing server, network, architecture. I consider myself a solid linux/network expert and have managed datacenters. When I look for a new/more exciting job, or for a pay raise, all I see are "cloud, AWS, devops". I never see "old school" sysadmin jobs e.g. as you say, we have a room full of linux boxes and we manage them with ansible/scripts/etc, but we design and maintain them ourselves, come join our team".
Because the old school systems just chug alongg with a minimum of administration, while those newfangled cloud thingamajigs need tons of adminning...?
I mean, that's one interpretation of your (admittedly, anecdotal -- but that's what I see too. Lots of anecdotes sums up to:) data. A valid one, AFAICS.
You don't need pet servers. Puppet or Ansible make your baremetal cattle.
This is cattle:
* PXE boot server(s)
* Image contains masterless puppet bootstrap.
* Server(s) asks git - "give me the bootstrap for my mac address"
* Server(s) gets a list of classes to apply.
* Server(s) applies classes.
Done.
I'm looking for a solution to make unattend installation possible.
Edit: Can use Ansible to make a PXE server. (Egg or chicken thing)