Many highly competent individuals like to understand the fundamentals and first principles of how systems work and how different problems can be solved, and thus find work that provides no transferable knowledge/experience to be less appealing.
Of course, there is a balance here. Sometimes frustrating tasks must be done. In that case, ideally the end goal is sufficiently motivating.
Right, but to those people, whether you use Docker or Kubernetes or custom scripting or a unicorn whose horn magically creates new instances when you need them is mostly just an implementation detail. It's no more interesting than exactly which compiler or DB you use; these are just tools, means to an end. If the differences are significant for your use case, sure, you evaluate and choose accordingly, but then you use the tools to get on with the job. The deep, interesting stuff is almost always in what you can build once you're into that territory.
However, I do think it's likely there are some fundamental principles at play that make tool 1 better than tool 2 in specific contexts, and I think it is valuable and interesting to explore that.
Perhaps we are getting into the scientist vs. engineer mindset. I would guess, though, that the best engineers undoubtedly have some scientific curiousity helping them excel in their craft.
One extreme is that I see companies hold on to toxic employees because they're brilliant, even if they are bad for the company long term.
While a very large company can get away with building a custom in-house ecosystem that takes months for a new employee to come up to speed on, most companies find it more palatable to use industry standard products/systems that a new employee is already familiar with and can come up to speed on quickly.
Of course, but who would you rather hire, someone with solid fundamentals who can get up to speed quickly with anything, or someone who job-hops to use the latest buzzword library/framework/language that may have a useful lifetime of just a few years or even less? Given an either/or decision with other things being equal, I would almost always prefer to hire the former, and then use whatever tools worked best for the job regardless of whether they were built in-house or imported. A job-hopping resume full of trendy buzzwords is a big red flag for me.
While a very large company can get away with building a custom in-house ecosystem that takes months for a new employee to come up to speed on
This argument gets made all the time in discussions about frameworks, but I just don't see it. In my entire programming career, I have yet to encounter this hypothetical nightmare architecture that takes months before you can become productive and has no useful documentation or examples even though the entire application is built around it. (YMMV, obviously.)
Being able to get up to speed quickly on a new framework or library is an essential skill for a working professional developer. Whether that is the latest shiny Web World buzzword or something internal in your new employer's environment makes little difference, as long as the internal design is also sensible and reasonably documented -- and if it's not, you have much bigger problems than who you're thinking about hiring next.