How ancient this all seems . . . until you realize people are doing the exact same thing by shipping "golden" VM/container images around instead of doing proper package and configuration management.
(Ugh. I'm actually pretty insulted by the incompleteness of the tools they write in an attempt to replace what they think we do. I was going through some puppet add-on to manage your ssh configuration the other day... it had a setting for "allowrootlogin" - but only allowed "yes" or "no" - no "without-password" - which is the only acceptable setting if you must allow root login at all, which you probably shouldn't.)
That's the thing... the uncool stuff that isn't making money goes away. Nobody cares. But the uncool stuff that makes money? Someone needs to actually understand it, long after the developers have moved on to the new technology of the week.
Really, the division of labor makes some sense. 90% of everything is crap nobody wants, that can safely be left to die of neglect. So when building a new application, it doesn't really make sense to engineer it to be scalable from the ground up; you're probably going to throw it out before you have to worry about scaling. And maintainability even more so. The way most companies run, five years from now, that app will be huge (and they can throw huge resources at maintaining it) or it will be dead, and the fact that it's difficult to upgrade libraries in it with known security holes won't matter. So let your developers run wild and use whatever unmaintainable crap they feel like using, why not? If it's successful, you can pay someone to deal with the problems later; if it's not, it won't matter, and you will have more developer time to spend on a project that might actually succeed.
But say you've got a hit; something that people care about, something that pays the bills. At that point? You hire people like me to prop it up. If you are smart, you hire another set of developers to re-write the system in a way that scales... but meanwhile, you need this giant pile of shitty php to keep serving the ads that pay your salary, so you hire sysadmins.
In the aughts, this was all about managing shit php code and corrupt mysql tables. I'm guessing five years from now, at the companies that manage to survive the coming carnage, people like me will be slogging our way through ancient and impossibly tangled VM images, trying to figure out all the weird versions of this and that and the other that the developer installed, and how we can set up something compatible that isn't full of known security holes.
Agree. It's important to iterate quickly and get the intent right and not sweat the small stuff, including whatever notion people have of what's "scalable" which usually is misguided if not a fantasy. Scaling is "a good problem to have" after all; it's only necessary if people are actually using your stuff in the large.
Having said that, migrating off of a terrible language is nearly impossible and always painful. Pick stuff with a good runtime -- Java, Go, Haskell, Erlang, C#, whatever -- and you have a chance of scaling if you ever need it.
It still seems wrong to me, though; it seems like if you've got several orders of magnitude more resources to build version 2 than you had to build version 1, you could re-do it from the ground up in something... better.
Call me a simpleton but I am happy with better package management and config systems.
Containers add even faster startup time and the option of isolation so you can have multiple containers which think they're writing to the same location but actually aren't. That's not terribly important in a one-app-per-system model but it's awesome for testing or running multiple instances of an insufficiently-configurable app on the same system.
Fundamentally (in terms of process, not technical implementation) that's all containers really are.
Run this, with this config file, and everything is pre-packaged for you in a bundle you can ship to your prod server if you like (or not, run it locally, whatever).
It's quite nice. I shared the skepticism ("so.. it's just VM's then? So what?") but having switched to a dev toolchain using it - it's actually a LOT simpler than developing locally normally.
Since you can pretty much use any config management system you like with containers, they also play nice with that too.
There is one other ancillary benefit in that they enforce a little bit tighter process isolation than you might get if you just installed a bunch of stuff on a given server. So that lets you do things like overcommit available resources (put more on the server than it can run at one time, but stochastically, it can probably handle) and it helps ensure you have "livestock" and not "pets" (servers configured automatically, repeatably, and with no concern with their demise rather than servers you need to baby).
They're pretty much just a (mostly) handy abstraction for a bunch of things that you probably want to be doing but is exceptionally frustrating to set up piece by piece.