There aren't enough humans for cloud-native infra
siliconangle.com
siliconangle.com
1. Point out how the world is changing so fast now, and that computers are super extra hard (databases are 500 components now, that sounds pretty hard)
2. Mention how you can’t just get competent employees and trust them to build anything!
3. Don’t worry, big consulting/solutions company (Cisco, IBM, Oracle, etc) has a solution to your deepest fears! Just contract out inferior products and consulting services at 10x the price of doing it yourself.
Pretty sure this is just a marketing piece, but regardless, the industry needs to stop thinking of software development and operations as inherently separate. Developers should operate the software they write.
Have you seen a webapp go off the rails? There's millions of weird fail-conditions. Can they learn what to monitor, at what frequency?
Security... 'Principle of least privilege' is at odds with "make it work". I've seen devs assume the database server has commodity connection to the internet. They never asked if it did or didn't. And I've fought with them on the dev AD stack with an attempt in keeping the GPOs in line with production. But it's just 'too hard'.
And I've seen devs go 'just spin up some more machines'. But unlike yesteryear where we provisioned hardware for cost for X years, aws/azure/gce "allows" devs to have a direct unlimited budget tied to the company. We sysads learned responsibilities, and how to budget, and how to requisition. Someone pressing a button on a dashboard, or writing a script doesn't. You'd be surprised just how angsty the C levels are with this cloud computing and developers with near-free reign.
Yes. That is part of the design of any system; observability is in everyone's mind when they are writing software. Software that fails mysteriously means less weekends and less time for drinking beer. Software that tells you exactly what it's doing means when something goes wrong, you fix it, and it never happens again. Or, it's so transparent you figure out the bug before you even check it in.
> Security... 'Principle of least privilege' is at odds with "make it work".
There is certainly value in having strong security principles, both in terms of policies and infrastructure. Infrastructure is key here; an ACL-checking proxy in front of all services, mTLS out of the box, the ability to write, deploy, and manage small services that are easy to audit... all help make security the rule instead of the exception. This is what all those "service meshes" and "policy frameworks" and "orchestration frameworks" aim to provide.
> I've seen devs assume the database server has commodity connection to the internet. They never asked if it did or didn't.
That's why the dev team should be operating that service.
> And I've seen devs go 'just spin up some more machines'. [...] You'd be surprised just how angsty the C levels are with this cloud computing and developers with near-free reign.
In my experience this has always been a utilization problem. One shared cluster for everyone is too complicated / expensive / dangerous / conflicts with empire-building, so every team has a bunch of t3.4xlarge instances that average 0% CPU load just because. Meanwhile, companies that care about their computers have seemingly-complicated orchestration frameworks so that they can extract value from every CPU core.
The good news is, a lot of what used to be manual (wrangling machines, consolidating resources, mechanical aspects of security) has some serious open-source investment. People don't need to schedule programs to physica computers anymore. People don't need to write authentication and mTLS code into their application anymore. You don't need to fill out a form and create a purchase order to get a TLS certificate anymore. Times are changing.
Your experience seems to have been shaped by supporting bad developers. We aren't all like that, though, and you'll find that there is less and less room for the incompetent these days. It's true of any field; someone builds you a house and the floor isn't level. "Don't build houses anymore," you say! No! Just don't hire the guy that did a crap job. Everything gets a lot easier when you set out to do a good job.
The flip side of this is the C levels who have absolutely no experience monitoring cloud spend but have heard the marketing pitches or smelled the "best practices" in the air and are sure that moving to cloud services needs to happen yesterday whether or not demand patterns justify it.
I’m a good programmer in my domain. I have enough ego to believe that I could probably figure out how to install things and get them working... for awhile.
But I know that in the long run I would fumble it so badly. I don’t follow security bulletins, I don’t know the first thing about how to set up database backups, I don’t know how to configure a web server, and so on...
Security dependencies/updates? Those are largely transparent and a matter of noticing and updating. Operational changes like updating a systemd config or something. They affect your platform but not your application.
Underestimating the complexity, variety, and difficulty of all the tasks of delovering professional-grade software is the biggest challenge in the industry.
It's like telling a soccer player to be his own cook, trainer, and psychologist. Specialization means letting the law of comparative advantage do its thing.
It has nothing to do with "what I'm able to do" and more about being optimally efficient.
It is hard. But a soccer player should know how to make pasta, cook meat so that he doesn't get sick, even if he isn't a gourmet chef. He should be able to do physical activity without harming himself, etc.
We should all be generalists to some degree. Where on that spectrum of specialization we should be satisfied is a valid debate.
I also don't disagree that there are costs/inefficiencies generated by being more generalists vs more specialized. But it also depends on the size of the organization you can rely on. If I work at a startup, I may need to be more of a generalist. If I work at FAANG, I can specialize because I can depend more on the specializations of others.
When you notice a problem, setup monitoring to collect metrics about the problem and alert on it before it becomes a downtime. Have ops write down their experience with that problem into a playbook, then have dev figure out which parts of the playbook can be automated, or if the system design can be iterated to remove the problem class altogether.
If you're a dev and you don't listen to ops, you're just as doomed as an ops guy who doesn't read the dev's manuals.
This was how things used to be in most places in the 80's and 90's - it was a disaster. Development and Production Support have conflicting goals, developers want to fix the bug/add the enhancement and release, production support wants to keep things going and have no downtime. Production releases and gate keepers became the solution, to mix them up again is bad.
Maybe if there was an infinite pool of developers that are skilled in all the applications that are being run and all the environments they run in then this would work but even then developers aren't good at keeping things the same, it's not their nature imho.
Edit: Not only that - the temptation to just pop in to production and fix the latest bug is too high, its only a little fix - what could go wrong :-) - if the developers have the keys to production then this happens. Gate keepers are good and make you test your code before its ever released.
This stuff is table stakes for modern applications.
Who still manually patches applications in production? Who deploys artifacts that didn’t come from a build/test pipeline? Who still SSH’s into a server and hacks a config file instead of applying the configuration change with a traceable commit / PR process?
Sysadmins who want to return the the glory days of the 90s, regaling us with tales about how everything we’re doing has been done before, and missing what’s different about it this time, and why won’t we just build .deb packages and hand it to them to run on a big baremetal server they will administer for us, that’s who.
I know, since my first 5 years of my career I was doing just that, working with crusty BOFH greybeards, before I went over to the dev side.
Ok, boomer!
Am I reading this right? Are they really saying that the main issue is hiring DevOps/SRE engineers at scale is hard - but that hiring and keeping an additional team of AI engineers who understand cloud infra is somehow more achievable for the average company?
I just don't see that playing out well for places where cloud and/or AI technical expertise is not a core competency. There are managed cloud providers who specialize in these areas. If your problem is literally that you can't scale DevOps, it's at least worth pricing out those services before committing to hiring an even more specialized role for infra purposes.
That being said I think the major issue isnt devops hiring. Its having poor SRE practices, no breathing space for engineers, and scaling for the sake of it, most of the time.
LOL, what? Maybe for the cloud provider... but what’s presented to the end-customer is most certainly not 500 components.
* a dedicated key vault instead of installing certs directly on the DB server * separate IAM / SSO * separate monitoring * separate logging and audit * separate vnet, subnet tooling * separate orchestration * separate object store / data lake * separate queuing * separate replication / backup * separate tools for streaming data, batch processing, job running
The only real additional vectors for a "Modern DBA" are CAP theorem, elastic scale, cloud security, multi cloud, and the sheer belwidering variety of platforms you may be required to support.