Hacker News Reader

If you're writing Ansible playbooks and keeping a deep understanding of the system in your head, just call yourself "devops" or "SRE" and roll with it....

More to the point, anyone who can write good, robust Ansible playbooks and who can actually deeply understand the system to the point of being able to debug a misbehaving system and reproduce problems effectively is necessarily a competent, qualified developer. You might not enjoy writing large new software systems (I don't) but you certainly could do it, you have all the skills and then some. If you're a qualified operator who is also a qualified developer, "devops" is probably a more accurate descriptor than "sysadmin" (even though, yes, in theory "devops" is not a job title but an organizational practice).

> It seems to be a favorite trope to paint sysadmins as a dying breed of monkeys who are only capable of keeping pets and doing manual tasks when they get around to lifting their knuckles off the floor. Who do you think is writing the playbooks for ansible? How do you think you can even write those things without first having a deep understanding of how the system works and having a set of procedures around them? Who do you think runs the systems your containers sit on top of?

I totally see where you're coming from, but I suspect we've both been doing sysadmin for a really long time. You might remember in say 90s to mid-2000s

- 'sysadmin'. Subscribes to 'login;', is a member of SAGE. Views his job as automating his job until there is nothing left. Probably used perl and shell, then Python and Ruby.

- 'sysadmin'. Person who sets up the server. Knows the basics of Windows NT or 2K. Doesn't really like to program. Automation is a nice to have, they don't script because they're 'not a developer'.

I choose to view 'devops' as a way of the former group from separating itself from the latter. The people who acted like 'DevOps' was a new thing probably just knew too much of the latter groups and didn't know enough Perlmongers and alt.sysadmin.recovery types.

I see that but I also see the opposite. Sys Admins blaming monkey devs for things like requiring all ports open or rdp access with root.

There's so much negative blame torwards the workers for mgmt decisions and the resulting shit.

Ex: we have one dev environment with 250 bare metal machines that each dev can ssh onto any of them. When we transitioned from ClearCase to git git was only installed on 10 of them.

So now sys Admins blame the devs for not requesting them all, devs blame the sys Admins for not knowing it when the reality is the mgmt decision to not prioritize dev prod parity is the problem.

There is definitely Sysadmins blaming devs, but not because they're monkeys.

The attitude that Devs have about Sysadmins comes from arrogance. Devs are arrogant that Sysadmin work is monkey work, and "they could easily do it if they had to because it's not nearly as complicated as writing code". It's a complete lack of respect for Sysadmins.

Sysadmins blame Devs because as soon as Devs are put in the position of doing ops work, they completely and utterly fail, but instead of changing their attitude about Ops work (i.e. that it's actually a real and separate skillset than dev work), they maintain their arrogance that the work is dumb and "I could do it if I really had to".

Both issues are caused by arrogance of the Devs... the way Devs see Ops work, and the way Devs interact with Ops people which is then reflected back at them by Ops.

And in your example, Ops has it right. If Devs cannot do a basic thing like provide specs on the environment/tools they need to do their own work (what, you really don't know that 'git' is a tool you use, even though you type the word 'git' 20x a day?), how is Ops supposed to do anything about that?

I wonder how much of a problem this is. Anecdotally:

I've spent my entire career - about 5 years beyond university - working in a cloud-only environment. My dev machine is an ec2 instance, whose environment my company provisions appropriately, and we have tooling for managing the application environment end to end where I rarely have to worry about linux primitives.

I can put together some pretty impressive things on AWS, reasonably comfortably, but I know that if I ever leave for a company that still runs their infrastructure classically, I'll be at a disadvantage. I've even stumbled around one-off linux environments for personal projects, most of the concepts are still foreign to me.

Funny, I come from the opposite direction. I've been using Linux¹ and FreeBSD since the early 1990s, and was using UNIX since the mid 80s (SunOS, AIX, ...)

I find it really easy to deploy things in the cloud. I especially find it easy to fix cloud deployments when they don't work, compared to my cloud only colleagues.

If we're not careful, I think we might start to lose some basic OS level skills.

There are stories, perhaps apocryphal, about certain tribes of South American natives who find axe heads discarded by another tribe and see it as the work of the gods.

They don't know how to make their own axe heads, and see them as mystical objects.

It's not anyones fault for not being born early enough to have ridden Linux from its infancy, but I would encourage people to learn the basics of the OS.

Putting together cloud offerings, that's only product knowledge.

1: https://en.wikipedia.org/wiki/Yggdrasil_Linux/GNU/X

We used to have four groups voting against management's worst ideas. We got rid of the DBAs, we got rid of QA. Now we're getting rid of Ops. And while that means that we have more power, it means we have a lot more responsibility, because it's just us against ignorance and bad management now.

And we all sit in a big room where we don't get time to consider our replies before giving them, which means if we make a rhetorical error it doesn't matter how valid our concerns are.

I really don't want to run my own company, but it's starting to feel more like I have to if I want anything like sanity.

Developers did not took your job, it was the machines...

Kubernetes strength is its declarative nature. I.e. the actual "tasks" (the imperative part), moved into the controller code and is being automated away.

As more and more operators/CRD (i.e. automatic sysadmins) get written, more jobs will disappear.

For example, you can imagine a database operator that just by stating your schema (as an CRD), will create the tables for you, the indices, do automatic backup/restore, do automatic migrations, create the monitors and the alerts. I.e. will completely replace the DBA.

> I.e. will completely replace the DBA.

It replaces the DBA for routine maintenance. It does not replace the DBA when you need an expert to help you plan your database architecture and figure out what went wrong in production.

It might also replace, what you call an expert.

As compute get cheaper, the design goal of those new systems is to replace human time with computation time. This is the key idea behind automatic machine learning, for example.

Image a system that needs to map logical schema or architecture into the best implementation. The system can create 1000's of combinations, and test them against your data/queries. And come up with the best design.

There already is a database operator that can do everything you have listed and a lot more ( I helped write it ), but it doesn't remove the need for DBA's. It just changes what they do on a day to day basis. I think this is what OP was saying. Just because you have an operator that can do most of what a traditional DBA does doesn't mean you can replace all the world's DBA's. Someone still needs to know _why_ a specific query managed to do a multiplicative join and lock all your tables for hours, even if the operator knows how to flag that query and reject it.

Sure, but in a day of a dba, or even over a month, how much does he deals with understanding deadlocks?

BTW, the operator that I saw are not there yet. What I want to see is a CRD for a set of input schemas, and an output schema, and let the operator create the most efficient query.

I suspect that, if an organization, especially one with its own hardware, tries to move to Kubernetes in a cargo cultish way and without understanding the nature of the undertaking, this could easily lead to disaster and people getting fired.

That's another way Kubernetes can take away jobs :).

I'm an old school sysadmin that recently has been making a buck doing "devops consultancy". In reality, my job is fixing stuff that the developers "automated" (because I don't really think an automation that is not performing any work is an automation at all). I've seen some scenarios that are hard to describe outside of my sysadmin expertise with something other than silly.

Not so recently I hear a lot of some Jessie guy, brilliant, who developed and amazing product and was a very good engineer. His team all made clear that this guy was so efficient that he was in charge of a lot of stuff; sadly he didn't documented a vital step of the development pipeline that was causing all the builds to fail and he left the company years ago. His team, that was at least capable enough to keep the ship afloat for more than two years without Jessie who was the driving force and main engineer, combed the documentation and code but couldn't find the problem, they appear to be pretty good coders and the codebase was clean enough that I do not question their capacity as developers, but the problem was too deep, mainly because as they told me, this Jessie guy was pretty great.

Well, actually the problem is that Debian Jessie reached EOL some time ago, and this guys recently they migrated the build server, losing all the image cache. They were unfamiliar enough with errors on the package manager and with the shadow of this Jessie guy that even if they found something about the backports url, they didn't thought it was related to their issue. Just changing the repo url was enough to keep them working again so I could take a look at the rest of the stuff.

That's just the silliest one I can think about, but just one of many. I'm certainly not over impressed with developers doing "devops" by themselves.

> will completely replace the DBA.

When you have paying customers or external SLAs, you need 2 departments:

1) DBA

2) Eng. Security

If you want to avoid hiring those people, there are things you can do:

1) design your app around a K-V store, like DynamoDB. Reporting (ie. aggregation) will be the tough part.

2) Have a sr. engineer review security weekly and file jiras that are followed up. Don't agree to any IT compliance standards without full-time security engineers.

To directly answer the parent poster, what he's talking about are the basic mechanical things that a DBA does, which is a small portion of a DBA's responsibilities.

For example, just deciding what to monitor and how to address the alerts on large tables alone justifies having a DBA. Coinbase and Gitlab have both had extended outages recently because of inadequate DBA resources.

When I talk to programmers, I start with, "Our masters have 400 days uptime. How does your new system maintain that level of availability?" Typically that's the end of the conversation.

Source: I am an experienced DBA.

> Who do you think is writing the playbooks for ansible? How do you think you can even write those things without first having a deep understanding of how the system works

The sys-admins don’t even know what ansible is. The devs are writing the playbooks and trying to convince the sys-admins that terraform and packer/docker are great tools while the sys-admins are still remoting into windows servers and clicking though graphical installer dialogues to configure the oracle servers “just so”.

As for the “deep understanding” that was replaced with ready built AMI’s and packer/docker configs ages ago. You don’t need a local expert when you have the end result readily downloadable from a central trusted source. (And embarrassingly the available images are often better than what the local sys-admin thinks is top of the line).

And at the end of the day, while it’s extremely cool to see someone utilize 20 years of experience to squeeze out an extra 15% of performance from your setup... well an extra server which adds +98% performance costs like 10$ a month, and the expert of 20 years is just not cost effective.

There’s a culture clash, and it’s not because expertise isn’t valued. It’s just that a lot of old sys-admins have spent their entire carriers optimizing against targets that are no longer relevant, and teams can get by perfectly well without them.

Truth is that you honestly don’t need a sys-admin or DBA on modern team. The ‘kids’ do just fine without them, and just spin up the systems on AWS/Azure/GKS with a blatant disregard for the fact that they “do not yet have the needed seniority to carry out such important tasks”. And if the horrible complexity of it all becomes to Much, they spin up 3-4 variants of the infrastructure and use the one that works best in performance and reliability testing, without ever needing to seek the alter of the almighty sys-admin with gifts of gold and silver in order to bless their endeavor and be allowed the extra couple of servers.

We don’t have coffee machine admins because coffee machines are trivial in the modern software company (even though they do of-cause need regular maintainer and cleaning and are extremely important). The same is true of sys-admins, they are no longer needed because the system procurement and provisioning are trivial (even if they also need regular maintenance and cleaning and are of-cause extremely important).

Your understanding on what sys-admins are/do is very limited/off-base. I'm a sys-admin for engineering department of a public university. We use Puppet and Ansible to manage a diverse array of workstations, compute servers, and storage servers all hosted on-premises. These machines exist to serve undergrad student lab needs, research compute clusters, and public and internal infrastructure.

There is a world outside of your bubble where servers exist to run a product.

I find it so interesting to see the sort of bubbles that people live in and believe in.

There is a big % of the sp500 companies with huge infrastructure inhouse, legacy system and sysadmins generating most of the money in the world and yet you think that kids that can script against a cloud API have them in their pockets.

I really suggest looking outside the bubble.

The “kids” are typically devops engineers with 5-7 years of experience. They get called kids by sys-admins because they prefer postgre to oracle, and Linux to windows. Because they don’t have the same 30 years of experience doing roughly the same thing in the same org.

Of cause there are systems-admins who adopt and are now working half of a devops workflow, but they seem to end up either becoming devops engineers or finding a new job where they are back to being Kings of a closed domain that developers only interact with though emails and audiences.