Comment by xorcist | Hacker News Reader

Comment by xorcist | Hacker News Reader

Parent Full thread

> why should a sysadmin who doesn't even know Python

Because a Python sysadmin has been through all the transitions of packaging systems, all the nasty corners of "backwards compatible" changes, and knows how underlying changes to the operating system will affect your code, what the storage behaves like under load, and why one tech is not "better" than another. If you really hired an admin (cough, "reliability engineer") for a Python codebase that doesn't know Python, well, that's a different question altogether.

> I have regression tests to catch if an upgrade breaks anyyhing

You don't know what you don't know.

When you can reason about the multiple ways the above statement can fail, congratulations! You are now a seasoned sysadmin, the scorn of junior developers who just want to get things done (who incidentally read a great blog post the other day about a new packaging system that we should immediately transition to and by the way it's all backwards compatible).

>You don't know what you don't know.

That's the point of regression tests. The sysadmin also doesn't know. Unless he's the one writing the tests (and IME he's not) or he's painstakingly regression testing everything by hand (trust me, he's not doing that either), making him a gatekeeper for all library upgrades achieves very little except adding bureaucratic friction.

Look, do you want a gatekeeper or not? For your small little web project you don't need it, and a little downtime probably isn't catastrophic. But as soon as you are under audit rules you need it, and we call this specialized role the admin. When you grow bigger this will likely branch out to a dedicated change manager, at which point you hopefully have other specialized roles for security as other things as well.

I understand this does not make sense when you are not more people than can fit around a table, but as you grow you will feel the need for more and more specialized roles to fit the changing requirements. The first specialized role is probably the sysadmin (devops, reliability engineer, whatever you call it) and he or she should preferrably be the one on the team with the most knowledge of how things work "under the hood" because that person is the one that can save you when things go haywire. Unless you trust this person to be more knowledgable than you are in those areas, as they rightfully should be, you're going to have a problem.

>Look, do you want a gatekeeper or not?

No, ideally not - that's the idea behind https://en.wikipedia.org/wiki/Continuous_delivery

Where gatekeepers are required (because regression testing is not yet fully trusted enough for continuous delivery), QA should be the gatekeeper, not sysadmins.

>For your small little web project

My comments are based upon working on projects with a turnover of > ~1-1.5 million USD / day.

>But as soon as you are under audit rules you need it, and we call this specialized role the admin. When you grow bigger this will likely branch out to a dedicated change manager

Every time I've worked with somebody whose role was "change manager" this role was introduced:

* As a response to repeated downtime in the past caused by some kind of idiocy.

* They were required to "sign off" on releases purely as an added bureaucratic step to cover some manager's ass.

* They never once prevented or caught a production issue.

* They always slowed down releases.

>The first specialized role is probably the sysadmin (devops, reliability engineer, whatever you call it) and he or she should preferrably be the one on the team with the most knowledge of how things work "under the hood" because that person is the one that can save you when things go haywire. Unless you trust this person to be more knowledgable than you are in those areas

Ironically the whole idea behind devops (which I fully agree with) is that it should not be a specialized role - developers and ops teams should be blended.

This is precisely because if the two teams are separate and one throws code over the wall to the other then things will go wrong. Then a manager will insist on a gatekeeper.

I think a lot of this debate argues for the sysadmin role being part of the dev team. The only real way to get both constraints (production stability and update to date fixes/features) is to have fast feedback between the interest holders of two sides.

In the python-specific case -- the requirements.in / .txt files for the virtualenv should be part of the software VCS, but the sysadmin should be able to edit & pin things just like the devs, so that they can bring their expertise to the container, rather than having to fight it.

---

Mind you, my opinion might not scale - I'm part of a small enough team that I'm holding both those roles, but I try make sure to spent time wearing both "hats", so that one role doesn't get more man-hours clocked.

IMO if a sysadmin wants to have visibility into the requirements.txt that's fine.

If they want to enforce a policy of pinning versions, that's very welcome (though I would do that anyway).

If they have specific, relevant comments about upgrades of specific packages - again, fine (though in practice they never do).

If they want to be a gatekeeper for changes to that file they can fuck off.

Reply on news.ycombinator.com