Hacker News Reader

Comment by snorkel | Hacker News Reader

I've had to suffer doing web dev work in a group in which sys admins didn't allow the devs to login on production servers. This policy was for the sake of "stability" and "uptime" when in fact the sys admins lack of understanding about the code and the stack they would in fact break shit all of the time. Least productive place I've ever worked for. Laughably the biz managers kept asking why we couldn't we be more "agile" and deploy new features everyday like Facebook, why can't we do that? It wasn't even worth answering.

View on HN

If your admins don't understand the code and crash things - that's an organizational problem that can be corrected.

Giving developers admin access isn't the answer.

I've seen both sides - where admins didn't undersatnd the environment well enough or the application in it and broke it, and where developers didn't understand the system well enough and pushed out code releases that broke things. Neither situation should exist if things are set up correctly.

Things should be automated and documented. You need systems knowledge as part of your development process.... it's not enough just to throw a PHP guy at something..... everything needs to be architected in a way everyone understands.

Several very important standards require that developers do not have access to production (PCI comes to mind if I'm not mistaken).

You think PCI law can tell the difference between a developer and a sysadmin?

Unless you're a really small org, devs and sysadmins have different roles and need to have distinct privileges to prevent them from stepping on each other. The bigger you get, the more important this is.

You have a few things from an internal control POV that you can do to help prevent "bad" things from happening. Examples include: privilege separation, change control, dev/test/prod environments, scripted deployments, automation, etc.

When devs bitch about not having the privileges they need, the biggest issue is usually communication. The SAs work for a different Director, and there is an adversarial relationship. Fix that, and most of your problems go away.

Fun fact: Amazon and Microsoft have done "devops" since before the word was invented. Google and Microsoft do not. As usual, the correct answer is "don't hire idiots" and "do have clear policies, tools, and audits", not "don't allow person X to touch machine Y"

Devs and admins have different mindsets and attitudes, and that's a good thing.

Devs are more aggressive and forward-looking. Admins are more conservative and status-quo-oriented. You don't want folks with an aggressive mindset to admin your stuff. You don't want folks who worship the status quo to do development.

why would you need to have access to prod ? Your development/staging environment (not local box) should be same as prod in most aspects except capacity so your push to prod is 2 step process developer -> staging -> prod

debugging is one (perhaps the only/primary) need. if you have stuff only breaking on production systems, it may be because of a number of factors that aren't easily replicatable in a testing environment. Sometimes the only feasible way (assuming you want fast answers because of live production downtime) is to get on to production servers and look at things there. In most cases read-only access should be fine.

Ideally, this is true.

But, I've worked in places that had bad resource planning, and didn't allocate the same kinds of resources in staging. When we had performance problems, there was no place else to go.

It can be hard to convince people to take their expensive hardware and IT time to set that production system up, and now, "duplicate it so we can debug". Eventually, I think you just give up and go somewhere else.

You can't reproduce a dev/staging environment with the same traffic pattern, user behavior, and request load as to a production environment. Google bots won't crawl your dev/staging environment and you won't know its impact until you see it. DDOS attack won't happen in your dev/staging environment. Click fault won't happen on dev/staging.

Developers probably shouldn't have root privilege on production but they need to have read access at the very least. They also should have write and deployment privilege on a few production machines for testing and experiment. Whatever mess they made there should be counted under acceptable loss.

No disrespect, but after a statement like this, "...deployment privilege on a few production machines for testing and experiment.", I wouldn't let you near my servers. That's not what a production server is for.

Reply on news.ycombinator.com