Hacker News Reader

Comment by lasermike026 | Hacker News Reader

Sys Admin/Engineer here. We had it right 15 years ago. You couldn't become an admin unless you started as a programmer. You needed a deep understanding of your OS and good knowledge of C. At that time the devs ("programmers") were less knowledgeable than sys admins. You had to have to skills to analysis, debug and fix their code. When sys admins stopped coding the result was weaker admins. Sys admins ought to code within their domain, engineer, and maintain their systems for performance and high availability. It is long over due that sys admins return to their roots and get ahead of the pack.

View on HN

I'm part of the recent sysadmin generation that had no programming skills and grew up doing what I consider more "application administration." Lots of clicking Next with no real understanding of what to do if that Next button quits working. Stack tracing? Procmon? Miserable understanding of DNS and networking, what's a /24? These were things senior admins knew and used once the tickets went up the chain because customers were screaming.

I was always (well, not always, once I'd proven myself) a senior admin/engineer, now I focus almost purely on AWS automation, and it bums me at out how many admins are completely happy with just being Next button admins who are absolutely AFRAID of touching a bash or zsh shell, or even a Cisco router without the UI.

I suppose it shouldn't bother me, it's done great for my career.

Are you talking about sysadmins on Windows? I can't imagine a Linux sysadmin being afraid of bash, zsh or a ios shell. My impression is that knowledge of some combination of bash, perl and python scripting is a basic job requirement.

I've worked for several MSPs and yeah, I mostly mean Windows. But we worked on just about everything. Especially Cisco/Dell CLI equipment (routers, switches, etc).

the root of the problem (the idea that sysadmins can't program) might come from the fact that every company whether furniture/clothing store or any other sizable non-tech business needs sysadmins. But only if technology is your core business you need good developers. This probably leads to generalization of sysadmins not knowing their bits/bytes. Sure guys working for ISP's and in datacenters are a different breed but they still get thrown into the same category because of the job-title.

I think the problem is, some companies are expecting a sysadmin to be the "IT guy". The one who is only expected to call contractors to fix that recurring network problem, and install enough Windows boxes for the new hires.

https://www.linkedin.com/pulse/devops-bullshit-joachim-bauer...

If a task involves "clicking next", then that task should not require any specialized knowledge to do. I totally agree that all developers should be willing do write code to automate "DevOps" tasks whenever they see fit. I know a lot of programmers who do this. These programmers always have a bunch of scripts sitting in their ~/bin, and you hardly ever see them clicking. I am a "programmer", but I love "DevOps" stuff, and mostly recently I wrote a large automation framework that got CI team interested. Tons of energy and time is wasted everyday where teams don't understand (and don't want to understand) each other's code.

> If a task involves "clicking next", then that task should not require any specialized knowledge to do

I would disagree with this point. If you're installing anything on a production box you really should know what that installer is going to do to your environment. Have you walked through the MS SQL Server installer wizard? You absolutely do need specialised knowledge if you're running that on a client's server.

SQL Server is probably one of the easiest "big" software systems available in the world. Without having much specialized knowledge you can setup a powerful HA cluster. It's mostly clicking next and following some basic instructions. Out of the box, it'll perform pretty well (just remember to click the "Optimize for Ad Hoc Workloads" checkbox in the properties page).

SQL Server requires vastly less "specialized knowledge" than, say, Postgres. I would be surprised to find that most SQL Server installs come with someone with really in-depth knowledge.

The same applies to a lot of other things MS ships, like setting up AD. Sure, knowing what you're doing helps, but MS sorta built an empire on making a lot of that stuff pretty trivial to do. For another example, look at setting up IPSec. A cute little UI you click on, versus well, the process on Linux.

Sorry no, it's an easy clicky pointy path to hell when you discover one month later no-one understood what the file path options meant (and there can be many when installing Standard+) and then the system volume ran out of space.

Or what accounts to run SQL's various services under and wonder why stuff doesn't work or their SQL box gets hijacked ("oh I'll just run this under the SYSTEM account").

The install wizard for SQL Server 2008R2 Standard Edition has around 20 pages (counting tabs on a page) of user selectable options and settings. You certainly do need to know what you're doing and understand the implications of each every one of those settings.

Sure if you're just doing a quicky install of the Dev Edition on your workstation you can afford to be a bit lax and click on through, but on a mission critical production box you need to RTFM or let a SQL admin who knows what they're doing perform the install.

I've been called in too many times to unf*ck a SQL server because a customer has declined one hour of installation support because their devs thought they knew better.... "oh yeah it's just clicky pointy install, why do I need help with that". Trust me GUI installers do not equate to "easy install" if you have no clue what the installer is asking you.

I'm a SQL (MS and MySQL) admin, amongst many other roles, for a webhoster with 19 years MS SQL Server experience (started on SQL 6.5 in '97), been there seen it all.

I mean, if you consider knowing that databases require disk space to be "specialized knowledge" then, I mean, I guess so. Otherwise the installer is rather straightforward in telling you what the paths are for. And creating a new DB also shows you which files will be used. There may be 20 pages, but most of that is ... pretty straightforward, easy stuff.

I've run SQL Server in life-critical production systems (911 call routing) and found it to be very straightforward. Just make sure the alarms are turned on and enabled for replication.

I've run it in HA clusters, both with a shared DAS and with the new shared-nothing system. A walk in the park. The only difficult thing I've personally run into with SQL Server was figuring out the Broker Service stuff and dealing with replication outside of a domain. And that was probably mostly networking configuration issues - the config wizards are pretty amazing.

I'm not implying that there is no specialized knowledge, and many SQL Server installs are probably suboptimal. But it's far from necessary to know much to go very far with SQL Server.

Compare to Postgres, where doing anything correctly requires mucking around with the config file, and, last time I used it, logging into the shell with a special user. And HA, well, good luck.

Sounds like you both came to the conclusion that to do things it's best if you pay attention and care.

I may be the only one around though that doesn't find config files to be 'mucking' or a form of mysticism that all should kneel before the user who can wade through them. That also, is not 'specialized knowledge'.

I managed to install Postgres from source, and have a standard setup. It wasn't so bad. Run the usual configure make, make install scripts, and go through a tutorial page for setting up the initial user. Ok, I have a fair bit of Linux experience but I am not a sysadmin.

Where do you even get a job being a Click Admin? I mean, I could use a hobby that pays about now...

> Stack tracing? Procmon? Miserable understanding of DNS and networking, what's a /24?

I'm primarily a developer, but I'd love to understand networking/low-level OS concepts like these in greater detail. Every time I start with a tutorial, I just get overwhelmed with how much I don't know — is there a book you'd recommend for a developer getting into devops (advanced networking, low-level OS architecture, configuring a machine from scratch, etc.)?

For networking, I think this is a great place to start: http://www.amazon.com/TCP-Illustrated-Protocols-Addison-Wesl...

For OS, I would recommend looking at Bach (on the design of UNIX) and Tannenbaum's books.

Thanks for the recommendations! Got 4 of them for less than $40 :)

The problem is that such a sysadmin is expensive, and what companies call "sysadmin" today is an almost completely different job.

Microsoft revolutionized personal computers because they made things easy and cheap. Their stuff was meant to be used by the average joe, not a greybeard who'd survived the Unix Wars.

At the same time, IT is always seen as a "cost center" to be reduced and outsourced. Unix? Too expensive, replace them with Windows, and if there's a problem, re-image the system.

The sysadmins you describe still exist, but they're called SREs by Google or who knows what elsewhere. Today's "sysadmin" is an outsourced phonecenter guy in Bangalore.

I still believe, a good or the best sysadmins are the ones who can code. A sysadmin should know the system inside out, including the software running over it.

As a sysadmin for the last 15 years, I'm fine with this, as long as you're going to pay me as a senior software engineer (if I have to know the app inside and out) who is also going to be on call constantly and doing infrastructure all day long.

If you as a business or employer want to continue to squeeze additional roles and knowledge into people, be prepared to pay for it.

Of course you should get what you are worth and the position needs.

My experience has been people rarely want to pay appropriately when they're demanding you do several roles.

Why would you want to get paid less? In my experience developers in Indy get about 80k. SysAdmins for IS and SQLServer get about 120k. They also don't have to work that long since the servers themselves are fairly robust.

> SysAdmins for IIS and SQLServer get about 120k. [My note: I presume you do mean IIS]

Presuming this is USD, but even in the UK as GBP - "In my dreams" - and I'm a developer as well as a sysadmin for Windows (IIS/SQL) and Linux (Apache/MySQL), 25 years man and boy but have always kept on top of my game :)

Outside of London you'd be lucky to hit 45K GBP (~70K US$) in a permy role with that skillset, perhaps 60-65K GBP if contracting.

The UK pays quite a bit less than the US. $120k /year for a good sys admin would be a deal in the states.

A deal in San Francisco or NY perhaps, but not most of the country.

Midwest pays about 80 to 95K for a decent sysadmin (Unix/Linux w/ 10+ years of experience), only the specialized shops (trading firms, any place with a lot of money flowing through them) will go 120K or more.

If you know of any place north of Chicago that pays 120K or more for a 20-year Unix/Linux engineer with C systems level programming talent, and won't burn you out in 2 years like the trading firms downtown, I'd be interested.

That was how I started a lot longer that 15 years ago - a common task would be oh we have just brought this expensive piece of kit (200% of my yearly salary) its its in that room hook it write any drivers then go and see Bob to work out how to digitize droplets in 3D

When I worked for a large Telco one of the reasons they employed me was to have an in house sysad for our Primes - we had the largest cluster in the UK running MR back in the early 80's

I rember one of my IBM mates at the time being horrified that we allowed to write all our own JCL

I've been at this for over 20 years and have zero programming experience. So, your blanket statement isn't true - most of the best admins I've worked with have had no programming experience but deep systems understanding.

I doubt you can fix today's devs' code, unless it's a very very junior dev. It's like saying a doctor should be able to fix a chemist's work.

Hah, I wish that were the case. Over the last decade, popular programming languages are just catching up to what existed in the 70s.

I don't know if it was intentional, but this comes off as an incredibly arrogant comment.

Agreed And my Dr is fixing my biochemistry you should see the amount of pills I take a day to get it back into balance.

Some do. They're called MD/PhDs. While they're somewhat rare, they do indeed exist.

Reply on news.ycombinator.com