Ask HN: Is this true? | Hacker News Reader

Ask HN: Is this true?

26 pointsglen42 comments

We were in the process of hiring a CTO. One of our programmers suggested that server guys are not normally good coders; and programmers are not normally good server guys. He suggested that, in general, the fields have bifurcated and it is uncommon to find people who excel in both domains. Would you say this is true? mostly true? or BS?

View on HN

Likely this sounds petty and glib, but I have yet to meet a programmer who had the kind of administrative sense that makes a 'good server guy.' The politics surrounding change management and policy enforcement require technology administrators to have at least one foot firmly planted in reality. This doesn't keep server people from being proficient coders, but it suggests there's an upper limit. And, from personal experience, since I switched from full time admin to (mostly) full time programming, most days I couldn't tell you what day it is. It really feels like a personality shift due to less demand for administrivia and more demand for creativity. Four years ago, I knew the location of nearly every object in our home, all the bills were paid on time, all of my socks were matched up and we had a meal plan. Suffice to say, I've started wearing mismatched socks and the bills get paid when someone calls. Much to my delight, the world has not come to an end. Maybe I'll live longer, too.

I spent the first part of my career as a "server guy" (along with networks). I ran tech at a popular ISP in Chicago called EnterAct.

I spent the next part of my career as a developer, first at a security company, then at a streaming multicast company I founded, then at a network managment company.

I spent the next part of my career as an almost full-time security researcher for hire, with a pronounced focus on high-end enterprise technology (SAN, WAN compression, replication, database, storage).

Looking at it from all three angles, I'd say: there's definitely something to this. Being a programmer gave me insight into how systems worked. Being a security researcher gave me even more insight (I had projects that were literally months spent reverse engineering equipment firmware and building protocol test suites).

I'm confident that after all this "insight", I knew more about how, say, iSCSI worked than any SAN administrator in Chicago.

So, how would I have done as a manager of a SAN array? Terribly!

The things that actually matter in administration --- understanding the processes that need to be in place to make changes, understanding what kinds of changes will occur, understanding what's a typical kind of failure that will just take a couple hours to diagnose versus what kinds of problems mean that your vendor is putting engineers on an airplane --- those things you learn by being in the operational/engineering role.

To excel in the operational/engineering role, you need to dedicate yourself to the things that matter for that role, and not allow yourself to get distracted by shiny things. Developers and researchers: prone to distraction by shiny things!

What I don't think is true is that strong tech people are predisposed to one of these roles or the other. Could I be a strong ops/engineering person again? I think so. But I know I can't do it while I'm doing this job.

At a certain point you're talking about SANs, FibreChannel routers, HVAC systems, UPS, virtualization, etc etc, and keeping up to date on all of this knowledge is a job in itself. So a guy spending most of his time doing "big IT" sysadmin stuff might not have the time to become a really great programmer.

That being said, I know plenty sysadmins that have gone on to become great programmers. And truly great programmers tend to love computers, both software and hardware.

Larry and Sergey were apparently also server guys: http://www.flickr.com/photos/bsmif/3465464623/

Nah, Larry and Sergey were entrepreneurs, which meant they did whatever they had to to keep their idea alive. Which often included hardware design, sysadmin, algorithms, infrastructure, graphic design, accounting, investor relations and marketing. They sucked at most of it, but they were "good enough" to turn their idea into a multi-hundred-billion-$ company.

Those corkboard servers you show, BTW, were removed from service in a hurry once the fire marshal found out they were building servers out of corkboard.

Sure, but the point is that in the beginning, they were arguably great at developing software _and_ building a server farm that worked. My main point was that until you hit the "enterprise" server stuff, you have the capacity to be both a good server guy and programmer.

Above a certain point, there are so many technologies you need to know that people need to specialize.

In 99% of cases going with an ex-enterprise sysadmin for your CTO for a early-medium stage startup is a horrible idea. Sysadmin needs are significantly smaller than programming/engineering needs until you get to a really big scale. So, while either is probably OK if they are overall smart and good at tech management, my advice is go for someone with experience managing programmers vs. someone with experience managing sys admins if you have to choose. Ideally you get someone who has been CTO at a startup before and has experience bridging this particular gap.

The third aspect of this is management. Someone who has been in management for 10 years is not going to be a spectacular programmer or sysadmin because they simply wont be up to date, but they've got a solid tech background and great managerial skills. These tend to be better later hires, but could be right as well depending on your situation.

Back in those days I was running infrastructure for a failed small business leasing market place startup called elease. Our cage was right behind one of Google's. I think it was Globalcenter's Sunnyvale datacenter. Google's cage (the one with the warping mother boards) would exhaust this wall of heat into our cold aisle, which I felt attributed to the 15% failure rate we were getting on our VAlinux servers. VALinux .. Those were the days.

Mostly true. Tautologically because programmers specialize in programming and server guys specialize in running servers. Different problem domains with different ways of thinking, tools, problems. And we are talking about real programmers and real sysadmins and not some young hobbyists, I guess.

I think this is a pretty good characterization of my friend -- he's a great coder, but I think he will always be a better sysadmin. Not because of some innate sysadmin trait, but because he likes it better. You always put more effort into what you like more.

True. It's about specialization. You could also ask if programmers make good database administrators. Highly capable/experienced programmers that develop for the Web (like many on HN) will have some level of knowledge of all facets of the network computing experience, including how routers and browsers work. However, their primary focus and highest area of knowledge should be programming, hence their title.

An analogy might be basketball. Growing up you learn to just get the ball in the hoop, but those that turn professional specialize in a certain position. Very few can play all five positions quite well (like Magic Johnson).

No offence but you why are you calling the position CTO if he is expected to code and do sysadmin work. Why not simply call him/her an engineer?

Just curios on the choice of title.

do you seriously think a non-coder can effectively choose good coders?

do you really think a non-sysadmin can effectively choose good SysAdmins?

I mean, there is enough overlap between the fields that a good sysadmin can usually tell if a programmer is completely BSing, and a good progrogrammer (well, a C systems programmer) can usually tell if a SysAdmin is completely worthless, but I wouldn't trust anyone who was 'pure management' to detect either.

I think that's BS. While I'm not a professional as either, I do both and enjoy both – I didn't have the resources to hire my own sysadmin, but now I've found that I enjoy it. I think to enjoy doing either requires many of the same qualities: problem solving, patience, innovation, curiosity and vigilance.

Then again, there are jaded "specialists" who don't dein to dabble in the "other" field, I'm sure.

Good point about them drawing on the same qualities. Makes sense.

And these qualities make sense in consulting and many other domains, too. So would you let a consultant run your servers then? (I know this is a bit over the edge, but...)

You can enjoy both, but sometimes, and mainly due to lack of extra hours on the clock. The focus needs to be maintained on often divergent set of problems.

Not all programmers are very familiar with hardware issues, or dependent on their area of programming expertise on network design.

I also found that many programmers are oriented towards creating new features, while administrators focus on running established tools at maximum efficiency.

While drawing on the same set of qualities, the interpretation of them can diverge.

Some people posses both sensibilities and the mind space to treat both disciplines well in environments and projects that scale. They are not common.

maximum efficiency, bah! a good sysadmin is focused on keeping shit from breaking. If that means putting up an active-active redundant system where you really only need the capacity of one system, so be it.

Those of us who get paged when the shit hits the fan tend to be pretty careful of breaking things; and quite often, we work on systems where relatively short downtimes can lead to losses greater than our yearly salaries.

That said, all the best SysAdmins have at least some background in C and NIX systems programming, and nearly all NIX systems programmers are also pretty good SysAdmins.

Agreed. Of course to a degree it matters what you mean by "excel".

In general, an expert program will not be a truly expert sysops because each of them is a deep field and developing truly deep expertise in either of them can be an almost all consuming task.

But, you can easily find people that are pretty good at both, and I would go so far as to say anyone who is good at one also already knows at least the basics of the other, they just might not be a true expert. Certainly anyone who is good at one has shown they have the right mindset and capabilities to become good at the other if they decided to.

I don't think it's BS if we're talking about professions rather than hobbies. Just because you do both doesn't mean that either one is a specialty.

All coders do some system administration as a matter of course, and the reverse is true as well. But to specialize in both is probably rare.

Well, I am a professional in both, and it's not at all BS. Yes, it's possible to dabble in both, but if we're talking expert levels here, then guys that are experts in both the programming and sysadmin fields are really, really rare. (I'm not an expert in either field. I'm just barely good enough to do it professionally.)

Respectfully, I think that once you start dealing with some more interesting problems, and doing so professionally, you might begin to realize just how much you don't know in one field or another.

When you don't expect much at the start, one person can do programming or sysadmin job first - if you're just hosting your own page without any special features. If you get to the situation where you have many servers to take care of (5+ let's say), then sysadmin simply becomes a full-time position that needs someone experienced with this environment. Someone who's mainly a programmer may not have enough real-world experience to deal with the potential problems, or might not know where to look for solutions. If you require quality, you'll need specialists at some stage.

Splitting the roles is also practical. At some point you may have a project deadline coming soon, when suddenly (for example) network fails - do you really want to send experienced programmers on a trip to the datacenter in that case?

Really, it's a question of how big you are and how much time does your infrastructure need...

I pretty much agree, though really, a programmer who cares enough can carry you well past 5 servers. A good SysAdmin can handle a ridiculous number of servers; I've worked places with north of 10K servers per SysAdmin. (of course, here the hardware monkey and the SysAdmin job was split; you need more hardware people than that, unless you are outsourcing your hardware to amazon or the like. )

That certainly used to be true for the most part. The lines have started to blur:

http://jedi.be/blog/2010/02/12/what-is-this-devops-thing-any...

As someone who self-identifies as both a systems guy and a programmer, I'm continuously surprised by how many sysadmins don't read any C code and by how many programmers can't effectively configure Apache, to give two examples. I really don't think I'd be as effective at my job as I am if I didn't have both backgrounds, didn't do both regularly, and haven't held positions where I'm expected to do both. Often times, it becomes institutionalized to have the developers "throw their finished products over the wall" to the systems team. I've found this doesn't result in a good environment, rarely do the programmers know about the deployment environment, and systems people don't get a chance to provide input on issues of scalability and maintenance.

One guy who was an early mentor of mine in systems work actually hates doing it, and prefers to program exclusively. I didn't know this when I was under him, I credited him with helping me get to where I am today but he'll hear none of it.

I do think it's true that is is rare to find people who do work effectively in both domains, or even like to. There is a sort of closed mindedness on each side toward the other -- come on guys, we're all engineers! I've had technical and non-technical people alike be surprised at where I can contribute (a memorable one is someone in the accounting department at an old job who was forced to admin a machine back in the day being surprised that I could code, and more recent one is someone being surprised that I was able to successfully contribute to dealing with IE layout issues -- in both cases it's been like "but you're a systems guy, you can't do X", because they've only ever known me to be in that role "officially").

True. While many programmers seem to think they can also do server stuff, as a 15 year veteran "server guy", I can say that I've rarely, if ever, seen a programmer who makes a good server guy. The things you need to focus on are just too different.

Programmers tend to focus on the very small details and seem to only look at the short term question of "if something is possible to accomplish". A server guy needs to think about the big-picture, with a focus on not only if something is possible, but the impact of a particular approach on the rest of the infrastructure and the long-term maintenance cost of any given action.

The contrast that you describe between the state of spirit of sysadmins vs. programmers seems to mean that programmers do not need to consider impact on the rest of the architecture and the long-term maintenance cost of any given code change.

The programmer you are describing are a kind of programmer who makes lousy programs. I've read enough of those programs to say: they are not pretty -- they do not work well in large scale -- and they cost a LOT.

I am developer (about 12 years of commercial experience, more than 20 years of total experience). I was tech-lead of admin team on last big project for about 3 years.

I can administrate servers (about 9 years of commercial experience, about 20 years of total experience with UNIX administration, starting from school), but I hate that. My choice is to create copy of the server, then make changes and write them down, then test them on the copy, then apply verified solution to production server (see http://vlisivka.pp.ua/en/modern_administration , sorry for my English).

I even implemented my own run-book automation tool, which reads documentation and then executes steps in automatic or semi-automatic mode (I prefer Documentative Programming style).

I often prefer to develop script or RPM package to make change to the system instead of making changes by hands.

I mostly agree with that. Probably because I consider myself a good programmer and am perfectly fine to delegate all sysadmin stuff.

I think the point is they are distinctly different skillsets. Programming is very different from setting up and maintaining systems. There's a tiny bit of crossover, but not all that much.

Sysadmins do need a basic sense for coding, however they tend use different technologies than the one people generally use to write web applications (though maybe they'll release Bash on Rails sometime soon)

I've found that the best sys-admins know enough programming to automate their jobs and debug application issues. They don't have to be the greatest programmers, but they're fantastic debuggers.

The best programmers know all the system settings, services, and behaviors that can affect their apps and thus have to know the underlying system.

I have yet to meet a truly great programmer who didn't thoroughly understand the system he was running, or a fantastic sys-admin who wasn't a pretty good programmer.

While a good server admin or a good programmer with (relevant) generalist skills and people skills can succeed as a CTO, you need a CTO that knows your market and your goals and your organization and your customers, and that can quickly determine what works and what doesn't.

While a CTO is solving today's problems with today's shiny and with shiny duct tape, you also need a CTO that can identify and can hold both your short term and your broader and longer-term organizational goals in sharp focus. You need a CTO that establishes clear goals and measurements and trends, and a CTO that can then push, pull, poke, suggest, prod, cattle-prod or ego-stroke toward those organizational goals. And you need a CTO that adapts.

If a server admin or a programmer has the requisite adaptability and people skills and technology skills and presentation skills and has an indefatigable focus on making your market, hire him. Or her. But you're not selecting for and not hiring for server admin or programming skills here.

You want somebody with the technological skills of and the focus of the Borg.

It depends on what you mean by "good" programmers or "good" server guys. Up to a certain point, yes, there are plenty of people that can do both well enough to get most jobs done.

However: unless the guy you're hiring happens to be an active developer for the operating system that you're expecting to him to admin, then there's a pretty low chance that he could be considered an expert in both fields. Each field is just too deep and broad, and requires too much time to maintain current knowledge in.

A programmer can probably figure out a particular sysadmin problem, and a good sysadmin can probably figure out a decent approach to a programming problem, but while they're figuring it out, you're paying them and potentially also paying for downtime or a slower development process.

I'm a Linux Syadmin. I build large student computer laboratories, build, configure, and maintain servers, with bits of desktop support thrown in there. I'm not a strong programmer. Sure, I've made a few iPhone applications and do a little bit of bash scripting, but I don't have the time in my job to maintain the skills that an excellent programmer would have. When I've spent a few weeks programming I get better, but when we don't have time to maintain that skill, I can lose it pretty quickly.

I've worked with some amazing python programmers and without using it day in and out there's no way I could have the in depth knowledge of all the python libraries like they do, so compared to them I'm pretty useless.

Are you hiring a CTO or a sysadmin?

Well, we ended up hiring a person already. I was just intrigued by the thought that server guys are usually not good programmers and vice versa. What do you think?

It's a matter of temperament rather than technical skill. Some people like to really get their teeth stuck into a problem and work on it for years at a time. Some people like to execute one discrete task and move onto the next. Both could be equally good at writing code and configuring operating systems. But one will be happier as a programmer and one will be happier as a sysadmin. They might even produce the same number of lines of code a year, one a single 10,000 line program and the other a hundred hundred-line scripts.

Programming is about problem solving. A good portion of system administration is also problem solving. I know several programmers who are excellent system administrators, and a couple of sysadmins who are OK programmers (but they don't really enjoy it). Personally I'm more interested in programming but I've definitely jumped into the sysadmin role on occasion to help solve problems the regular sysadmins were struggling with.

In general? No. No to tue, no to mostly true, no to BS. Programmers are not necessarily good coders, and sysadmins aren't necessarily good administrators. Instead of worry about that validity of the generalization, what do you want out of your CTO? What unfilled role in the team are you trying to fill?

I think it's worth realizing that it's tribal behavior as much as anything else. Corporate fiefdoms develop. Staff protect their turf. In my opinion many people are unaware of the likeliness that this is the result of evolutionary biology.

That said, the value of some level of specialization clear. We have to commit a lot of energy to the things we will become really good at. However that doesn't come without problems. In the scientific community there is concern that it's becoming harder to do cross-discipline research.

While we must accept specialization for practical reasons, and realize that there are also social and biological factors, there is value in encouraging cross-disciplinary experience, especially in leadership positions.

The programmers may like to critique the potential CTO as not as good a programmer as them, and the sysadmins may do the same. But I sorta think you want a CTO who can see the forest for the trees.

(Cue criticisms of dumb CTOs....Now! ;-)

I didn't read your question. Just came here to say that if you want me to read your post, you need to write something a little more descriptive in your submission title than "is this true", or "look at this", or any other such non-content.

True IMHO. Get a server guy, that's where the rubber meets the road. Ultimately your business succeeds or fails on whether your service is secure abd available.

Programmers should be required to have been "server guys" because they would have a better appreciation for building in logging, management, and other troubleshooting aids. The sysadmin doesn't have the benefit of the debugger when something goes wrong in production. The more information you make available (on demand or as a matter of course) the fast production issues will be resolved.

Reply on news.ycombinator.com