The danger of having system programmers around | Hacker News Reader

The danger of having system programmers around

utcc.utoronto.ca

60 pointspreek28 comments

This, needless to say, is what those people who talk about the virtues of "diversity" are trying to say. And the virtues of a diverse team of specialists is, in turn, the theme of most caper films (e.g.: Sneakers).

Especially in emergencies, it can be really great to have three or four people attack a problem from different angles. What you need is someone reading the docs while the system programmer is running strace() and the tuning guru is reallocating processes and memory and the customer liaison is on the phone trying to figure out if they really need to be running this particular enormous task on every page load.

The alternative, if you're by yourself, is to try to multiplex the roles, but it can be extremely difficult to pull yourself out of systems engineering mode while you are in there.

This reminds me of what my electrical engineering friend once did when his washing mashine stopped working. He got out his multimeter, opened up the back panel, and finally traced everything to an open circuit: the lid was open.

Ah, yes. I spent five or ten minutes this year examining a washing machine problem my wife reported. The problem: her husband had turned off the hot water feed while he tightened the connection, then had failed to run it back on. I believe she forgave him his negligence ...

This is a general programming issue, not a system programming issue. Not so long ago at work, I had to implement a regex search over a Lucene document store. Now, the thing is that we used the latest version (at the time) of Lucene, 3.0. All the examples I found online and most of the documentation referred to version 2. Naturally they didn't mentioned they were using version 2, I just had to scratch my head a few times while I tried to realize why the regex API depicted there is not only different, but not even merely existing. Turns out one of the design goals of Lucene 3 was to separate stuff from the core Lucene to external libraries. So, after finally searching through their SVN I found the source files for the regex libraries. I ran into some problems with the way Lucene parses regex queries and relates it to terms and again, as in the article, after hours and hours of searching through documentation I resorted to just reading through Lucene's code, which indeed turned to be fruitful for less the time I invested in finding and reading documentation...

I really was not sure what to learn from that episode. The logical conclusion seems to be to try to read your way through the target source code, if you have it, before investing the time to find and skim through documentation. On the other hand, it feels like a really bad lesson. Time will tell if this moral holds up.

I've had this go both ways before, too. I think the point to take away from this sort of experience is that the best you can do is apply your best judgment and re-evaluate as you go along. Often opaque documentation makes sense after you gain some context by reading the source; it works the other way, too. With experience, you gain a sense for what path to go down; sometimes that (subconscious) heuristic is totally wrong, which is why it's important to consciously re-evaluate on top of that. The current approach isn't making you any practical headway, even though you're learning a lot about the system in question? Take a step back and try looking at the bigger picture; what are you really missing? If you can, sleep on it for a bit.

Most of all, if you're getting it right more than 50% of the time (and I'm finding it hard to imagine a way you'd do worse than that) there really is no point beating yourself up about it.

My friend and colleague lost his job over this type of behavior... He'd simply go of the rails for days trying to figure out a problem. When he got his next job, I advised him to always think about how the company makes money from his labor. Sometimes it makes sense to "go deep", but often it's cheaper to let it go. If you're dead set on figuring it out, do it on your own time, not the company's.

The real trick is finding a company and industry where it always makes sense to go deep.

I don't know. I think the key is just knowing how to prioritize. Sometimes it makes sense to go deep, especially if the problem is causing a lot of expense. Other times it's important to work on other problems.

Right now we're refactoring our codebase from one language/framework to another. Identifying which parts are still maintained and need the refactor vs. which parts would be a waste of time because a developer hasn't touched it in years is important.

This is especially important if you're one of few/sole engineers on a project.

"that's what she said."

Despite the "incomplete and frequently wrong" nature of documentation, I still think a careful reading of that documentation is probably the best place to start.

If it doesn't hold the answer, fine, we can look elsewhere. But it does hold the answer often enough that reading that funny manual first all the time will save time over the course of a career.

>>I still think a careful reading of that documentation is probably the best place to start.

I used to work like that in the mid-1990s.

When I had a problem, first I thought a bit, then I looked at the documentation -- and as a last option I searched (at the time, mostly Usenet).

A year later I'd become realistic (or lazy), I searched before reading the documentation.

And after another year, I searched before even thinking about the problem... :-)

First, there is no such thing as a "systems programmer". This is a useless, career-limiting, HR-invented categorization. When I made the mistake of signing on with a few recruiters on a job hunt a while back, they looked my recent job experience and labeled me a "systems programmer". This was differentiated from things like web design and GUI. It is wrong wrong wrong. We are software developers. It should not matter what layer of architecture we are coding in. Every layer has so-called "business rules"; everything has MVC. Label yourself, and you limit your thinking and your evolution.

This article does have a valuable lesson, in spite of the miscoloring. What we see in this article is an example of overthinking the problem. With experience, your debugging/problem-solving skills will get better, and this kind of mistake will not be often repeated. It took me many years to get out of the habit of speculating and to get closer to actually debugging. I had learned that it was faster to look at what was really happening than to guess at it. It helps even more to come to a detailed understanding of your system, but if you do not have time for that... Never start at the bottom first: start with the symptom.

Are you sure about this? I tend to think of a Systems Programmer as one who is more skilled in a particular area of the stack (closer to the OS). For example I wouldn't generally hire a self described "Web Programmer" to work on the guts of VMWare if I worked there. Sure, a good programmer can pick up on that stuff but a good programmer can also pick up basket weaving, it doesn't mean basket weaving is a useless career limiting hr invented categorization.

I wouldn't generally hire a self described "Web Programmer" to work on the guts of VMWare if I worked there.

When people describe themselves as something, it is to indicate what they are interested in doing. Hiring a self-titled "Web Programmer" to do VMWare work is probably not a good fit, not interesting for them. But it is not unreasonable to find that they are capable of doing the work if they wanted to. We need to be careful about categorizations.

I see your point about basket weaving, but when I went to school, I did not see a class for it in my CS curriculum, and I also did not see "systems programmer 101" or "web development 101". These are applications. In CS, we learn how to navigate in all these fields, whatever their rules... that is, out in the field, we can learn enough of basket weaving to create a software model for it. The point is that we can solve the problem in software.

Really, it is because of this nonsense that I gave up on recruiters entirely and learned to go directly to the technical hiring managers. The good ones do not use requirements checklists with bullet points like "knows C++", "has programmed in linux", and "knows basket weaving".

Edit addition: This thinking would be career-limiting on me because I have adopted a mindset that boxes me in, focusing in the wrong things.

At the University of Maryland College Park, Systems Programming is, or was, course CMSC 415. Maybe you should have gone to a better school or, more likely, have paid attention when you were there.

I have never heard of University of Maryland College Park.

I had a course in OSes. And a course in databases. And a course in graphics. And a course on compiler design. So on and so forth. All part of the curriculum. They taught us to be computer scientists, not systems programmers or web engineers or whatever fancy title is nice. I should add that the university I attended is one of the top in the US for CS.

Also, in interviewing candidates, I did see that there are several schools that do teach about the toys rather than about CS. I generally ignore that and look for good developer qualities instead.

Your second sentence is pretty personally insulting, beyond the call of simply disagreeing with me. Please consider an edit?

And who are you to patronize other people because of their university choices? The university I went to didn't have a systems programming specific course either. So what's the deal here? The only good university is the University of Maryland College Park?

He made a good point, people often dismiss candidates because they see some buzzword they don't like. A good programmer is a good programmer regardless of what he calls him/herself. It's more important to assess a programmers competence, desires, and fit within a company than discriminate on what "type" of programming you think he should be doing.

I'm not really sure what point you are trying to make. If someone self-titles with "Web Programmer", and by your own admission that denotes an interest level than that seems like its limiting their career in a way they desire. The OP is refers to himself as a Systems Programmer which seems to fit just fine into your definition of self-titles.

This sounds much more like you're projecting frustration with recruiters rather than making a useful point on categorization.

For brevity, the main takeaways from my posts here are these:

(1) Avoid overanalyzing when debugging. This is true for all programmers.

(2) Divorce identity from skills to avoid personal limitation. Say "I can develop web pages", not "I am a web page developer".

(3) Side point: I am not convinced specialization in CS is helpful, and in my experience, it proves to be a hindrance to integration.

Sorry for the confusion.

For point 3, I think it's a benefit. I sell myself as a particular kind of programming specifically because I do not want to work on lame projects that don't interest me at all. I think it's perfectly valid to self-identify as a "systems programmer" or "web programmer" because, unless you're under certain situations, you probably don't want any ol' programming job. These days I even specify what languages I want to work in because I'm so nitpicky.

For sure there is. I'd call the group of engineers "systems" hacker/engineer/programmer if they are working on work on OS, database, networking, and distributed infrastructure kernels.

These type of programming requires a distinct skill set and usually more experiences than for example web frontend development (PHP, JSP, etc).

Anecdotally, I can recall exactly how to manoeuvre the TSS register in real mode to successfully boot a VMM guest but I couldn't for the life of me recall how to make a speckled background in CSS.

Asymptotically there's no such thing as a systems programmer but that's not really how the real world works.

I think we are well aware of the arbitrary definition, and it misses the point. A software developer hacker/engineer/programmer solves problems, using the software as a tool to that end. There are surely several domains in which they may work, but the real skills -- the ability to put together code, debug, etc and not the memorization of the business rules of each domain -- are what makes the programmer.

I can understand if a hiring manager wants somebody who has some experience in some OS, in using Dreamweaver, in knowing the Java language, or in some other platform. These are just toys, though. I agree that that experience is a bonus, but the real core skillset has nothing to do with any particular toy. The hiring manager will easily miss good developers with such a narrow-minded focus.

And certainly, strong experience in any of the items of this list should not suggest that the developer cannot also do GUI programming, web programming, JavaScript, or any of the like. These are simply environments. Everyone has to learn the local company's codebase or some business's specific rules.

I have never seen an educational institution use these terms. Nobody gets a serious degree in "systems programming" or "web frontend development". If they do, avoid that institution!

What you want in a developer is the ability to learn, the ability to think, and the ability to communicate on things that make some hardware operate. But call them a "systems programmer", and we do a grave disservice to our industry. Distinct skill set? No way. Don't let the pencil-pushers carve us into tidy little imaginary fields. They do that because they are trying to measure the work. As someone who has experience in all the levels from our computer's digital circuits up to databases and sideways to some pretty graphics and sound, I can assure you that these definitions are ridiculous. I provide maintainable solutions.

I am not a systems programmer. I am not a web developer. I am not a game designer. I am a software developer: I can do all these things and more -- and specifically, I personally have, so I know the differences.

I do not hire people because of the toys they have used; I hire them because they are capable of learning and extending mine.

The other critical factor is that "digging straight in" is _fun_, at least for people with this mindset.

Reading documentation, on the other hand...

unfortunately it's not always easy to recognize when a programmer is doing the right thing, because then things are running smoothly. it can seem like the programmer isn't special because hey, it's easy to read the manual, etc, whereas going deep to figure something, whoa, good thing we had that guy on the team! same thing with simple and elegant versus complicated designs.

Funny, my experience with systems programmers has been that the real dangerous part is the unmaintainable and unportable code they write.

Wait, I left out the punchline: for projects that don't involve systems programming

You think I'm trolling or trying to be funny, but it's actually a real trend I've noticed.

I've seen some advice for chemists formulated like:

"With a few weeks of hard work, creativity and dedication in the lab you can often save hours in the library."

Reply on news.ycombinator.com