Not only coders are hard to recruit | Hacker News Reader

Not only coders are hard to recruit

blog.sysadminsarena.com

53 pointsalexgotoi68 comments

The premise that technical people with in-demand skills are hard to recruit has very little to do with the screening process. The screening process is not a difficult problem to solve.

The real difficulty in recruiting in this space is finding and engaging these people, not testing them.

If this is a Hacker Rank for sysadmins, then great. I'm sure you'll have plenty of users but we're still a long way away from solving the bigger problem.

> The real difficulty in recruiting in this space is finding and engaging these people, not testing them.

Tons of people in this space hang out on various places online, where you can simply advertise or do some PR and many do, is it really that difficult?

When was the last time you engaged with a company with a view to working with them because you saw an ad they happened to post in the right fora?

The monthly hacker news "who is hiring" thread.

Great example. HN is the perfect place to find people right? I mean, I should know.

HN who's hiring, like most online job advertising, is a crapshoot. It's a once-a-month post where you are drowning in noise. Finding a very large group of potentially relevant people for the roles you are hiring for isn't as straightforward as it sounds. Engaging said group and convincing folk from said group to come work for you is exponentially more difficult.

I guess this is sarcasm. I was only responding implicitly to your initial question:

>>When was the last time you engaged with a company with a view to working with them because you saw an ad they happened to post in the right fora?

So, yes, hn is where I engage with a company with a view with working with them where they happen to post in the right forum. Suddenly your point is that it isn't a good place to find jobs? Maybe not, but that wasn't your original point. You're moving the goalposts.

Apologies. Wasn't intended to sound sarcastic. I genuinely do believe that HN is the perfect place to find great people but it's still incredibly difficult and if you're hiring at scale it can be super tough to engage enough people on a place like this.

Places like HN and relevant IRC channels are great if you're hiring for one or two roles but beyond that then it's a bit of a lottery.

Beyond a handful of roles you already gave up on finding brilliant people. You can't reliably find the best of the best in industrial amounts, no matter what you do. You can (probably) build stable supply for competent people at best.

Regrettably, HN is the only public resource that I'm able to recommend anymore. AngelList had promise, but never really delivered.

I've hired 3 people from an IRC community I run, they all turned out to be exceptional (sysadmins and developers).

I've never been hired from IRC though.

Don't get me wrong, it's entirely possible if you do it right. I've hired quite a lot of folk as a result of interacting with them on HN alone. My point is that it's a significantly more difficult problem than screening those people.

Agree 100% with this.

At the past companies I've worked in finding the talent is the real issue. At my previous company it wasn't so bad as we had an in house recruitment team who generally dealt with sourcing, and generally we had a bigger budget for these kind of things.

However now I'm working in a smaller startup and we're having a lot of difficulty finding the right candidates without bowing to the extortionate cost of recruiters. We've tried the monthly who's hiring, angel.co and other such sites but it's a very slow process, and it's a lot of work to trawl through profiles.

Could do with some disruption in this space!

Agree that the sourcing is hard. We have this in mind, too. As i wrote in the Medium article, we will offer a practice mode for sysadmins. If we will have traction in this are, we will take into consideration to offer a sourcing product also. Until then, we focus on delivering a testing product for companies and a practice product for sysadmins.

Indeed Peroni, it's difficult to find and engage high talented people. But we are trying to solve another problem here. We want to optimize the time spent by hiring managers testing the potential candidates. When you have to test 15 candidates (1h per candidate) in a week, you need to take of production 1 technical person (most time a senior one). This is 15 hours per week. The biggest problem is when they look great on resume and during the interview you realize that is not so good as the resume stated. This is where we want to help.

As I see it, in my case, a part of the problem in finding good people to recruit was the grind through the 100s of applications and only a couple being good enough. This grind added up and I had to settle for less mainly because of time constraints. What if I managed to weed through double the amount of applicants by dismissing the ones that don't handle basic / medium tasks, would have had the time to engage more more quality engineers. It's just a game of odds and I hope I can tilt it.

The Cisco Certified Internetwork Engineer (CCIE) certification used to be the gold standard amongst network engineers 10-20 years ago.

Becoming a CCIE required a passing a notoriously difficult hands-on 8 hour in-person lab examination[1]. SysAdminsArena reminds me of a remote version of the CCIE Lab Exam for sysadmins.

Fun fact: the CCIE program was established in 1992 and was originally going to be called "Cisco Top Gun"[2], but they had to choose a more serious name.

[1] https://learningnetwork.cisco.com/community/certifications/c...

[2] http://www.bradreese.com/blog/how-the-cisco-ccie-program-was...

20 years later -> SysAdmins Top Gun :))

There's been a lot of criticism of hackerrank.[1][2]

Are you doing anything to make sure your service gets more respect?

[1] - https://news.ycombinator.com/item?id=12825953

[2] - https://news.ycombinator.com/item?id=12667174&source=techsto...

I don't think I would enjoy HackerRank as an interview, but as a user I do enjoy being able to practice computer science problems.

Just wrote an answer above which i think is good for your concerns. What'do you think?

(Speaking as a coder here, although I've also done sysadmin through most of my 20 years of working in the industry).

This would be quite nice. Going through the job searching process at the moment, the "aptitude tests" etc that I've seen are very good at blowing cobwebs off my obscure analytical skills, which I rarely if ever use, yet fail completely at getting any gauge of my ability to get every day stuff done in a clean, smell-free way.

As a result I'm now doing kata training on Coderwars and am getting more used to this style of testing. It's actually good fun, and I think it'll help increasing my ability to just spit out code. In real life scenarios though, most of the time is spent analysing a problem or process flow.

You just mentioned some of reasons why we are making this product. If you have more ideas, please share with us. Thanks

I think if that's the approach SysAdminsArena is taking, it's quite good. More advanced problems could be replication, caching, etc.

That's the approach and we are glad that you support it. Great topics for more advanced problems.

Glad you asked that. As i saw in the treads mentioned by you, the common problem is that the tests are more about theory, and less about practice. Which is true. Sometimes, in interviews, you are asked about algorithms that you studied in college and never used before. It's very difficult to score high in this tests.

We propose to eliminate this gap by having real job, day to day, scenarios like: install apache, mysql, php, clone site, install database etc.

These are recurring task in the daily activities of a sysadmin.

What do you say?

Those are pretty baseline things that get automated away pretty quickly. More useful to me (as both a practicing devops type and a manager in that capacity) would be tests gauging ability to write bash scripts to do various things, or simulated problems that need to be diagnosed/fixed using command line tools (e.g., strace, netstat, etc.).

Indeed, those are pretty basic tasks. We will have 3 levels of difficulty and the examples mentioned are in the easiest category. In a early phase of a recruitment process, an experienced sysadmin would solve in less than 3 minutes and this is helping screening process when you have a huge number of applicants. But for latest stages of recruitment process, we will have more advanced scenarios for the tests.

Definitely your suggestions are great and we will discuss having tests in this area.

Cheers!

> an experienced sysadmin would solve in less than 3 minutes

Why do you think so? And, I'm hopping you don't rely on timing on this level.

An experienced sysadmin would have little experience with basic tasks, since he solves them around once or twice per job. And you never know the environments they tried it before, that may be completely different from what you are presenting them right now.

I'd say "installing X" is a test for encyclopedic knowledge, and a very unreliable test for expertize. That's the kind of test that gave bad names to IT certification at the early 2000's.

What scenarios would you suggest to have?

No idea, but the certificates that focused on those things became enduring jokes.

As an ex-sysadmin turned coder, "sysadmins aren't paid to install apache" could be as true as statement as "programmers aren't paid to write code". The meat of the value of a good sysadmin is problem solving, learning, and communication abilities (perhaps a "duh" but the same qualities I look for in a programmer.) While there's certainly something to be said for a senior experienced sysadmin being expected to have an insane knowledge of their systems, at least for more entry roles, I've seen many of the same problems in sysop interviews as dev interviews, and having a hackerrank section for it may only exacerbate that. I'm simply not convinced that no matter what you guys ask from a programmatic angle you can solve the deep problems of interviewing that these soft skills are much harder to poll for, resulting in an uneven reliance on less effective signaling of tangential talent through e.g. hackerrank score.

Anecdotally, this sort of "filtering" would have precluded both my own hiring early on in my career (so I'm naturally sensitive to this), and that of probably two of the three best junior sysadmins I've had the pleasure to working with having come out of entirely non-technical fields, while allowed in both of the "disaster cases" that come to my mind.

Don't let this ramble be a "it's impossible stop trying;" but perhaps a "more of the same won't solve the problems we can already see endemic across our field", as the great-grandfather post brought up. I simply worry that the availability of "traditional hackerrank brute-force-evaluation" will simply result in more people leaning on it than trying to think more critically about their interview process.

So this is to have a test VM they have to fix? Excellent! I heartily approve of this method for finding a sysadmin.

This is a thread where I talk about how we did this: https://news.ycombinator.com/item?id=8332104

We used to use it a lot - a trivial Java webapp running in a slightly misconfigured Tomcat with a slightly misconfigured Apache in front on a slightly misconfigured Unix box - though it's fallen by the wayside because we never quite get around to the faff of setting up a slightly broken test system for them. So having that as a service would be pretty good.

If this is what you're doing: I suggest not just keystrokes, but that they keep a log of what they're doing and thinking. Because we never expect them to fix everything wrong with the box, but we do want to know how they approach it.

Hi David, thanks for your thoughts. Very useful the thread provided. And YES, this is what we are doing.

Keystrokes is just one of the metrics we are about to collect and analyze. We are also thinking about number of commands wrote, how many were correct, how many wrong, also a log of the entire test. In future phases, we may take into consideration pair working or capturing a movie of the test.

What other ideas of relevant metrics do you have?

So, if I will follow best practices and will prepare backup, then restore backup locally (to check that backup is OK), then will play with backup until problem will be solved (to not harm target server), then prepare runbook, and then will just execute it at target server, I will pass your test? Just two commands.

Yes. you should pass the test. This is one of the scenarios we are thinking. you get the backup location do the restore. done you passed.

I'm talking about copy-pasting of solution.

Ok. Maybe I misunderstood you. If you are thinking of having the tests leak and people just finding solutions on Google I'm just thinking of dynamically generating the problem. How about 3 characters missing from the config file for apache at random. You have to understand what is required and then try to fix it. Does this solve your case?

I had similar idea to your and found no solution to problem of copy-pasting. Good problem sets are hard to create but their solutions are easy to copy. Moreover, this is the good practice to try to solve solution locally, then copy-paste solution to target server.

I hope, you will find a solution for that.

I don't know of any relevant metrics at all for this sort of thing :-) What we did was mostly review the log of their thinking as they went, then talk to them more to work out if they think like a sysadmin. We want sysadmins who can think, and that's the opposite of a process susceptible to Taylorisation.

Note that in my example, I don't expect them to solve all the problems. I do expect them to have a good try at it, doing it as an open book test with Google to hand. (Open book tests are always hard.)

Speaking of sysadmins, a note for yours. You are currently serving blog.sysadminsarena.com over HTTPS with an HTTP to HTTPS redirect, but are not doing the same for your website. In fact, you don't have an HTTPS endpoint for your website at all. Going to your website by modifying the URL from the blog entry (because clicking your logo takes you to the front page of the blog, not the website) results in a `ERR_CONNECTION_REFUSED`.

Just an FYI :)

Thank you!

I'll fix it tonight after work with LetsEncrypt most probably. The blog is hosted on medium.com so they provided the ssl termination. But there should be no excuse for not having https on a website in 2016.

Now this looks like a real interview task, not installing apache (which I as a non-sysadmon could do easily as well)

Interesting problem you're addressing but I've been working on this problem from a completely different set of metrics. Fundamentally, I really don't care how a problem is fixed as long as it doesn't have bad repercussions. If a web server is "slow" users don't care why - giving freedom to admins to approach higher level problems while still measuring them is something I'm confident that can be done while I take a number of issues with assessing anything based upon commands entered even in realtime (I'm mostly concerned from a security / bad or illegal practices in prod perspective, to be frank).

The approach I'm working on is easier to do technically but requires some more costs.

With that said, glad to see I'm not the only one that's been frustrated with hiring devops / sysadmin / SRE roles. Google's even commented how hard hiring for SREs is for them and I think we have a huge gap between people stuck in the 90s approaching systems and those that modernized.

We are also glad that there are other people that are working on this issue. Good luck!

Isn't everyone hard to recruit? The difference in perception of coders might be that the evidence for a bad recruit is obvious in the code they produce. There could be many bad recruits in all other areas but are less obvious.

I generally like the idea of clearer quantification of a persons output but it should be done very delicately. As in, if you are going to measure everything someone produces, maybe that data should only be available to that person and it's up to them how they share it.

The uncertainty principle likely applies to people too. Hard to measure people without influencing them. Trick is to do it in a positive way. Managers measuring people for the purpose of firing them would be perceived, and have a negative effect on morale, than measuring people to identify opportunities for education, training or skills development.

Great idea to offer the data to the one that is producing it and he/she decides what to share. Thanks

Feel free to ask anything or provide feedback. Thanks

I am not sure you can do this I believe sysadmin/sre/devops person wears too many hats. What are you going to test ?

We are going to give real life scenarios based on the various roles you are testing with.(basicly a ssh login to the test machine) And we plan to asses the candidates way of solving the problems based on what commands and what process it uses to identify and solve the problem.

Please allow me my honesty but that's a stupid way of thinking about candidates and the roles. Or I do not get it. So you are basically going to judge the candidate based on what criteria ? If they use vi instead of cat to append to a file ? So decide what is the best way of solving the problem ? Or you give +1pt for using tail instead of opening the whole file in VI ? Are you aware of the fact that there are many types of systems and products that sysadmins use ? They do not behave the same. It makes no sense to use "top" on Solaris even the number for "uptime" reported by Solaris are different. What about Windows sysadmins ? What about stuff like Veritas ? What about docker ?

When I used to work in OS support my actions were optimized for the assumptions of a default minimal install and being able to describe actions to a person of questionable knowledge over the phone and interpret what they would most likely try to read back at me. The certifications were along those lines too.. While the exact nature of the certifications was ridiculous and would be even more ridiculous in terms of judging a general user's skills, dropping an administrator on minimal installs with broken ttys, etc makes a lot of sense. If you can't recover a system when vi is not an option then you have a problem, if you spend a lot of time recovering a good term when you could have used tail, then you are also not ideal.

The only suggestion I would make is that a good tool is present and working then I don't expect a penalty. I expect tools to not be present accordance with standard minimalist setups and to pay extreme penalties if I need to call a package manager or move to a less minimalist install to complete the task.

I strongly disagree. We should not test edge cases and specific scenarios. We are trying to test the candidate and his abilities in daily tasks. If I say I use vi instead of "commands" I will probably solve most of the "text" editing tasks faster than an admin that only knows how to get around using commands.

I can come up with tons of scenarios when where you do not really know the internals you can get stuck easily. Like if I do chmod -x on the chmod binary. Does it prove anything if the candidate is not able to solve such task ?

While I mostly agree with you, I can speak from experience with how we handled the first point:

We were interviewing a new admin for the sysadmin/maintenance team and her answers to "how would you solve this problem?" were "Open that file and check that these variables are set and, if not, set them". One of the more jerkish people there went down the rabbit hole of "How are you going to edit that file? What if you don't have vim? Okay, what if you don't have emacs? Nano is gone too" leading to the response that made us hire her: "Well, in that case the system is probably completely hosed and we need to stop configuring things and focus on recovery. But I think you are looking for me to tell you a sequence of commands that involve cat. If you give me five minutes I can check stack overflow and get back to you with those"

Which I personally think should be the response for stuff like that. Understand there is an alternative and a way to resolve the corner cases and know how to get that info if needed.

Brilliant answer :-). I totally agree with you. But this is the problem..you cannot implement such a thing to a tool like sysadminarena.

Actually I believe being a sysops/devops or whatever you want to call it is one of the "most" challenging jobs in the IT industry. You get zillions of technologies you have to administer and maintain but it's impossible to master all of them. What makes a good admin is the ability to learn,adapt,ask for a help and knowing where to look for an answer. It's easy to look up the parameters if you know the fundamentals.

Interviewing for me as a devops person is one of the most frustrating experiences. You are often asked questions that are specific to the environment of a company you are interviewing with. It's unbelievable how many times I've been asked about parameters of tar, du etc. It's totally stupid. And imagine interviewing with a company that uses only a specific subset of technologies and they heavily emphasize on that.

I would argue that basically any tech job (and probably other fields) is about knowing how to solve a problem (I really do wonder what the medical doctor equivalent of stack overflow is).

And just targeting a limited subset is kind of reasonable. Because, odds are, you will be. If you are running infrastructure for a web server you probably will have decided on a few solutions and have time to experiment with new ones on a new platform. Scientific computing? Same deal. It is the same logic by which coders should learn a range of languages and tools but will probably use a very small subset at any given job.

And I think that the semi-automated tools COULD detect stuff like that. One point for each solution presented and maybe something akin to those god awful rubrics for the more complex ones. So going with the tar approach (after twenty years, I know xzvf and czvf and can use a test file to figure out the order):

1 point for a correct answer 0.5 points for typing "man tar" or "<google> tar" And then a subset of the remaining 0.5 for each letter and the order of operations. So something like tar xvf foo.tar.gz ./foo would get you most of the credit.

The big problem with stuff like this is that it should ONLY be used for an initial filter, but it will inevitably be used as "Only the top N people matter".

Hmm, it seems like you are looking at how you should triage an applicant if they let you test them very painfully. I am interested in games I can play where mastery also serves as a potential initial triage and could lead to interesting discussions.

Just as I like project euler gamified as a fake ideal programming project, I would like a systems game where the point is optimizing for the worst case with minimal tools.

Add an IDE, a choice of editors, all kinds of softskills requirements that are more significant to an employer than these base primary skills. Now, I wonder why I am working for free as I am no longer just practicing our core/universal skills in an idealized/timeless environment. The better place to do that form of showing off work realistic skills is in OSS projects.

Thank you for your input and suggestions, highly appreciated.

Do you work as a sysadmin? If so, how many times did you recover from a broken tty at work? (I mean, is it a number bigger than zero?)

I've been one for a short while, and get to play devops from time to time. When I see a machine so broken that it can't give me a tty, it's time for a reboot (if last command caused it - actually don't remember it ever happening), full reinstall, or trash. I won't lose any time recovering it. If there's some important log on the disks or anything, mount the disk on another machine.

Now, I've recovered a broken machine plenty of times at home. I've edited files with sed and pipes because I broke every other editor available. I have never actually edited the memory with sed, but I could if I had to. But all that is at home, where it's fun to create a crazy configuration, and totally not fun to throw things away. It's fun, but completely unproductive.

> Do you work as a sysadmin? If so, how many times did you recover from a broken tty at work? (I mean, is it a number bigger than zero?)

Before then I was an sysadmin for a little while, but I don't really see what difference the role makes. I've never had a job where I didn't at some point need to deal with a legacy serial console of some sort misrouted through incompatible aggregators, etc, in order to deal with a networking problem and/or auth problems.

The idea that I would blind reboot a system in a failure mode I can't diagnose is not something I would normally consider until I've exhausted all means and only if I know a fair bit about it. I'm sure some of my colleagues did things like that as some were developers with relatively little system knowledge, but only a few were stupid enough to mention losing failure data through laziness to senior staff.

> It's fun, but completely unproductive.

? If I had a page for you should have diagnosed and every time you hit a swiss cheese failure with that attitude than I wouldn't use computers at all anymore.

We are aware that there are more ways to skin a cat. I think it's fair to evaluate a candidate based on wether if he can solve an issue or not and the amount of time it took. I agree it would be stupid to score based on individual commands and we didn't plan to go that low level. But we plan to count the number of commands with typos or wrong parameters. Don't you agree this would be a way to differentiate between 2 people that solved the same problems in the same amount of time?

Sorry I have to disagree here as well. I do not think the time or number of commands or typos should be measured. I do not think you can do any quantification here. I believe a lot of sysadmins (talking about unix) do w;uptime;inspect syslog as soon as they connect to the system. Most of the time it's just not necessary to do this but for a lot of sysadmins it's a habit that actually saved their asses several times.

Does it make sense for a sysadmin to know a numeric notation for chmod ? Probably. Would I ask that on an interview ? Nope If you need to ask why then you have probably never worked in a heterogeneous environment.

1. How exactly are the sysadmins graded? (because in the end, let's be real...this tool will most probably be used by HR folks to filter out candidates and they'd not care too much if you send over a screencast/replay of what the candidate did)

1.1) Sys admins generally have their own set of tools (which would be customized per company I guess)...would these be installed by default? (I'm assuming you are provisioning docker containers / candidate / test and then destroying them after use).

1.2) I've used the ones that are sent for developers and it's been a pretty terrible experience at best (it's buggy, won't accept valid code etc.). If this is the case for sysadmins, it's even more worse (but then again, I guess you can put a spin on it and claim it's for the best as they'd have to deal with unpredictiability).

2. How will different technologies be integrated to test candidates on? (for example, testing knowledge on multi node kubernetes cluster and so on).

1. I'm thinking the best grading sistem would be yes / no ? rigth ? If you manage to solve the problem and tests pass you pass. 1.1 If you get root access on a machine you have the freedom to add whatever tools you feel fine. 1.2 I don't think doing a terminal emulation is the way to go. 2. Advanced tests are on our plans but we haven't got everything figured out yet.

When will be the beta version available to test?

We proposed to have it available on mid December. Subscribe on www.sysadminsarena.com and we will keep you up to date.

When you have a very low supply of high quality people, and at the same time a huge demand for those same people, it is a major problem. You claim to solve this problem, by making it even harder? How does that help?

Kesor, we do not claim to solve this problem. We propose a product to help assessing easier the technical skills, not to improve the sourcing. In the recruitment process flow, our product comes after there is a short list of candidates (screened based on their resume) and want to perform a more deep screening.

We know there is the gap mentioned by you. Hopefully, in the future, we will be able to provide help in this area too.

Reply on news.ycombinator.com