Hacker News Reader

Comment by davidgerard | Hacker News Reader

So this is to have a test VM they have to fix? Excellent! I heartily approve of this method for finding a sysadmin.

This is a thread where I talk about how we did this: https://news.ycombinator.com/item?id=8332104

We used to use it a lot - a trivial Java webapp running in a slightly misconfigured Tomcat with a slightly misconfigured Apache in front on a slightly misconfigured Unix box - though it's fallen by the wayside because we never quite get around to the faff of setting up a slightly broken test system for them. So having that as a service would be pretty good.

If this is what you're doing: I suggest not just keystrokes, but that they keep a log of what they're doing and thinking. Because we never expect them to fix everything wrong with the box, but we do want to know how they approach it.

View on HN

Hi David, thanks for your thoughts. Very useful the thread provided. And YES, this is what we are doing.

Keystrokes is just one of the metrics we are about to collect and analyze. We are also thinking about number of commands wrote, how many were correct, how many wrong, also a log of the entire test. In future phases, we may take into consideration pair working or capturing a movie of the test.

What other ideas of relevant metrics do you have?

So, if I will follow best practices and will prepare backup, then restore backup locally (to check that backup is OK), then will play with backup until problem will be solved (to not harm target server), then prepare runbook, and then will just execute it at target server, I will pass your test? Just two commands.

Yes. you should pass the test. This is one of the scenarios we are thinking. you get the backup location do the restore. done you passed.

I'm talking about copy-pasting of solution.

Ok. Maybe I misunderstood you. If you are thinking of having the tests leak and people just finding solutions on Google I'm just thinking of dynamically generating the problem. How about 3 characters missing from the config file for apache at random. You have to understand what is required and then try to fix it. Does this solve your case?

I had similar idea to your and found no solution to problem of copy-pasting. Good problem sets are hard to create but their solutions are easy to copy. Moreover, this is the good practice to try to solve solution locally, then copy-paste solution to target server.

I hope, you will find a solution for that.

I don't know of any relevant metrics at all for this sort of thing :-) What we did was mostly review the log of their thinking as they went, then talk to them more to work out if they think like a sysadmin. We want sysadmins who can think, and that's the opposite of a process susceptible to Taylorisation.

Note that in my example, I don't expect them to solve all the problems. I do expect them to have a good try at it, doing it as an open book test with Google to hand. (Open book tests are always hard.)

Reply on news.ycombinator.com