Hacker News Reader

Comment by alexgotoi | Hacker News Reader

Hi David, thanks for your thoughts. Very useful the thread provided. And YES, this is what we are doing.

Keystrokes is just one of the metrics we are about to collect and analyze. We are also thinking about number of commands wrote, how many were correct, how many wrong, also a log of the entire test. In future phases, we may take into consideration pair working or capturing a movie of the test.

What other ideas of relevant metrics do you have?

View on HN

I don't know of any relevant metrics at all for this sort of thing :-) What we did was mostly review the log of their thinking as they went, then talk to them more to work out if they think like a sysadmin. We want sysadmins who can think, and that's the opposite of a process susceptible to Taylorisation.

Note that in my example, I don't expect them to solve all the problems. I do expect them to have a good try at it, doing it as an open book test with Google to hand. (Open book tests are always hard.)

So, if I will follow best practices and will prepare backup, then restore backup locally (to check that backup is OK), then will play with backup until problem will be solved (to not harm target server), then prepare runbook, and then will just execute it at target server, I will pass your test? Just two commands.

Yes. you should pass the test. This is one of the scenarios we are thinking. you get the backup location do the restore. done you passed.

I'm talking about copy-pasting of solution.

Ok. Maybe I misunderstood you. If you are thinking of having the tests leak and people just finding solutions on Google I'm just thinking of dynamically generating the problem. How about 3 characters missing from the config file for apache at random. You have to understand what is required and then try to fix it. Does this solve your case?

I had similar idea to your and found no solution to problem of copy-pasting. Good problem sets are hard to create but their solutions are easy to copy. Moreover, this is the good practice to try to solve solution locally, then copy-paste solution to target server.

I hope, you will find a solution for that.

Reply on news.ycombinator.com