As I see it, in my case, a part of the problem in finding good people to recruit was the grind through the 100s of applications and only a couple being good enough. This grind added up and I had to settle for less mainly because of time constraints. What if I managed to weed through double the amount of applicants by dismissing the ones that don't handle basic / medium tasks, would have had the time to engage more more quality engineers. It's just a game of odds and I hope I can tilt it.
Ok. Maybe I misunderstood you. If you are thinking of having the tests leak and people just finding solutions on Google I'm just thinking of dynamically generating the problem. How about 3 characters missing from the config file for apache at random. You have to understand what is required and then try to fix it. Does this solve your case?
I'll fix it tonight after work with LetsEncrypt most probably. The blog is hosted on medium.com so they provided the ssl termination. But there should be no excuse for not having https on a website in 2016.
1. I'm thinking the best grading sistem would be yes / no ? rigth ? If you manage to solve the problem and tests pass you pass.
1.1 If you get root access on a machine you have the freedom to add whatever tools you feel fine.
1.2 I don't think doing a terminal emulation is the way to go.
2. Advanced tests are on our plans but we haven't got everything figured out yet.
We are aware that there are more ways to skin a cat. I think it's fair to evaluate a candidate based on wether if he can solve an issue or not and the amount of time it took. I agree it would be stupid to score based on individual commands and we didn't plan to go that low level. But we plan to count the number of commands with typos or wrong parameters. Don't you agree this would be a way to differentiate between 2 people that solved the same problems in the same amount of time?
We are going to give real life scenarios based on the various roles you are testing with.(basicly a ssh login to the test machine) And we plan to asses the candidates way of solving the problems based on what commands and what process it uses to identify and solve the problem.