Hackermeter (YC S13) Wants To Kill Your Résumé And Replace It With A High Score | Hacker News Reader

Hackermeter (YC S13) Wants To Kill Your Résumé And Replace It With A High Score

techcrunch.com

57 pointsrjvir57 comments

It seems like all "code exercise" sites like this have the same annoying problems which I'll generally describe as lack of attention to detail in the exercise descriptions.

Take their Fibonacci exercise as an example. In the first section they say, "The first line of the input will be an integer N (1 <= N <= 100)" which specifies how many test cases follow. Then in the next section they say, "The first line of the input will be an integer N (1 <= N <= 10000), specifying the number of test cases"

So besides the fact that they aren't even consistent in specifying the expected range of N it isn't even clear why you would need to specify the number of test cases in the first place vs just reading one from each line until EOF. That makes you think maybe they want you to put in some basic error checking which again since they aren't consistent with N turns into a trial and error exercise if it even matters at all.

There is also no indication about the version of interpreters or compilers being used to check the submissions. If I choose to write my solution in python is that python 2 or 3?

Combine all that with an implied scoring system based on speed or number of tries and you have to wonder if this is really measuring anything relevant or if it is just filtering for people who's default assumptions happen to be the same as the person who wrote the lame exercise description.

I agree: I have no idea how the scoring system works. Am I supposed to get it done in as little (real world coding) time as possible? As few keystrokes? Is it supposed to be done in as few cycles (on their test machine) as possible? Should it be bulletproof, or just work?

I had to seriously do a double-take, I thought this was straight trolling, not a real company.

I doubt any serious developers would consider doing this, there's already a way to demonstrate your programming capabilities in the real world, either by contributing to an existing open source project or maintaining your own projects on github.

A number of start-ups have either already failed with a very similar business model or failed to go exponential growth. There was a good "Lessons Learned" post by one of them recently here on hacker news that got comments by both hiring managers and hackers. Why the rehash, again? What is this start-up really doing differently?

I personally wouldn't be motivated to use it either to find work or to hire. I think, when hiring, you really need to do a face to face interview. I've found resumes work well enough for pre-screening that it's not a pain point for me. And, it's not like there are so many candidates out there looking for work that it's impractical to do face to face, or at least telephone/skype, interviews with all the good candidates. I learned the lesson that face to face hiring is needed by hiring someone based on a co-worker's recommendation (even though he did badly in an interview, I looked passed it based on the recommendation). It's so costly and painful to make a hiring mistake that there's no way I would deviate from the current model and try a new method or service.

> it's not like there are so many candidates out there looking for work that it's impractical to do face to face

That really depends on your target market. For the most popular technologies (.Net, Java etc) in the most popular areas (London, Glasgow etc) for low-to-medium positions, you'll get tons of decent candidates.

Can I get a link to that post? I found this[1] one, but the post says straight up that in person interviews are deceptive and shouldn't be trusted.

[1] https://news.ycombinator.com/item?id=2346119

you're probably referring Work for pie: https://news.ycombinator.com/item?id=6148832

I wish this company well, and perhaps they'll be successful, but I don't think they've nailed down some ultimate formula for ranking and hiring programmers. This system sounds very easy for low-quality programmers with lots of time on their hands to manipulate - you sign up with a throwaway account with some fake name, get the list of questions, take your time searching for the answers online, and then sign up with your real information and have a carefully prepared way of typing out answers that get you high scores. If this achieves any measure of traction, you can bet that this will happen a lot.

This isn't an easy problem to solve and is something that companies have been struggling with for a long time, so no disrespect intended. But based on my understanding of how this works I don't think it will be very likely to yield great results.

Or even easier: never mind doing it yourself, just outsource it to some guy that already has shown the ability to achieve a high score.

Thanks for the critiques, and yes, it's not an easy problem.

There are countermeasures. For instance, who said we'll let you see the whole question bank? And if you type a close enough approximation, out come the plagiarism checkers.

But when somebody's gone to the trouble to make a database online of every problem, I'm happy, because if that's so we must be successful enough that we have plenty more resources to game their gaming.

Some direct competitors:

- https://www.hackerrank.com/ (aka https://www.interviewstreet.com a YC company)

- https://codeeval.com/

- https://coderwall.com/

- https://www.employiq.com/

- https://www.mindsumo.com/ (college student only)

- http://www.codewars.com/ (in beta)

http://initialround.com

http://www.topcoder.com/

Isn't topcoder still the king of this kind of thing. I don't think they particularly market it as being geared at hiring but companies definitely hire from there (I know UBS likes to) and your score does reflect on your ability, at least as far as being able to code competitively.

Edit: hyborg beat me to it :)

The best way to find good people is not by passing people as a value but by passing them through reference.

I'm one of the co-founders and I'm still saving that quote.

As for the answer to the implicit question...the best people are passed by reference when they're already recognized. We're aiming to help people show that their skills have value when they don't have a reference.

The number is just, well, a pointer to a broader portfolio - and we'll only be making that portfolio more expressive over time.

Depending on your industry and location, recognition doesn't always come easy. Considering that a majority of employers will have a non-technical HR employee only makes things worse. I think that there is definitely a spot in the market for a company that can successfully break this barrier and improve the marketability and recognition of job seekers. The key is that a service like this isn't the be-all and end-all, but instead should serve as a supplemental indicator of a candidate's ability.

I agree. Referrals are the best ways of finding good engineers. Just because one could (or could not) answer questions correctly on a test doesn't necessarily mean that he or she is a good (or bad) engineer. Everyone has good days and bad days.

There are also plenty of ways of circumventing this. Since a test taker is evaluated not just by his answers, but also by the process of inputting his answers, that itself could also be faked away.

While I don't think the point of this idea is to evaluate the candidate's personality, I'm wondering if the founders thought about this. Is this going to be left to the companies to figure out on their own, or are the founders going to assist in this? What I don't like is the idea of encouraging employers to filter candidates by coding skills first, then personality. As a project manager, I've dealt with my fair share of really intelligent but lazy, egotistical engineers and thinking back I wouldn't really hire them had I known about their personalities earlier. I would rather hire engineers with B+ coding skills and A+ personalities, than engineers with A+ coding skills by B+ personalities.

Here's my problem with the idea that you'll always get the best candidates from referrals:

You're the hiring manager. I'm a professional programmer who works with you. Let's say I personally know and can vouch for the ability of 100 programmers. There are 7 billion people on the planet. What's more likely? That your best candidate will come from my 100-person address book, or that it will be one of the other 6,999,999,900 people in the world you're not looking at?

Now it is likely that someone from my address book will be, on average, a better candidate than a random selection from the rest of the world, but all that shows is that professional references will produce "above average" candidates.

EDIT: I suppose the difference is whether you want the "best way to find good people," what the parent said, or a "good way to find the best people" which I argue referrals might not be.

As a younger developer, I struggle to imagine our senior developers - who have been doing complicated things for 5-10+ years - ever stoop to a number to quantify their experience. I suppose that senior developers are probably not the target audience, but I would never want a company to expect this, or a score to define my experiences as a developer.

A technical screening can be handled in a ten-minute phone call for free - with the dignity of the applicant intact.

As an experienced developer, I struggle to imagine any of the talented younger developers I've worked with being well-represented by a number that quantifies their experience.

I take issue with any hiring system that attempts to narrow the field of view of candidates. This includes any single-number "scoring" system, as well as hiring cultures that focus on single methods (e.g. "puzzle questions") to the near exclusion of other hiring criteria. If your organization is doing any of the above, it's shooting itself in the collective foot.

The problem is not the talented younger developers. It's the not-so-talented ones. At least they keep us old guys in business to clean up the mess afterwards but this is quite a problem in some fields. The talented younger developers are doing quite well by themselves. I get to meet them in small groups every now and then and some of what I find blows me away.

I can't really see this being any use at all for finding a really talented, experienced developer. Any half-assed developer should be able to do these little puzzles with a little research.

The really hackerish stuff can't be tested for in a format like this.

Best of luck, I recently received an email that WorkforPie.com was shutting down (http://thenextweb.com/insider/2013/08/02/developer-hiring-st...) and they had the same sort of model.

I get we always complain about the developer interview process, but are these business models actually solving the problem?

"...Lack of coder personality" That's more that just a mere gripe.

Many a job has been won or lost based on the character of a prospect. In fact, I've lost count of how many times I've seen otherwise very technically skilled people lose out to basic inter-personal communication. Above all else, a seeming lack of tact.

Indeed. Perhaps an option to include a video and explain an interesting problem? Definitely something we're considering.

I like that. An "in your own words" monolog can go a long way to help recruiters understand the type of person they will be dealing with.

That also means the recruiter(s) need to understand the questions they're asking. Technical merit is all well and good, but how well can a recruiter process the answers they're getting? This can't be a bullet point questionnaire with blind matching of answer "C" to question "1". Sometimes solutions aren't so black and white.

A developer standing alone can be evaluated by one set of metrics but when added to a group of other developers many of those metrics can be overridden by others and become meaningless. In simpler terms, teamwork.

I do not think developer grading is impossible, but there is so much more context involved. The "score" is dependent not just on the company, but the time and place in the developer's and the companies own life cycle. The perfect fit is always subjective -- these are organic life forms not rigid machines.

When you reduce a person down to a scalar, you're inevitably losing most of the signal in the different variables which go into that calculation. I am extremely skeptical that it's possible to come up with a model which makes that scalar useful to a hiring manager across a pool of candidates, even if you're some kind of data-science messiah. My .02 is that applicant scoring this way is inherently broken from a signal to noise perspective. (In fact, I bet if you just took GPA as the ranking metric, its performance would be superior as it's more competitive/harder to game, despite GPA being widely acknowledged as a subpar hiring metric - at least in my experience.)

Coding is a small part of a developer's job. Given a simple task, a programmer can code to it. A software engineer can create a proper design that expands out to solving a wide range of problems, a design that is easy to maintain and flexible enough to deal with problems.

I don't want to see someone's first solution to a problem. I want to know the solution they came up with after 5 mental iterations and a few hallway discussions with peers.

I wonder if the better approach (for a company like this one) would be to hire 4-5 good programmers, and 4-5 bad programmers, and let candidates write the tests; assign the candidates that most successfully identify the on-staff good coders from the bad ones the highest score?

It sounds about as silly as this scoring does in the first place anyway...

So the natural follow up question here is, do you have an interview process that will give you this information for a given candidate?

This feels more like a candidate for acquisition and integration in some HR platform, rather than a standalone effort with a real future.

Only people like software engineers would stoop so low to subject themselves to measures like these.

Aren't technical interviews (which are just politically correct way to test IQ) enough of a humiliation one has to go through?

Let me rephrase what you said in a better perspective. I don't think anybody would subject themselves to unnecessary testing, regardless of the profession or industry. If this becomes an industry norm, then tough cookie for us engineers.

Interviews are a necessary and evolutionary process of finding good employees, and we know that so that's why we subject ourselves to them. It's not a form of humiliation because it's the norm. How else would a manager figure out if you got the skills and character to do the job? It's not a perfect process by any means, but it definitely helps.

Here's where this startup tries to take things to a different level. I get the idea behind it, but I cannot fathom why anybody would want to subject themselves to unnecessary testing. What's the ROI? Why give out personal information and have it stored on some database for some undetermined, possibly indefinite amount of time (forever? 1 year? 5 years?)? Am I going to be paid to do this? Can I no longer get a job if I don't do this? If this is purely voluntary, I have absolutely no incentive of taking the test.

Perhaps these questions should be something the founders should consider.

In South Australia (and potentially elsewhere) university admissions for medicine have started to involve something of a personality test rather than purely academic aptitude. The workplace, and especially dealing with patients (colleagues or clients in this case?), is more than a calculated diagnosis.

I was disappointed to see that they limit the password length to 30 characters. Perhaps one of the exercises should be to hash passwords that are longer than 30 characters.

Bcrypt will support up to 72 characters. If you are going to limit the password length I'd expect it to be something closer to 72 than to 30.

It seems like as time goes on YC is funding far more startups that are direct competitors with each other. Hackermeter's pitch looks almost identical to that of InterviewStreet (YC S11).

Makes sense from an investment perspective. For the verticals that they care about this allows them to cut down on their exposure to execution risk.

Sounds fun, but everyone solves the same problems (ok, ideally not: "sets of custom-built, rapid fire coding challenges built by each employer. You can use Hackermeter’s pre-provided challenges, or build your own")? How long until copied solutions corrupt the scores? Better check performance on a held-out set before interviewing/hiring.

Let's not get into plagiarism-detection arms race; consider automatic refactorings / pretty printers with variations, and human input to vary variable names.

Oh Yay, another useless measurement of a programmer's abilities, very much as useless as the puzzle interview made popular by Microsoft and then Google. So you can measure somebody in an artificial sprint environment that has nothing to do with actually delivering a product. The real measurement is how somebody does delivering actual products to market.

Not going to help you with problems like when your boss says oh we have brought a mega expensive bit of kit 4 to 5 times my annual salary.

Just hook it up to the spare PDP we have and go and see Bob and work out how to use it to analyze the data from his experiment tracking high speed droplets droplets in 3dspace.

BTW I had to write a driver to network with a rtos (RT11) just to talk to the device. I was 22 at the time and joined the company straight from high school.

Please make your pages load faster. A 1.1MB header image which blocks other assets from loading doesn't do well for you.

Thanks. Will be fixed.

Hey xianshou, I've been trying to figure out how to privately message you guys about this, but your info@hackermeter.com email doesn't seem to be working. Your online code editor is leaking environment variables pretty severely; stuff like API keys.

Thanks. info@ is fixed, so you can send messages there now. Please send the leaks and we'll plug them.

Sent.

I don't know if this is common knowledge or something, but how are all these companies implementing a secure way to execute arbitrary code? Sounds like a non-trivial thing to get right.

Usually there is some kind of sandboxing in place. That way you can limit what the code is allowed to execute. For e.g. in PHP you would probably use Runkit[1] for this.

1. http://php.net/manual/en/book.runkit.php

Build a virtual machine, run it, have it output a single bit and destroy it.

If you can root their VM blind, without feedback, you win all the challenges.

Check out http://codewars.com (only supports JS, and Ruby). They offer problems with a voting system for solutions.

Perfect opportunity to let me Auth with GitHub wink wink

Will they be normalizing high scores against Fabrice Bellard?

Have you considered adding Scala? You're already running jvms for Java, and you could do incremental compilation with SBT to keep everything responsive.

Is this mainly for hiring people overseas? Otherwise couldn't you just pull the candidate in for an interview and look over their shoulder?

"I am not a number, I am a free man!"

I am NaN!

Reply on news.ycombinator.com