ammon

This is all just really hard (and counterintuitive). We've tried take-home projects, with and without asking the engineer to make changes live. This is actually very controversial (a lot of engineer are, reasonably, against the time commitment). And (we found) to get an equivalent level of consistency (the 1st step toward accuracy) the project needs to be pretty big. The work really needs to be of the level of what engineers do in a full day or more on the job. Shorter projects than this introduce the noise of how much time the candidate spent on the project. You can introduce a hard time limit, but then you're back in interview stress territory.

Large take-home projects (and trial employment) totally are better ways to evaluate engineers than interviews. Unfortunately, they require major time commitments from the candidate that many engineers are not able to give. Most (about 80%) engineers select a regular interview if given the choice. I do think they are a good option to offer, but they can't (unfortunately) replace interviews in the majority of cases.

We've also tried debug sections (where we give the candidate a program with bugs in it and ask them to fix test cases). This works great as a portion of the interview (but misses some people with other skills, so it can't be the entire interview).

being happy with the job and not having left after 1 year

Interviews absolutely should be grounded in trying to predict how a candidate would do on a job. That's the whole ballgame. The question is how to best do that. First, you need to run a repeatable process (my previews comment). Second, you need to look at the right skills. The approach we take is to track a lot, and figure out what works the best over time. What we've found to be most predictive (so far) is a base level of coding competency, plus max skill (how good an engineer is at what they are best at). So (beyond the coding portion) we don't actually care very much about what a candidate is bad at. We care about how good they are at what they are good at. To give as many candidates as possible the opportunity to show strength, we cover a number of areas. This includes back-end web development, distributed systems, debugging a large codebase, algorithms, and -- yes -- low-level systems (currency, memory, bit and bytes). We do not expect any one engineer to be strong in all of these areas (I'm weak on some of them). But they all are perfectly valid areas to show strength (and we work with companies that value each of the areas).

We've recently moved to a new interview processes organized around this idea of max skill. It's working great in terms of company matching and predictive ability. However, it seems we may have underestimated the cost to candidates of being asked about areas where they are weak. There's more negative feedback here than we've seen in previous HN discussions, and I think that the interview change may be behind that. I'm taking that to heart. I think we can probably articulate it better (that we measure in a bunch of areas and look for max strength). We're also running an experiment now where we ask engineers are the start of the interview which sections they think they'll do best on. I'm excited about this. If engineers can self-identify their strongest areas, we'll be able to make the process shorter and much more pleasant!

So, the bit shift question: that come up down one branch of a system design question that we used for a while (we've since moved to a more targeted version that is more repeatable). The (sub)issue involved adding a binary flag to a large data blob (this came up as part of a solution to a real-world caching problem). Adding a single bit flag to the front of a 1GB blob has a problem. To really add just one bit, you'd have to bitshift the entire 1GB. This is clearly not worth it to save 7 bits of storage (ignoring that that would not be saved in any case). You can just use a byte (or word), or add the flag at the end. When candidates suggested adding a bit flag at the front, we would follow up asking them how they'd do it (to unearth if they were using 'bit' as a shorthand for a reasonable solution, or if they really are a little weak in binary data manipulation). This was one small part of our interview. By itself it in no way determined the outcome of the interview, or even of the low-level systems section. Plenty of great engineers might get it wrong. But I don't think it was unfair.

We did not reimburse you for expenses?! Can you email me: ammon@triplebyte.com

We actually put a bunch of effort into consistency/repeatability checks. Every interview is recorded (video), and we re-watch and re-grade a percentage of them to measure the consistency. A long-term experiment we're running is comparing qualitative scores (code quality, good process, how good did the interviewer feel the candidate was) with quantitative features (which tests passed, how long did it take, what design--picked from a decision tree--did the candidate take). We calibrate the qualitative scores with the recorded interviews. So far, quantitative scoring is winning (when judged against predicting interview results at companies). We're waiting, however, until we can see which better predicts job success.

We work less well with very specialized positions. However, the approach of identifying strong general programmers, and then matching the ones who want to work on compilers with compiler jobs works surprisingly well (still often results in higher offer rates than the approach that companies themselves take of filtering first on compiler experience, and then checking technical strength)

Having designed the new IR in Google's V8 is a very strong signal. The problem is that it's vanishingly rare to encounter someone who did that. The much more common case is companies just giving strong preference to anyone who has worked at Google, or graduated from a top 10 school. Again, these are real signals (the average skill of folks who have worked at Google or graduated from MIT is higher than the population at large). But, it's a very crude signal (there are plenty of bad ex-googlers, and great folks who went to a state school). In fact, because the vast majority of programmers don't have credentials, there are almost certainly more strong programmers without credentials than with. For that reason, we choose to specialize on directly identifying programming skill, without using credentials.

tl;dr Credentials carry signal, but recruiters at companies have gone overboard, giving that weight above all else. We push against that.

I'm sorry that the interview went poorly! I'm especially sorry about the lateness. I just pulled up our notes from your interview, and I was your interviewer! So that makes it especially bad :(

I'm not an expert in all areas. At the end of the interview we have a section where we let the discussion go into whatever technical area the engineer wants to talk about. It sounds like you're an expert in an area where I am not. In those cases I try to ask questions and push deep, but (depending on the topic) that can be hard.

edit: removed discussion of the specific topic discussed. Sorry, folks below are right

Expertise is something our model handles less well (it's much harder to standardize). This certainly results in us failing some great people (and it sounds like that may have happened here). I'm happy to talk about this more. Give me an email at ammon@triplebyte.com.

So, those are not (unfortunately!) the primary way that we match engineers with companies. It comes down to questions about how important it is that engineers are strong in academic CS (some organizations think very, others think it's useless fluff), how important programming speed is (some companies think very, others want to see careful testing and think speed is sign of shallowness), or whether it's important that engineers are strong in http / web systems (some think this is the core of what most engineers do, others want problem solving and intelligence and think web programming is teachable). Some companies want the best engineers, regardless of english level. Others value communication over engineers. It's simple stuff. But getting the data is HARD (companies themselves don't know how much they differ), and matching engineers on these criteria significantly boosts pass rates.

This differs substantially by company. Some companies have one general interview process, and handle team/role assignment after making offers (for some reason this is common at very large companies and very small companies). Other companies (often midsize) have different interview tracks for different open roles. In that case we just treat each interview track like a different company. We gather an initial guess about what the track wants by talking to the hiring managers, and then refine this with feedback data on candidates. We've found that bringing up example interview questions and asking if it's important that candidates can answer them is the best way to get at what skills an organization values. There's still a lot of noise in this (a hiring manager can't speak perfectly for the the entire organization), so we refine it over time.

I'm sorry you did not like our process! For what it's worth, we've tested a bunch of types of questions, and relatively easy questions answered quickly provide the highest accuracy / time ratio. We also cover a bunch of topics (to give everyone a chance to show their strength), so we don't expect everyone to do well on all the questions.

We suffer from adverse selection bias (as does any online job application). Many companies don't even hire from their online applications for this reason (instead using referrals). The truth is that almost all of the people we don't interview would not have passed (we have a small control group that we always interview, to measure our false negative rate). We do certainly make mistakes (our false negative rate is not 0). But I am proud of our process. It's more open than any other interview I know of. We routinely find awesome self-taught programmers who no other company would interview.

I don't think the problem the author is highlighting here actually has very much to do with recruiters. The recruiting industry _is_ full of scamminess (spam to both engineers and companies, false-flag recruiters who claim to represent famous companies). However, as broken as the industry is, I don't think it's behind the author's problem here. He's talking about interview randomness. He was heavily recruited by Amazon, then failed the interview.

This is not a rare event. We have data on this from running Triplebyte. Almost no engineers passes all their interviews. The top cohort on Triplebyte goes on to pass about 80% of their interviews. These are strong, strong engineers (successful startup founders, ICPC national winners, Google staff engineers), and they still fail. Almost no one can consistently pass more than 80% of interviews.

One reason for this is just incentives for companies. False positives (bad hires) are far more expensive to a company than false negatives (rejecting a good person). So companies rationally bias toward rejection. To a certain extend we just have to accept this as rational. (Although I do think that companies tend to assume that everyone they reject is bad, and discount the cost of false negatives).

But I DO think the situation can be significantly improved. Interviewers vary significantly between companies. Some companies want everyone they hire to understand low-level systems and concurrency and thread scheduling. Others think this is irrelevant and want everyone they hire to be very fast and productive in modern web frameworks. Some think that CS is important, others think that it's bookish and irrelevant.

All of this makes sense (different companies do different things). The problem is that you can't tell from the outside what a company wants to see (oddly what the company wants often has little to do with the type of engineering they're doing. It seems to just be an engineering culture that develops over time). And companies themselves often are not aware of how what they want is different from what other companies want. Each company just says that they want to hire "excellent engineers", and think that that means the same thing to them as to everyone else. It's the cognitive bias where everyone thinks that the things they are good at are the core of the discipline.

This adds noise to interviews. At Triplebtye, one of the big things we do is measure this (by evaluating engineers already working at companies, and updating our model with the feedback on each candidate we send). I'd love to see this sort of approach take hold across the industry (and companies get better at advertising honestly what skills they care about).

The low resolution of the stereogram is interesting. If you look at a wall at an angle, you see large steps, rather than an angled surface. Looking into this, it's caused by the fact that the program limits pairs of constrained pixels to integer values. The difference of a single pixel in feature placement has that large an impact on stereoscopic depth perception!

I also added a render mode that uses interpolation to allow constraints between non-integer locations. You can press 4 to see this. It works well (removed the banding), but introduces artifacts as rounding errors are propagated across the screen (Magic Eeye images seem to show these artifacts as well, so I bet the algorithm used for those books takes this approach).

You're totally correct and I am wrong :)

EDIT

I incorrectly multiplied the tax rate to the salary difference, and said that this was a 5$ difference. This is totally wrong (as multiple people point out below). I'll leave my original embarrassing comment as a cation to not write stupid things :)

==

I just added in a mention of the tax difference. But the numbers are 13.3% in CA vs 0% in WA. That's significant, for sure. But 13% tax on $33k is about $5k. I don't think that changes the conclusion that the salary difference does (in many cases) cover the housing costs.

You're likely correct about the price for a similarly sized home. People who want large homes should probably not move to the Bay Area.

Totally fair point. CA state income tax may take an additional 13% out of the Bay Area numbers. I'm adding this into the post now... Thanks for pointing this out!

You might expect that companies in big markets with great growth would be more competitive (more people would want to work there, and it would be harder to get an interview / offer). But we see that this is often not the case, making it a great strategy.

Our next blog post is going to be about this (how engineers decide between offers, and how they can perhaps do it better)

When people compare salaries / the cost of living in different cities, they often fail to account for the fact that many people don't spend their entire salary in the local economy. If you are trying to save money (or pay for college, donate to a cause, or buy a Ferrari) this costs the same no matter where you live. You should only apply the cost of living adjustment to housing (and maybe food). When you do this, living in the Bay Area starts to make more financial sense.

I don't think that we should excuse this behavior just because it's common. We push back against this pretty hard at Triplebyte. I can also say (having run a huge number of interviews over that last year) that getting a personal, honest email to every candidate is just a matter of making it part of the process. Writing an email takes a small fraction of the time of doing an interview.

Bootcamps charge tuition. They do this because getting placement fees from companies requires consistently referring very strong candidates (a higher bar). They are competing with universities.

This is roughly what our data shows.

We also work with a bunch of great self-taught folks! Perhaps I'll blog about this in the future!

We see about a 50/50 split between the two groups. (Of course a lot of the people without a technical background are still analytical thinkers. There's probably pretty heavy selection bias for that)

Our dataset is not large enough to really answer this. My opinion is that quality of the bootcamp (how selective they are, and how well they present and motivate students) has to matter.

Interesting questions. Bootcamps are clearly incentivized to do this. However, they do not seem to be particularly good at it. Algorithms are over represented in interviews relative to most jobs, and yet (as our data shows) bootcamps are not very good at teaching this. Now, we are measuring algorithmic skill by asking candidates to actually implement non-trivial algorithms. We've observed that a lot of interviews involve what is essentially trivia about algorithms, and it's possible that the bootcamps are better at preparing students for this (we don't measure this skill so I am not sure). I think that this gets at the answer. If you make your interview go deep, it gets increasingly hard to specifically prepare for it, to that point where preparing is actually becoming a better programmer. Rather than a 30-minute question that covers knowledge of sorting algorithms (easily learnable), have your candidate spend an hour building a collision detection systems using a axis-sorted list of rectangles, and reason about maintaining this sort as objects move around. That's the theory. In practice there will always be some noise.

Good point. About 50% of the bootcamp folks we work with have some sort of technical background before the bootcamp. The other 50% do not, however, which is really interesting! (I have no idea how this compares to the bootcamp population as a whole. I would guess that folks with a tech background are overrepresented in the group that passes our screening step)

We don't expect every applicant to be good in all those areas, and we let them pick in which area to spend the majority of the interview. I totally agree with you that no one has all those skills (except of course our esteemed interview teem!). We explicitly look in all those areas so that we can find people who are great in all sorts of ways.

I think it's possible that these things just fundamentally take more time to learn well. I'd be happy to be proven wrong, however.

Pretty close. I don't talk about this in the post, but you are right. The college grads had more variance. You occasionally get a college grad who has been working on open source since they were 15, and actually looks a lot like a, engineer with 3 years experience.