How to Interview Engineers
blog.triplebyte.com
blog.triplebyte.com
Lately, I've been asking "given the starting and ending times of two calendar appointments, determine whether or not they conflict." No loops, no fancy algorithms, no tricks... and asking it has not been a waste of time. I get people writing doubly-nested loops over all of the seconds in the two intervals, comparing for equality; I get multi-line predicates full of redundancy that still yield false positives and negatives; it's depressing as hell.
Then I started interviewing people for engineering and security engineering roles. That's when I realized that unless you put a lot of effort into curating your candidate pipeline, you interview engineers who literally (without exaggeration!) cannot complete anagram toy programming questions in their "favorite languages." That was a sobering moment for me.
I'm in agreement that a lot of tech hiring is suboptimal, selects for noisy heuristics and is even (sometimes) sadistic. But I've also met engineers - typically people who have not done interviewing or hiring much - who think that reports of fizzbuzz failures have been greatly exaggerated. They have not been exaggerated.
I haven't interviewed anyone in over a year, but back when I was doing it weekly I used to mentally sigh with relief when the candidate I was talking to could actually approach technical questions, even if they ultimately didn't give winning answers. It is a sad reality indeed; the bar is low enough that you have an edge by merely engaging with the problem and reasoning about it with technical competence in an interview.
How much say do the people working on a team have, in who gets hired? I mean people who'll actually be working with the person day to day, not some manager.
If the answer is 'we tell recruiting agencies we're hiring for vague role X, Y and Z', then why are you surprised?
Is the work any interesting? No? Generic corporate codebase perhaps? And you're getting generic people applying? How strange :)
ps. This is not directed at you in particular. I just find it strange when people don't do an inch beyond 'we pay money', and then expect a mile beyond 'i do just enough to get paid'
Retaining staff doesn't mean that you aren't hiring at .2 headcount per annum.
Assuming that there's a basic process of CV review (by team members who would work with) followed by non technical phone interview. You will still have people who fail fizzbuzz. I'm not talking about small errors reading the problem - but the inability to even put a for loop in pseudo code onto the page.
The fact that the test is required is somewhat silly - but it's hardly limited to dull corporate roles.
See, you are assuming a basic process of a job board and anyone applying for the position.
I am saying you are so entrenched in mediocrity that you can't fathom there are much better ways.
I will say this though - they all require a fundamental shift from 'fill seats with people for minimum amount of pay, to do uninteresting work, with least amount of complaining' to actually giving a shit about people you work with, yourself, and your life :)
I have conducted literally hundreds of technical interviews. To me, too, it was a jaw-dropping realization that some candidates can arrive who literally have no idea how to program a computer. I don't know how they get through school (with high GPAs, even), held their last job, or impressed their phone screener. I don't even want to speculate. The sad fact is that trivial, two-minute, easy-peasy exercises can and will expose some candidates as being unable to work through them, even with the most friendly and patient coaching.
So I always ask at least one trivial question that can be solved in a minute or two with a couple of basic relations and a Boolean connective.
This point cannot be underemphasized. "Thinking aloud", not just in front of another person but under a very stressful situation (for most people) that also basically almost never occurs otherwise in daily life -- is a specific metacognitive skill that for many people can only be (even adequately) learned by either (1) careful training or (2) surviving many repeated failures, and finally getting at least partially acclimated to these kinds of confrontations.
Yeah sure, perhaps some of the people who "bombed" that calendar test really were the walking fraudsters the interviewer (who knows they'll get to stay in their well-paying job regardless of the outcome of said interview) makes them out to be. But most are probably simply nervous, and due to a variety of psychological factors (imposter syndrome, among others) are simply momentarily blanking out, and having suffering from a very common form of mild anxiety attack which makes the problem seem much more complex to them (at that moment) than it otherwise would, under more natural circumstances.
I've literally started nervous people with "naively print the string 'hello'"... After warming up, the best one eventually completed my max-difficulty questions.
I would add that it takes significant experience and preparation to interview people effectively. This is a difficult-to-learn soft-skill that not everyone has.
Too often folks just get dragged into an interviewer situations with no planning or coordination.
Is there evidence for this claim? I'm inclined to believe it but that's experiential. It's also something I've never had to work to acquire, but that could be a cultural thing (my family and friends talk a lot about thinking) or a personal history thing (I have a minor learning disability, and it has forced me to develop strong, conscious meta-cognition).
It's not a given, tho, that being able to talk about thinking is distinct from being able to think and being able to talk.
But talking through a problem forces you to use verbal reasoning. A lot of programming can be done well with non-verbal reasoning skills.
Personally, quickly sketching some timelines on a piece of paper would be the fastest way to get the correct checks. Talking it through would force me to convert my mental visualizations into words, and that'd make me stumble.
people who "talk to themselves" are labelled idiots. mental dialogue is supposed to be mental.
call it social conditioning if you will. spelling out your thoughts for someone else to hear them just so they can take note of them for evaluation is the polar opposite of normal human interaction.
Interviews aren't ordinary job situations: they are contrived situations where people are making life-changing decisions based on a very short interaction.
you don't know the circumstances that 30 year old programmer with a 2 year old child at home is facing. You could also be like certain firms and immediately nix him on the "merits" of ageism.
A female programmer might have travelled 2000 miles to just receive an abstraction question out of left-field over an irrelevant concept to the job. She flew out on this uncertainty, and she's in a rather discriminatory field.
26 year old dev has probably built CRUD apps all his short career. and, you're asking him to implement (isBinarySearchTree Boolean) on a whiteboard on the spot.
I've been in many stressful situations at work - none of them involved finding algorithmic solutions.
Not once in my career had I had to fix a problem in an hour, let alone 10 minutes.
I have had senior people get mad at me when I couldn't give an answer during a meeting - that's the closest to an interview type scenario where you need the answer now. But even then, I did not stress that I'd lose my job, which is similar to the stress the candidate is facing.
Armies train people by having them run obstacle courses with guns being fired all around them. Do a "realistic" interview, then, to really test how someone does "under pressure". Have them sitting alone in the room, and a screaming angry manager runs in yelling obscenities at them about how the production system is down, it's costing the company millions of dollars, and if they want to have a job they'd better get it fixed ASAP!
And have that same manager stand over their shoulder the whole time, yelling yet more obscenities. If the candidate doesn't get it fixed in, say, five minutes, then obviously they lack even the most basic qualifications to work as a programmer and you can reject them. Heck, save money and don't even do it on-site -- get their phone number and do it 3AM on a weekend, just like a real on-call situation!
Or... maybe the whole "under pressure" thing is just an excuse people use to cover for the fact that they like making people squirm, enjoy the feeling of power they have from knowing someone else's future is in their hands and that person knows it, and want to savor by making the trained monkey stand up and dance on command... or else.
Unfortunately, people who like the "under pressure" idea of interviewing seldom realize that sooner or later they're going to be the monkey and someone else will be the organ grinder.
Sure -- it's just the tenor (and for some people, sheer intensity) of the psychological stress experienced during interviews is, for many people, basically orthogonal to the stress of real-life work situations -- or even genuinely dangerous (even physically threatening) situations in life, otherwise.
With the latter, it's like "Oh shit - the client's gonna get real pissed if I don't figure something out real quick".† Or even: "That guy looks like he's about to lunge at me - I better think of something!" For which your brain and your glands have benefitted from millions of years of evolution in support of mechanisms for pumping just the right kind and amount of juice into your system to "figure something out".
But with interviews it's more like: "This person's evaluating me, using some hidden criteria. Given that the problem is slightly tricky, I literally don't know if they're expecting to power on at all costs, or, secretly, that I simply admit that I don't know where to start just yet. Not only that, something tells me he's not stating the problem quite correctly -- they do that, no all the time, but way too often in these kinds of interviews. And on top of that he's just being plain overbearing... and now he's starting to fiddle with his phone. On top of starting 15 minutes late. Like he never really wanted to talk to me in the first place. If I ask too many questions -- or even just one instance of the wrong kind of question -- I'm screwed, and gonna have to hit my inbox for leads again. Oh great, now he's starting to offer 'hints', completely derailing my flow of thought and whatever self confidence I thought I had. Fuck it -- I just should bail and go work on my personal projects. No one will be paying me $150k but at least I'll be learning something."
† Where "the client is gonna get real pissed" is roughly equivalent to "if I can't get these rocks to flake in just the right way, we aren't gonna have any more arrowheads and were' all gonna starve!", to a first order approximation.
understatement of the year.
The problem with giving technical interviews is you are testing for someone who is an extrovert that can bullshit under pressure. That's not what you want. You want someone who is smart and can solve problems with a compiler.
I have not interviewed many people yet, but already during one lunch (!) interview (where I don't ask any questions), a guy was so proudly talking about his achievements at his previous job in quite generic terms, that I've got an impression he was teached the whole story, rather than actually lived it.
Later he was not hired. According to (trusted) others in the interview loop he did not know how to do any basic shit.
Asking about favorite projects is easier for extroverts that can bullshit under pressure than technical interviews.
Edit: that should be 'overemphasized', of course.
I don't think they are fraudsters, they are simply not too good programmers who can't solve simple problems by themselves. Many dudes believe themselves to have technical talent just for memorising done basis. That does bit imply problem solving ability.
Momentary blackouts happen - rarely.
I am about to start work as a senior engineer for Apple with 4 1/2 years of experience, after passing two back-to-back onsite interviews with them, and I hardly have studied specifically for technical interviews. I possess an MS in mathematics.
Don't presume anything about an individual just because the person happened to not answer a particular question - the person could be actually much smarter than you. It could have just been a bad day, or a number of factors.
This never happens in normal work, even when working under pressure with a deadline that has to be met and not enough time to make it.
It's just a completely different setting and purpose. In one case I'm solving a problem for the sake of being judged and that judgement may impact my whole life for the coming years. In the other case I'm solving a problem because there is a practical need for it and I'm just doing my job.
It has nothing to do with being a good programmer or not, it has nothing to do with my actual ability to solve a problem either.
Funny thing -- these folks have, y'know, actual data on this stuff:
http://blog.interviewing.io/you-cant-fix-diversity-in-tech-w...
Choice quote:
As you can see, roughly 25% of interviewees are consistent in their performance, but the rest are all over the place. And over a third of people with a high mean (>=3) technical performance bombed at least one interview.
(assuming the list happens on the same day)?
What I like about pklausler's example is that it allows for people to find edge cases without too much technical knowledge. If I have a candidate who doesn't ask about the inputs (e.g. "are the appointment sorted?"), I'd be wary. Engineers are often given vague specifications they need to clarify or account for.
Unfortunately, this can also occur because the interviewer isn't clear on their own goals in the first place. They asked the question before really knowing what data points they want to come away with.
One could even argue that relying on the order of the items in this case is going to result in a worse design overall (it's one additional thing the caller can get wrong).
I like the problem, and I'll probably use it in future :) It'll be a low bar for those times where you're getting the impression that a bigger task will be beyond the candidate, and you want to cut your losses early.
FWIW, I've seen similar complete failures to reason when applied to the even simpler problem of testing to see if a point is inside a rectangle. Some people seem to have a very underdeveloped one dimensional measure function in their brains.
Instead of this:
"Given two appointments, one that starts at time A and ends at time B and another that starts at time C and ends at time D, how can I tell if they conflict?"
More like this:
"I need to run another errand while I'm in town for the interview. I can either do it after the interview, or if I do it first, I'd better be done before the interview."
That gives me something concrete to think about so I can make sure I am on the right track and won't miss my interview! And from there it's a pretty easy path to working out the detailed formula. If I jump into the abstract description right away I'm a lot more likely to confuse myself.
B <= C or D <= A
I've always asked something that involved a loop, to ensure that candidates understood how to write a for loop.
(I have a variant of the above, a "more complicated" question, that involves maintaining two pointers/iterators; that removes another huge chuck of candidates…)
numbers.reduce((a,b) => Math.min(a,b))
Then again, you might run into a joker who has just memorized how to use reduce for finding the maximum or minimum element in a list of numbers.
If they can also write, or at least quickly approximate, the for-loop solution too, then I'd say I've gotten an excellent signal: not only can the candidate write a for loop, but he or she might also have some experience with a functional paradigm from somewhere.
We used to ask people to right a function in C that converts an array of doubles from degrees to radians. It was amazing how many people had no idea and who had sailed through CV/resume screening, telephone interview, and 'technical discussion' part of the interview with ease.
I guess the candidates could 'talk the talk' when we were asking them questions on their CV, or general high-level questions on, say, threading or design patterns; but when it came to the crunch they couldn't write code.
Maybe the candidates were just good at generating CRUD applications and binding data to forms, and have rarely needed to process data. Not denigrating that aspect of software developement at all - it's just not what we need in our department.
If someone has been doing maintenance programming in a company without excellent culture that values code quality - their brains start to rot pretty quickly.
Five years later, they know how to debug, estimate, work on a team, just not really develop anything from scratch or think critically.
If you want people to do real development, those are hard to come by. If what you really need is someone to maintain some bloated codebase and make it worse, but not too fast, then most people will do just fine.
Another thing is: what do you really want from a candidate? What led you to start asking them trivia questions in the first place?
The really good ones will be put off by trivia because it brings them zero value. If you ask questions that actually pertain to the field you're hiring for, it'll go over much better.
For example I applied to work at a major bank, and got asked a bunch of security questions, which exposed me not understanding how main in the middle attacks worked. The interviewee had requested that I go home, do a bit of research and email them the answer.
This brought them value because they found out that's an area I have little knowledge in, and it provided me with value because it pointed me in the right place to go research.
Trivia puzzles don't add value to anyone. Frankly if you're doing trivia and are upset by the response you're getting, a part of that is your fault :)
You don't want to work with people who can't solve trivia like this. Something like man in the middle can be learned, basic problem solving takes time to develop.
The places that have garbage and years of technical expect far less from their devs because they can get far less. These are also the places that prevent tickets for refactoring and never let devs pay down technical debt, then they get mad when development is slow. They value the next piece of "functionality", even if it as simple as new entry in a menu more than they value their future. When there is so much technical debt that adding that menu entry takes a skilled developer 60 hours this becomes the norm instead of a problem to solve as a business.
I would rather not hire a developer familiar with such situations and failing to fix them. Whether the failure was their ability to recognize or communicate the problem doesn't matter the problem existed and they were near it and it wasn't fixed. If I start having those problems I want people who aggressively deal with technical problems in my technical positions.
I can of course be persuaded to hire such a person if they explain what they did and they seem highly competent. Perhaps, they did do all the right things, did go above and beyond and the culture was still so toxic as to prevent progress, I have been there before. But the interviewer would need to convince me and I think that is the point of an interview.
I'm not necessarily saying that such things don't exist, but not my cup of tea.
Say the first appointment goes from a to b and the second goes from c to d. What does it look like if the appointments _don't_ conflict? This happens just when one appointment ends no later than the other begins; in other words, when b ≤ c or d ≤ a. So, the appointments conflict just when not(b ≤ c or d ≤ a). We can then distribute the negation using [De Morgan's laws](https://en.wikipedia.org/wiki/Negation#Distributivity) to arrive at this: the appointments conflict just when b > c and a > d.
You're saving a few characters to make the code less obvious. Please correct me if there's some other benefit.
If two appointments conflict, that means that there is a time when they are both active. Thus each of their end times are after both of their start times. Of course, each appointment ends after it started, so that just leaves that each appointment ends after the other started. So if the appointments are a to b and c to d, then they conflict iff b > c and d > a.
(I've corrected bumbledraven's mistake of a > d vs. d > a.)
a = 1, b = 3, c = 2, d = 4.
Do they conflict?
b > c and a > d
with conflicting appointments from 1-3 and 2-4:
a = 1, b = 3, c = 2, d = 4
won't that evaluate to:
3 > 2 and 1 > 4
true and false
false (no conflict)
I could be wrong, but I think you have to do all four comparisons as in aldarn's comment:
https://news.ycombinator.com/item?id=14641043
(no fair peeking!)
which I first understood to be a conversational statement rather than a statement of logic (which I assume GP is referring to), in which case it should be (b > c or a > d).
1. Figure out which appointment starts first
2. Check if first_appointment.end > second_appointment.start
So:
boolean AreAppointmentsConflicting(int start1, int start2, int end1, int end2) {
// first and second refer to the start time of the appointments.
// The first appointment is the one with the earlier start time.
int first_appointment_end, second_appointment_start;
if (start1 > start2) {
first_appointment_end = end2;
second_appointment_start = start1;
} else if (start1 < start2) {
first_appointment_end = end1;
second_appointment_start = start2;
} else { // same start time
return true;
}
return first_appointment_end > second_appointment_start;
}
return (start1 > start2 && end2 > start1) || (start1 < start2 && end1 > start2) || start1 == start2;
The third answer would get massive upvotes for brevity on LeetCode. The first answer would be preferable for readability in an actual code base.
But I can understand the objection to the third version too. I don't think the problem is that the expression is too simple - simplicity is generally a good thing! - but that it's easier to reason about non-conflicting times instead of conflicting times:
If my other meeting ends by the time this one starts, we're good.
Or if this meeting ends by the time the other one starts, we're good.
Otherwise we have a conflict.
So now if you like the step-by-step approach, we can write a function that seems pretty straightforward and easy to understand:
boolean AreTimesCompatible( TimeRange thisRange, TimeRange thatRange ) {
if( thatRange.end <= thisRange.start ) {
return true; // that one ends by the time this one starts
}
if( thisRange.end <= thatRange.start ) {
return true; // this one ends by the time that one starts
}
return false; // they overlap
}
// Time ranges are compatible if that one ends by the time this one
// starts, or if this one ends by the time that one starts.
// Otherwise they overlap.
boolean AreTimesCompatible( TimeRange thisRange, TimeRange thatRange ) {
return
thatRange.end <= thisRange.start ||
thisRange.end <= thatRange.start;
}
boolean AreTimesConflicting( TimeRange thisRange, TimeRange thatRange ) {
return ! AreTimesCompatible( thisRange, thatRange );
}Half the time I pick a good solution and half the time I've rushed in and picked the wrong one.
If I have picked the wrong one, I know within 30 mins that it is a flawed approach, but have usually explored the problem sufficiently to pick a good solution. The interview unfortunately ends after 30 mins.
I should also add that I'm not much better at this after 13 years of professional development than I was when I left uni, when it comes to manipulating linked lists and arrays. My systems design is a lot better though.
Unless the interviewers haven't given much thought themselves. "Hey Joe, we have to interview a guy in 20 min. and I can't remember where are those damn interview questions. Gather up a few for me, will ya?"
The dimensions of appointments are [startTime, endTime, location]. If you have points in one appointment that intersect with other appointments, then you have a collision.
It isn't though. First assume that all datetime stamps are in UTC, otherwise we have opened a can of worms that still trip experienced developers. Now draw a "timeline" and start enumerating the different cases. Then you can make a reasonable solution. But I would only expect this from experienced developers or people with a higher degree.
To which meeting does t1 belong? One, both, neither? Can we set up an event at t1 which is not in conflict of either? How about two events at t1?
if (endtime[1] > starttime[2]){status=conflict}
work? Assuming time is encoded in epoch format.
[appointment 1 start] [1 end] <-- some time --> [appointment 2 start] [2 end] (case 1 - no overlap, appointment 1 first)
[appointment 1 start] [appointment 2 start] <-- some time --> [1 end] [2 end] (case 2 - overlap, appointment 1 first)
[appointment 2 start] [appointment 1 start] <-- some time --> [2 end] [1 end] (case 3 - overlap, appointment 2 first)
[appointment 2 start] [2 end] <-- some time --> [appointment 1 start] [1 end] (case 4 - no overlap, appointment 2 first)
So you need to do:
if (appointment1.end > appointment2.start AND appointment1.start < appointment2.end) OR (appointment2.end > appointment1.start AND appointment2.start < appointment1.end) // conflict
a = Appointment 1 start
A = Appointment 1 end
b = Appointment 2 start
B = Appointment 2 end
The complete set of orderings are thus:
aAbB - no overlap
bBaA - no overlap
abAB - partial overlap
baBA - partial overlap
abBA - complete overlap of one appointment inside the other
baAB - complete overlap of one appointment inside the otherbarrkel has a thorough explanation here:
https://news.ycombinator.com/item?id=14641485
Or, as I realized later, it seems helpful to me if I look at it as a practical problem instead of an abstract one:
This actually isn't correct. Consider the events (0, 3) and (4, 6). The end of the second (6) comes after the beginning of the first (0), but they don't conflict. You want `and`, not `or`.
EDIT: oops, just saw you edited it. Good catch :)
Your notation there, where instead of a pair of start/end pairs you have a list of tagged times, reminds me of a good approach if one is doing a generalized version of the problem: given a list of N appointments, find conflicts.
Make a list of tagged times, where a tagged time is a triplet (time, 1, name) if appointment named "name" starts at time "time", and is (time, -1, name) if appointment named "name" ends at "time".
Sort the tagged time list with time ascending as the primary sort key, and the start/stop tag ascending as the secondary key.
Now to find conflicts you simply scan through the tagged times list, keeping a running total of the start/end tag values. If the running total is greater than 0 when you begin to process a given entry, that entry has a conflict with an earlier appointment, and the running total is how many earlier appointments it conflicts with.
As described above, this lets you print a list of what appointments have conflicts with earlier appointments, but it doesn't give an easy way to say which earlier appointments conflict. If you want to do that, it is straightforward. Just add a set data structure, and during the scan of tagged times add "name" to the set when you encounter an appointment's start, and remove "name" when you encounter an appointment's end. When you find a conflict, the set contains the names of all of the earlier appointments the present appointment conflicts with.
The above assumed that two appointments do not conflict if the ending time of the first is the same as the starting time of the second. If that should be counted as a conflict, just change the sort so that the secondary key is sorted descending instead of ascending.
Can I assume the times are in a sane date format/datatype? If not, start with e1 = toSaneDateFormat(endtime1), s2 = toSaneDateFormat(startime2) (Fill in if interviewer is interested).
Then, as you say, a check for overlap is easy - but maybe one wants to be more fancy, like: if (timeDelta(e1, t2) < timeToWalkFromAtoB, or < 5 minutes -- they should be considered an overlap? If they are on different continents, maybe < 24 hours should be an overlap?
At any rate, I'm guessing (hoping) this leads to discussions about representing dates, and what the business logic is (eg: physical meetings - you can't teleport from one location to another).
[ed: And as others have touched on, if you deal with timestamps/raw number types - be careful that you don't end up with appointments in "wrong" order - I'd say a sort() aware of date-objects might be your friend here.
ed2: In fact, if you can assume a sane date-type, and timeDelta, you could probably assume an "interval" type, and simply ask for overlap?(appointment1, appointment2) ... ]
In my experience interviewers are usually looking for the technical solution despite the business oriented solution usually being much more applicable (and thus relevant) in the day to day role.
Key thing to remember here as an interview candidate is to clarify with the interviewer the scope of the question and the nature of the answer they're looking for. If for instance the interviewer starts with the simple technical solution and then probes the business aspects this might be a nicely rounded question.
e.g. in J:
'a b c' =: 50 60 3 4;3 50 49 99;2 10 10 12 NB. 3 different sets of appointments
3 :'ok`nope{~0>*./-~//./:~_2]\ y' every a;b;c
┌──┬────┬──┐
│ok│nope│ok│
└──┴────┴──┘ overlap = a.start < b.end and b.start < a.endMother of god
It is as you put: depressing as hell. And this is among white collared, educated, demographics. Imagine how it must feel to be an uneducated demographic in most other parts of the world...
This is only a problem if the interview/hiring/capitalism process is meant to be fair.
> Imagine how it must feel to be an uneducated demographic in most other parts of the world
The less privileged here certainly had jobs in HS and college where you just filled out an application, the manager made sure you weren't a convict or on drugs, and you got the job. The "uneducated demographic in most other parts of the world" is probably not solving algorithm quiz questions on a whiteboard.
It's also a problem when those same companies want to bitch and whine about a "talent shortage" or a lack of diverse candidates.
so much this. Why do companies get to receive all sorts of incentives to hire when candidates they reject walk into other great tech firms, make bank, and generate boatloads of value?
Maybe some should just pay more, too. There are plenty of firms just being too cheap to hire great talent, including some large, well-known ones.
How many operations can a modern CPU perform per second: A) Thousands, B) Millions, C) Billions
We'll accept C, and B with explanation. I'd say roughly 75% of the people I've asked (who have gotten through a phone interview) cannot answer it with any ability.
People whine about the difficulty of interviews, but honestly, almost everyone we've ever hired have said the interview was pretty straight forward, and nervousness is the biggest issue. Sucky people whine about reversing a linked list or finding a cycle or whatever.
But this kind of thing may only be mentioned in one sentence in an intro computer systems course, and most young people wouldn't care about how many operations a CPU can do.
I also realized how different schools emphasized computer architecture differently for CS graduates. I remember having to design a simplified PDP-11 in Computer Architecture. We had to take assembly language and such. But that was one specific college and I shouldn't assume all of them do that.
Eventually I moved away from such questions and took a more collaborative approach such as "let's solve a problem together". So I'd stand beside them at a whiteboard and we think through problems. I think that puts candidates at ease and it reveals more of the skill and personality traits that were relevant for us.
No Overlap -- EDa < SDb OR EDb < SDa
Overlap -- SDb <= EDa AND SDa <= EDb
I have this written on a PostIt in my binder.
The solution is a simple predicate. Assuming both events are internally consistent (i.e. end time after beginning time) they won't overlap so long as the beginning of the first comes after the end of the second OR the beginning of the second comes after the end of the first.
Seems like a good weed-out problem.
Disclaimer: no prior experience with the problem.
Trial employment is expensive for the company.
[...]
Trial employment (and large take-home projects) are expensive for the candidate.
I can't help but think there's way to mitigate that expense among several (dozen) companies.
Triplebyte is already interviewing for other companies, why not set a low bar and hire everyone who passes it for a week? Charge your clients accordingly. If you have 25 clients hiring for overlapping skills, have them all pay to hire a person for a week.
40 hours of of labor / 25 clients = 1.6 hrs of labor per client, or about what it costs to do a single technical interview.
Meanwhile, the interviewee gets to do a single, focused project for a week, but effectively interviews with 25 companies!
I can get on Dice right now and find over 100 openings for python/java/golang/javascript/c++ engineers, certainly all of those companies would benefit from something like this.
More power to Triplebyte if they can figure out how to take those projects and turn it into a client deliverable on top of defraying interview costs.
I recently went through a round of employment where I quit my job at the beginning.
Between updating my resume/social networks, brushing up on academic CS, finding leads, scheduling interviews and follow-up interviews and managing/negotiating offers, it was absolutely a full-time job.
I can't imagine finding a new programming job while still working at the old job.
> it be weird to take a week off just to work for another company
If trial weeks become commonplace, it starts looking like a form of structural lock-in.
Isn't this basically what bootcamps are about? (at least from a job seeker's perspective)
In practice, only those without jobs have the time to attend one... and if one isn't actively working in the field, then 2-3 weeks might be better than 1 week.
Surely a company could objectively learn a great deal about a candidate by looking at a week's worth of git commits and evaluating the final product for correctness/maintainability/optimization.
But still, taking TOO much of the humanity out of hiring ignores the fact that ultimately, a person has to work with other people, so you're right that it's not perfect.
It seems to me like an enormous amount of time wasted for both interviewers and interviewees.
When considering someone for a development/engineering role there are a few basic questions to figure out:
1. Does this person have competence within our technology stack?
2. Can this person communicate clearly?
3. Is this person productive/can this person be productive in our environment?
4. Does this person have characteristics that will allow them to succeed in our environment?
Making people jump through hoops like some sort of dancing monkey while they are on a job hunt is needlessly cruel and is a waste of everybody's time.
If you regularly see people who can't succeed with what you consider basic engineering questions, you either have a recruiting problem or you yourself have a communication problem.
If you expect an interview candidate to expend 8 hours or more working on interview tasks, you should be presenting a unique enough opportunity to justify it.
If you expect an interview candidate to whiteboard a solution to a coding problem that they received in the room, that should be part of your everyday work experience and you should go through the process yourself in front of them so that they have an understanding of what your expectations are.
If you want to review their code and talk with them about it to determine competency, provide an example of your own code and a set of questions and answers that would be successful.
Don't put engineers through a gauntlet of challenges that you would have trouble successfully navigating under the pressure of feeding your own family. They have more than one company they are interviewing with and each company has their own focus. Put them in a position to succeed and give them the chance to do so. That's what you want out of your own people - the ability to succeed given a reasonable amount of guidance and clear parameters for success. If you can't provide the guidance or the parameters for success for an interview, how can anyone expect to succeed in your actual working environment?
A distant acquaintance interviewed for a secretary position. It didn't go well, because she was asked a general knowledge question (something along the lines of: "name some of the planets in the solar system"). She was furious with how irrelevant and unfair this was. It makes me wonder though, would I have hired her? I probably wouldn't have asked that question, but still. When someone is missing some really basic knowledge - even if it's irrelevant - it's not a good sign, per se.
These are part of the warm-up, I'll ask one or two right at the beginning of the interview and then move to the real questions.
Examples are: "How does the autorelease pool work?" (Objective C before everyone switched to ARC.) or "How do you ensure that an object is garbage collected," (C# and Java.)
These are more about judging a candidate's real knowledge versus stated knowledge. Someone who's spent 8 years in Objective C better know how to use the autorelease pool; and someone with at least a year of experience in a garbage collected language better know how to ensure that an object is collected.
You can call `GC.Collect()` and hope. But far as I know the GC will do a best effort and gives no guarantees.
Of course that's besides that fact that I would need strong arguments to accept a `GC.Collect()` anywhere in the code. But to call it just to ensure an object is collected? Oh boy..
Depending on the case you could add some code to the `Dispose(bool)` method to know when your object is collected, but also in that case you risk flipping a Bozo bit.
Similarly, if a programmer fails the question "in the alphabet, which letter comes after A", I'd be worried, and rightfully so. I'd give them a chance to recover from that, just to make sure it wasn't a once-off, but it's not excusable.
Please keep in mind that resumes nor any other data from the candidate can be trusted, because a huge amount of candidates lie or are mistaken about their own skill. I once interviewed a candidate who claimed to have 10 years of SQL experience but couldn't write a query on the order of "select * from table_name". Fully half of all people I have interviewed failed to convince me they could write anything more complex than "Hello World".
I am not advocating hours and hours of needless hoops, but a candidate for just a about any programming job should be able to Write a working fizzbuzz implementation. Questions on the order of fizzbuzz can and do eliminate a huge number of possible candidates.
That's not a problem that is unique to software, yet nobody asks directors to prove their budget forecasting skills during an interview, or has their tech writers go through the process of building automated indexes or varied pagination.
If you suspect someone can't write fizzbuzz, send them packing. Don't waste their time or yours. They've already instilled a lack of confidence which would mean that they would be facing adversity on day 1.
After my experience interviewing it seems reasonable to expect that no one can write fizzbuzz and the only way to know otherwise is to have them write it.
Why are you so opposed to the idea of meritocratic interviewing? Why not check a few things that can be easily and objectively checked? Have you ever interviewed people?
We aren't business people: we do actual productive things.
In an interview? Not that I've ever seen. Reviewing a portfolio is not what we're talking about.
> Chefs might be a good model too; it absolutely the case that before hiring a chef they have to plan a menu and cook.
If a chef is expected to plan a menu and cook as part of their interview process, they are going after a very high end job. They know the kind of food that the restaurant serves. They are provided with the tools, equipment, and ample prep time to properly consider their course of action. And let's not pretend that every chef has to go through a live cooking trial for every job, or that most high end chef's would not throw a hissy-fit and walk out the door if you assumed that they would show up to an interview ready to plan a menu and cook a meal.
--- "what? You've never heard of fizzbuzz? It's a simple game that..." cue a terrible explanation of the game missing key requirements that the interviewer expects the interviewee to just know.
So now the developer not only has to come up with a set of code under pressure, they have to do so with a set of unclear requirements. And what the heck, why not take away their daily development environment to ensure that they are more uncomfortable and let's have them do it on a whiteboard so they have no hope of resolving simple syntax errors that arise because this particular developer floats between four different languages at his current job.
What you see as a game to weed out a littany of liars and cheats is seen entirely differently from a job-seekers perspective. "I have 15 years experience building complex systems and this jackass wants me to write kiddie code on a whiteboard to prove what, exactly?"
I'd rather not work for somebody who operated off of the assumption that I was a liar and a cheat upon our first meeting. The only reason people subject themselves to this kind of treatment is because they _need_ a paycheck.
hire specialized contractors to do the things you need done.
ive never been asked stupid fizzbuzz but have so far not been able to sufficiently fail at a job. weird.
- Don't ask them to whiteboard, especially don't ask them to whiteboard some absurdity
- Don't put a panel of 5-10 people against a single candidate, but if you must absolutely (why?) do not do it on the first interview
- Don't ask time-wasting stupid questions about manhole covers, boiling bagels on mars, overlapping clock hands, or how many eggs fit into a phone booth
- Don't put someone into a socially awkward situation
- If you ask a knowledge question and the response is something like "I don't know, but I would google it" that should be acceptable if they are demonstrably capable of using what they find
- Do ask them to walk through their own projects, their code, their past work/contributions, etc
- Do try to effectively convey the corporate culture so the candidate can know if they are a good fit or not ahead of time
- Do try to accurately convey what their actual job duties and day-to-day work will be
"Well they're a great candidate otherwise but they totally failed the boiling water on alternate planet question, so let's keep searching" - believe it or not, that's a real mentality out there. I don't get it and never have.
I had a phone interview with Google and was asked to come up with a variation of a soduko solver. Couldn't get it there and then, but half an hour later walking down the road, a really elegant solution came to me.
As long as you make it clear that the actual number they come up with is irrelevant and you just want to see how they think (and you're also open to pragmatic answers like "why the hell would I be trying to calculate the weight of an Airbus A380 when I could just look it up on wikipedia?") I don't see why it's that bad a question.
Subtext: "Did you create a famous open source tool, write a book, have patents in your name or architect some amazing system at a big company? Doesn't matter! Our hiring process prefers code monkeys who can solve our puzzles instead of system thinkers."
> An interview can result in a bad engineer being hired and later fired (a false positive).
Subtext: "A hire can either be good or bad. It's never management's fault if it doesn't work out."
I agree that the article gets this wrong. It reminds me of Diego Forlan's time at Manchester United.
Diego Forlan is/was an Uruguayan striker at Independiente (Arg) where he scored 36 goals in 77 games (a very good rate). He then moved to Manchester United (Eng) in January of 2002 and was anticipated to take off. He made 18 appearances in the 2001-2002 season but failed to score. It took him until October to net his first Premier League goal.
He was loved by the fans as he has/had a great attitude and had some delightful near misses. He also famously scored twice against Liverpool[1]. But he continued to struggle at United, scoring only 17 goals in 63 appearances until he was sold to Villareal (Esp) in August of 2004. In his first season at Villareal he won the Pichichi Trophy (golden boot for La Liga) and the European Golden Boot. What a turnaround!
Forlan went on to have a terrific career and is considered one of the best Uruguayans to play modern football (soccer). His time at United was certainly a misstep but it wasn't because he was bad. And it wasn't because United were bad. It just didn't work. Sometimes it just doesn't work.
It really does draw parallels to other jobs as you say. I recall a company I joined that had a great mission, interesting work, friendly team and lovely atmosphere but something about it just didnt gel with me. I still have no idea what it was but I guess sometimes it just doesn't work.
I suspect candidates that wrote the Linux kernel, Rails, Modern Effective C++ etc... don't need a recruiters services. If whatever candidate wrote is not so big as to land them jobs almost automatically then was it really that big as to demand consideration separate from college degrees and other ignored history?
I think we dislike the idea of our history being ignored because we all feel that we are special, but in reality very few of us are.
How would one know? Usually we know a friend who works somewhere and they say the company is shit. Or you work somewhere and someone doesn't work so you end up thinking they're shit. It's not often that you have both sides of the story.
>very few wrote books,
Books might indicate a good communicator and independent ability to complete a task but it's not an indication that someone is a good team member that fulfills a particular role.
>verfy few created a foundational open source project, etc...
I contend that this actually risks being a Balotelli. Good in the right atmosphere, but very independent and often stroppy when they don't get their way. They can be terrific in their element but often they're not good enough to build a team around.
If both the interviewer and interviewee have overlap they can discuss then the interview is most productive.
Well yeah, there's that unfortunate fact. The problem is, a lot of interviewers can't tell the good from the bad -- and it can only take one bad question to effectively sink the interview.
Every time I hear that question, I die a little inside. I'm not sure it's possible to answer it honestly without taking yourself out of contention.
"Well as I'm interviewing here I would obviously see taking this position as a step foreword for all the reasons we've discussed. Naturally I don't see this position as the job I'll work until I retire, and obviously I'll be looking for at least some mild expansion of responsibilities in the future. Whether that happens here or somewhere else in 5 years remains to be seen, but staying here is certainly an option depending on how things are going after 2-3 years. On that note, exactly how are company name's technical and managerial tracks structured..."
In that paragraph I've been honest, I've let them know that I don't intend to do menial labor forever and have aspirations, the minimum amount of time-loyalty they can expect from me, an assurance that I won't job-hop, and I've shown further interest in their company as well as the initiative to ask questions of my own if I haven't done so earlier.
Edit: Reading it here may sound like I want to be a manager in 3 years (which is crazy for a junior dev), but that's where manner and vocal tone come in.
Obviously shouldn't answer and say 'I want to work at Major Competitor X, doing Y', but being honest is good for everyone involved. Everyone knows people stick around for maybe a few years and move on. 5 years is a long time, and you may not be (probably not) working for the same company, but its a question about character, motivation, and goals. Not company loyalty or some sort of trap question.
It depends on whether you are interviewing the job or the job is interviewing you :)
I would argue that not knowing answer and not knowing question are two different things. I may not know answer to a questions, but know that the question exists. I personally am comfortable not remembering things that I can easily look up.
That seems really weird. Academic CS isn't really necessary for most programming jobs, but I can't see how it would ever be a detriment.
Great programmers should hopefully recognize and implement a proper balance between practicality and rigor, as necessary. They should also exhibit self-awareness and self-restraint, which sounds like the real problem for the guy you mention.
Those people were just getting a bit better at building things. They were building experience. It just so happens you found their bad code that they'll realize was terrible a few years from now.
We can't simultaneously hate our own previous code and expect everyone else to delivery perfectly when they first start (or even at mid-level).
A recent example: https://news.ycombinator.com/item?id=14267555
There's also an immense cost to not having CS fundamentals in terms of performance and wasted time; I don't want to pay someone to try and solve a version of the halting problem, or matching opening and closing delimiters with regexes.
Conversely, I've been hired, without a formal interview, twice merely for having gone to CMU. I guess it can work out either way.
In senior positions CS knowledge is less relevant day-to-day, because your experience overrides a lot of that theoretical stuff. However, whenever I apply to companies, the questions about basic CS usually result in me turning down interviews. It would be a poor use of my time to memorize that stuff again only to not use it beyond the interview.
I had a really bad experience with this recently, where I had a final interview which went really well in 3 out of 4 interviews. In one of them the interviewer was expecting a very specific solution, and I believe that I said something like, sorry I don't exactly remember how tries work, and I believe that is what resulted in no offer.
At first I was disappointed greatly, since it seemed like a great opportunity, and I had a great interview and experience with the other 3. But in hindsight, I realized I would really not enjoy working with this person every day.
In my experience, people are often hesitant to hire people who are higher on some (perhaps tacit) measure of nerdiness. Only when people want to win badly enough and it's clear that others are doing better, will many people start questioning such biases.
Depending on context and delivery, overfocus on academics could also be interpreted as snobbery, giving an impression that the candidate is difficult to work with, or that they think less of potential colleagues who don't have the same formal training.
Knowing how to code quicksort or reverse a binary tree isn't, but an understanding of normalization definitely is...
- it works well sitting around a whiteboard, which is a nice friendly dynamic that emulates how we really work
- it doesn't require detailed/arcane recollection of specifics, is more conceptual. Who wants to fail someone because they don't remember a parameter or some specific details of how TreeMaps work?
- it can't be faked - no-one can talk their way through a database design discussion unless they really have an aptitude for it
- its interesting. Who doesn't want to spitball around, say, modelling what Uber's database might look like behind the scenes.
I would say there are plenty of decent programmers around who can't (or haven't) done any real database design work, but if its a core skill (SaaS app development) that you really need, then its an easy one to detect at interview.
Remember that FizzBuzz was originally proposed as a response to the number of CS degree holders who allegedly could not code their way out of a paper bag.
I don't care if someone can derive the hierarchy of Turing degrees from first principles. I care if they can do the job I'm trying to hire for.
I imagine they correlate an over interest in CS concepts with said group.
After 1-2 technical phone screens, we send applicants a coding challenge for which they have 48 hours to complete. The coding challenge is representative of some of the work we do (e.g. Given our API, create a D3 visualization of X) and should take applicants several hours to complete.
In submissions we look at everything from overall program structure and approach to applicants' attention to detail and usage of good industry practices.
This coding challenge quickly weeds out applicants that interview well but code poorly and is an efficient way for us to see the quality of work we can expect from them (and conversely, the type of work they can expect to do with us).
After a successful challenge, we bring applicants in to meet the team. At this point, it's largely just about culture fit.
The 48 hour time limit is a relatively arbitrary timebox and isn't a hard limit; we value quality over speed.
I check my ego at the door, I try to make the person as comfortable as possible, small talk, help them out as much as I can, provide them with answers or explanations when they get something wrong. I've never asked an algorithm or data structure question, I've rarely asked a patterns question. Most people can't answer basic every day questions.
Overtime my goal in interviews turned into making sure the interviewee left with more domain knowledge than when they came in hoping it will help them do better somewhere else.
Ended up creating a pre-screening service that asks the same basic questions and in the end provides the user with a study guide based on how they answered. Currently piloting with two local recruiting companies and I'm finding the same similar statistics of failure/success. Maybe that makes me the issue :)
Strangely enough I warned candidates days before the interview of what I was looking for and still almost nobody bothered to brush up on the couple of basic methods needed to do this 1 task.
If you have a 90% rate of "utter face palming failures", the problem isn't with the people you're interviewing, it's with the interviewer and/or the interviewing process.
Resume says 'I'm an expert in SQL'. Great, lets start some every day foundational questions. What is the difference between an inner join and an outer join. Why might we use a varchar instead of a char data type? Why do we use indexes? What is the purpose of a foreign key? A good majority of the candidates can only answer the join question.
'10 years experience, senior dev in <main language>' can you write a function that returns the largest number in an unsorted array? Most struggle to even pseudo code it. Tell me something, anything about interfaces. Basic security question on SQL injection/xss/csrf/hashing/encryption. Without a doubt most have only heard of SQL injection and then they get that wrong. Most say hashing and encryption are the same thing.
I do blame myself, the failure rates are incredibly high. But when resumes looks great and onsite can't pseudo code a simple loop, or answer basic questions about something they claim expertise in, what can you do? Require and call references to make sure the the lead dev with 12 years experience and decent companies isn't lying? Because the result of most of my interviews appears that people inflate their resumes and flat out lie.
I ask this because some of your "foundational" questions are not as simple as you might think (I could talk for literally hours about XSS and the only conclusion I could give you would be "there's no sure-fire way to prevent it", for example), and others border on pop-quiz material, which is not really a great way to evaluate someone.
Another question that's important to ask: you say you're working with recruiting companies. Have you tried not doing that? Recruiting companies don't exist to find you qualified people, they exist to spam you with résumés. Recruiting companies have been known to flat-out alter résumés to make them better match the set of keywords you said you wanted. And my own experience is that recruiting companies are the source of a huge percentage of headaches in hiring. Cutting out the middlemen can and likely will drastically increase your success rate.
Even among people who did a degree that involved implementing these algorithms, 99% or more will never do it again once they set foot off their campus, because they'll just use a library that already has it implemented. Then they'll free up room in their brain's working set for the stuff they actually do have to implement. And the longer they go like that, the longer it'll take and the more difficult it'll be to find where all those tree algorithms got paged out to in their brain, and the worse they'll look on an interview which consists solely of asking for that stuff.
Meanwhile, the inexperienced and likely unqualified (given what people claim about the average CS graduate) person who just walked out of their college graduation still has it fresh and ready to regurgitate onto your whiteboard, and passes with flying colors: great technical skills, great "fundamentals", A+++++ hire ASAP!
The solution to this is to stop using proxies for the thing you want to know, and start actually testing for the thing you want to know.
If a company were to offer a base salary at a mere third of the Google base salary for the same job, why should an applicant have to copy Google's depth of computer science abstraction in the coding interview?
Is the onus on the developer for even entertaining such a demanding question at a low pay, or does the current economy favor employers in such a way that developers have no other choice? Or rather, is the knowledge of available jobs asymmetric enough to where employers can pay pennies, piss, and peanuts for A-level employees?
EDIT: But during the entire process keep in mind they are hopefully interviewing your company. Far too many companies forget this aspect and the entire process has become lopsided in favor of companies making candidates jump through hoops.
Also I learned to be more picky. "Want to see my code and apps? Show me your code, show me how you work!" Eclipse required? Bad workflows with weird limitations? "This is how we do it here, it works for us" attitude? High discipline environment with daily standups? Cheapskate workplace/outsourcing center? Sorry, I'm passing.
Can anyone from any of these companies confirm this? This strikes me as really odd. So only one employee from FB, Apple, Dropbox, and Stripe needs to interview a candidate that Tripplebyte say is a "Go"? I'm having a hard time believing that.
I'm considering using triplebyte, but also want to control where my applications are sent to.
I would argue that as a candidate I would rather interview with as many of my potential coworkers as possible. Interviewing is a two way street. The more exposure to the actual people I would be working with allows me to make a better-informed decision. How is reducing my exposure better for me the candidate? It seems its really better for Tripplebyte though and maybe the company looking to hire if they are short-staffed but not the candidate.
I think the idea is that if you do the first round with Triplebyte instead of with Apple, Facebook, Dropbox, or Stripe you are effectively doing the first round with Apple, Facebook, Dropbox, AND Stripe.
So now I need to start from scratch to get all of the information I'd need to make the decision of what position I am interviewing for fits. What makes this doubly difficult is I've been "fast tracked" to the interview process where those discussions aren't expected.
The more I read about triplebyte the more I think their heart is in the right place but they are creating a market for lemons.
If triplebyte can create a system where the final interview is a cultural fit interview of no more than 2 hours and both sides feel they have enough information to proceed, then they are onto something.
Any thoughts on that?
I ask things like, "what's the most interesting bug you've encountered?" or "what have you been excited about learning recently?" to try to find a topic we can have a conversation about in a domain they are familiar with and interested in.
How has it played out when you've done that?
I agree with this. Better yet ask a multi-part question that has an easy first part and a harder follow on.
And this goes doubly true for any company that reaches out to me first. Like, if I apply to your company, then yeah, you have an expectation of me being interested in the company. But if you reach out to me, then maybe you're the one who needs to sell yourself to me.
I don't really get that attitude by employers. Whenever I'm interviewing candidates, I really try to sell them on the role. I try to tell them what's awesome about it, why they should want to work here over literally anywhere else in the world, and why you WANT to sit next to us for the next six months and work with me and my team on projects. Part of my job as an interviewer is to not only gauge your ability and interest, but to convince you that we're the best place for you to be.
Or... do interviewers really think I selected their company is the only company I even considered talking to? We're engineers. We have options. You (the company) have to meet me at least halfway.
Incidentally, I've often wanted to ask my interviewers to solve a tech challenge. I don't want to join a company where the team is incompetent and I'm constantly cleaning up after them. Oddly, that's never a consideration that companies allow candidates... ;)
No "reject" decision can ever be reconsidered or questioned by anyone, ever.
First, we defined the skills we're looking for i.e. Programming / SysAdmin / Cybersecurity.
Our process goes as follows:
1. We ask candidates to answer a quizz by phone, with questions in the 3 chosen fields. Duration = 1h.
2. We ask candidates to solve remotely with Google Docs 5 real-world problems asking for skills in Algorithmics, Data Modeling, Object-Oriented Programming, Software Design, and Parsing. We also add a bonus question like "how many people can get into a train". Duration = 1h.
3. We have a HR meeting at our office with a non-technical member of the team. Duration = 1h.
The quizz helps us to remove of our list the candidates that do not have the technical culture we're looking for, and to see what they already have worked on.
The 5 problems helps us to see how the candidates can code, the bonus question helps us to see how they approach new issues and manage their stress.
The final HR meeting is very typical: we try to see if we would be happy to take a 6 hours long flight with the candidate.
Most candidates do not go through first step, and roughly half on them do not go through second step.
This simple process definitely helped us to reduce the time spent on hiring and to make the really good candidates shine.
You ask trivia questions for a whole hour? That seems pretty gratuitous. I think most people would tire of that after about 15 minutes.
>"We also add a bonus question like "how many people can get into a train". Duration = 1h."
You ask Fermi questions? Between an hour of trivia questions and one "bonus" Fermi question your process sounds pretty horrible to me.
Interviewing candidates is a two way street. There are many candidates who would not want to work for a company that thought asking a candidate an hours worth of trivia questions and a bonus Fermi question was acceptable. For many this would be a red flag.
Can't speak for everyone, but I'd turn away and walked out the instant I hear this or similar BS (which is totally unrelated to "approaching new problems" or "stress management" in programming at least)
I can explain what everyone of them is, and why you'd want to use them. But I've never written any one of them and I'm not going to spend the time to memorize something I can look up in my CLR book or stackoverflow.
It's embarrassing that tech interviewing is still stuck in a 1990s mindset.
So you'll either have them built in to the language you'll be using, the company's using the wrong language, or you'll be tackling problems where they're not really needed.
Google Docs are horrible coding environments. In fact, anything besides the candidate's native editing environment makes for an obtuse and off-putting experience, all around. I mean, yeah, the candidate can suck it up and make it work if they felt they "had" to. But on balance it's just unnecessary mental gymnastics -- and completely avoidable source of awkwardness and all around unpleasantness in the interview experience.
And how would I use that, as an interviewer, in a phone screen, realistically?
(I don't believe that I could — reliably — get them into a Google Hangout, for example.)
(We used https://coderpad.io/ ; it was alright, and certainly better than Google Docs.)
Lastly...why are the engineers coming up with the hiring process anyway? Is this common in other professions?
Candidates would be watched, would know they're being watched, but would not see themselves being watched, which would probably take a lot of stress out.
You could even test many candidates at the same time this way. And you could setup a system where each candidate could ask questions if needed, without being heard by other candidates (like in language learning classrooms).
This must have been tried? but I wonder why it's not even discussed in the post, as an alternative?
You're sat in the conference room, given a machine that has a screen monitor program running, and a set of tasks to perform on the code. You are told where to find the code in the introduction on the sheet of instructions. You are allowed to use any resource via the computer (so nothing you've brought in like a phone or laptop is allowed; basically, all work has to be "shown"). Then...go.
Outside the room, the dev team watches the process. We want to see how you go about each problem, what resources you use, etc.
It's always surprising the number of people who go for an hour or more and do nothing. Just sit there (or seemingly sit there - there's no keyboard or mouse activity).
When I interviewed, they actually gave me an online test; the job was supposed to be for javascript, but the online was in PHP (which my prior position used, and I had been using for years). Passed that, and came in for the on-site part. I thought it was odd that they first tested me for PHP. Did a bit of face-to-face interview, then they wanted me to do the watched javascript portion (single page app).
I first sat and read the problems, formulated some possibilities in my head, then jumped in. 30 minutes later I was done, and popped open a new file to write something to that effect (figuring if they were watching, they'd come back in). I sat for several more minutes, then they came back.
"How's it going?"
"Alright...umm, hey, I'm done."
"No way! Are you serious? Are you sure?"
Odd...
"Yes."
Apparently they had never had anyone finish the test that quickly, and get it completely right - especially someone who also had PHP skills, and had already demo'd them in the first test, which was also done correctly. I found this rather odd, as neither test was that particularly difficult.
Apparently, though, it's not uncommon in our industry. I'm not sure what to make of that...
Personally, I feel just as much stress knowing that the interviewer can review all my typos and mistakes in minute detail, leaving me to write code in my local editor and simply copy-paste working sections into the provided editor.
Amazon made use of a service similar to what you've described at one point, however they required access to your machine to verify you weren't "cheating" in the interview. You can read one of the (many) threads here:
This is a strict specialist search - here, too you should make a decision depending on your requirements and future outlook: Do you need a generalist that can branch out and specialise, or strictly a specialist that only does the thing you need right now?
(Also let's not argue which is better, specialist or generalist - deep down, each of us already knows and made the choice ;) )
Or you can ask someone to beat a Go bot that is ranked over 5 kyu.
Or you can ask someone to find adjacent zeroes in a matrix of numbers in under 15 minutes.
All those exercises, however tricky, do not correlate much with how smart you are or what your programming abilities are.
What on earth does that have to do with building (actually, in most cases, maintaining) a CRUD web app?
What does that have to do with n-tier web application architecture?
Keep in mind writing code is the BARE MINIMUM required for a job. If they can't do that, they shouldn't be on-site in the first place. If you have your candidate taking a day off work and taking a day worth of engineering talent off engineering-related tasks, and you're spending that time asking them to write code, you've failed to apply the filter higher up in the funnel.
Ideally, you verify their coding ability BEFORE you bring them into the office, whether via a tech phone call, live-coding session, or take-home assignment. Before the haters start saying "but they can fake it!", hold your horses. Once you've established that the candidate can code, when you bring them on-site, you quickly VERIFY your previous conclusions. You quickly VERIFY that the coding assignment you gave them was completed by them. This should take no more than 15-20 minutes. Maybe you just have them walk through a chunk of code. Maybe you ask them to recreate their answer. Something. Anything.
But if you send in three different engineers, and they all spend the entire hour asking the candidate to solve programming tasks, you're literally evaluating the BARE MINIMUM of the hiring requirements. And there's so, so, so much more to being a good engineer than writing code.
Outside of a QUICK verification, your on-site questions, imo, should NOT be writing code (or even pseudo-code). They should be about previous projects, architecture, personality, professionalism, punctuality, communication, teamwork, autonomy, and collaboration skills -- all the non-tangibles that answer the basic question of "Do I want to sit next to this person for the next 6-12 months?"
If your idea of a good interview is to ask someone to generate a calendar between two dates or sort a list or find the minimum integer in a list, you're completely, totally missing the point. You're evaluating the bare minimum to get them in the door -- and if you're doing that in-person, you're wasting your time. You should have filtered that out before they walked in the door.
Remember, these candidates are often lying to their bosses and calling in sick or using up valuable PTO to take an entire day off work to come sit in your office. Don't waste their time with redundant tests and things that you could have figured out before they walked in the door.
(And dear god, some of the questions in these comments are... awful... at evaluating anything except, I dunno, getting the right answer. Yikes.)
The ability to spell is one of them.
> Ask questions as close as possible to real work: This is achieved by making interview question as similar as possible to the job you want the candidate to do (or to the skill you're trying to measure).
This "or" thing in parentheses is really important. Interviews should not be "as similar as possible to the job." They should be all about the skill you want to measure. And there's quite a difference between these.
People can be trained, assuming you have time and resources.
The point of standard algorithm interviews is to measure a particular skill: intelligence. The belief behind this is smart people will tend to solve problems well in the real world. It's okay if they don't know all the skills they need; they can learn them. (And if there's a particularly challenging skill, then maybe you need to add that into the process.)
> Avoid hard questions: If a candidate solves a really hard question well, that tells you a lot about their skill. However, because the question is hard, most candidates will fail to solve it well. The expected amount of information gained from a question, then, is heavily impacted by the difficulty of the question. We find that the optimal difficulty level is significantly easier than most interviewers guess. This effect is amplified by the fact that there are two sources of signal when interviewing a candidate: whether they give the “correct” answer to a question, and their process / how easily they arrive at that answer.
You don't give examples of what qualifies as hard vs. easy, so it's a little difficult to judge this. But generally, I'd advocate for people asking harder questions.
When a question is easy, a little thing -- small point of confusion, etc -- can make a big difference in whether the candidate seems to have gotten to that answer well or not.
When a question is hard, this is when you can actually see a great differential between great vs. good vs. okay. You can offer help and see how effectively they pick up on that advice.
If your interviewers are just throwing out a question and then sitting back and seeing what happens, then yeah, candidates won't be able to tackle a hard question. But that's not what they should be doing.
To be clear here: hard does not mean "ask about some obscure data structure [skip lists, etc]." It means a challenging question involving common knowledge (hash tables, strings, arrays, maybe binary search trees).
> 5. Ask every candidate the same questions: [...] there is no justification for asking different questions to different candidates. If you evaluate different candidates for the same job in different ways, you are introducing noise.
I think this is overplayed a little.
If you're doing algorithm-style interviews, any question should be basically evaluating the same skill (problem solving skills). If I ask my favorite question and you ask your favorite question, but they're evaluating the same skill, then it doesn't really matter. There's nothing to make one question better than another. Why not let each interviewer ask the question that they like?
Now, if every interviewer asks a different question, you won't be able to give a well-defined metric on what good performance looks like for that problem. But that's okay. You can't really do that anyway.
When companies give well-defined metrics, they get too rigid. The candidate takes slightly longer to complete the code because the candidate thought things through more thoroughly, and now they're pushed into a lower performing bucket. And that's wrong.
These standardized metrics sound good in theory, but they don't work well in reality (for algorithm interviews, anyway).
For example, my guess before running these experiments would have been that simply looking at progress (how far a candidate gets through a problem) would be a bad measure of interview performance. I'd expect that things like style and how communication how careful a candidate was being would render pure progress a mad metric. However, when we compare pure progress to a subjective score the interviewer gives the candidate (ignoring specific reasons, does the interviewer think the candidate is a good engineer after the interview), we found that pure progress is more predictive of success at companies! (To be clear, we don't only look at progress at Tryplebyte. We've also found other things to be predictive.)
The same is true (among our candidates at least) for question difficulty. Easy, straightforward question (write a command line interface to store and retrieve key-value pairs) are more predictive of success at companies than question that try to target intelligence (calculate how much water would collect in a histogram)
Additionally, what matters is not natural talent but the set of skills and techniques an individual has built up using those talents, compensating for their weaknesses and taking advantage of their strengths. I've met plenty of very, very smart people who flounder the first time they encounter a code base they can't hold entirely in their head at once: someone who was less "smart", with a smaller working memory, but who has been developing skill with abstraction and system metaphors since CS 101 is often a better actual developer.
The myth that developers need to be "smart" is pernicious, and the cause of most of the really horrific code bases in this industry.