What every PhD should know before interviewing with a tech startup
blog.routific.com
blog.routific.com
Defensive anyone?
Notice the assumption here, that their 20-minute task is a perfectly reliable indicator of how the interviewee would behave in general.
The interviewee was right to leave them without saying a word.
God I get sick of the smugness. You can cut it with a knife.
I think the interviewee did make a mistake during his interview, in that he didn't read the room. They weren't looking for anything "cutting edge" and they know that they can't understand the in-depth and complicated work you did for your thesis. So what the interviewee should have done was focus on simpler and easier methods to start off with and kept it general. They already know you're smart from your degrees and from your CV, the focus of that segment of their interview was to see if the candidate had the communication skills available to express their ideas.
We have the same meetings with our stakeholders and during our project meetings. I don't explain the complex methods and algorithms used in-depth. I just keep it general and tell them how everything fits together (if I have performance indices then I also add that in to show accuracy). If the stakeholders want more detailed information like specifically how it breaks down and why I used X method instead of Y, then I can answer their questions and (if they showed continued interest) be willing to put together a report for them detailing the literature reviews and information related to that topic.
20 minutes isn't a reliable indicator of how someone can do their job. However, as the interviewee or the person in the "hot seat", it is your job to know how to sell yourself and to use your time effectively to maximize your chances of getting an offer. The decision makers (or in this case, the interviewers) has the hard task of figuring out which candidate is right for them. Your job is to make that easier. Once you get the offer, then you get to decide if you want to accept it or not.
And then after you get the offer, you reflect on your experience with that company so far. If the interviewers seemed to be focusing on different things than you're interested in or if it seems like they're going to be miserable to work with, then you politely decline the offer and move on to the next thing. If it seems like they have a condescending tone towards you and your work, then you can always just say "thank you for your offer but I'll be declining".
Here are some details that were cut out of the post:
- we have an extensive engineering interview process, typically 3-4 sessions. First chat, tech screen, take home assignment, and half day hackathon with a few engineers - sometimes we do the initial chat and tech screen together so we had a 40 min chat, and 20 min simple tech screen in this case - proposed of simple tech screen is to decide if this person knows basic coding/problem solving (eg Josephus problem) - goal is not to write code on the white board, but rather talk and discuss approach, general problem solving
This candidate was specialized in OR, so we asked an optimization related question (given an array of products and prices, how to spend all your money exactly). Instead of a discussion, the candidate dove into writing mathematical proofs, and after framing the goal of the exercise and a few hints, he ended up writing a 10-level deeply nested for loop...
Obviously I'm not trying to make generalizations, but I've interviewed over 100 candidates in my career, and there is a strong negative correlation with PhDs in particular, hence the article. Understandably, because they are trained in other methods, typically unsuitable to startups, as others have commented. But if they want to get into startups, I hope these tips would help.
The whole point of a PhD is to NOT approach problems like that. Someone with a PhD has literally been trained to approach problems in a methodical, thoughtful manner, writing a thesis on the singular topic over a long period of time.
Is the problem here a fundamental problem with the PhD manner of approaching problems, or is it a problem with your startup mentality and hiring processes?
A more accurate description is: The startup risk tolerance is extremely high, thus it rewards quick and dirty over well thought out and solid.
This is often opposite to academic work, which needs to be well thought out and rigorous to an extent, though not necessarily in code quality. The code just needs to work for the experiment and be correct. It doesn't need to be maintained, extended, and debatably, communicated. This is also probably why startups tend to hire younger people, rather than older developers who've worked for places where the risk tolerance is very low and thus they take the time to make sure things are very solid.
Maybe it is an artifact of the fact that I work on computer systems research (where the focus is on finding the simplest solution that works) but the comments here make me wonder where you (collectively) are finding your PhD candidates!
I do agree that is often OK for PhD code to be terrible ("least publishable unit"), but that is also true of non-core production code at startups! ("Minimum viable product")
>Many PhD students can be perfectionists. But they need to remember that there’s an important compromise that needs to be made between the best solution and a good solution that can be used in practice.
Code quality doesn't seem to be a concern for Dr. O at least.
Appropriate older developers should have an even better handle on when (and how) to write good code and when (and how) to write bad code. Moreover, their bad code will be better, and produced faster.
Young coders are hired mostly because they are cheap and don't understand stock options.
Perhaps you misunderstand the agile process? Quality isn't something you ever actually achieve through the Feynman Algorithm (write down the problem -> think real hard -> write down the solution) regardless of whether you spend two hours thinking real hard or two years. Quality is achieved through iterative, continual improvement and polish. Otherwise, waterfall software would be the pinnacle of quality.
In other contexts "nice to use" is a better measure of quality, and I definitely agree that some iteration can help there. Even so, I'd caution that iterative development and a planned overall architecture aren't necessarily incompatible. And when following an iterative approach you have to be pretty careful to optimise for something that most users like, rather than a just bolting on everyone's pet feature.
If you are hell bent on the algorithms and data structures interview model, you just might not have PhD level problems to solve.
It's unclear what problem they gave the PhD was about. It could have been a hard optimization problem or some coding test.
What I got from this article is that the company shouldn't be looking at PhDs but rather at coders. At least that's what their interview seems to be geared towards.
Similarly, considering the author's self-declaration of being a TechStars alum, I also could pass judgement about them being a second-rate company since if you are willing to go through an accelerator, and you have not gone into YC, that probably means something. Luckily I know better than that. But I bet this superficial judgement would be more statistically accurate than painting all PhDs with the same brush.
I went through a similar struggle at my first interview out of school, at least in regards to an interview.
As I did non-majors courses at a community college before an intense 2.75 years at a full course university, I sometimes had to compromise on my course selection, as some things were taught in semesters I was unable to take them in.
In the interview(consisting of multiple interviewers, this was my first) in my first I was given the task to create a small class that would hand out read or write access, allowing many reads, and one write, and never a read and a write at the same time.
I thought for a moment, and started dictating my solution. Unfortunately, due to the above, I was never able to take an Operating Systems or Parallel Programming course. Thus, I had never even heard of a semaphore (I now consider that inexcusable, but perhaps understandable). As I set to work implementing two locking list structures, the interviewer would constantly interrupt me, trying to steer me elsewhere - a place I could not have possibly even known about - an unknown unknown if you will. This ultimately resulted in what was, to me, and I'm sure to him, a very frustrating 15 minutes and maybe 3 lines of code.
All this to say, I think sometimes there are just vastly different expectations on each side of the table, and it can result in mutually poor experiences in the interview room.
I would like to hear his side of the story. Maybe he was like "what kind of ridiculous problem do these people ask, I cannot wait to get out of here"
I deal with this a lot as a programmer who doesn't have a comp sci degree. I know what a semaphore is, I just don't call it a semaphore and I didn't know that it was called a "semaphore". I have been happily chugging along calling these "control variables"
Sounds like two weeks before a paper deadline!
There is so much wrong with this sentence.
I think that most people and companies would have been quite concerned with an interview ending in this fashion. I also think most people would have taken a moment to reflect about the sequence of events that led to an interview ending in such a manner.
Not these folks though, their first thought was we "we need to write a blog post." It's amazing that they chose to take a one-sided interpretation of an awkward event as a marketing opportunity, complete with a picture of "Dr. O"
Just like with UX design, you should design your interview process with the people you're interviewing in mind. The interview process isn't valuable in and of itself; it only matters to the extent that it fulfills your actual goals—evaluating and recruiting suitable talent. If you are rejecting or scaring away candidates with PhDs and you believe it's because they approach your interview incorrectly (even though they might actually be a great fit to the company), change your interview process.
In this case, it feels like the company came up with an interview process and then decided that the interview was an end unto itself. That is not a good approach to building a strong team. Just imagine how a similar outlook would look like in your sales process—would you ever write a blog post about how clients with PhDs weren't understanding your sales pitch correctly?
Of course, it's possible that the candidates you rejected would not have been a good fit. But if you believe that, you would not write a blog post like this—instead, you would focus on improving your upstream recruitment process.
Either way, the important attitude is the same: you should treat this as a defect in your process and you should debug it. Find the root cause of the issue and fix it by changing your system.
While it looks like this company wasn't the place to try it, I do find myself admiring the student who had the fortitude to keep on working through the problem while the interviewers looked on. I hope he finds a better opportunity soon.
That's such an uncharitable interpretation of the post.
It's a job interview, not a final exam. You're not supposed to sit there thinking for half an hour in order to produce a flawless solution. Maybe the interviewer was making progress, maybe they completely misunderstood the scope of the problem, who knows? The interviewers can't read your mind. Ask clarifying questions, make sure what would be the output of the algorithm in simple cases, then think about it enough so you can outline a solution.
This is such a blatantly false statement. Nobody cares if you come up with a shitty answer in an interview. They expect working code which is optimal. You won't pass the interview if you write a naive, working solution. It's literally all or nothing. Maybe it's time companies make it clear to candidates.
The truth is if you can develop an optimal solution super quick during the interview, you're better than someone who can't. It might be like
optimal solution, quick > working solution, quick > no solution.
The problem is if you can't do optimal quick, then you end up spinning your wheels and end up with no solution in the allotted time.
As an interviewer, it's important to set expectations ahead of time so the candidate knows what they're getting into. It sounds like that wasn't done here. But it seems like the cat should be pretty far out of the bag about this style of question for anyone who bothered to prepare in advance.
Come up with any solution, no matter how bad or inefficient, and then improve it. That's better than trying to find an optimal solution and not being able to come up with any solution at all before you're out of time. Then, you've demonstrated an inability to solve the problem. To put it in black and white terms, you've failed. An inefficient solution is much better than no solution at all. And it may be good enough if the data set is small.
Also, if you give me an answer in 10 seconds, efficient or not, you will impress me as someone able to make rapid progress. You can then improve it, putting you in a better position than someone who ended up with the same final answer but took longer to give me any solution at all. Working code is critical.
Discussion is important. If you've chosen an array over an ArrayList, I want to know why, such as an ArrayList of a primitive type being inefficient, or that we know the size ahead of time. Conveying that you know what you're doing is more important than a specific choice. Because if you know what you're doing, I'm confident that you can handle a different problem that turns up if I hire you.
It sounds like you are passing on good candidates due to them not being fluent in blabling. Which is fine if the job requires that ability (half selling to customers position), but is not if you need engineers to build system.
And then startups complain that there are no good people - they exists but get ruled out due to non technical presumptions like this.
That ability is not the same as being good engineer or problem solver. It is pretty much orthogonal. E.g. You can be good and both have it or not have it.
Basically,you test social skill for position that does not necessary requires that skill.
It has more to do with whether you think out loud (which I do and have a bit of advantage in such hiring process) and less to do with overall attitude of perfectionism when giving away finished work. I would not hire such person as consultant or support, but I worked with such people on positions that don't require much speaking with customers and they were great to work with.
Going back to the example of a doubly nested loop to determine if there are any two numbers in an array that sum to zero, it should take at most 1 minute for a smart engineer to figure out that it works. They don't have to speak before they've evaluated whether it works. The metric I use is: how long did they take to get to a solution that works, even if inefficient?
Back in the day my grandma was testing my foreign vocabulary. She had to learn that I knew the translations, just not at the speed she was used to.
Not every brain works in the same way.
If I ask to determine whether an array contains two numbers that add to zero, a competent engineer should be able to give the naive answer -- a doubly nested loop -- in ten seconds, or at most a minute. If someone takes a lot of time to give me any answer at all, it's a concern.
But seriously, giving programming problems like these to a PhD is probably going to scare the hell out of them, not because they think it's hard, but because they'll find it indicative of the work they'll be doing if they get and accept the position. Hopefully such faker filter questions are followed up with ones with real technical meat.
But this question lets me learn a lot about the candidate:
- How quickly did they come up with any working solution?
- How long did it take them to reach the optimal solution?
- Do they know time complexity?
- If they're using Java, did they use a plain array or ArrayList? Do they understand boxing and performance?
- Do they understand the range of an integer? I'd tell them that the array represents the population of the earth every year from 1900 till today, and see if they realise that they need a 64-bit integer.
A competent engineer or just one who's practiced HackerRank questions and mastered the cliche choreography of the SV interview format?
"Think out loud! We want to know how you think! Ask clarifying questions! Start with the naive solution! But oh also please do all of these things within an arbitrary time restriction known only by me if you want to impress me."
If you don't know the answer to a question, start talking about what you know in the general vicinity of it and see what happens.
You might get a hint for a direction, or the person giving the test might drop the question entirely because you brought up something more interesting (so careful there!), or they might decide that, while you cannot answer that particular question you're still sufficiently knowledgeable.
As long as you keep talking good things will happen compared to silence. A bad answer is a lot better than no answer, always. Also good is asking suitable questions back. If you're lucky you can derail the professor from giving a test to having an inspiring conversation about the subject, without grade loss.
So, this is a skill that is (indirectly) taught and practiced in academia. Thus I don't get the point of academia people having problems with this.
Let me get this straight, you invited 100 engineers in for an onsite after qualifying them with a phone screen and their resume? You conducted 100 Google style whiteboard algorithmic puzzle interviews just to find 1 single engineer to work for your startup?
These facts also resonate and could be generalized as something might also be terribly wrong with your "precious" interview process as well.
Don't get me wrong, I kind of enjoy doing those things personally, but it doesn't seem like a great way to evaluate developers.
But rather than whinging about it, we opened a free "Student Incubator" program [0], where we work on ML projects with selected talented students, teaching them about collaboration in fast-paced SW projects.
No. No. No.
The difference is that production quality software needs to be usable by other people -- often many other people -- instead of just the author or a small group of researchers. In many cases it must work close to 24 hours/7 days per week under highly variable conditions.
This means it generally must have fewer and less severe bugs than a proof-of-concept or research prototype. This often requires extensive testing by end users or QA folks standing in for end users.
There are many examples of shipping commercial products and widely used open source programs that clearly meet this criterion but are certainly not easy to read, maintain or extend code.
Modularity in particular often fails to produce either readable or extensible code, rather producing numerous layers of confusing modules and/or objects -- "lasagna code" instead of "spaghetti code." It also can fail to produce "production quality software" as I define it above.
I'm an engineering Ph.D. candidate focused on optimization and forecasting (tldr: large amounts of data analytics). I also code extensively in R. As a Ph.D. candidate, you're supposed to be the expert with the cutting-edge knowledge in your field. You're supposed to know what options are currently in development, the pros and cons of each methodology, and the significance of the final output and how it fits in to the big picture.
Most PhDs focuses on developing and implementing best methods (theoretically) and continuously improving accuracy of the final results. Methods that haven't matured enough to be implemented today, but will tomorrow. Timing and speed is still important in academia, but (and this is my opinion) more focus is spent on implementing the theoretical approach correctly.
From my observations, industry environment focuses on implementation speed (not the theoretical approach development but rather just "coding it in") and making it work to "solve" the problem. Focus is spent on implementation speed over maximizing accuracy. Therefore, simpler and less computationally intensive (and in most cases, easier) methods are preferred over more complicated methods.
I guess to sum up my point, academia focuses on cutting edge technology and continual development of some of the best methods available. That's how you get more of your research published and further your academic career. Private industry would love to implement cutting edge technology, but focuses more on implementation speed which can sometimes impact model accuracy. From my understanding of the article, it seems like the candidate they interviewed wanted to show the extent of his knowledge whereas the interviewers were looking for the general "overview" and discussion approach and thought process (his mistake probably for not reading the situation right, but just what I gathered).
Oh also to add as a P.S., most PhDs spend time focusing on theoretical problems and methods. Which usually means that knowledge regarding full stacks, infrastructure design, supplemental technology (e.g. D3), etc. can be a bit lacking at first compared to someone who spent the equivalent 4/5 years working in the industry and who has gathered experience using those technologies. Also PhD is really a "solo" mountain/task to conquer. While you do have colleagues and lab mates with you, everyone is usually doing their own research. So unless you spend time outside of your "work time" contributing to team coding projects (which is highly improbable as you'll spend most of your waking time working on your thesis), you probably won't get too many people with coding collaborative experience.
Overall, the post is pretty standard to what everyone needs. I'd chalk this up to inexperience in the workforce.