Stanford CS9: Problem-Solving for the CS Technical Interview
web.stanford.edu
web.stanford.edu
I do somewhat blame the whiteboard culture, but I guess that's the one that has given the best results so far. I've no doubt that it will change soon.
Given that technical interviews a common method of partially evaluating engineers/developers, it doesn't strike me as particularly odd for a practical course to exist that helps students maximize their ability to get a job. As you mentioned, it's really only one class out of dozens of others they'll take over several years. Myself, I took a number of electives that had far less practical impact on my career trajectory.
To take an absurd example, imagine if for some reason Law firms started requiring an Irish Stepdancing component to their hiring process. I'm sure broadly speaking, you'll see the most dedicated lawyers practicing away on nights and weekends, and the best Law Schools with the best students will hire the best Irish Stepdancing teachers. But at the end of the day, Irish Stepdancing has very little to do with practicing Law. Similarly, if solving stupid whiteboard problems has so little to do with being a Software Engineer, that they can't be learned in the 40+ other courses or internships/PT work, that a student will have to do in their time at University, then the question is why are they even done?
Again, nothing wrong with someone trying to get it right but at the end of day we all lose.
On the correlation of irrelevant things to performance in a demanding job you know law school is at best as related to the practice of law as Bar review courses right?
https://www.edx.org/course/how-win-coding-competitions-secre...
It's a little unbelievable.
Industry is sane. They ask stuff relevant to actual work or real basics. They don't make a gameable process full of rules and techniques.
* Is attractive enough (compensation, lifestyle, etc.)
* and has a larger supply of qualified applicants than desirable positions
Will result in applicants preparing for however they are interviewed, tested, etc. This isn't unique to software engineering interviews. People prepare for interviews or exams in finance, law, medicine, etc. They all have a system that is gameable. It might be newer in software engineering interviews, but that is mostly because CS is now incredibly popular and top talent is compensated very well.
That you think interviews and exams are interchangeable concepts illustrates the problem. Nothing wrong with a rigorous filter at the gates of the profession, but lawyers don’t need to spend months studying for each company’s half-assed version of the bar exam each time they change jobs. Our industry is dysfunctional because, unlike law and medicine, no one trusts the credentialing mechanisms.
The bootcamp style of skilled labour generation is possible simply because of the demand-supply gap.
You can't learn and practice surgery yourself. On the other hand, you can teach yourself software engineering. You can be practitioner with nothing more than a computer - and so many people are.
That doesn't mean CS degrees are a hoax - it just means the field is more accessible to those without college degrees (though its definitely going to be a more difficult path).
At least in Europe things are a bit more strict.
Would you disagree if people from the US did not need to pass the bar to call themselves lawyers, or the equivalent for doctors? Is it just because I compared this specific thing from the US in a negative light compared to EU?
In terms of PE / CENG status its more who you know that what you know :-)
Sorry if I forgot anything, fellow Europeans.
Core tech behind computers is more than just a network protocol that got lucky.
In case you missed it, Robert Metcalfe is an electrical engineer.
It is quite different to have spent 5 years doing nothing other than building up engineering knowledge, and just learning a few things to get the job done.
I'm always very weary of companies that tell me I'll earn more money because I have a PhD completely irrelevant to the job I'll be hired to do. It signals that they're valuing the wrong thing, and that doesn't make me want to work there.
Engineers are only those with an university degree from an university acknowledged to actually be teaching engineering.
Then if they want to actually make use of the title, get the credential, which yes requires an yearly contribution.
Finally, in any kind of project where human lives are at stake, the engineer signing for the project has to be validated as such.
I happen to think that our profession is no different from lawyers, doctors, mechanical engineering, civil engineering, and so on.
Even the FAANG companies aren't immune from bad hires or people who burn out.
At the same time, the "person who can work with the business user on the software, think about the architecture of it, identify the design necessary, come up with the estimate that actually matches the time frame that it will be done in with a reasonable error... and produce software that takes X as input and provides Y as output while being aware of where the edge conditions may exist and ask for clarification on how it should work"... I believe there is a distinct lack of that portion of labor.
Furthermore, there is also a lack of people who are able to move from the first labor pool to the second, and a lack of mentors who have the time and ability to help that group move to the second.
I don't think its incredibly difficult to hire an entry level person as long as one sets the bar low enough and has people within the origination who are capable of providing the design. On the other hand, it is very difficult to find the people who can give the necessary instruction to the entry level people to allow them to become productive within their ability.
As an aside, I also find that within the entry level group... there are a sizable portion that have the attitude of "I learned language X and that was hard enough, I'm going to stick with it and not learn anything else." That X can be found for all languages and none have the monopoly on it. However, it is disconcerting for me to see those individuals... I started out as a C programmer, and then Perl (full stack web - some JavaScript in there) and then Java (enterprise), and then Java stand alone (swing application)... and while I'm still a Java programmer, I can see other languages looming on the horizon. Java will become the COBOL, and while there are still COBOL programmers out there, its not something that one wants to get stuck in for another two or three decades waiting for that last app server to be turned off before they can retire.
Why would culture developed skills that make you rewarded less?
The ability to teach juniors is not searched for nor rewarded either, where social skills are even sometimes treated as something that makes you less good as programmer.
There's also a sizeable portion of companies that have the attitude of "we want someone who's really passionate and good at learning. oh and they need to be expert in language x. oh you're not an expert in x? sorry, we can't wait for you to learn it. we don't care if you learnt something else"
That's what it's like for many other in-demand and well paying industries (she's mid-20's and makes low six figures in San Jose). Stop pretending like everyone else has to deal with the same bullshit as you, hiring in tech is BROKEN. Other industries do it better.
People's lives are dependent on how well she does her job. Are the hoops you have to jump through justifiable for how "important" your job is?
Nursing
Medicine
There is not one single shared course between the two degrees. If you can find a course shared between medical and nursing students in all of Europe I'd be surprised.
Nursing is taught at many universities with medical degree, first and second year students tend to share some of the lectures.
Also, because of that, people with nursing degree are able to acquire a medical degree by taking the extra lectures and exams afterwards.
It is not examples that one finds online, rather on corridors and schedule notes.
What I can provide as an example is that nurses and doctors have equal access to many master and PhDs. Check "condições de acesso" in some of the links from the page and you will find "enfermagem" as accepted degree.
http://www.uc.pt/fmuc/gabineteestudoavancados/formacaoposgra...
In the vernacular, with regards to salary, it is used to describe a range between 100 and 200k. So low six figures tends to mean closer to 100k. That’s very consistent in usage.
It is a little odd though, I would agree about that.
Can you explain how other industries do it better and come up with a better, plausible, non-fantastical way to interview programmers? I'd love to know how to interview people "better", but it still has to be something where I can later present evidence that my decision is based on, in some kind of report. It would have to allow multiple people at my company to interact with a candidate in a half day or less. Copying what another industry does would be great, if it worked, because it'd help convince people it was a good method to try out.
> Are the hoops you have to jump through justifiable for how "important" your job is?
I never thought a 1 hour phone screen and a few hours onsite was particularly onerous, and based on how much the job pays, it seems well worth it.
> People's lives are dependent on how well she does her job.
I'm not sure why you seem to think that saving lives correlates with strictness of interviewing, especially with the 2 jobs being so drastically different.
This is simply wrong. I suppose if you've spent your whole life as a developer, it may seem like it would be done likewise other industries, but it's not the case.
Typically, how you might "prepare" for interviews in other industries is...learning about your potential employer. And maybe reviewing your accomplishments. That's it. And most people don't even do those things. None of this hours and hours reviewing textbooks from your college classes, since you've been in the same job for 5 or 7 years and have forgotten algorithms that you never use in your job. It's simply perverse.
Software development is the only industry I'm aware in which technical preparation for job interviews is not only advisable, but increasingly necessary for interviews at all levels of the career hierarchy.
Even in other engineering disciplines, this kind of bullshit is unheard of for any interviews except maybe your first interview out of college. And in industries like law, medicine, etc., if a potential employer asked you to perform some random task from a highly technical and complex subject from med school or law school, it would be very unusual.
Actuaries aren’t expected to prepare for and pass a vector calculus and linear algebra exam every time they interview, but they are required to pass a proper exam consistently administered and graded.
This might be a big improvement in our field as well.
In music, you often get handed some sheet music or asked to prepare something.
In the cooking industry, you throw them at a kitchen and tell them to impress you.
In the programming industry... We ask people to write toys and logic puzzles. That isn't what they'll spend their days doing, that's just practice, like scales.
A violinist might be asked to play a technically difficult but short piece during an audition, even when most of the music they would play as a member of the orchestra would be less challenging and take a much longer time to play, with plenty of time for rehearsing ahead of time. Likewise, a software engineer is ideally asked to solve a small but technically challenging problem during the interview.
No. Compared to what we do as software developers, the correct analogy for a violinist interview would be to ask them to whiteboard all kinds of obscure music theory principles, like the set theory underlying serialism and Arnold Schoenberg's twelve-tone technique.
I'm not a developer, so maybe this sounds absurd, but as someone who has good understanding of how medicine works, maybe this industry could use a few standardized exams/certifications?
I think it's possible that, because a practicing physician will have passed the MCAT, USMLE Step 1/2/3, and have obtained a license, they don't have to deal with this sort of bullshit in interviews. It seems like the idea of a "licensed software developer" may never exist, but maybe that's why the hiring process is the way it is.
I doubt it. for example having OCJP doesn't prevent interviewers from asking basic Java questions. It is like they are telemarketers with prepared script and aren't flexible enough to skip some parts no matter what.
It has always shocked me that people don't instinctively think that doing this is an absolute necessity for every single job interview. I have very often been met with a totally blank look when I've asked candidates what they think it is the company I'm interviewing them for does, even when phrased as nicely as, "Tell me a little about what we do and why you'd like to work for us" - surely not a hard question, you would think!
Not so much hard, more that it sounds like a complete waste of time and could be signaling to people that this is one more non-technical/HR interview they have to get through before getting to talk to people they'd be working with and who can fill them in on details specific to their prospective job or team. Whatever you're trying to figure out with that question, there's gotta be a better way to probe that, that doesn't make it look like you're wasting the available time.
I normally for public companies read the annual report and accounts.
There are many reasons for candidates to be excited to work for a company:
- friends
- great team fit
- technological challenges or preferences
- work-life balance
- location
These and others contribute to an employee doing his job with passion. (note: money is not one of them. there is a study that showed that above a certain level for a person, no amount of extra remuneration will result in an increase in productivity)
And I imagine this is the case even for pinnacles of vision like SpaceX. I imagine employees at SpeceX do not care about the future of SpaceX , they care about putting man on mars, making space available cheaply, pushing the boundaries of human exploration and of technological capabilities, Musks vision and enthusiasm. But not SpaceX as a company.
People should really stop expecting companies to have emotions. Companies do not care about people so people should not care about companies either. People care about other people and society at large.
Job preparedness training is looked down upon within universities because it's the banal economy poking its nose where it doesn't belong. The industry or a professional association can and should develop job-preparedness-focused training programs to augment (as in law and medicine) or even replace (as in HVAC repair and precision machining) CS undergrad. Nothing wrong with that. They just don't belong in academic departments. Those are for something else, and industry's training needs shouldn't be subsidized.
An employer should value a relevant trade school degree more than it values a Bachelor's, unless it turns out that well-rounded intellectuals with exposure to the underlying discipline perform better than those with direct training in the relevant activities.
I don't think one can ignore the disconnect between the stated charter of top universities (basically to do research) and the motivations for the majority of undergraduates who actually attend them (prestige, which hopefully leads to a high paying job). I doubt that all donors give these institutions donations to further their research goals. I know for one that if I were given the option to focus the impact of my donation, I'd readily choose it be applied to a practical, skills-focused program within the research-focused university that I attended. Furthermore, the undergraduate programs of top private universities in the United States aren't funded by tax payer dollars - they're funded by sky-high tuition. As the OP shared a class applying to the undergraduate curriculum at Stanford, a tax money argument isn't relevant here. If the course were geared toward Masters or PhD students - different story.
Finally, universities seem incredibly resistant to re-evaluating their place in society in the 21st century economy -- at least in the U.S. From childhood, a college degree is pushed as a necessity for financial success. Top universities like Stanford are sought out not just for their academic rigor, but for the network and prestige afforded to their alumni, who can parlay these things into a strong career path and a high paying job. Univerisities seem deaf to this reality. I'm sure they understand it, but for whatever reason are reticent to acknowledge that their role in society and to the economy has evolved since their founding.
I think Stanford should be applauded for offering this course - would love to know how many students registered.
That's not to say we ban career training (or office buildings), but some space is and ought to be reserved for other things.
Students seeking higher education for purely economic reasons are wrong to do so, and a gauntlet has been thrown to the business community to launch serious competitors that more directly address its training needs.
I have trouble understanding why there can be no middle ground between the status quo and pure vocational training. As I mentioned in a comment below, most arguments against middle ground that I've heard boil down to, at the core, nothing more than circular-reasoning like, "this is the way it's always been, and we want it to stay this way because things have always been this way." Why don't universities make an addendum to their charters to maintain the goal of being world-class research institutions AND provide pragmatic, real-world training? What sacrifice would be made? Would the quality of university research truly decline?
Whether students are wrong to pursue higher education for purely economic reasons is a matter of opinion, not a matter of fact. What is a fact is that many students do pursue higher education for purely economic reasons, because society tells them from a young age to do so, and it's reasonable to assume that a degree from a top institution in a technical field will lead to a high paying job. The business community certainly could do more to shoulder any perceived burden for professional training and/or certification; however, traditionally research-focused universities should also work to better support the motivations of the majority of their undergraduate (and paying) student body, as Stanford has done here.
What universities provide (liberal education, basic research, etc.) is scarce among what surrounds them (capitalism). Time is finite; the more of it we spend on workforce training, the less of it there is for an undergraduate education's current content.
Maybe the current content is not worthwhile, so we may as well gut it and reuse the facilities/framework for vocational training, but surely you recognize that doing so makes the current content less available.
>traditionally research-focused universities should also work to better support the motivations of the majority of their undergraduate (and paying) student body
Customer-service mindset is a poor fit for education. If "the customer is always right," why bother with teaching?
Or to have job seekers complete an online test proving their skills?
https://www.indeed.com/forum/gen/Job-Interviews/Has-anyone-h...
What is unusual is that we essentially ask people to set aside and forget everything they know about how to do their jobs, and instead perform a separate set of skills which, since they are't used on the job, must be carefully re-acquired whenever it's time to go interview.
No other industry I'm aware of does that.
Interviewers don't share the questions they are expecting you to prepare for.
For example a couple of years into my first job I got to work to be told we have just brought an a0 digitizer ( cost about 2x my then salary) id like you to hook it up to the PDP 11 and get it working. I should point out there where no drivers or software.
I even had to make the serial cable up then write the drivers using some of the more advanced bits of RT11 to run two processes with the driver interrupting the main program when data came in on the rs232 port.
That being said, to prep is better than to not prep.. just like the SAT. The problem is that these things favor those who make the time.
Part of interviews aren’t even testing skills, but testing prep. Of course, you want to test for innate genetic talent as well.
Good interviews do that, but as with gattaca, we see that desire and sincere interest counts too over genes.
Preparation, sincerity, and genes. That’s what I like to see in potential employee, in that order.
Sometimes, of course, one attribute really outshines the others - but its order in priority should always be factored in.
1. Source: individual who tried just showing up for 75% of his 20 year career.
Plenty of fields do - performing arts, film, trading, etc. all involve some form of short, intense, expensive activity to which you show up prepared and work in a burst of superhuman activity. But software isn’t like that at all - you get a problem and you dig into it, mull it over, research, etc. at your leisure until it’s done. We’re novelists, not actors.
Also, you mention research. Research and preparation are synonymous.
This is wrong. If I want to handle a problem by researching it, I have a goal in mind ("solve problem X"), and I'll look for things that will help with it.
Interview preparation specifically avoids that approach. Instead, you're supposed to become familiar with every possible problem, in case it comes up during the interview. Most of them, obviously, won't, and from a "research" perspective that means that nearly 100% of the time you spent in preparation was wasted.
If you're writing software under the kind of time pressure where consulting reference material is unacceptable, something is deeply wrong. Most software engineers, most of the time, should be able to research topics as they come up rather than prepare (beyond the standard preparation in college).
Making long-term plans, building consensus around them, etc. is important, but is nothing like practicing for whiteboard interviews.
Much like programming. You program for all relevant edge cases, as one might be used at some point.
People who don’t prep are horribly lazy and are terrible at enumerating edge cases. The have buggy code that fails at some point. Laziness is by far the way worse quality? Though not the only sin.
Lack of sincere desire to be helping the team succeed and no innate talent are bad too.
And most people, most of the time, can and should research them as needed.
>You program for all relevant edge cases, as one might be used at some point.
Yes, because a program doesn't get to pause execution and defer to the human mind to ask "hmm, what do I do in this case?" A programmer does so all day every day.
>are terrible at enumerating edge cases
Any attempt to pre-compute the edge cases of all possible programming situations will be hopelessly inadequate. Enumerating edge cases requires analytical thinking in the moment, essentially the opposite of preparation.
In my experience, having been an average programmer (and maybe still being, I hope that by now I'm at least slightly above average) who worked in The Netherlands at a "system integrator", a web dev "sweat shop" (some projects using 10 year old tech, relatively high staff turnover) and even a startup, such companies don't ask for whiteboard coding, however take home projects are quite common.
I also interviewed at some more "hot" unicorns in Japan, which do have these kind of questions (using a dehumanizing online test in fact).
How do they happen to be developers then? Doesn't being unable to implement a FizzBuzz quickly imply they can't code anything that is more complex, by definition?
By the way although I do understand programming (OOP, FP, SQL) and can develop a considerably complex app from scratch in an IDE like Visual Studio with ReSharper or IntelliJ Idea that would expand code snippets, offer auto-completion and highlight errors on the fly I doubt I can code anything more complex than a "hello world" using just Notepad and expect it to compile and run without errors immediately - I use to make small typos and miss some syntactical elements all the time thanks to my ADHD (but modern IDEs solve this problem). Also, a person standing behind my shoulder watching me code and counting time would decrease my performance by an order of magnitude at best. Does this count as a failure from your point of view?
> FizzBuzz test is actually pretty great interview question [...] you should be able to do that without thinking
If you are going to code for me, I'm not interested in what you can do in 1 minute, I'm interested in what you can do in 1 year. I want you to spend more than 1 minute just thinking about any problem that is remotely worth solving. And that's the underlying problem: FizzBuzz is not representative of the problems I hire you to solve, and solving it fast is not representative of the approach I want you to take for solving the problems I hire you to solve.
FizzBuzz can be a reasonable point to start a conversation about approaches and style. But that's true of almost any piece of code - I used strcmp() for a number of years.
However I don't see how they guard against an outdated skillset? My company employees five Infor Sys21 RPG programmers who thought the term "web service" was synonymous with SOAP until I ran a workshop.4 out of 5 worked most of their career for IBM and all have bachelor's or masters in CS, EE, or Math. With a weekend of refreshing they would definitely pass a whiteboard for me (although not necessarily with the most optimized solution). They are all over 50 btw. And work slow and steady (but efficient) with no burnout in site.
I can come up with a "good" solution for nearly every question related to algo on leetcode and have very in demand skills as well as been the top SO poster and core open source developer on a Java platform that is #1 in a Gartner magic quadrant. At 27 and am on the verge of burnout that as happened as a result of getting large Adderall / Vyvanse scripts (legally) and being on the computer for nearly 14 hours every day.
I read this and I wonder if you will understand what I'm saying when I tell you that it is specifically BECAUSE proponents like you fuck themselves up beyond comprehension that I consider whiteboard tests to be a massive red flag when I'm interviewing at a company.
Your career should be a source of service, joy, growth and challenge for your entire life.
Don't destroy it for yourself (and others!) right at the beginning.
So much for the fitness test, maybe we should have one for employers as well.
I have nothing in my GitHub profile. Some toy repos of no particular significance, one contribution to Rust that I've made before realizing it takes too much of my spare time... and that's it. I don't think that should make me un-hireable ... people that have kids tend to not have much spare time besides work, learning new stuff to stay relevant, and taking care of said kids. Not to mention, my company has me going through some bureaucratic process for any sort of open-source contribution that I intend to make.
- Code-hinting (semantic analysis) in an IDE - Product management - Low-level JIT codegen & optimisation (for ARM Neon & Intel SSE) - Some datascience (technology/ product evaluation for a 8-digit aquisition) - One of the early contributors to a web standard - Distributed systems, bigdata (mostly graph-processing at quite big scale - Facebook-big)
and probably other stuff I can't remember right now. I worked in half dozen different programming languages. I'm pretty sure I've "stayed relevant" more than the average Joe, but you wouldn't know it from github.
Given that the person is presenting StackOverflow as an example of their profession, comments on SO would be very telling for how the person would be responding to critiques of their code and their responsiveness to questions. Much can be seen in the attitude and professionalism that would be seen in email. Things like:
* Do they write in complete sentences?
* Do they use spelling and grammar that would be appropriate to send to a client? A director?
* Do they range against all that is wrong with the world over little things?
Likewise, with Github there are issues that they have logged on other projects - do these issues contain sufficient information about the problem so that the person working on the issue can diagnose it? For issues on their own projects or projects they contribute to (and have taken ownership of the issue), are they responsive and provide useful information to help someone provide a good bug report?
I'm not too concerned with the code. That's easy to fix. But a person's attitude and providing these sites as an example of their professional mindset is something that can be difficult to get at in an interview when they are only trying to put their best on display.
Same thing with me but I don't think it's hard to spend some weekends and write something to express your motivation an illustrate your coding skills.
> people that have kids tend to not have much spare time... my company has me going through some bureaucratic process for any sort of open-source contribution that I intend to make.
I understand your point perfectly but what I don't understand is how does it make sense to hire a coder whose code (a real (though not necessarily big) app code, not a 5-minute puzzle code) you have never seen.
Well, let's break this down a bit:
- if you hire a junior, all you care about is smartness & maybe culture fit. You get both of these better through interview than GitHub repo - if you hire a senior, you hire for social/organizational skills, problem-solving skills, diverse expertise etc. GitHub is not particularly relevant
GitHub makes sense when you look for someone with deep specific expertise in a narrow, open-source-related technical area. Which is rare. I think it's more often used to check that "this person has coding style preferences similar to mine", which is IMO not really a great idea.
Rehearsing at a relaxed pace over a single weekend should be more than enough, particularly to get used to discuss your ideas and using a whiteboard.
I am pretty sure that they will perform better than people who just spend a single weekend rehearsing.
Gaining deep understanding is the trick, and it comes from constantly being exposed to CS-like problems. This doesn't need to become something intensive like studying before exams, in fact having a deadline might be really bad for it as you won't think about each thing enough time and the pressure makes you less prone to wonder about the problems.
It just needs to get you passively thinking about a problem, mostly abusing on your "background brain-threads" and taking the time to make yourself deeper questions around the problem and the data-structures used, like which problems are similar? which are the properties of the problem that enabled some approach? what's key for the DS to be useful on that problem? are there lower/upper bounds to the solution? why something doesn't work?. Some quick research later on the problem might show you a better way, and then the right question to ask is what did you missed to come up with that, did you ignored some property of the problem that the "right" approach exploited? was it just something you didn't knew about?
Going for those kind of questions will make you better at analyzing problems and come up with some strategy to solve them, and when you have that clear getting it written shouldn't take much effort.
Every one of them is the same: you go in behind a screen, and you have to play those few minutes of music better than anyone else that day. It's hard to do because they choose the hardest stuff, but you know what you're up against.
It's very imperfect. It doesn't reflect most of what you're going to be doing in the job. You might play one of those pieces in a given season. The bulk of your time is spent listening to the players around you and playing in sync with them, understanding what the conductor wants, etc. There's a lot more to being a good orchestral musician than being able to nail your part for Don Juan completely alone without context.
It's not great, but it is consistent. Evert tech interview I've had in person has been wildly different. Even since I learned pretty early on to ask about the process. You still have no idea. It can be anything from an entire dev team grilling each other in a pissing match that had almost nothing to do with me, to 1:1 with someone grilling me about the finer points of a PhD topic they just finished years studying to a casual conversation over coffee just to get to know you to live coding exercises where you have an hour to write a 3-d car driving game to a conversation with part of the team where they tell you about their real-world problems and ask you for ideas about solutions to getting paid to work on a live project. All for web developer or data engineering positions.
I remember being in one interviewer role a while back for a dev ops person for the team. I got thrown into the mix at the last minute because the other interviewers realized the day of the interview that they don't know anything about dev ops, and I kind of carried that portion of things for us. They had literally nothing to offer to the conversation, so it was unprepared me interviewing a highly competent dev ops pro. After 20 minutes, the other interviewers left and said, "I really don't know why we're here. I don't understand any of the words either one of you are using, so just carry on without us."
I felt really bad for that guy. He was really really good, and the only thing we got out of it was that the team didn't know why we were hiring for that position.
The amount of variability is truly insane. I hope that whiteboarding culture doesn't win the standardization contest because even worse than in the case of a violin audition, there really should be no performing or bravado in an engineering team.
But even standardizing around that would be an improvement over the utter chaos right now. Because then it's consistent. Perhaps misplaced, but at least consistent.
I wish that instead of teaching classes about how to win the CS technical interview, schools would teach classes about how to interview for technical roles. How to design effective interviews that will help find the best person for that role. There's a ton of research around this. It's not just "whatever seems to work for us." And we need to be better.
Aside from security, I think this is the biggest challenge we face as an industry. Everyone interviews differently and almost all are done poorly. But the vast majority think they are good interviewers, and they're just not. It also won't improve on its own, because there's almost never any effort at honestly evaluating the hiring practices or decisions.
I’m also working on the space with www.onsites.co to solve how broken the interview experience is.
shakes head
Our society is waaaay to litigious if an interview is going to be overshadowed by the legal team like this.
If people at Google were interviewed by juggling raquetballs, there would be books, courses, etc. on how to juggle better. People are willing to pay for this prep because they want to work at Google. Eventually most everyone interviewing at Google would be very well prepared to juggle.
Obviously if the process was a 100% fair assessment of knowledge and potential and people do better than you then good for them.
As someone who gives a lot of interviews, I'd love a better way to assess a candidates ability that is less gameable, but we haven't found one yet.
Other companies are much more specific in their interviews...you actually get a specific JD with expected skills that you hope will form the basis of your interview, and not all JDs require you to be a distributed systems/algorithm wiz.
This is exactly the concept of an IQ test, right down to the stack ranking.
They're pretty well understood by now, and they take less than a day, which compares pretty favorably to "months".
As part of the application, I was instructed to provide 2-3 paragraphs on each of half a dozen questions that covered various aspects of software engineering. The in person interview asked a predefined set of questions about the material that I had provided. That part was more of a "demonstrate that you have the mastery of the material claimed in the written portion and that it wasn't produced by someone else or some other source". This also tested communication skills.
No weight was given to GitHub contributions, hacker rank, leetecode rank or whatnot. There was no whiteboard.
While this isn't a fabulous job at a tech company, it is one where good engineers can and do find themselves at without needing to have the proper shibboleth to get into one of those tech companies.
For another job application that was more of a sysadmin/programmer bend, a simple backup script was the assignment (took about 1h to get all of the edge cases). A portion of the interview was a demonstration and review of the code (in which I had to answer questions about the code that I had written).
While mock interviews can be helpful in the communication skills department and reducing anxiety, there are many other ways to test the person rather than use a proxy such as contributions to a public repository or foo rank websites... and also without resorting to whiteboard for various algorithmic tricks that you either know or don't know.
- Preparation not required.
- Preparation not beneficial.
And it shouldn't be surprising that they achieve that well. Consider these scatterplots comparing students who took the SAT without preparation to students who prepped: https://infoproc.blogspot.com/2012/02/test-preparation-and-s... .
As further described at the same link, research quite consistently shows that the effect of preparation on SAT scores is small, and...
> One of the most remarkable aspects of this line of research has been the lack of impact it has had on the public consciousness.
Again, this shouldn't surprise you, because it is a design goal of the SATs not to show an effect of preparation on scores, and they do their own research to ensure that preparation in fact does not have much of an effect.
Normies and free thinkers just take a pass.
However, golden wingless dragons seem to do just fine.
If, one hundred years ago, Oxbridge had floated the idea of sanctioning a student to embark on a program of study designed to anticipate questions that might arise in a job interview, it would have been as preposterous as suggesting that the Queen should take her meals in a pub.
And yet today, we have one of the preeminent universities not just of the United States but of the world offering a course that appears to amount to Kaplan for a tech job.
These kinds of courses and prep has existed in similarly popular fields for many yeas: law, medicine, finance, etc.
And a computer science degree is deliberately, emphatically NOT a professional training regimen towards a career in software engineering.
Some medical schools will use NBME (National Board of Medical Examiners) subject exams as final exams for various third year clinical rotations. Through a combination of experience on the rotation and studying material from textbooks and review books, the exams serve as good preparation for the USMLE step 2 exam since the test style is largely similar.
I will never subject one of my potential hires to this nonsense. These concepts are valuable and academically interesting, for sure, but for the practical kind of engineering that's done at most companies it's simply not needed. Either that or it's already implemented in a library.
What a waste of time.
What I'm saying is: unless these students just want to sit around repeating their DS&A course(s) over and over again the rest of their lives, this sort of interview, job, and prep course is counterproductive in the long term.
Required viewing: https://www.youtube.com/watch?v=zDEpeWQHtFU
Take home assignments are easy to cheat on and take a lot of time.
Asking questions about someone's experience is a great way to find someone who is a great conversationalist that can't code.
Asking someone to program at a computer in a limited amount of time falls victim to the same issues that the competitive programming, whiteboard interviews currently do.
In fact, if you want to hire a great software engineer, you can't do that reliably with any of these interview techniques. If your interview can be studied for, you aren't going to fight great software engineers. At best, you can find someone who will do the job.
A mediocre software engineer.
Does performance on these tasks correlate with performance on the job? Is it easy to measure this performance in interviews reliably? Then it's going to be used by companies.
Technical interview performance has ~0 correlation with performance on the job? I find it hard to believe. Is the mean performance on the job of a software engineer that fails to pass Google interviews as good as one that does? Why do most companies want to hire Google software engineers if that is the case?
As for my second question, it should have been "is it easi-er to measure this performance reliably than other skills that may be relevant"? It sounds like this would be hard to answer. But this kind of interview seems to be best suited for repeatable evaluation (although, of course, you do have the problem that people will prepare and bias the results, but I don't see how you can avoid that if you want a repeatable way to measure performance).
Finally, "and companies use them in spite of that". Any guesses why?
Google had a study that has been linked to death around here which showed the scores hires received on their interviews didn't correlate to job performance ratings. There are of course biases and flaws in that study, but it underscores another separate point: studies should show they do correlate, not the other way around (prove a negative).
> Finally, "and companies use them in spite of that". Any guesses why?
Bandwagon effect, cargo cults, ego boosting/confirmation bias from applying a hazing ritual, laziness all come to mind.
Maybe not, but the ones who lack the ability to do so will also tend to be the coworker who constantly asks you and others inane questions, dragging down the team's productivity as a whole.
I don't believe cheating is such a major concern. The applicant should be able to talk about their solution and all the choices they made if they actually wrote it and understand the solution.
Then if your company is large enough, your novel challenge gets leaked and everyone knows it ahead of time. Solutions start being sold or distributed. Then your interview is meaningless.
I'm all for a viable alternative to the whiteboard interview, but I haven't encountered one yet.
Beyond that, if you do use a single question, you're now playing a game of whack-a-mole: I used to ask this question about the challenge, now I can't because it is leaked and people prepped for it. How do you scale that across an engineering organization? Do you have weekly meeting with every interviewer informing them of the questions about the challenge that are now blacklisted? It's going to be infeasible for any reasonable sized organization.
The challenge with this idea as well is that the risk of false positives is much higher than the whiteboard interview. The current interview process is designed around minimizing false negatives.
False positives are really costly to an engineering organization. It's not good for morale when engineers are being hired and fired in the same month on a regular basis. It's takes time to onboard, and slows the team down. It's also not good for the company reputation - would you apply to a company who had a reputation of firing a good chunk of its new software engineers within the first month?
A big part of the interview problem is that everyone wants to think that they are solving really important problems and are changing the world, etc.
Companies need to be honest about what they are doing and who they need to do it. Sometimes you have a hard problem and need a specialist or a brilliant problem solver to conquer it. Most companies need middle-of-the-road people who are reliable and consistent but not necessarily geniuses to maintain the things that have already been built. The idea that everyone needs to be a rockstar or a 10x-er needs to die. This overly inflated notion of what even some of the top tech companies are doing is the reason for all of this craziness.
Extremely few genuinely hard problems are solved by profitable companies. Some exceptions happen. But they happen(ed) in the R&D sections of those companies. Bell Labs, Google, Apple, Intel, Facebook-ish. But those are not the money-making day-to-day jobs that power the profit engines of those companies.
Those think tanks do excellent work, but they are not significantly different from research PhD programs. They just pay better. It's always been the .0001% of top people who solve problems theoretically, and then lesser people find ways to implement those solutions.
For any genuinely novel or hard problem there are ways to suss out how much a person has been coached, and how well the problem is truly understood. You don't have to change from week to week. It's not that hard.
In my own field, I can give a take-home challenge: here's a relational dataset. I want to work with this as a data warehouse. Give a rough outline of what changes need to take place, and how you would design the warehouse. Ask questions as needed as you go through the exercise, and be prepared to defend your design choices at the tech interview.
In less architecty roles, perhaps I'll ask a person to do some pretty basic data transformation as a kind of a fizzbuzz. But the tech interview is just to ask a series of questions that are pretty open-ended but have a list of points I think are relevant and assign points based on how many checkboxes they hit. And yes, that list can and will be leaked at a sufficiently large and desirable company, but it doesn't matter. Because the checkboxes are there to remind you from interview to interview what you were looking for. They are not a replacement for your assessment of the person. And you, personally, can change your mind as you go.
The point of an interview shouldn't be to find out if a candidate knows exactly what you know. If that's the design of your process and the only thing you care about, then you are going to have problems. The point of an interview is to let the candidate tell you what they know and for you to have some baseline for comparing that to what they need to know for the role.
In some cases, all the candidate needs is basic programming skills and the ability to memorize stuff. And in that case, you don't need to worry about if the candidate cheated, you need to worry about how honest your team is about what it's doing and what you need as a manager or coworker.
It's like security through obscurity. It never works. You need a better process.
Right now it's often "all or nothing", meaning companies are very careful about who they hire and some people just have a hard time getting to prove themselves in the act. Even if they don't get the job, a 3-month paid gig is better than unemployment.
But it really depends on what the position in question is.
Like, things get out on time and shipped, right? I may misunderstand the intent of the comment though.
But if they are sub-contracting out their work AND the work is done correctly, on time, etc. then it's all gravy. Yeah, you should probably hire that 'friend' instead, but if the interim works, it works (speaking very generally here)
Not to be too salty about this, but I remember friends from college who are terrible teammates (no significant contributions to group projects, barely understanding the project material, but knowingly do so cause others will finish it for them etc.) but go on to get job offers from the Big Four. Having these technical whiteboard interviews will not filter for such soft skills that are needed to succeed in the workplace.
Mock interviews, resume writing and critiques, life at startups, how to answer team/behavioral questions, and panels of employers and employees are all covered with sample whiteboard/coding problems only being given one week of attention. This is simply preparation for how to professionally navigate the job landscape.
I always thought that this is a good question to ask https://github.com/alex/what-happens-when but then again probably not for all roles. It's good because even if the interviewee has seen the question, you can really gauge how much they know.
If a candidate has shown an ability to solve several hundred medium+ DS & Algo questions on the spot, that gives him an edge up in day to day to programming. Not everybody can do that.
You can supplement that with other areas (concurrency; distributed systems; execution latency, memory management etc; quirks of and design ideas behind your favorite programming language).
I think this class can be a good prep for day to day programming.
I've actually become a better programmer by practicing interview problems, because they ultimately are a bit like the arithmetic involved in solving more large scale problems.
So much for there being a staffing shortage in our field. These types of interviews imply there is a much higher bar that exists to weed out all the chaff due to oversupply or companies won't settle for anything less than the most academically best people.
The point is - there are a lot of unfilled positions. Companies notoriously complain for not being able to find "good" software specialists, because... people with minimal clue on how to judge relevant skills introduce increasingly more cut-throat filters and processalize everything.
You can do both the content and the prep, as I am sure is the case with Stanford CS.
While it's unique to have a course specifically named for this and am sad to see it as well, I don't think this is as big as people are making it out to be, nor is it a canary in the coal mine for higher education. To me, this only signifies the growing shift in education, as it becomes more available to all, being about preparing for your career as much as learning itself. The simple fact is that most attending college today are not doing so just out of the desire to learn alone, but also want to, well, have a nice career in their respective industry. To be able to make a living.
While I would also agree with cautioning about education for career alone, I think it's important that modern education focuses on both. It's the reality of an increasingly college educated world.
I suspect those would be more appropriate for a 'data scientist' / 'researcher' position - for a generalist, the interviews were very standard and not crazy at all.
In hindsight, if I had taken the time to review basic CS algorithms ahead of the interview, I would have done much better.
That said, I'm not sure I'd be a better employee at Google if the only difference between my being hired or not hired is the fact that I reviewed some basic data structures and algorithms.
Maybe there should not be cookie cutter interview at all, but more of harder look at which skills and personality traits are really needed for this or that position.
And I tend to do good on whiteboard interviews.
It's kind of sad that there's a trick that needs to be learned to solve technical interviews without testing the candidates' inherent knowledge, but at the moment, this could seem like the most logical way to get a good job for a Stanford CS student with no knowledge on technical interviews.
Shouldn't we focus on creating great engineers instead of training them to make a good impression on an interview?
I think is completely immoral. Becoming good at something is a matter of consistent practice that brings experience and expertise over time. There is not other way around it.
People might pass the interview but their hat will fall in a matter of weeks. This is PUA for interviews by a University, this is really depressing.
The only thing that gives me hope is that great engineers will always solve problems no matter what. Tech hiring is going to become much harder.
We hear there's a shortage of good people in SV / USA. What if you plainly ignored all companies that whiteboard? There should be plenty well paying jobs left.
disclaimer. a major tech company whiteboarded me twice (codility + live coding via google docs). They were really selective, none of my friends passed. In the end the job was lousy so ultimately the test selected more for willingness to jump through irrational hoops than productivity.
I'll take your word for the other fields as I can't speak directly to interview vs exams there.
My github is active, I work on things that a potential employer as well as fellow developers would be interested in and I also am trying to grow by showing bad, good, and some code that I'm very proud of on my github. But, the thing I noticed at the two aforementioned interviews was that they hadn't even taken a look at my github.
It seems many new grads are being drawn to highly-visualized resumes with fancy templates, often featuring a bar graph for skills ranking (some ranking themselves as "8 out of 10 in JavaScript", which comes across as laughable and out of touch).
My feelings on memorizing data structures aside, the resume lessons may at least help some grads get interviews that they wouldn't have otherwise, and that's a good thing.
A binary tree search has little to do with figuring out the pros and cons of which api to code against.
How about a scenario where you get to try to find a bug with legacy code?
While the system is admittedly problematic, does it really come as a surprise that for extremely competitive jobs at the entry level, such artificial hurdles need to be used to filter the plethora of black-box applicants? I can't speak to more senior roles, but for entry-level, this only seems like a natural (and practical) response. It happens everywhere, not just CS:
For any competitive school admissions, SAT/LSAT/MCAT what-have-you are all poor indicators of student performance, and boil down to trainable tests.
In finance (both quant finance/and positions like sell side banking, private equity), people constantly practice math/financial modeling/accounting, for interviews. Many of the more selective quant shops (for undergrad hiring, I should mention) have much more difficult problems than MS/GOOG/AMZN technical interviews, and I would argue the type of competition level math/probability are just as unrelated to their jobs.
In management consulting, people spend hundreds of hours practicing for case study interviews. Just search consulting interview prep, and you'll find an egregious number of prep companies.
Some people brought up law and medicine. First, I don't think the grueling nature of the medicine certification is even close to what it takes to be an entry level developer. In law, while "technical" interviews are rare, often times employers filter you by much cruder means: pedigree (both the school you attend, and at top echelon law firms, clerkship status, and top-top grades within those schools). Is that any better?
The common denominator here is the high demand for all of these schools/jobs, and the problem always seems to come down to, how to optimally recruit -given- resource constraints. Maybe I'm cynical but almost any reasonable entry-level interview can be "gamed" or trained for, and most firms are either small and lack resources to extensively evaluate their applicants holistically, especially if they're only in undergrad, or they are large and get way too many applicants, so some proxy is necessary.
This is a decent primer (and people need primers!) and a good resource overall but I'm very skeptical that this is terribly helpful for most interviews. I liked some of the problems but most are about a notch lower in difficulty than required for the offers Stanford students get.
With that said, it's clear that the students that take this class are going to do well in their internship and job search. I'm honestly just becoming more convinced that ability to perform in these interviews is an inherent quality of top students (no matter the original field!) at this point.
It was interessting to do so, i think it made me better and i will try it again.
And yes there are people who don't care enough to work for companies like google and there are.
Also while i did more interviews as the person who is interviewing: I feel that i like to have more better prepared people to interview.
If you failed to complete the problem or took longer than expected to complete it because of extra complexity entailed by a more "correct", out-of-scope solution, it may especially cost you - even if in the end, your solution was "smarter".
Maybe the other guy noted that there are a couple of ways to solve the problem, and asked a clarifying question about it.
I've found that the best way to handle it in a conventional whiteboard interview is to mention it when appropriate and offer to add the functionality later.
"Now, here's the part of the code where I'd deal with odd N. Since we're guaranteed even N in the problem statement I'll skip this for now, but we can come back to harden this function later if you like."
This timing!
I'm going to interview on Tuesday with a startup in SF. I've spent better part of last 3 weekends studying and solving DS&A.
Any advice from HN on what could I do in ~14 hours of prep time?