given a day or two, sure, I can go review, find that solution, and provide an implementation and explanation as to why it matters for a particular problem domain, but an hour-long phone interview isn't gonna check that effectively, unless they really want to see the initial stage of me silently reviewing maths literature to awaken knowledge in cold storage
on contrast, I _can_ explore systems design stuff fine in that space of time, because systems design _is_ what I practice regularly in industry, because that's what I'm regularly asked to do.
At my last company, we stopped doing system design entirely because we found it was completely uncorrelated to whether or not a candidate did well on the job.
Our project-based interviews were the main hiring criteria, and we originally planned on using system design interviews for leveling. What we found is that doing well on the project-based interview was a much better signal than the system design interview. When we used the system design interview for leveling, we under leveled certain people and over leveled others. We discovered that it would have been more accurate to only use the project interview for both making a hiring decision and leveling.
Ultimately, what we had trouble with was formalizing the evaluation criteria for the system design interview. With a project interview, we formalized the requirements of the project and everyone on the loop evaluates candidates against those criteria. With the system design interview, only one person ends up evaluating each candidate and a single person's judgement is highly variable. Unless the candidate did something egregiously wrong, it was very hard to say whether or not something was strictly "bad" or "good" and it was impossible to write down the list of "bad" and "good" qualities ahead of time.
Did you have any trouble with candidates declining the project? My first instinct as a candidate is to turn down take home project based interviews. I almost always say no to them (especially if it’s for the 1st round) for a few reasons:
One: In general they seem like a poor use of my time. I would rather do 5 one hour coding or design interviews at 5 different companies than a single 5 hour project at one company.
Two: I’m skeptical that anyone is actually reviewing the submitted project. And the evaluation criteria are usually unclear. For instance is it worth it to add linters during the build process? Or to introduce caching for api responses? Will the person evaluating the project notice details like that?
Lastly: Every coding or design interview carries over to the next one. Preparing for one is preparing for all of them. In contrast I’ve seen projects (for a backend role) range from “build a simple weather api, any language” to “build a sudoku solver specifically in JavaScript”.
Our process was to have someone "setup" the interview by introducing the project and helping the candidate get set up. Then the candidate would work on their own. Periodically, someone else on the loop would stop in, see how things were going, answer questions, etc. At the end, the last person on the loop would go over their code with them. This was a very important step. Hearing someone talk about their own code is just as important as the code itself. It's also interesting to see how candidates debug things, should any bugs arise. Finally, we would debrief. We'd bring up their code in a meeting room, test their project, ensure it meets the requirements, find bugs, etc. We'd discuss the code, see how clean it was, whether or not it was nonsensical, etc.
Selecting a project for a project-interview is non-trivial. It's definitely possible to pick a bad project. For instance, sudoku solver sounds like a bad project. The evaluation criteria is too strict and it sounds like there is probably an "optimal" answer. Basically, it sounds like one long leet code question. A good project question has a number of qualities:
1. The project needs to be a language of the candidates choice. You want to see how they code, how they think about problems etc. If you give them a skeleton, you're constraining their thinking and will get a weaker signal. It's also very informative to watch a candidate setup a new project. Good engineers will easily setup a new project in the language of their choice. Weaker candidates will suffer.
2. Evaluation criteria needs to be clear. We had 5 requirements that the project needed to meet. It was objectively easy to say whether or not the requirements were met. Failure to meet any of the requirements was essentially an auto-fail. I've seen project interviews where they don't tell you exactly how your code is evaluated and then run a test suite against your code. Forgot to test a negative number? Auto fail. How is that reflective of work done on the job? Such opaque criteria are terrible.
3. The project needs to be well-defined, but still somewhat open ended. We had 5 criteria to meet, but it was amazing the variety of responses we got. New grads would struggle to get all 5 done in time. Experienced engineers would use the time to go above and beyond. Going "above and beyond" wasn't really necessary. We could still judge skill just from the basic requirements, but it was cool to see how far some people took the project in the short time given them. This is a reason something like "sudoku solver" is a bad question.
4. The problem needs to involve some sort of data model / state tracking. A lot of software engineering involves tracking state, so we found it's important to see how people actually do that.
5. The current team needs to do the interview themselves. For instance, going back to a sudoku solver, how many people could sit down and do that in a short period of time? How many people spend their time thinking about discrete math problems like that? Personally, I don't like doing that, so it seems weird that I would require that of a candidate. Everyone on the team should feel comfortable that the project is achievable and reflective of the general work done on the team.
I'm consistently shocked by coworkers I've had, who I knew had to pass challenging leetcode interviews, who seem to not only know nothing about building real world software, but don't really have a firm grasp on algorithms.
Yes they can process a leetcode question requiring dynamic programming in record time, but when mapping a real world problem to a dp (or any other similar solution) are completely at a loss.
If you think co-pilot is cool now, just wait and see what it does in 5 years. A team of 10 tomorrow can probably replace a team of 50 today.
So maybe the interview process in 10 years is more like: "build an auction site like ebay in the next hour."
This is very "Monkey's Paw" and I can absolutely see something like that taking hold, as sloppy and stupid as such a task might be.
I had an interview with a YC company who asked me how to do something, and then I did it, and then they asked "how could I make it a one-liner" - immediately thought was "I'm pretty sure I don't want to maintain a codebase full of fun little one-liners for everything" - managed to get it, but it definitely looked like crap.
If I were to get asked to build eBay in an hour, I'm sure my auction model would be 1) shit, 2) very prone to social engineering type attacks (fake bids, etc)
Shoutout to the interview I had where I was asked a tricky graph question, reasoned my way to the optimal solution, then failed because I "should have recognized" that it was a variant of Djikstra's algorithm from the beginning. As we all know, being able to regurgitate an existing algorithm shows far more promise than being able to logically create it yourself!
What will it do? It can't write exponentially better code. Maybe 50% better, so that I would need to edit it's outputs less often. The only thing I could see it doing exponentially better would be writing _less_ code. Telling me "stop! close this file. close this project. you are working on the wrong thing!"
Technology can jump sharply but then go relatively flat. We still haven't gone back to the moon, 50 years later. We still don't have at-home nuclear power plants.
I see diminishing returns from LC hards and greater other than as a guardrail to block management from hiring senior people who are practically no better than the junior people.
Somehow companies are going to have to now filter through a lot more candidates to fill positions, and for right now leetcode-like tests are the quickest way to get a gauge of an engineer’s performance (on leetcode).
Don’t get me wrong: I think leetcode hiring depending on how it is done doesn’t result in getting the best talent.