Byteboard assesses for on-the-job engineering skills
blog.google
blog.google
The serious, playing-to-win version of this approach replaces most of the on-site interviews, and generates results that are more trustworthy than those interviews. It takes less time from candidates, reduces inconvenience, and yet is cheaper and faster to run for the company. I'm still amazed more people haven't figured this out and run with it.
I say this because I think the skillset required to "choose a good engineer among thousands of applicants" is quite different from "lead a team of engineers to develop a product."
Some are better than others, but point being, if one can't say "based on research, this is a good way to interview," why would you risk changing what seems to work? Why would you risk offloading your "good enough" strategy onto another company that, to you, probably is just guessing as much as you are?
Isn't data a potential solution to this? One issue is that the best data would have to be gathered on timescales of years, and not just a few years.
Fundamentally it is a human evaluating another human. Process and rubric can assist in eliminating bias but as human interaction is complex, a bit of pattern matching and whooly experience is going to go a long way and excels in these domains.
Let's also not forget that things that interpret data are subject to bias as they are built by humans too - see Amazon's case where filtering was accidentally sexist
Assuming that you've already invested 50 hours practicing algorithm problems[1], how does this approach take less time from candidates?
If I'm interviewing with a dozen companies, it would take far less of my time, to do the initial phone screens with all of them. A typical HackerRank/Leetcode-style pre-onsite phone/online interview typically only takes 45 minutes to 1 hour.
If I had to spend 5 hours on average implementing a custom project for a dozen companies, that would take me 60 hours in total. Even with the time spent studying, short 1-hour algorithm problem interviews take up less time.
[1] Not to mention: I don't think time spent practicing problems on Leetcode/HackerRank is a complete waste. Sure, you're not going encounter hard algorithm problem everyday at work. But it builds certain skills that might be valuable eventually, as a developer.
Does anyone have an example of what a Byteboard interview looks like, maybe a sample project? I feel like there are some details missing here about what makes Byteboard as helpful as the article says it is.
From what I can see, it's a lot like the type of content I've experienced smaller companies asking, which I'm for because it does require some amount of open-ended engineering that fits into what applicants would be expected to do every day. There's a made-up feature that's required, and you discuss how to implement it (and then actually implement parts of it), that sort of thing.
What I think is still a bit disappointing is that this is being pitched as a fix for screening, not for the actual on-site interviews where the memorization questions usually come into play anyways. It feels a bit disingenuous and contradictory.
So evaluators have literally any level of experience? Can we, as a society, please agree to stop using this actually meaningless "up to X or more" language?
I get that they want us to have the impression that the evaluators have experience, but it seems like they weren't confident enough to commit to any real statement to the effect. E.g. most evaluators have at least 10 years experience. Even "at least 1 evaluator has 15 years of experience" would be a stronger statement, although it would be more obviously empty.
However, until they actually test it out for themselves, both they and the talent pool are not going to have a clear idea on whether such practices work well to improve the hiring process and hence source the right people.
But whatever maybe interviewing and the process of recruitment is bound to have some false positives/negatives which are completely fine given the fact that the pool is humungous and that the candidate may in the future seek to establish themselves by improving upon their lack of skill by filling in their knowledge gaps and the companies should provide adequate support and morale boosts for such employees, those who strive to achieve better knowledge and skillset given enough time and backing.
Solving this hiring problem is probably more tougher than the leetcode puzzles that the candidates get. Surprisingly, not even Google can solve this.
Sure, it makes things easier for the interviewer since they'll get better candidates but does this change anything for the interviewee?
They'll still have "to find time to comb through my college computer science books, practice coding theory problems like implementing linked lists or traversing a graph, and be prepared to showcase this knowledge on a whiteboard." All that is up for game during the brutal onsite interviews.
Not that you would usually implement such a thing yourself anyway. So these leetcode style interviews are actively selecting for people who practice 40 year old CS solutions that don't work well on modern hardware, and selecting against folks with real-world experience.
[0] https://www.youtube.com/watch?v=YQs6IC-vgmo [1] https://ticki.github.io/blog/skip-lists-done-right/
> Not that you would usually implement such a thing yourself anyway.
Google has data structure libraries. They write low-level code like kernels that has no library of data structures. They implement programming languages where they need to implement data structures from scratch. Lots of need to do it.
I think most binary tree implementations require pointers, so they aren't amenable to this type of optimization. I haven't implemented a skip list myself, so I could be wrong here and happy to hear about why if I am.
[edit to add]: I totally get why this would be applicable at Google. The part I find silly is that most companies other than the FAANGs usually just use libraries made by the FAANGs and don't have much reason to focus on this level of comp sci in their own interview processes, yet they still read articles like this and copy Google anyway.
"Fancy algorithms are slow when n is small, and n is usually small." -- Robert Pike
Cons-based linked lists continue to be the container workhorse of MLs. In imperative languages, arrays are preferred because allocating a chunk of memory is even simpler.
> So these leetcode style interviews are actively selecting for people who practice 40 year old CS solutions that don't work well on modern hardware
These claims are extremely common, so I'm not being snarky here, I want to understand your reasoning.
You're doing the interview, they're gathering data, then discussing it in a meeting:
What do you think is being measured?
Can you describe a utility function they are applying to that measurement?
> Not that you would usually implement such a thing yourself anyway.
But engineering isn't like shift work. The complexity of tasks you encounter are likely Pareto distributed. Thus something like 20% of the tasks you work on will justify 80% of your pay.
In a leetcode style interview? I think they are attempting to measure aptitude at solving algorithmic-style problems on a white board. I think they are indirectly measuring ability to perform under pressure, ability to vocalize while thinking, ability to function in unnatural environments, and either time since graduation from a CS major, or time spent practicing / gaming the system on leetcode (the premium feature even lets you see which questions your employer will likely ask).
I think they should be measuring things like: ability to select good frameworks for a solution, ability to reason about how such frameworks are implemented at an algorithmic level, when and where to add an index or avoid a shuffle, how to debug, ability to write readable code, ability to coach more JR devs, ability to learn from more SR devs, etc.
> Can you describe a utility function they are applying to that measurement?
I'm not sure I can. I see this as a very multi-dimensional problem, so I think the utility of given skills would have to be considered in regards to the existing skills within the team.
Honestly, I feel like "software engineer" has been split in two: R&D is done within the FAANGs (and maybe a few universities) and the results are open-sourced. Outside the FAANGs, something more like "software development" happens: FAANG libraries are glued together. _Very_ occasionally, you get a smaller company that has a legitimate business reason to write their own library or database or whatever, and an engineer is lucky enough to convince them of that.
I'm also not trying to be snarky - I've been stuck on the latter side for most of my career, because I prefer to work in smaller companies, and as such have not had as much opportunity to practice "interesting code" as I would have liked. Perhaps this is due to my market (Denver), but I've also worked with companies in the bay, and I've seen a tremendous amount of over-engineering when off the shelf parts would have worked.
So I'm legitimately curious:
1. Am I interpreting your questions correctly in that you think the ds & algo whiteboard interview selects well for desired attributes?
2. Do you solve those kinds of problems on a day to day basis?
3. If so, do you work at a FAANG?
I don't mean to pry, so feel free to disregard any of those questions if your are uncomfortable answering them on a public forum, or message me privately. Thanks!
So do I bring my dog and wife and books and video games and...? Cuz between the job and that stuff I don't have time (or energy) to make anything at home.
I mean, I've been wanting to learn Rust, and ML, and containers/cloudy things... I have some ideas for side projects... I just don't have time and usually want to do the opposite of "think real hard to make good code" in my off time.
Truthfully, I came away a bit jaded about the whole culture around technical interviews and whether most companies actually have a genuine desire to change their interview process.
Still, we both really want to see something like this succeed. My current bet is still on https://www.woventeams.com/ but very interested to follow this as well.
FWIW, this was exactly what we were against at Headlight, as is Woven and (presumably) Byteboard as well.
There are plenty of practical interview questions to ask candidates, but it requires a lot more thought and preparation than I think most companies want to do.
The reason (imo) that quiz/trick/etc. type questions are popular is that there's a clear "right answer" most of the time. No nuance required. Introducing nuance requires humans to reason about things, and that takes up time (and thus money).
The irony for me is that when I did encounter these questions, I already knew the answers from research (i.e. the egg drop question, how would build an M&M, etc).
You described one (long) interview cycle, with apparently an offer (since you seem to know the compensation) that you declined.
What I wonder though is whose pain points are being alleviated: job seekers or employers?
While it sounds like this is better for job seekers than technical interviews, it actually sounds like it will compound the pain. Specifically because this is yet another method of validation one must pass to even be considered for a job. Take the various hurdles a typical job seeker must overcome:
1. Get through the initial filter.
Referrals, networking, and recruitment companies are potential solutions. Byteboard just tacks on another item to this list that must be checked off. It does not replace having to do any of these other things.
2. Pass the technical screen.
Crack open those college textbooks, cram leetcode, shore up your portfolio, and now, also pass the Byteboard test. Unless every company uses Byteboard or Byteboard guarantees eventually getting you a job, this again doesn't replace anything.
3. Pass the onsite.
Culture fit is culture fit. That one is unavoidable and reasonable. However, everyone knows that onsites still contain a technical portion. Sometimes a significant one. Byteboard only replaces the pre-onsite screening portion of the interview process. That means GOTO Step 2. You still need to study and pass the onsite technical interview.
The copy reads well initially. Digging deeper though, it starts skewing much more heavily towards employers:
* employers get a better technical screen (filters better)
* employers save time (no resume filtering + technical screening)
* employers save a lot of time (fewer, higher quality onsites means fewer hours spent by the team interviewing candidates)
Which all leaves me wondering, what is the value proposition to a job seeker?
The current interview questions from companies are based on tricks (like two pointers, start at both ends etc) you need to know, memorize, and be very good at figuring out what trick to use in advance .
As a job seeker that has learned there are tons of tricks and its impossible to memorize them all. I for one would welcome anything that tries to end that shit once and for all, which this claims its trying to do.
1. all companies use Byteboard
2. Byteboard replaces the onsite technical portion as well
If you look at Byteboard's description of their own value proposition (https://byteboard.dev/how-it-works):
> How does Byteboard fit into our hiring process?
> The Byteboard interview is designed to replace one or all of your pre-onsite technical interviews. It provides you with a strong understanding of your candidate across 20+ essential software engineering skills, so you can make decisions with confidence about which candidates to bring onsite.
You'll notice that they are only attempting to solve #1. Hence why I spoke about where in the funnel Byteboard is trying to insert itself. They are not attempting to solve #2 which means job seekers still have to do everything you listed. Only now job seekers have to mentally prepare themselves for potentially going up against Byteboard's black box rubric:
> review each anonymized interview for the presence of 20+ essential software engineering skills, which are converted into a skills profile for each candidate using clear and well-defined rubrics.
Oh and since one of their value propositions is tailored interviews:
> end-to-end service that includes the development of unique questions, an interview platform, candidate support, interview evaluation, and skills reports
You don't get the benefit of something like Triplebyte (not that I'm sold on them either) where you only need to be vetted once.
The top comment by tptacek sums up what needs to happen pretty well. As I see it now, Byteboard is very clearly marketed towards employers and maybe that is the best first step towards changing the status quo. However, right now, I do not see any value add.
Sadly it's not going to happen over night. I am just glad they are trying and honestly I don't see how they can come right out and replace the entire interview process for companies over night.
They need to prove to more and more companies that if people can do this real world stuff then you should hire them and not focus on trick questions.
I am already thinking I will prefer companies using this new system as long as they stay true to their word and keep it real world.
The reality is that there's far more steps actually required than that. I want to see if they were going to go through all the details; ask what all the details were, etc.
But even more importantly, the entire time I'm talking video games and tv and movies and cars. I'm trying to distract them from actually doing the steps. Even better is that as the steps escalate, the more difficult to steps become until you make it to WSUS and you literally cant install wsus successfully. Ask Microsoft, not me, why wsus is so terrible.
The fun thing about this interview process is that it's not trivia bullshit. Like "explain the role of a windows server" or "what is a windows domain" or "How do you backup active directory" this is all ridiculous questions that dont gauge if someone knows anything about anything.
I also think there is a lot of unconscious bias in the technical interview in most companies and not a lot of people like to admit it. We have a similar take on the problem in our own product: https://medium.com/codesubmit/were-all-a-little-biased-a4749...
The trend is definitely going into project-based assessments, at least here in Europe. I think most candidates would prefer that kind of assessment if companies started to invest time developing them. Most of the time they are either a nonsensical mix of brain-teasers or just take way too long to complete.
If I'm understanding this correctly, hardly anything new in the space.
Employers should be required, by law, to compensate interviewees for their time at a rate equivalent to the job for which they are interviewing.
(Macdonald’s, Goldman Sachs, line cook, VP legal, same law, everyone, everywhere)
Yes, the candidate is taking a risk by interviewing. But the company is also taking a risk, and spending considerable engineering/recruiting time on a candidate that most likely isn't qualified. They're also footing the bill for travel, if needed. And because most don't make it, they end up investing 100 times over, for every candidate that they actually hire.
That sounds like a fair trade, to me.
1. It strongly incentivizes bad behavior for bad hires. A completely unqualified candidate would be extremely motivated to obtain an interview, since even getting to do a first round of screening might result in them being paid $100 or more.
2. It doesn't sufficiently disincentivize bad behavior for employers. Companies like Google spend A LOT on interviews. A few hundred bucks more per candidate is not going to significantly burden them enough to change any practices, especially when the cost of hiring a bad engineer is so high.