A hiring test I'd like to run

I interviewed at Twilio. My resume and phone chats made it abundantly clear that my professional experience was 95% backend in languages like Java and Go.

I looked at their Glassdoor where people wrote they were heavily biased to the algorithm type questions, for better or worse.

In reality I was given only a single coding interview my whole onsite, and it was to build a JavaScript SPA, including routing/linking, without the use of any framework, just vanilla JS.

I was allowed to Google whatever but 50 minutes is still a crazy time crunch to figure out a lot of the stuff that needed to be figured out.

It felt like a massively unfair way to evaluate me. I have honestly never felt more screwed over and time wasted by an interview than at Twilio and I would highly recommend avoiding interviewing there.

My overall point being, is there’s no silver bullet. It’s not necessarily about this magical process will work well and this one will not. It’s about the small details, especially things like making sure all of your interviewers actually read the candidates resume and that everyone knows what type of role they are interviewing for, and that the interview is designed to discover both the candidates strength and weakness. I’ve been on both sides of the process and I know too often shortcuts are taken and churning the pipeline forward takes priority over doing a diligent job and respecting the time of every candidate you engage with.

The very fact that the foundation of tech recruiting is inexperienced recruiters, with minimal tech knowledge, who are incentivized by sales type numbers, and churned through themselves at a comical rate, speaks to how the industry views recruiting. College athletes get recruited, top law school graduates get recruited, engineers aren’t recruited but relentlessly spammed in hopes of finding more bodies to feed into a process straight from a Kafka short story, the problem starts there and no a-ha process improvement will fix that.

The single best interview experience I've seen was given by Oracle when I worked there a few years ago.

The onsite was a panel interview where candidates were given a laptop with a configured IDE and codebase with a specific set of bugs. Their task was to debug the various issues and complete as many as they could with the understanding that it was not expected for them to solve them all or even most of them. There were also tests to help them. And a different section of the interview asked them to right some tests and some code for again brain dead easy coding algorithms like reverse a string.

I thought it was great because it closely resembled what being a dev is like and actually tested for the skills we cared about. No you didn't get to use your own custom vim configuration but I'm not sure how that would have really worked even if they wanted.

This reminds me of the story of the young intern that went around a few years ago. They sheepishly admitted to their boss at the end of their summer internship that they had just googled all the Excel things that were required, that they hadn't actually known any of it.

"But your generation KNOWS to look for help, and knows where to find it! That's where you were doing it right," was the boss's response. And that's heartening.

At my previous company, the technical interview was 2 hours long. The candidate was given a choice of about 10 very small, simple projects. Then we sat at a pairing station and paired on the problem. The candidate can use google, their favorite editor, write tests, anything they want. The goal is not to finish the project, but just code with them for 2 hours. It's the best technical assessment I've yet seen in an interview setting.

One caveat is this company really believed in pair programming, and virtually all code was created in pairs. So that's part of why this interview approach worked.

Some of the tests in my current company work like this. We don't expect you to have everything memorized. We give you a problem and then say, "feel free to use any reference you want, help, man pages, google, stack overflow, whatever."

Some people still stubbornly refuse to look for help.

Some people act on the first google search result even though it's obviously wrong.

Some people don't know how to read a man page.

I find it very illuminating, but I don't really have enough data points to say how successful it is.

> I'd tell them that at the start, obviously. This isn't The Secret Rules For Getting Hired.

> "I know you've never used Blender," I'd say, "And this job doesn't require it. But we want to see how quickly and accurately you can learn to use something unfamiliar while under pressure."

> Or whatever.

The main problem is that the level of pressure that people feel under interview conditions will vary dramatically. Personally, I would find this a very bad test as I get horrendously stressed at interviews yet somehow manage to portrait a facade of calmness (to be fair I've been told once that I looked remarkably calm).

To the usual refrain of:

If you can't handle the pressure of an interview then maybe you are not a good fit for our company

I normally reply:

If day to day working at your company generates interview like levels of stress, then please do not offer me a job

Well this is exactly like a test I went through at one time. It even used the same reasoning pretty much.

The job was fullstack node and react, I went in and they said we want you to look at this game written in Python which is really slow and tell us what is wrong with it (I hadn't done python in years and back then wasn't very good at it). We will sit with you. I said I was game, they said lots of people weren't - one guy evidently got offended and said he didn't do that language (I can definitely think of one friend of mine who is totally competent who would get offended at being asked to work in a language that did not have anything to do with the job he was applying for)

The game was using a big dictionary in an array and you had to look up words from it and it was slow (about 7 minutes to do searches). So I thought to myself, damn I cannot remember the name of that algorithm for this (binary search), ok let's stall a bit maybe I will remember the algorithm name.

So what's the first thing you would do?

Uhm how about checking a profiler. So we checked and found some simple things we could optimize. And then I looked through the code and I saw there were parts I could cache etc.

So after all those things were done the time to do a search was a little bit more than halved and I still couldn't remember "binary search", so I stared dumbfounded at the screen for I think about a 3-4 seconds and I started to say "uhm" and then the guy sitting with me I guess ran out of the time he wanted to spend and said "ok well there is an algorithm for this blah blah blah" showed how to do it in python. Thanks for your time.

I actually didn't want the job but still felt somewhat let down that I couldn't remember.

I'm a mechanical engineer by training and still occasionally do some entry level hiring in it.

My favorite question in recent years has been posing an advanced project - not unreasonably difficult, but one I know they won't have covered in school. Make it clear I don't expect them to have the answer, but have them talk through their approach to this project with me. "How would you get started?" "What would you need to know?" "What resources would you use?", etc.

For us it's been a great filter - both a gauge of their current knowledge and ability to tackle an unknown.

This is very specifically a terrible idea.

I passed an analogous interview to this once when I was much younger, and then subjected several applicants to it thinking it was a useful filter.

It's not, it's abusive psychological torture that rewards sadistic amateur psychologists and interview administrators. The only way to succeed is to submit to the interviewer by asking for help. It filters for people who know to implicate others in failure while taking credit for small successes, and bullies people into submission.

The interview I passed was a variation (I learned after) on the bridges of konigsberg, where instead of asking for help, I asked if it was impossible. The person administering it hired me on the spot, but even this was wrong because the place was full of people who thought the reason they were there was because they were intelligent - something they didn't have control over - and they acted As pettily as psychotically as you would expect.

The problem on teams isn't individuals not asking for help, it's managers who don't have the confidence of their teams to ask.

If you are reading this and I put you through that exercise 15+ years ago, I apologise.

We do this at retriever.co for our virtual assistants, in two areas:

1. we have applicants do a data transform with an esoteric excel function that they've /almost surely/ never used, and prompt them to find videos / tutorials on how to use it if they don't know (which, we assume, they wont)

2. we have people answer fake customer support tickets that require them to do some on-the-job learning about the Internet Service Provider industry -- to a level of detail that requires that they read through and grok a page on wikipedia explaining up-time calculation.

While neither of these pieces of knowledge are necessary for their day-to-day at Retriever, they allow us to stratify candidates' ability to learn new skills and grok new information.

100% would recommend!

This made me laugh - I first applied for temp jobs around this time and there was a typing test.

"Wow," said the person who had left me alone with a computer that felt old in the 1990s, "I've never seen anyone score 50WPM before."

"Uhh... I could type a lot faster, but the letter T is sticky and I had to delete a lot of Ts"

She looked mortally offended by this.

I was given a job as a dishwasher which I was sacked from after one shift.

I partially agree, and it depends on what you are interviewing for. In the end, getting a job done, on time, without error/bugs is generally most important to the people you are going to be working for.

As for algorithm questions at some FAANG or wannabe startup, it can be rough. Personally I tend to blank on a lot of terminology, similarly on names a lot of the time. Some trivia games kill me, I can see the answer's face, the movies they've been in, but just blank on the name.

All the same, in my life and career, I've managed to learn, adapt and create both simple and complex systems in new languages and platforms.

On the other side of the table, I tend to ask maybe 3-5 trivia like questions only to gauge where a person is at, and let the conversational parts determine if the person would be a good fit. I think you need some of both. I feel drive accounts for more than even skill, knowledge and experience.

> Are they able to read a manual?

Uh, I can read a manual (in fact, I enjoy reading software documentation), but it's going to take me longer than an hour...

After a couple of years working as a web developer, I decided to go to College. I took a "Web Development" class. I failed the final. The assignment was to code a website by hand. I used BBEdit to pretty print my hand coded HTML and it inserted a <meta name="editor"> tag that I didn't catch. I dropped out and went back to getting paid. Been doing it for 20 years now, though I use Prettier to do my formatting these days.

This is very similar to what I do for roles that are supposed to be independent and to work with customer software. I ask the interviewee to do something with a technology or software they don't know, but give them no other constraints. They can use whatever programming language, whatever OS, whatever IDE, whatever existing libraries online and even IRC/Slack if time permits - the goal is to see that they can solve a problem that is new to them without panicking. In real life they'll have a lifeline or two in the form of colleagues, but those colleagues might also not have the answers.

Obviously a terrible solution for a standard product engineering position, it is only relevant for special roles.

I don't mean to be derisivs but at that point - testing for reasoning, understanding, personality - Why stick with a software tool?

You'd be better off arranging to play a strategy, preferrably collaborative, boardgame.

One day I'll use Pandemic as an interviewing tool. I wonder of it'd work as a pre-formal interview stage

First test of any software dev: Do you know how to look for answers on stackoverflow?

Second test of any software dev: Do you know how to ask a good question on SO?

I had that same automated Microsoft Office test at a temp agency in the 1990s.

Funny thing, it had a bug. As long as you didn't let go of the mouse button, it would let you open and navigate all of the menus. So I simply browsed the menu UI until I found the correct choice. 100% score.

I've always wanted to run one where we bring them to a room to interview. Maybe take a break. Move their stuff to a slightly different room. Pick back up right where we left off. Maybe change out the furniture and people outside during the interview. Introduce them to numerous employees, who will respond to names swapped amongst each other. Another break - switch interviewers. Pick up right where the first one left off - insist that this employee has conducted the entire thing. If they drove there, it would be best to move their car.

I'm not sure what position I would be hiring for, but this is the interview I would like to conduct.

Other possible ideas would be a background check that is either insanely deep (so you're 3rd grade teacher was unavailable for a recommendation) or raises issues that they have no part of, "we see you've spent time in North Korea."

I love this.. Definitely going to set something like this up.

And anyone who does go for the help gets an immediate offer.

I hate people too stubborn to admit they don't know.

I hear a lot (and believe myself I suppose) that I would rather hire someone that knows how to learn than someone who currently knows thing x but can't pick up and learn things on their own. But I feel like it's VERY hard to figure this out about someone. It's pretty easy to test how well someone knows a particular technology or tool (given some variance for operating in a high-pressure scenario etc.) but it seems pretty hard to tell if someone can learn new things quickly, and learn the right things. I feel like this is what algorithm tests are trying to do, but I also feel like a lot of algorithms are based on having a key insight into a problem that is, at least in my experience, pretty hard to have when you're in a room with strangers on a timer and being judged, so if you don't know the particular algorithm to apply, good luck. And also it's pretty easy to go study algorithms and hope you remember how to implement DFS if that's what they ask you in the interview. And there is some non-zero value to just knowing about different data structures and algorithms for certain types of work for sure. I don't think many of us web developer types will be implementing DFS ourselves, but I run into issues pretty often where having knowledge about different data structures and algorithms comes in handy.

So I can see the appeal to doing a test like this that has nothing to do with a particular skill, where you're trying to evaluate someone's judgement and how quickly they can absorb new ideas or figure something out on the fly etc. BUT, I just think there are too many factors that can really muddy this up, and I still don't feel like there's an objective way to measure "smart and learns quickly". Even "learns quickly" is somewhat vague; I personally wouldn't expect someone to pick up, say, functional programming in an hour just by saying "go read about it." Some things take longer to learn than others.

Anyway, there's no silver bullet. I still feel like we end up making pretty subjective decisions about other people when interviewing them. I don't think we're even all that aware of them ourselves. I would say though that if you're interviewing somewhere, you're getting useful information about that place by how they interview you, and so I wouldn't feel too disheartened if you feel like the interview is unfair or doesn't work for you. That's probably a good sign that you wouldn't enjoy working there anyway.

I think the primary issue with interviews is that the interviewer doesn’t share expectations. I think it’s totally fine to evaluate how a person asks for help, as long as it’s clear they aren’t expected to solve the problem.

Further, you’d need some standardized way to evaluate how well someone asks for questions. Otherwise you’ll have interviewer bias.

Which is why designing interviews are very difficult.

Interviews should be personalized. That they are conducted en masse with little consideration for applicant background is an artifact of the difficulty of adequate personalization. Targeted ads are valued precisely because they are targeted, presumably increasing the chance of a match. Has anyone heard of any startup companies in the personalized interview space?

I was pretty happy with the interviews I went through for my current position, although that might be selection bias because I passed them. They all shared some important attributes:

1: Fairly open-ended, there was no "bzzt-wrong" moment like the Office 97 case in TFA, so I didn't feel a lot of pressure. It was definitely enough rope to hang oneself, so the interviewers could form an actionable and objective judgment, but as a candidate, I didn't feel that I was on trial, more just chilling and talking tech with some other nerds.

2: The underlying skills (understanding one's audience, breaking a large problem into pieces, making reasonable assumptions) were obviously directly applicable to the position. I never felt that I was doing senseless work, even when the questions were clearly contrived, because the connections were self-evident.

3: Despite the above, the actual questions didn't require a lot of domain-specific knowledge. Even the one that sorta did, could be answered by assuming that it was similar to a more common system, and that's how I approached it.

4: In every case, there were "plan B" options and accommodations for having a gap in one's knowledge. "I don't know this off the top of my head, but here's where I'd go to find out and here's the guess I'm going to use in the meantime" was a valid answer. Perhaps the best answer in some cases.

The result is that I work with a team of people who are super adaptable, can think on their feet, but have a keen interest in pushing towards correctness as soon as real information is available. Whenever I think about possibly going elsewhere, I remember that most HR departments don't select for any of those attributes, often quite the opposite. (And those places tend to be the clients whose problems we get called in to help solve, as a result.)

I experienced an extremely simple (and IMO effective) version of this.

For a dba-adjacent job, I sat down with a dba and was shown how to connect to a sql server instance and change some particular setting. At the end of our hour of discussing other technical matters and my resume, he asked me to demonstrate recall of this simple command.

There must be a serious dearth of candidates in my area, because the last time I was subjected to a time-constrained test was almost three years ago, and it was a Codility challenge, so I had some time to prepare for it.

I quit my job recently, and in order to find another I went full-auto with the applications - I've sent close to thirty of them I think, so a long series of recruitment processes followed.

They were all mostly the same - I either received some homework to do or had a 1-2h technical interview during which I was asked a set of questions which were easy for the most part. And that was it.

My point being: I've discovered that I can't really relate to the stories other people posted in this thread. Does this mean I wasn't ambitious enough regarding my applications, or are the others subjected to a difficult job market? Maybe both? Or none? I really don't know.

I keep hearing and reading horror stories like this one on HN and other website. I never did interviews at big companies, nor did I do any in the US, in 15 years of freelancing, I only had once a technical interview, with a C++ guru, I failed most of the questions but still got hired because "you still did good, kept calm and asked good questions".

In my last gig, it was for a work on a Java application. I was upfront that I only did a bit of Java in school, but that I was more of a C++ guy. The Java guru hirer was like "Yeah, if you know C++, you'll get Java quickly" they were more interested in my experience with video and image processing.

I basically avoid to do HR-based interviews. Short-circuit them when you can.

That's how Holberton School (we train people to become software engineer) select and train students. We are a college-alternative where students learn by working on projects, in group.

The whole education is flipped compared to a regular one. We give students problem sets and they have to find the answer. We never give lectures (we have no teachers), just guidance so that they can get started on a topic, but never enough that they can actually achieve the tasks we give them.

> Are they able to read a manual? Candidates (no prior experience in coding needed) need to answer questions on a specific command that can be found by reading the manpage. At first we provide the manpage, then we show them how to find a manpage, and finally, we don't provide any guidance, because at this point they should be able to do it on their own. In the curriculum, once students, we have a quizz to makre sure they can answer basic questions on the topic before jumping into the coding part.

>Can they formulate a search query? We ask trivial questions that can be answered just by Googling. Might sound like a no-brainer to the developer community, but not everybody is good at this.

>How do they assess whether the tutorial they found is suitable or reliable? That's the tricky part, but basically, because the school is not providing material that contains the answer, candidates and students have to navigate the ocean of information/tutorial that can be wrong/incomplete of correct. By pressing a button, students work (code at this point) is automatically and instantly corrected, so they can easily figure that what they built is correct or not.

> What steps do they take to make sure they're finding - and learning - the right information? On this one, we as a school, are the one providing this guidance. Basically when students are doing their projects, we define learning goals. Getting coding part done is great, but we also want to make sure that students have been able to grasp concepts that are key to becoming a great software engineers. So students look at these "learning points" before, and by the end of the project, they must be able to discuss all the points.

Haha, that's funny, and an awful idea. At a previous employer, as a hiring manager, I offered candidates an option of bringing in their fully-configured IDE in their own laptop with the language of their choice and I'd give them the problem to solve. Essentially, act in the environment that you've already mastered.

Only one guy did (I had maybe 10 a month) and for some inexplicable reason he couldn't figure out how to create his `main` equivalent in his language in his IDE. He must have been incredibly nervous because we tried "perhaps you could sketch out your thoughts in broad strokes and then we can return to the IDE" etc.

I just got interested in the way the school 42 works. Especially the month-long "test" that they have to accept tuition free students.

The idea is to make you complete a lot of programming projects in languages and tools you normally don't know at all. When you start. The evaluation does not only take into account if you finished project but also if you helped others, knew how to look for help (basically: google or other students). Its goal is to select students who would work well in a peer-learning project-based-learning paradigm. This, in my opinion, is much closer to real-life conditions than any academic test.

Our team uses hacker rank or something similar (I forget) to weed out bad profiles (most of them). We combed through their question base to remove things that felt too specific. But I’m happy with the result, because there’s nothing you can’t google, and googling is allowed.

We’re a python and R shop. You can choose either test. As a non R user, I can get a decent score on the R test, because I can read docs. And that’s great.

We have also dabbled in game based assessments, and while I have qualms with it, the same kinds of skill sets seem to be the primary differentiator of success.

My 2c on two types of assessments people probably normally disregard

Another take on what the author proposing is to simply not care about the output of the task, but focus on how that person approaches the problem (even if they don't manage to complete the task at all).

Honestly, I wish more companies would just tell you in advance of the interview what the specific prompt is going to be (and perhaps the evaluation criteria too).

Sure, you could "cheat" by Googling the answer in advance or asking someone for help. But if the question's sufficiently open-ended, it's not that hard to suss out in follow up questions whether you have a deep first principles understanding of the subject area or you're just repeating something you read.

I don't know about this. I don't think I should spend most of my time learning completely new tools.

If people spend most of their time learning tools, that encourages tools which are easy to learn but lack deep and powerful functionality. I think it's fine, and even good, for tools to require focused learning to understand, and in exchange have a worldview and tool use that is really powerful. Like Emacs or vim or Excel, for example.

See also Kobayashi Maru

"a test of one's character or a solution that involves redefining the problem and managing an insurmountable scenario gracefully"

I do hiring interview in my company and one of the test we like to do is to give the interviewee a problem out of his area of expertise and open a browser and an editor for him and tell him that he is allowed to search for things. Most of people I interviewed do not use Google help they are afraid you judge them negatively for it even if you say you will not.

This reminds me so much of Vernor Vinge's story "Fast Times at Fairmont High". It's set in the near future, and one plot point is that high school tests cover this skill--to read through the manuals quickly and get things done with new tools. I recommend it, and the related book "Rainbows End".

I like how one of the comments is about someone getting a job by telling the interviewer they knew what a computer was.

This is an excellent idea!

rather than give them the application, just show them the file (location, name, and extension) and ask them to make some useful revision to it

fwiw this describes my experience of interviewing at Databricks

If the test doesn't reflect the job you'll be doing then it's not a good test of whether or not you'll be able to do the job. If you're applying for a role that expects you to know an application then looking in the help is cheating. If using the help is acceptable in the job then it's not.

It's also worth noting that people don't like "proxy tests" for the skills they'll need to use on the job. If the test only tangentially assesses a skill then it's going to be influenced by other skills or flaws and won't give an interviewer a good idea of how good you are at something. This is why people complain about whiteboard exercises in developer hiring. Unless remembering and explaining an algorithm on a board is something you'll actually be doing in the role then it's not a good test - it sort of assesses how good you are at explaining an algorithm, but it really assesses memory, presentation skills, etc. Someone with brilliant whiteboard skills isn't necessarily a good developer, so hiring them on because you were impressed by how well they explained something in front of a board feels wrong, especially to people who aren't good at the tangential stuff.

In the case of the suggested hiring test in the article, it doesn't test someone's job skills. It tests how well they look for help, but it also tests how well someone copes in unfamiliar applications, or with esoteric UIs (in the case of Blender), or if they've actually used the app that's being used as a proxy. None of those things are necessarily a good assessment of whether the person can do the role they're applying for, so the test fails.