Technical interview methods pale in comparison to playing Factorio with someone | Hacker News Reader

Technical interview methods pale in comparison to playing Factorio with someone

erikmcclure.com

412 pointsblackhole344 comments

We used this a couple times at Sandstorm back in the day. At the end of the interview the candidate would play Factorio cooperatively with the team for a while.

I think it is remarkably effective at identifying the kinds of skills and personality traits that a software engineer actually needs to have in day-to-day work. You can find out if someone is self-directed, how fast they work, whether they produce clean designs or spaghetti code, whether they are good at cooperating or tend to go off on their own, etc. Some people will just sit and watch and do nothing unless instructed... that's bad. Some people will build stuff, but with obvious efficiency flaws and "bugs"... also bad. Some people identify what needs doing and get it done effectively but without trying to be perfect... that's good. Factorio essentially compresses real-world work patterns into a shorter time period, giving you the chance to see how someone works in the space of a couple hours. I don't know anything else that does that.

But, obviously, it's problematic. A person who has played the game before will have a big advantage. A person who doesn't play video games at all will have a big disadvantage. For these reasons, I don't think I could seriously recommend this approach be adopted more widely.

But then, I don't know of any approach to interviewing that I think is fair. Everything has problems:

* Regular interviews bias towards people that are charismatic, not effective.

* Coding interviews bias towards people who can code under pressure with someone looking over their shoulder, which is almost nothing like real-world coding.

* Puzzle-y algorithms questions identify people good at clever solutions but that's hardly what most people need to do day-to-day when coding.

* Take-home assignments bias towards people who have lots of free time, and still don't really tell you how that person works with others.

* Looking at GitHub history biases against people whose lives are too busy to code in their free time (e.g. maybe they have kids) and who weren't lucky enough to have a previous job that touched a lot of open source.

* Looking at past work experience misses brilliant junior programmers coming out of college.

* Don't even think about using ML for this, that doesn't solve biases, it just hides them in a black box.

It seems like there's no right answer here.

There really isn't. At the end of the day every company I have been in has: the asshole, the smart guy, the idiot, the guy everyone is scared of, the guy that can't be fired ("the originals"). Then you just kind of multiply them randomly depending on team size.

These hiring articles are a huge pointless waste of time.

My company had all of those rolled into one. Thankfully he burned himself out with the technical debt that he had created and quit. He was smart and could probably beat me at hacker rank type challenges. I know to focus on maintainability over cleverness.

Makes me feel lucky to be in my job, and a jolt of gratitude every-so-often is a welcome thing, so thank you.

I work with 8 other senior or senior-ish engineers, we are all basically taking a slight paycut (not too much, maybe 80-90% of rates we might get elsewhere) to work for a non-profit making interesting stuff for kids, and everyone is genuinely nice and fairly hard-working.

I wonder if the "taking a paycut to do something actually interesting and worthwhile" is part of it.

Hi, could you share a little of what you do? It didn't even cross my mind I can work for a non-profit, still get paid (not top dollar) and have a good community impact (kids in your case). Thanks!

I work for a grant-funded educational software company that makes science simulations for K through college, but mostly middle school and high school.

Hire quickly and fire quickly with a generous severance. That’s the only method that I’ve seen works effectively. Netflix does more with 2000 engineers that companies with 10-20x that.

This also lets you take chances on engineers that weren’t “perfect” but had a lot of positives to offer.

> hire quickly and fire quickly

This kind of mentality is not conducive to a well-functioning nation with people who have dependents and need stability.

Saying that it isn’t conducive to an entire nation is a bit of an over-dramatization, don’t you think?

I think a generous severance makes up for that. Plus people would know what they are getting into, just like Netflix. If you don’t want the risk/return you don’t have to apply or accept a job there.

> Saying that it isn’t conducive to an entire nation is a bit of an over-dramatization, don’t you think?

I don't think so.

> I think a generous severance makes up for that.

That's your opinion. But what you think is generous isn't necessarily what others think is generous.

> Plus people would know what they are getting into, just like Netflix.

I disagree.

> If you don’t want the risk/return you don’t have to apply or accept a job there.

The risk/return needs to be clear though and hirefast/firefast isn't always clear.

If the nation is well-functioning, getting fired isn’t the end of the world. If your job is independant of your healthcare and you have several years of allowance to seek another one it’s fine to try out a few jobs before finding what sticks

I would reverse it.

The nation is not conducive to business if it can't provide healthcare and safety nets for those without work or in between jobs. In the US we are held hostage with healthcare based on our employers - if you're even lucky enough to have a stable job with benefits.

To say nothing of the universities.

Show me one coutry who pays unemployed benefit at the level of the job you need to leave because of that insane hire fast fire fast.

In France your jobseeker's allowance is 80% of the average salary of your last 12 working months. Whether your last employer was for 10 years or 10 days. It starts slowly ramping down after the second year of searching I think

French unemployment has been well above US unemployment since the 70s. This isn’t a free lunch. If the state makes employing people more expensive employers will employ fewer people.

Given sufficiently generous unemployment benefits (ie. mincome), I fail to see how that's a bad thing.

Because most people who do work are being productive, doing useful things, serving their fellows. And being unemployed if you’re of working age and without some other productive role is awful.

Unemployment is bad for society generally and for the unemployed. Maybe one day we’ll get to fully automated luxury gay space communism but France is not there. Those extra unemployed people are having their talents and productivity wasted by economic mismanagement.

"Hire quickly and fire quickly" has other issues than healthcare and safety nets though, moving across a country and spending weeks trying to find a place to live (possibly losing money on rent because you can't find anything to buy yet) only to have to do it again in a few weeks' time is difficult.

Less of an issue if you're full remote, or you provide good accommodations, I guess.

Indeed. And of course the biggest advantage of them all: living in a big city. One of the main reasons for the salary difference: it's a lot easier to get raises if there are 10 companies offering the same job

Not to mention that most apartments have mininum lease durations, and penalties for breaking said leases. So if you move for a new job, sign a one year lease (because that's all that's available, or you can't afford to pay extra for shorter terms), and then a month later you're handed a box and a security escort...

I definitely think that if every employer used Netflix's strategy there would be issues, but if the job market had more churn stability would take on a different meaning. Imagine that it was expected to change jobs frequently if you or your employer was unhappy; I'd bet lots of people would be way happier.

The problem is of course that even if you have every reason to think that some other place would hire you, you need a degree of slack to begin with. There aren't any easy answers there, but that's where I'd say an expanded social safety net should come in.

In the SF bay area for these roles the job market is much more fluid than most places

Past patterns can be an interesting contrast. During WWII generals were hired and fired quickly. Because the practice was pervasive there was no big stigma to being fired and there were always alternative positions to try. It also helped that there was a general structure which was not explicit or required but was broadly exhibited. That is, new hires were expected to show basic proficiency within two weeks and to make a significant contribution with in six weeks. Since a relatively quick test of skills was built in to the management structure there was less pressure on initial evaluation.

These arguments are silly. If you’re competent you’ve got nothing to worry about. If not, then something to work on.

Incredible hiring risk aversion is what makes finding a job for everyone a shitshow. Definitely not an efficient market as it stands.

If it weren’t for an easy to hire organization that gave me a chance, I’d likely be homeless.

It isn't a company's responsibility to create a well-functioning nation.

Says who, Milton Friedman?

Surely companies have some minimal responsibility to act ethically, and surely behaving in a way that causes social dysfunction is not ethical.

It's the responsibility of the government to create a well-functioning nation, and to use corporations to this end. But to suggest that people or the corporations have a direct responsibility, is to shift the responsibility away from those who are on 'our side', and to those who are either neutral or its primary victims.

Surely every stakeholder has some degree of responsibility in the proper functioning of any system.

By your reasoning, I am free to degrade public infrastructure such as roads because it is not my responsibility to build them.

Equivalently: I am morally justified in committing crimes since I am not a police officer.

I have a responsibility not to violate the law or vandalize the roads, but I don't think I have a responsibility to drive extra carefully or patch up potholes as I see them.

To me, this seems like a very cynical scheme. We tell "normal people" that they, personally, ought to BUY AMERICAN to prevent jobs from moving abroad. When they eventually do, we can tell them they should've done more and that it was their fault. It shifts the liability on to the victims, rather than the perpetrators.

But the responsibility of ensuring that a country has a good trade deficit is upon the government, if anyone! There's no serious way I can prevent it, so the only real point of saying it's my responsibility is to diminish the government's responsibility for failing to prevent it.

To suggest that the companies have a moral obligation, is to suggest that an abstract body corporate can have a conscience. If it could, there would be no need to ban slavery - just expect that the honest businessmen of America cease to traffic in them.

>I don't think I have a responsibility to drive extra carefully or patch up potholes as I see them.

We agree. We also agree that businesses do not have the responsibility of fixing society.

However, they do have the responsibility not to degrade it.

When you add in "responsibility" like that, it becomes very vague. What does it mean for a company to have morally transgressed? Does it mean that it has violated its conscience, or does it just mean that it's an unfortunate behavior we would do best to punish?

Companies are inert objects. They do not have consciences. I can't say that a company does an evil thing, any more than I can blame a rock someone else threw for hitting my head. It hurts, but I can't very well blame the rock for it. Companies are just like driftwood in the ocean; their motion is defined solely by the forces acting upon them.

What is left, then, is to suggest that society should go after the companies that don't do what we want. But this can never be done by individuals. It has to be done by unions, governments, or something of the sort. It cannot be done by the companies themselves, and it cannot be done by individuals.

When you suggest that companies are doing something wrong, and not the government for failing to regulate them, this is tantamount to suggesting that the responsibility for punishing them actually lay on the individual. But to do so makes the primary victim into the perpetrator of the aggression against him. It is a terrible thing to do.

>Surely every stakeholder has some degree of responsibility in the proper functioning of any system.

A civilization enforces responsibility through the law.

>I am free to degrade public infrastructure such as roads because it is not my responsibility to build them.

Yes. And either the government should set laws to prevent you from misusing the infrastructure, and/or the people should shun your company for being a bad actor. The former being consistent for infrastructure that is owned and maintained by the tax-funded government.

>I am morally justified in committing crimes since I am not a police officer.

Morality and legality are different things. Neither people nor corporations should be judged by the government through the abstract, subjective morality of individuals, only by agreed upon, written law, and at its loosest which is a court's interpretation of the law.

You as an individual can hold anyone responsible for anything under any criteria, but then you should state you're speaking for yourself as per your own moral code. As a society, we have rules for that.

If you want to make your moral code the law, then that should also be explicitly mentioned as such instead of speaking as if it's the status quo.

>Yes. And either the government should set laws to prevent you from misusing the infrastructure, and/or the people should shun your company for being a bad actor.

It seems we agree. Similarly, governments should encode the civic responsibility of businesses into law.

Whether or not they have actually done that is a different debate.

Netflix doesn’t act unethically. They are upfront about how they hire and fire and people select into applying there and jobs there based on that. Same as CostCo has much higher standards for employees than Walmart and much higher benefits. If all employers were like Costco lots of people would be unemployed who currently have jobs. Having an employment model that is not perfectly average is not unethical.

It would absolutely work -- assuming everyone has access to healthcare (preferably universal coverage).

Unfortunately this cuts at the moat that big corporations have set up not to mention health insurance industry and big pharma. Which is why it hasn't happened yet here (though it has in many other countries).

Everyone has access to healthcare in the US, under ACA and Medicare.

Only when the pay doesn't compensate for the instability, which in Netflix's case, is a lot. You could work two months and have enough saved to last you an year.

Why? COBRA covers 3months of health insurance; ACA covers more, and unemployment insurance and severance pay gives you months to look for another.

The only major problem would be relocation, so I'd suggest working a few months at a new job before relocating for it.

You have to pay the employer's share for COBRA, which is usually thousands of dollars per person per month. Right when you've lost income from the job.

The Netflix model makes a ton of sense, but doesn't apply to most other companies, bc Netflix is a unique case. They pay top of market, all cash. They have massive name recognition and obvious longevity. Probably most uniquely - they only hire senior people who have a track record of experience they can look at.

When you're a startup that nobody's heard of, half your engineers are going to be junior, and half your comp is stock options that will probably never be liquid, the offer doesn't make any sense. Are you just going to give offers to 20 random new college grads and coding bootcamp alums planning on firing the 50-75% that don't cut it? Good luck ever getting junior people to accept offers again. Do you think someone with 10+ years of experience is going to take a pay cut to take the job with a random startup with a decent chance of getting let go if the manager isn't feeling it after a few weeks?

With 15 years of experience and a family to provide for, there's no world where I'd consider that type of hiring/firing culture for anything other than a proven brand name with top of market comp, environment, and coworkers. Did an interview loop w/ Netflix a couple years ago, seems like an amazing place to work, ultimately went a different direction for non-culture related reasons.

Start ups need different kind of people than big companies. With a startup the biggest mistake someone can make is not moving fast enough, and the startup dies. In a big company the code base always gets too big, so a person who works slower, but is able to improve the code base while shrinking it is worth more than a person who finishes fast, but blows up or slows down the code base. Interviews have to search for the right style of people.

That has major issues with immigrants, who wont apply in a fire quickly scenario.

You can only do this if you have the name recogition and compensation to attract enough candidates who are willing to take this risk. Plus, if you have long on-boarding processes this could become quite expensive.

Yeah. I used to know someone who worked at a call center once. The onboard training was like 40+ hours. They had a problem with people who only got hired so they could be paid to watch videos before quitting on the first day, but there wasn’t anything they could do about it.

This approaches biases against those whose residency status is dependent on employment (e.g. H1Bs).

There’s no silver bullet when it comes to hiring.

Which is fine because H1Bs shouldn't be used for most positions. If something actually requires an H1B candidate then you use a different process.

H1Bs are one example, but isn't living in a high-rent area another? In either case, you can't maintain your physical position for long without continuous employment.

H1B is only meant to fill positions that companies "say" they can't fill with american citizens. It has just been abused to hire people cheaply.

Let's focus on solving problems in america for american citizens first, then we can worry about everyone else.

You probably don't need Hadoop, and you probably don't need 10x engineers.

> This also lets you take chances on engineers that weren’t “perfect” but had a lot of positives to offer.

This is not true. An AS/HFA/Autistic person wouldn't survive very long at Netflix, because they biologically struggle with the goal-setting/performance review game. These individuals may have talents that are biologically unavailable to neurotypical people.

The Netflix approach selects for a very specific type of individual: an extreme degree of rote knowledge and the ability to effortlessly apply that knowledge. There are people who acquire this knowledge and talent via unconventional means, and Netflix is able to grab those people up.

I can personally confirm that there's at least one autistic person in a senior engineering role at Netflix.

n=1.

In addition, that individual is likely wasting mental resources gaming performance reviews, or their manager has identified their struggle and is protecting them from the ceremony. Systematically, Netflix is not equipped to deal with them.

Everyone either wastes mental resources gaming performance reviews or ignores them as a sideshow to actually doing a good job. The better the performance review the closer these two approaches are. Unless you have experience of Netflix’s culture and management you are less qualified to opine on whether Netflix is equipped to deal with autistic employees than someone who knows an autistic Netflix employee who is successfully working there.

> Everyone either wastes mental resources gaming performance reviews or ignores them as a sideshow to actually doing a good job.

You can find out that this is a false equivalency in DSM IV. Spectrum individuals struggle with certain executive functions, and performance reviews align incredibly well with those metrics.

You've claimed the anagolous equivalent of "everyone gets tired climbing stairs, paraplegics are no different."

> opine

I hate to tu quoque, but I'm one of those individuals and I do struggle with performance reviews - as would most (not all) adult diagnosed individuals. Literally page 5 of my AS/HFA workbook. I'm lucky to be managed by someone who gives me qualitative reviews, instead of quantitative ones.

I have also been diagnosed. Autistic individuals aren’t paraplegics in this analogy. They are dramatically shorter than average. Working under a disadvantage is not the same as being unable to do something.

It’s not like the population of people who take performance reviews seriously rather than as bullshit hoops that have to be endured or gamed is large.

I have two questions here:

- what is the shortest time they keep you before firing you? 1 week? six weeks? 2 months...?

- I suppose that the level of internal documentation must be stellar. Can someone comment on this?

While I get some of this sentiment, wont some sort of social deduction game like 'Throne of Lies' be better at identifying potentially good team members?.

I'm not familiar with that game, but the point of using Factorio is it is a game where you have to build complex systems and maintain them, which has many parallels with real-world software engineering. "Throne of Lies" doesn't look like that kind of game.

Throne of lies and social deduction games like them are about finding out peoples real intentions from what they say and do, also aspects like using and understanding subtle signalling to collaborate with ones team.

In my software engineering work I do a lot more “writing documentation to be as clear as possible” than “social deduction and subtle signalling” ^^;;

In fact i would hate to work at a place where you have to do social deduction

> Some people will just sit and watch and do nothing unless instructed... that's bad. Some people will build stuff, but with obvious efficiency flaws and "bugs"... also bad. Some people identify what needs doing and get it done effectively but without trying to be perfect... that's good.

Some personalities are better suited for programming, although I think hard work can overcome anything.

I remember an old blog post [0] of mine that used Hammerstein-Equord personality quadrants of smart/industrious, smart/lazy, stupid/industrious, stupid/lazy and describing it for how that effects tech team planning.

[0] http://www.prepend.com/2009/05/programmer-classifications-by...

> Some personalities are better suited for programming, although I think hard work can overcome anything.

I am not sure about the hard work part. Some people don't seem to think the right way for programming.

I studied a one year IT masters conversion course back in the dot com boom, and chose the best / hardest course I could find. The first 3 weeks of the course was an intensive Java programming course. There was an exam at the end of this. They suggested that anyone who didn't pass this exam, don't do the rest of the course, but they didn't actually force them off. I was friends with at least two people that failed that exam, but continued anyway. They struggled and I don't think they passed the final exam despite plenty of hard work.

I think hard work and studying is probably better than natural aptitude and no hard work.

I’ve worked with people who didn’t care at all, it was a job, but they studied and worked hard and were pretty good.

Of course, I’d rather work with someone who had both.

I had a pleasant interview experience one time, that avoided a number of those problems.

I was asked to bring in my laptop with some code I had written. I did, we talked about it. The interviewer asked me to add a simple feature. It's code I was familiar with (as you would be after working for a company for a bit). So no stress. No gotchas.

No time wasted with yet another technical test with a throwaway app. Setting up a new project is always a bit of a faff.

Admittedly not everyone will have some code ready, but then a normal take home test would be the fallback.

It seems like there's no right answer here.

I think you've done a good job identifying the most common options. Since none has proven decisively better than any other at aiding hiring decisions, how about letting the candidate decide which sort of interview to participate in?

> But then, I don't know of any approach to interviewing that I think is fair

The companies that think deeply about interviews are aware that their interview is biased, and they do so intentionally. At a certain point watching 5 good engineers not meet the bar is better than having 10 bad engineers get through.

Maybe the 'right answer' is to let your needs drive the bias of your interview. Do you have a position that needs to deal with immense pressure with management breathing down their neck? Bias your selection with whiteboard interviews. Are you looking for a principal engineer that can drive adaptation and change in a large org? Bias your selection process to get that.

tl;dr Maybe the issue is the one size fits all thinking to the tech interview.

You really hit the nail on the head here. I couldn't agree more.

However, I think playing this game would be a fun and interesting interview.

We used it at Cloudflare a few times too (as suggested by former Sandstorm people). I still remember getting the purchase of the game approved by engineering leadership. Was pretty humorous where it went from wanting to understand to whatever approved in like 30 seconds.

I agree with your assessment of the pros/cons of the various interview methods. I agree that there is no right answer.

Huh I actually had no idea it was tried at Cloudflare!

A reminder that the goal of an interview process is not to be fair. This is not a sports event or a court case.

Mostly your goals align with "being fair", but not always, and that's OK.

I agree, so upvoted. The most important question is: "Would I like to work with this person?". That is unfortunately unfair.

> A reminder that the goal of an interview process is not to be fair. This is not a sports event or a court case.

Given the power that companies have over individual workers, I'd say that a job interview should be treated as seriously as a court case. So yes, making it fair should be an explicit goal.

A company's goal isn't to hire every good potential employee, it is to hire a number of good employees, and avoid hiring bad ones. If Factorio wizards make good engineers, who cares if only 1/10 good engineers have even played?

Who cares if they are all white 23yo frat bros? who cares if no woman can get hired so long as the company makes hires it likes. Or gay, or Jewish, or whatever busy is in your selection.

I'm not asserting there's a specific bias in this particular interview choice, but I am asserting that if you were taking a fairly narrow part of the population you better be working damned hard that you don't have hiring biases that are not relevant to job performance

> A person who doesn't play video games at all will have a big disadvantage.

Even more so for a person who can't play video games, e.g. blind people. Many of the other interview techniques, for all their faults, at least don't (inadvertently) discriminate against people with disabilities.

> person who doesn't play video games at all will have a big disadvantage

My thoughts exactly while reading the article.

I've honestly simplified my approach to interviewing over the last few years and I'm pretty happy with the results.

I just have a conversation and ask them to tell me about a favorite code project they've worked on. Could be school project, previous job, whatever.

And just let them talk about it. I'll prod for more detail if I need to give a little nudge. Ask about problems they solved when doing it, etc.

The _only_ thing that I'm looking for in this process is whether or not the person is truly interested what they're doing. This comes through when talking about something they're excited to have worked on, seemingly small problems that they had to really work to solve, etc. Geeking out, if you will.

The single most important trait I've seen in programmers that I've worked with over the years is "level of interest". There's too much that you have to learn on a daily basis to be successful in this field. If you're not interested in learning new things and finding solutions to the problems that you encounter, you're not going to make it very far.

Beyond that and basic competence in the core skills that the job requires, I don't look for much more.

Thus far, everybody I've ever hired who has passed this little conversational test has been an excellent team member.

This.

I was a hiring manager for 25 years, and this was exactly what I did.

I made very, very few mistakes, and actually had some pleasant surprises.

> I made very, very few mistakes

How do you know? How often did you follow up on rejections to see whether they should have been hires?

Actually, for the company that I worked at, and the level of folks we hired, I often heard about how they went.

The only one that made me sad, was a chap who used to show up on the Adobe Photoshop splash page. We offered, he declined.

In the aggregate, I think things turned out OK for all of us. There are some folks that I have to admit, probably did best by not accepting us.

>There are some folks that I have to admit, probably did best by not accepting us.

Yeah, I hear that. I remember interviewing someone who was _good_, but not at this narrow thing that we were hiring for. She seemed SO hopeful that we would give her an offer, I wanted to tell her directly "this job is going to waste 6 mo of your life and offer you nothing I return"

My most fond recollection was a young chap that was a "second choice." The first declined, and this guy had very little of the exact skills we needed, but he had a real eagerness.

He turned out to be an amazing hire. I think that he far outstripped the first choice, and he has gone on to do quite well, after leaving our company.

I like to think that I ran a fairly good shop. We kept people for decades.

When they finally rolled up our shop (after 27 years), the person with the least seniority had ten years.

Same here. There's no silver bullet question, and no high pressure situation with me.

My approach has just been to have a conversation, and try to move it along naturally. What do you think of concepts? What useful STL did they add? Why did you go with this design? Ask me something.

If the person doesn't really have the level of interest required, they will run out of things to say. Like a cave left unexplored. The good candidates will go down all the caves with you, even if they don't quite know what's there.

In particular, the times I've had dud hires it was not possible to know from the standard fare of coding questions and IQ questions. When I was doing that obviously the people I hired passed. But for instance one guy really didn't want to be a programmer, he just knew how to answer the questions. He ended up stalling on his first assignment, and eventually decided to change careers. Another dud also lacked motivation, perhaps because he was asked to do something not quite in his area to begin with. He ended up not doing anything at all.

Apart from those two, dozens of others have turned out ok. I never ran into that guy who couldn't do FizzBuzz, despite not asking. I also don't think it was all that bad to let go of the duds within a quarter, they never got so ingrained in the team that we depended on them, and they could plausibly move on to another job without mentioning us on their CV.

I’ve been job searching lately and it is brutal. I’m fairly social normally but interviews terrify me. I freeze up, lose my train of thought and I don’t think my enthusiasm for my work really comes though. It’s probably not as bad as I imagine but it’s really frustrating.

If you will permit me some trite but genuine advice: I have been interviewing and hiring people for well over a decade and it's not much more fun as the interviewer. Just pretend you are having a conversation with a friend about a project you really enjoy. It's kind of like worrying about how you will perform in a race. The training is already done when you show up to the starting line. The race is the easy part. You know what you know, just demonstrate it. Let the chips fall where they may, as they say. If you fall short, so be it. It's kind of the Zen approach to interviews. Don't try to hard, just do what you know how to do and find that place where you can be genuine.

Same here. Motivation and will to learn beats almost everything in a ever changing world of IT.

Look for curiosity, interests and blend them with your product.

> A person who has played the game before will have a big advantage. A person who doesn't play video games at all will have a big disadvantage.

There's also a "similar game" advantage. For instance, a person who played games like OpenTTD before will have a much easier time with Factorio rail signals, since the two kinds of rail signals in Factorio work very similarly to two of the several kinds of rail signals found in OpenTTD.

so, you've pointed 7 flawed existing approaches as the reason to come up with and use an 8th one. At least those 7 have actual connection to programming. Your 8th one has only imaginary indirect connection only through your internal interpretation. It is just like those box-building/etc. games they play at Scrum training which have nothing in common with reality.

And many flaws in the 7 aren't intrinsic flaws, instead it is flaws of the wrong "one size fits all" application:

> Looking at past work experience misses brilliant junior programmers coming out of college.

yeah, don't use that as negative for junior programmers coming out of college.

>Take-home assignments bias towards people who have lots of free time, and still don't really tell you how that person works with others.

the same, if candidate prefers that way - let them, if not - use other approach

etc.

> * Don't even think about using ML for this, that doesn't solve biases, it just hides them in a black box.

Can you clarify: what is "ML" here? "Machine learning", ML the language, MatLab, "machine language" (a.k.a. assembly), Matt LeBlanc? I'm out of ideas...

Machine learning

I think the answer is that it takes time to get to know someone and their skills. There is no quick simple test. You can give them multiple tests over time if you want.

I think that if you have money for recruiting then you can screen with an initial test, then pay them for a few hours or days of additional tests. And then ideally there is a contracting period that is an additional test but on a real project.

> Regular interviews bias towards people that are charismatic, not effective.

A cooperative multiplayer game is also going to be a similar soft skills check that favors people that can communicate not just well, but also charismatically.

That said, a possible advantage to doing that in a game, as opposed to a regular soft skills interview (or worse trying to force such soft skill checks into the pressure cooker of a whiteboard interview or puzzle questions) is that you maybe provide an environment where the interviewee can be dopamine hacked into believing they can relax and you maybe get a glimpse into how the person's soft skills check out on the other side of that boundary where interviewee knows that they need to be "on" and acting.

I'm not actually sure if that is good or bad, but a lot of my takeaway from this article and other comments in these threads is it sort of feels like there's a lot of excuses for "pick a game that seems technically similar", but most of the "good feedback" that seems to be coming back from these experiences resembles what history tells us about regular interviews versus technical interviews: soft skills matter a lot more than what you can gauge technically in an interview. Most technical interviews are just soft skills interviews under the pressure cooker of taking a non-standardized pop quiz your professor decided you need to take because you showed up to class in only your underwear (and also you are flying for some reason in this nightmare). Cooperative games possibly present the opposite of the pressure cooker: dopamine hacking the interviewee an excuse to let the pressure seem like it is off while you are still doing that same sort of soft skills check.

(Aside: problem solving is classically a "soft skill". "Behavioral interviews" were as much about problem solving decades before the software industry invented the "technical" interview.)

(Further aside: This sort of dopamine hacking is very familiar to anyone that's seen Fraternity/Sorority rushes. Soft skills are hugely important for "group fit"/"team fit" in college social groups as they are in employment groups, and seeing how potential new members interact in "fun"/"game" situations often tells you more about the person than if you sat down and interviewed them one-on-one in a suit and tie or equivalent and they know they have to be "on". Rushes include both of those things for many reasons.)

> I don't know of any approach to interviewing that I think is fair.

That is true, because it was my motivation to build a wizard for behavioural interviews designed to remove biases and be objective. In practice, most of the time people are not even sure what they are looking for - I know it when I see it - and default to gut hiring. This leads to teams made of people that think like you, so you're basically paying 2 salaries for one brain.

My formula for removing biases is in 2 steps: first define explicitly the behavioural traits you desire, and second, use falsifiable questions to determine the truth.

Because we're here to help each other, if this is relevant to you, I'd be happy to share this solution with you.

> if this is relevant to you, I'd be happy to share this solution with you

Yes, please!

I hope it is ok to share this here: https://freyasense.com/recruiting/

Give me a sign and I would be happy to give you a tour

> Coding interviews bias towards people who can code under pressure with someone looking over their shoulder, which is almost nothing like real-world coding.

I don't understand this. Why instead play a game, which, as you said, could have disadvantages because there are people that do play and people who've never played. I guess the thinking goes that by easing into a game you can see clear behaviors of people. But you can also try to ease them into coding while you are looking over their shoulder. The interviews I do, and I've seen done at my previous company and the one I'm currently at, it is similar to pair programming, and yes, there is people that is super nervous at the beginning, and that seem to need a lot of hand holding at the beginning, but after coding and having a few small wins start to ease at the test and then they can surprise you with very nice coding or interactions on solutions.

Please don't. Better, I'll remove the please and go JUST DONT FUCKING DO IT.

Hiring is broken because most people hiring aren't good at it and the people working in tech are pretty much idiots (I'm focusing more on SV-startup, 5hour discussion about aeropress, put people down because they chose vim or emacs - people kinds). From 'cultural' fit interviews (who the fuck cares if you are a sikh or an atheist, whiskey drinking or think alcohol is evil? You are hired to do a job, not much more). This 'i have my social life at work, so I want people I can hang out with' shit has got to end. I work remotely for 10+ years and I can see in interviews/meetings the people that seem to base their identity and life around work, and the ones that actually are well rounded people that don't care about which coffee grind size you use on your morning crap of joe.

I have been a professional boxer and dabbled with MMA/Grappling. I can say most things in the article can apply. So.. at the end of the interview just get them in gloves and duke a few rounds? I would be sued out of existence, but for some reason, between take home assignments, 10h whiteboard drilling and whatever the fuck is hip now (leetcode or what not), interviewing is broken because people are idiots. I have been interviewing people for the last 5 years. If in an one hour talk that is semi technical, you can't understand if the person knows what the fuck they are doing or not, then the problem is you, not the whiteboard or the take-home or the paid assignment. Just admit most people interviewing others for technical positions have not fucking clue or training, BUT, hey, they are smart people from Standford or MIT, so 'how hard can it be' and we get the clusterfuck that it is now

Thank you for writing this. It bothers me that the genealogy of leetcode has been completely obscured as it comes from competitive programming competitions whose primary demographic were who? Those smart people from Stanford or MIT. Once you realize the "who", it becomes even more clear for me why we're dealing with 'i have my social life at work, so I want people I can hang out with'-type people when we're preparing for interviewing. Not to mention that crowd is least likely to have any experience communicating with, say for example, Black folk, amongst other demos, as peers, you you get all these issues surrounding application of "culture fit".

> as it comes from competitive programming competitions whose primary demographic were who? Those smart people from Stanford or MIT.

I mean, it comes from every intro to algorithm and data structures classes at every serious CS program.

> Not to mention that crowd is least likely to have any experience communicating with, say for example, Black folk, amongst other demos, as peers

Not sure what you are insinuating here... That folks from these schools or who do good at algorithms don't see black folks as their equals? That sounds incredibly racist.

> I mean, it comes from every intro to algorithm and data structures classes at every serious CS program.

If you wanted to indicate the level of generality here you wouldn't have used the adjective "serious"

> Not sure what you are insinuating here... That folks from these schools or who do good at algorithms don't see black folks as their equals?

I said "least likely to have any experience communicating... as peers". Hard to talk about "culture fit" when you don't have experience gauging certain folk who're far from sharing your background. That's just simple empiricism, which is why the reciprocal notion is true for many Black folk as well. De facto segregation has ruinous effects.

> If you wanted to indicate the level of generality here you wouldn't have used the adjective "serious"

I've seen a lot of programs that have "IT" or other technical terms in their description but that would not prepare a student for a real Software Engineering position. Same thing with a lot of bootcamps.

Serious in this case means any accredited 4 year public or private university. So, not boot camps, not sketchy for-profit crap like Trump university. I went to an unremarkable 4 year public university and they focused quite hard on data structures and algorithms. I thought it was the same everywhere else...

Given how accepting techies (in the US and even Europe) have been of "Chinese folk" and "Indian folk", I don't really see why they would be any less accepting of "Black folk". This stereotype really needs to die.

Big agree to everything here. I've seen my fair share of office romance, and wow, folks, it sucks. When your work is your family, friends, your whole life, it's gonna be a dating pool for people, too. When you start sleeping with one of your "family" members, that's fine, whatever. But when you inevitably split, work becomes a soap opera, everybody feels the need to pick a side, banal decisions (like attending a meeting or not) become political. Shit.

And to think, I've been told by a department head that the way to get business done with a certain co-worker is to go out for beer with him. So that's my fault for not knowing / doing that. No matter that I don't drink. No matter that I'm not cool enough for this dude to want to go drinking with. Big ol' fuck off, loud and clear. Wouldn't it be better if he could just respond to my emails like everybody else is expected to?

As someone who is actively interviewing, this really resonates with me.

yes, I agree.

Honestly I was looking at this and thinking it's not too far off from introducing yourself as Tyler Durden and then testing him.

> people that seem to base their identity and life around work, and the ones that actually are well rounded people that don't care about which coffee grind size you use on your morning crap of joe.

These don't seem related. I happily base my identity around my work. I don't care about my coworkers personalities or inclinations at all, as long as they're not assholes and they're competent.

Maybe I'm misunderstanding what you mean by identity? I don't hire people because I want them to be friends.

What I mean by basing your identity about work is that your personality, your life, your social circles, etc etc is based around work. Is like someone basing their identity on their sexual orientation, or the sports club they support. Their work is their life, almost a cult like following, and use that work for making others part of that circle (think cultural fit shit, about people not wanting to hire/promote people because they don't hang around for friday evening beer social hour, or in a kind of upper management thing, not wanting to hire people that don't play golf)

I understand your frustration, but I somehow need to figure out who to hire and who not to even if it can't be done reliably.

I need to ask some technical questions and I need to give simple coding exercise on a coderpad.

You would be surprised how many people can smooth talk themselves through any non technical person but can't do shit when faced with an actual IDE and a small problem to solve.

The real broken part of this is that there is minority of candidates who just can't develop and go to a huge number of interviews looking for one place where the process slips up and lets him pass or maybe be very lucky with the questions. Even though they are minority the constitute majority of interviews just because of how many interviews they try to attend.

I did mention that a one hour tech talk. I don't think you should willie nilly hire anyone just because they are a smooth talker.

Quick example, Hiring a mobile developer: You talk with them, if Android, ask how Kotlin interfaces with the JVM and what they think (good or bad) about type erasure. iOS? Ask/talk a bit about memory management/ARC. Anyone that has experience with these languages/os (unless hiring for very junior position) should have some thoughts and opinions about it. They may say something you don't agree with (ARC is crap because X or Y) but they will know what they are talking about. You can also dig in to know if it is something they read online (ARC is automated reference counting, meaning each 'allocation' increases a ref counter. Ok, that is good, so, why use `autorelease` or `release` and when to use each? etc etc kind of questions)

You can do the same with backend (framework vs libraries) or frontend (react + state libraries) or whatnot, but if in an one hour tech chat with someone, you can't understand if they actually know they can at least code similar to giving them a fizzbuzz or whatnot, it says more about the interviewer than the person being interviewed.

One technique I use is I ask them what are the things they know well. Hopefully, there will also be something that I know. I then start digging in the area they have already advertised they know well and try to judge what it means when they say they know something well.

Also if a candidate can master even one thing very well, it gives hopes they are at least somewhat interested in development and can learn other things, too.

What I found is that people who are not interested in development but rather do this because it pays well almost never learn anything very well. They just don't have drive and curiosity needed to dig one topic long enough after what they have learned was enough to complete the task they were assigned.

I don't know man, I don't think the coderpad thing should be your litmus test. Would looking at their github and talking over something they've written suffice? Why not? I get too nervous to code in front of someone on coderpad. It sucks, my mind goes blank. But otherwise I'm a pretty goddamn productive programmer. I've interviewed with a lot of places and the only place I could get traction with agreed to forgo the coderpad and just look my github projects over. Everyone else loved me from talking to me but were dumbfounded when the coderpad shit came up. I feel like I'm kind of fucked in this field I've worked incredibly hard for the past decade in.

> Would looking at their github and talking over something they've written suffice? Why not?

For a simple reason. I don't want to select only people that have github projects. I think it is wrong.

If interviewers grade candidates based on github projects then I can bet there will be market for people to fake their projects. It is just reality.

What about creating incentive for people to work entire days, at work as well as on a private project? Is that ok?

You landed your job based on github project? Good for you.

Good lord, this is an awful idea. I've played a ton of Factorio, and I'm a decent engineer, and while the skillsets are similar conceptually, the actual execution and mechanics of them are very very different. Factorio involves a lot of care with spatial layout, tricks to optimize loading the conveyer belts, ensuring that the right ratios of pre-cursors are sent, etc etc.

You could get a candidate who perfectly understands design patterns for programming, because they've studied programming and know how to program using programming languages and programming tools, but they don't translate that very well into the spatial environment of Factorio. Debugging and optimizing in Factorio again are conceptually similar to doing so in a programming language, but the actual techniques and patterns available to do so are very very different, and you have to learn them. Most of us do not figure these things out from scratch by ourselves.

As others have pointed out, it also is begging for a monoculture. Plenty of people who are good at programming don't give a shit about video games, and will either do poorly in this interview, or will skip it altogether. The result would be hiring one type of person, and not having any other skillsets on your team. Even as someone who would do well in this interview, I would run screaming, as I do not want to work in a monoculture.

Also, you might end up with a bunch of industrial engineers who may not code (yet).

> A senior developer should be able to explore the UI and figure out a goal, then establish a plan for accomplishing that goal.

I'm confounded on why the author thinks being a senior developer (say, in embedded systems) would make someone better at exploring a game UI compared to an intern-candidate who puts 20+ hours into gaming per week...

I would probably walk out of the interview if someone suggested I play factorio as part of the hiring process - I have been intentionally avoiding it - you might as well offer me a free hit of crack cocaine while you're at it.

Factorio isn't a game about shooting things. It is a systems-design game. Your goal is to orchestrate the processing of several different resources (iron, copper, coal, oil, stone) into several different intermediate products (gears, circuit boards, sulfuric acid, radar systems, etc), in service of a final deliverable (a rocket launch, or if you're ambitious, a continuous series of rocket launches).

To that end you must manage a variety of subsystems and how they interact with each other and with the components that you build. You will be rewarded for consistency, for modularity, for automation of frequent tasks, for reproducible designs, for looking ahead and preparing for scalability. You will be punished for failing to plan ahead. You will also be punished for trying to scale too fast, too soon.

The interview is a bad idea, yes. Even if all candidates were experienced with the game, as a showcase of how to build things, you'd still need many more hours than a typical interviewee is willing to provide, to showcase the approach. But it's not a game you get better at by focusing on "gaming".

> Factorio isn't a game about shooting things.

Thank you for providing additional information, but I already had a good idea of what Factorio is; that is how I know I would be addicted if I were to ever try it.

I was specifically criticizing the idea that a senior developer should automatically be more proficient than an intern at navigating the UI of a game - even when that game is Factorio. It's seems preposterous to me, even though it's a tiny detail in a larger essay.

> navigating the UI of a game

Why does it matter if it's a game? Navigating the UI of GitHub, JIRA, VSCode, etc. are all important. Being able to learn new skills is a good thing to judge people by. Seeing how quickly they pick up tools and apply them to solve problems is a great way to see how good a teammate someone can be.

People just get hung-up over the fact that it's a game -- I don't see anyone complaining about the fact that you have to figure out how to use the HackerRank UI during an interview for some reason.

My wife wasn’t a gamer before she met me. There are a hundred little things that gamers take for granted that non gamers don’t know to expect. You have to explain things like “there are lots of places you can go here, but if you’re a gamer you know it’s all fake: there is only one way to progress and there will be green lights at that spot. Why green lights? Who knows, green is just a standard message games use to say go-here, you know about it because of all the games you played in the past. Also, for some reason there will be items hidden if you wander around”

You say Factorio isn't a game about shooting things, yet probably the best way to be useful to a team game you've just joined is to fill your inventory with rails, drills, and combat equipment and then point useful at the nearest unexploited ore patch and go forward until all the red spots between the patch and the factory are gone. Doesn't prove anything about your programming ability, but since the factory is always gonna be running out of some resource anyway you'll be perceived as useful

> You will be punished for failing to plan ahead. You will also be punished for trying to scale too fast, too soon.

Repeatedly playing a game helps one internalize and "have a feel" of when the timing is right. I agree with you - it's a terrible way to hire people, for the reasons you state, and because it unfairly favors people who have played Factorio before.

The thing about game engines is that they can only represent a simplified approximation; for example driving the "same" car model in 2 different games feels very different. Therefore game resource management and complexity are nothing like real-life. Would you want to work with a Director/Project Manager on the strength of how good they were at StarCraft 2 in an interview? After all, it's about resource management, planning and timing, right? </snark>

I assume everyone is curious and can easily filter garbage to find useful ideas. But I’ve been surprised the number of times juniors got stuck and weren’t able to Google and find libraries that are top on stack exchange.

Or designs with alternative analyses that honestly leave out the top toolkits for a project because they found the gartner report and didn’t go any further.

I don’t consider these junior/senior types of things, but there are things I have to realize vary across people and not everything is intuitive or easy to everyone.

I am a senior engineer and have been programming for close to two decades. I picked up Factorio a few months ago and felt overwhelmed. I couldn't finish the tutorial. I think my problem is spatial relations--it is something I've always struggled with (how to arrange furniture in a room, solve a jigsaw, etc).

I don't think this is a detriment to my programming ability at all--spatial relations really never enters into architecting a large system. Of course you need to think about design constraints but none of them exist on a physical plane.

Factorio, on the other hand, requires actual spatial relation ability. You need to visualize how belts intersect and how to best position things so they dont interfere with each other. This is where I struggled.

Well according to the thesis put forward here you aren’t a developer they’d want to employ .

As a job seeker you need to identify these companies fast so you can avoid wasting your time on them.

That's not really how interview processes work.

If factorio interviews fill all roles with competent personnel, reduces hiring of incompetent people, etc., then it could be "good enough". Sure, you miss a lot of people who would also be good [enough].

They're not saying people who can't play factorio can't develop; it's the opposite. The hypothesis is surely that the can't+can't group has a large overlap.

Why not use starcraft instead and filter by apm and multitasking? You are sure to miss out on a few good candidates but you will filter out a lot of people who cannot manage 3 bases really effectively while watching the minimap, while designing custom build orders that can survive in the current meta. It’s like interviewing boxers on their blitz chess playing ability because you want “people who can think ahead under pressure.” Please.

Skipping candidates “just in case” is like not hiring women “just in case” they distract the men. Those defensive hiring practices destroy diversity, diversity of thought, and the chance for a human to be evaluated in a fair or rational way.

I don't think an arbitrary choice of test is "like not hiring women", but other than that yours is a valid criticism (personally I'd suspect it to bias for wealth).

I somewhat buy the anticipatory response elsewhere that essentially it's no worse than some of the other arbitrary choices -- basically, every style of assessment will miss some good candidates.

Also a senior engineer. I tried factorio demo out for maybe 30 minutes, but hated the fact that I actually have to walk the engineer around and do menial tasks like resource collection myself. Why can't I just do stuff with mouse cursor like in most other top-down strategy games? Maybe the game changes in later stages, but the demo seemed to be full of nonsense busywork that is prevalent in survival games.

Independent of the thesis of the article, which is obviously terrible, it does get much better very quickly.

The "gather resources yourself" stage is like the first 5 minutes of the game, more so to force people to automate; I actually think it's an intentional stylistic choice to demonstrate how they are different. Like the first technologies one gets automates all of the resource collection in the game (and later tier resources can't be collected by hand). It becomes a pretty good point and click strategy game very quickly. In fact after the first couple hours you can basically play the game from the map screen if you want, using blue prints to put down large buildings and spidertrons as units to move around, without moving your main character ever again.

Like this is how the mid game looks (user designed blue prints, bots build it automatically, etc): https://www.youtube.com/watch?v=M0k1YeMm-6o

The manual resource collection aspect becomes automatic once you have a basic factory setup.

There's also a mod called long reach[1] which allows you to do most things without walking around. If you want to give the game a second chance I'd look into that.

To give an analogy with programming. The demo is more like setting up your programming project, which can be a bit tedious. While the normal game play is more like thinking about the software you're building, defining good abstractions, thinking about how to structure your code both on a macro and micro scale, keeping things DRY etc

[1] https://mods.factorio.com/mod/themightygugi_longreach

You might like shapez.io

> and do menial tasks like resource collection myself

That is PAINFUL in Factorio for a reason. It forces you to automate it away as soon as humanly possible. Which is the point of the game.

>Why can't I just do stuff with mouse cursor like in most other top-down strategy games?

Because thats not what the game is. If you want that, play something else.

> and do menial tasks like resource collection myself

They've actually changed the early game so this part isn't necessary anymore (assuming you're talking about mining ore).

I agree with this - having watched the trailer and played the demo, Factorio looks to me like a game of managing logistics of transporting resources around, featuring a conveyor belt system that only works in 2D.

In another game, I'd consider that an annoying micromanagement feature.

The conclusion of the article somewhat contradicts the title:

"What's the takeaway from this? I don't know. We certainly can't switch to using Factorio as an interviewing method - you might as well just give a candidate a take-home assignment. At the very least, we can do better than whiteboard interviews."

So it's not a description of how to use Factorio for interviewing, beyond a simple assignment of Factorio tasks to seniority levels. It does however make an interesting comparison between Factorio and software mechanisms.

This is a great way to end up with a monoculture, with all of the fragility that’s associated with it. That might be a fun way to hire engineers 1-5 in a startup but it will hold you back in the end. Many good engineers have no interest in games, and won’t present well in that arena. You’ll get lots of false negatives.

“Fizzbuzz is all we have...” is a myopic take. Interviewing is hard but not as hard as the article makes out. There is a legitimate debate between take-home and on-site interview practices, but at the end of the day you want to see a work sample that is as realistic as possible from your candidate. Companies that are good at hiring understand this. Factorio has a tenuous (at best) correlation to the work you’re asked to do.

It’s like doing puzzle questions because you “can see how they think about problems”, but interviewers are really just selecting for people that think like they do.

(And for context I like Factorio.)

> but interviewers are really just selecting for people that think like they do.

No. We don't care how you get it done, but that you try in a way that would be worth paying you for.

My job isn't a calm safe place, it's the chaotic result of two departments and the proverbial rubber hitting the road in one spot. It's not intentionally bad, mean, hard, or long, and we weed out people who don't cooperate well, but every crazy thing you can imagine has happened.

But even in previous, easier, positions people were still put on the spot by being assigned to work with customer support on some issue way up the stack from their normal job, which required to be on a call with 20+ outsiders, from CEOs to SREs, and representing us and our department properly. The people on HN who talk about giving people calm interviews to see their true selves don't get it, the performance under a small amount of stress is far more interesting. Do you get mean and snappy, get entitled, shut down, or what?

If being asked, kindly, to solve a basic leetcode is too much then there's no way you'd be suitable for anything else we do.

How can this not have a huge bias in favor of gamers? Might work for them because that's the kind of culture fit they need, but I doesn't look all that generalizable for other companies.

You only have to understand how to move your character using WASD, the concept of Inventory and pointing at different places on the ground with a cursor. Failing to understand this simple set of rules would disqualify a person from most jobs ever invented.

But its biased against people who can't use mouse and keyboard :)

But on serious note, People who played the game could just hide the fact that they know the game mechanics. And act like they are catching up faster then normal person would.

Isn't that a common criticism of every interview style? Whether it's whiteboard questions, take-home projects, soft skill questions, lots of people will lie and say they've never seen the question before.

At this point, you would be foolish to admit that you’ve seen a question before. Everyone is faking it, and if you be honest, you’ll just keep getting harder and harder questions from the interviewer. The only time you should not fake it is if you’ve been asked that question at the same company. I think they would keep track of that and if you didn’t mention it, they would look deceitful on your part if they catch you.

> you’ll just keep getting harder and harder questions from the interviewer.

Well, there is a counter strategy that works even in this situation.

The counter strategy would be to just know way more interview questions than the interviewer, and know the answer to every question that they have, so that they run out!

When I was full time applying for jobs, a couple years ago, it got to the point with me. It turns out that people interviewing others, only have a couple questions that they use for interviews, and it is not that hard to know all of them.

>People who played the game could just hide the fact that they know the game mechanics. And act like they are catching up faster then normal person would.

This is a universal problem with all interview techniques. Interested in any ideas for how to fix it.

It's even a problem with "personal interviews," because some people have had more experience with optimizing one-hour first impressions than others.

Most interviews in other industries try to figure out how much a person knows at that point in time. Software interviews try to do the impossible and try to figure out how quickly someone learns. That’s because this can be gamed, as you mentioned. I know people that have memorized 800 leetcode questions and got jobs at all the FAANGs, even though they were average developers. For the record I’ve never gotten into FAANG (although I have gotten into FAANG-adjacent) so my evaluation should be taken with a grain of salt.

The best interview is one that can’t be gamed. Just like Hollywood auditions. Give the people the script and see how well they do. Tell the interview candidate you are going to give them 3 questions and they pick the one they want to do, even if they know it already. This strips away all pretense that they are figuring it out on the spot. At least then you are on a level playing field.

It's not about understanding the set of rules, it's about applying it in real time while playing. It adds a huge mental burden on the interviewee if they aren't familiar with it already. And the older you get, the harder it becomes.

>Failing to understand this simple set of rules would disqualify a person from most jobs ever invented.

I think you dramatically over-estimate most people's literacy with games and movement in a virtual space, let alone most people's hand-eye coordination. How many people can touch type these days?

Why would the amount of software engineers that can touch type be down?

You literally only place your fingers on the WASD keys and you move your character in a 2D (!) virtual space. With Minecraft it would be a completely different story, but Factorio is really accessible.

> You only have to understand how to move your character using WASD, the concept of Inventory and pointing at different places on the ground with a cursor.

You make it sound so easy. Once I gave my PS4 controller to my dad to play a driving game. Even though he's a better driver than I am, it did not go well.

Using a controller for someone who has never used a controller for and mentally mapping the act of driving to it is probably more difficult then handing a mouse and keyboard to someone interviewing for a software development job.

Game controllers are different beasts altogether. It's always fun watching a hardcore PC gamer struggle with a DualShock (or the asymmetrical weird thing they use on Xbox) for the first time. I know what you mean, but Factorio is really much slower and much easier to pick up than a driving game on a controller. I'm not saying it's a great choice for a one-hour interview, but anyone should be able to do whatever they want in Factorio after a couple of hours.

Yeah it's not about "failing to understand", or "being able". It's about having advantage if you are already familiar with similar things. It's about those few extra things that you don't have to spend a single thought on anymore, giving you more headspace to think about the problem at hand. And it's also about feeling less stressed because you are in a rather familiar environment.

Can you really not see how such an interview would bias towards Gamers?

Worth pointing out that McKinsey developed their own game for demonstrating problem solving in interviews.

I don’t think anyone is arguing that McKinsey has a bias towards gamers.

I will argue McKinsey has a bias against humanity and good things, but that's off-topic :-)

I don't think it would have a huge bias towards gamers in general. the real issue would be people who already happen to be good at factorio. if you play with experienced people, it's easy to pick up on a lot of efficient patterns without really understanding why they work.

then again, I guess that's also true of software.

Reminds me of questions related to sports.

I have zero interest in sports and am clueless to the rules of baseball or football or basketball. I hated when I got questions that assumed I had any such knowledge.

Granted this was way back when when interviewers like to toss brainteaser questions to cargocult Google/Microsoft/etc...before they started doing leetcode to cargocult Google etc.

I'm D tier programmer who would easily land a programming job if factorio was the skill metric.

If you're actually D tier, and you're good at Factorio, why are you a D tier? You clearly have the mindset for problem-solving. What is holding you back?

If GP had the answer to that, they wouldn't likely be held back.

Software development as a career is a lot more than just learning, design, applying logic. We still have to manage projects at a people/organizational level, interact with team members with various backgrounds, understand what people are asking of you.

There's a lot to being a successful developer that is probably so easy for many individuals that they might not consider that a person might be IQ smart, but unable to handle such a task at a professional level.

I'm in a similar boat to the the GP: I do really well on logic tests and programming challenges, but I struggle with the the soft skills necessary to be a good team member.

> you're good at Factorio, why are you a D tier

Because these actually have very little in relation.

There are very few off by one errors opportunities in factorio

I'm not a programmer, I only know enough C to do simple embedded micro designs. Electronics is my jam, so I probably could get much better if I took the time.

Do you control trains with circuits?

It might, if they were actually suggesting this. Title was intended to race to the top of Hacker News.

>What's the takeaway from this? I don't know. We certainly can't switch to using Factorio as an interviewing method - you might as well just give a candidate a take-home assignment.

It’s unclear if this article is satire or not. Given that you will be spending 20 hours of at least two people’s time (one candidate, at least one employee), the space of alternative interviews is quite large. You could, for example, spend 20 hours observing them perform an actual software task. This piece doesn’t consider alternatives at all, merely asserting that this particular 20 hour game is the best that can be done. I see no reason to believe that comparing the play of a practiced factorio player to that of a brand new player would carry much signal at all. That means that a company carrying out this type of interview would need to ask candidates to study by playing the game.

The article clearly states that this is not a realistic thing to do.

This winter break I spent an embarrassing amount of time on Satisfactory on Steam, which is essentially a nicer (and probably easier) modern refresh of Factorio.

While I played, I came to similar conclusions to the article. Building factories in this game is very very similar conceptually to building software. I ran into exactly the same problems that I ran while I build software like:

- If I don't plan ahead enough the scope and layout of the factory, I end up with a spaghetti mess that is very difficult to rectify (technical debt)

- If I plan too much ahead, I overwhelm myself without even starting. The amount over-engineering and over-preparation becomes counterproductive and demoralizing.

- Starting new factories is fun. Maintaining and extending a factory that is starting to show serious design issues is a chore. I always tend to want to scrap the factory and start fresh.

- While designing a factory, modularization is key. You can go with a "monolithic" factory where you provide all possible materials as input, and try to build everything as output. It is very efficient transportation wise, and can centralize all management, but it can and it will become an unmaintainable mess. You can also design factories as "microservices", where each factory is a very compact, clean and scoped. It will only produce nails, or rubber, or copper wire. When you need to increase production of that item, you just duplicate the module (horizontal scaling). It seems fantastic at first, but the issue is now transportation. Dozens of micro-factories have to communicate with themselves to combine and produce more complicated items. The physical distance makes planning transportation a logistic and construction nightmare. So you have to find the right compromise between monolithic and micro-services.

I think I agree with the article that you can extract a lot of information about how good a person can be at software development by the way he plays Factorio/Satisfactory. Not so practical though :)

+1 on this. I could never get into Factorio with the simple graphics and 2D top-down.

Satisfactory is freakin incredible and it is easily the best new game I have played in the last 10 years.

If you enjoyed Satisfactory you might also want to try out Dyson Sphere Program. It doesn't provide the veritical building but it is a different take on the factorio/satisfactory style of gameplay.

I tried DSP, very fun game but the performance is terrible. i7-9700K and RTX2080 and I get 20 FPS before using even 1/10th of the first planet. Can't recommend it currently.

I've only played the demo of factorio, but on the same computer (AMD Ryzen 5, RX540) DSP runs much better for me - there's a noticeable slowdown on save now that I've covered about half my starting planet but other than that it's stutter free.

Interesting, while i haven't built a sphere yet I have factories going on multiple planets and haven't noticed any performance issues. I'm on a a 2700k with a 970GTX. Not suggesting you don't have issues, but perhaps there is a fix?

I am the complete opposite of you.

I could never get into Satisfactory with the 3D graphics, but I absolutely love Factorio.

Satisfactory is really great, highly recommended from me as well. I think the verticality really adds a lot of fun over Factorio where you are restricted to keeping everything on the same level. Can make some really confusing spaghetti though...

The loss of 3D was one of the biggest things I missed when moving on from industrial mods for minecraft (the inspiration for factorio).

I recently got an RTX card and played around with the minecraft RTX demos. I decided to look further into behavior packs, and apparently the scripting API is much more capable than it previously was. The community of people making content is basically nonexistent, but I now think it's just because people have moved on.

Everything is in place to have added blocks, oregen, ticking tile entities, power generation/distribution, multiblocks, etc. It would be nice to see a behavior pack that picks up the torch of gregtech/industrialcraft.

Unfortunately, most of the unofficial GT versions have lacked proper vision. That is, vision in a way that is not aligned with GT proper, and what really draws players of industrial games in.

Anyway, here is a promising example. I hope to see more industrial content for minecraft for windows 10.

https://mcpedl.com/advanced-machinery/?cookie_check=1

Yeah, I love both factorio and satisfactory, but factorio ended up being the one I dumped more hours into.

Satisfactory ended up hiding too many details as you move around - the first person view and the very large structures made it really hard to untangle the mess.

Satisfactory felt like writing regexes - Basically write only, hard to read and parse afterwards.

That said, man the first person view is fun when you're actually doing the building. Just not nearly as fun once things have gone wrong and you need to debug.

Don't know if I agree on the height issues... every machine has ladders, they give you a tower fairly early on, and you can always walk on conveyors

As for debugging, I've found that planning a build on paper, keeping in mind production ratios, eliminates the need to experiment and debug, and then building simply becomes figuring out how to route conveyors

I've never had issues troubleshooting by getting to a high place and looking at the big picture, but to be honest since I started planning my builds on paper it barely even comes up

Disclaimer: I've only played the Factorio demo.

If that's the analogy, then I don't think it captures the core difficulty of software, which is unknown unknowns. That is, you write something that depends on an external system working a certain way, you follow its spec and depend on it working that way, and then it just ... doesn't, and you have to find exactly where it deviates, possibly making up some complicated kludge.

To capture that, there would have to be e.g. some hidden logic about which direction the output comes out that you have to deal with and work around until you understand it.

IANAProgrammer, to some extent I create my own GIGO issues which are like small instances of poorly defined behaviour. Sometimes you'll realise your whole world has ground to a halt and it's a conveyor backed up because one item arrived at an inserter and blocked it.

I play a bit of DSP but have at least 3 different mitigations which are akin to input sanitising. However, if you just stick a filtered inserter by a conveyor to grab errant materials then it can comeback and bite you ... though I've found either I can route them back to the proper channel or I can get away with the technical debt by throwing enough storage at them and leaving it until the thing needs refactoring.

You can have too much realism! I like that every green-circuit board works, rather than getting failures of systems that use that item with some random errors (that increase of your closer to the sun, say). I think that would be too frustrating.

I think this part is captured by Satisfactory "Tier" system. As you progress in the game, you unlock Tiers which allow you to build new materials and structures. The issue often is that you optimize the hell out of your factories to build something that is needed at you current Tier, and when you unlock the next one, you suddenly have to refactor everything to build the new items, that require different ratio of materials and inputs.

There's not a lot of unknown unknowns in factorio, but one thing which it really captures about software development is your own decisions coming back to bite you, or emergent complexity from notionally simple rules. As your factory grows you'll find yourself wishing you'd built things differently because suddenly something needs to expand beyond what you expected or some part of it is right in the way of a train network. Train networks are really tricky to make work effectively, it's very easy to have deadlocks appearing if you make a mistake, and you might not notice until you suddenly don't have any power because the trail with the coal is stuck waiting at a fouled up junction or something like that. There's a lot of opportunity to debug problems as they appear even in something you designed entirely yourself (and in a multiplayer game it's even more so because you might be detangling someone else's build).

A lot of the process of learning the game is basically learning patterns to reduce this. At the higher level of megabase design it's basically about designing a modular system so you can scale things out as efficiently as possible (and there's lots of possible variations of this).

I think this could potentially be the Biters - the enemies in the game that will periodically attack your base. You can guess that they will attack your base, but you are never sure quite when and it's always a concern in the back of your head.

I haven't played Satisfactory since the water update, so I have no idea if my approach still works, but IMO the easiest way to design an efficient factory is to not bother trying to balance anything, and scale out in only one direction as needed. Feed everything along a central stack of belts, and multiplex inputs/outputs via splitters/combiners so any individual machine line will always have throughput.

It doesn't make for a very exciting build, but it's very rote and dependable.

Also - build the entire thing high up in the sky.

Saying satisfactory is a nicer factorio is ignoring everything that makes factorio a good game.

As someone who has played a bunch of Factorio but only heard about Satisfactory: Can you elaborate on that?

Factorio is very much like programming on a good team, automation, expansion, refactoring, tooling used to solve your problems.

Satisfactory is cool, but at no point do you get the "ok so I built this entire factory and now I can just replicate it modularly" - no, just place every, single, block, by, hand.

To me factorio is a rethinking of the RTS game, satisfactory is like a really nice minecraft mod.

Factorio didn't have blueprints either, early on. I suspect Satisfactory will add them as it is still in early access

I think its possible, and I might switch over my opinion if such a thing existed - it'd still be hard to optimize it to build worlds of "factorio level" complexity.

They've said they won't have blueprints.

Now, granted, they might change their minds -- but it's hard to imagine how blueprints would work. Factorio is played on a 2D plane, with few obstacles in general and none that can't be removed; Satisfactory is played in a 3D world with immutable obstacles.

The game is tuned towards not needing blueprints. You don't need large quantities of anything, just a small amount of every item -- the complexity scales up, but the scale itself kinda doesn't. Yes, some players will attempt to turn the entire output of the map into turbo-motors, producing exactly the right amount of every ingredient, and will build their factory in the sky to avoid dealing with the terrain--

And yes, blueprints could be useful in this specific case. But that's not most players.

> satisfactory is like a really nice minecraft mod.

Which is ironic considering a big part of the original inspiration for Factorio was factory building mods for minecraft.

Factorio is (kind of) infinitely scalable. There is no max map size, and the game engine's performance is so incredibly good that it can support absolutely insanely massive factories. You can go as big as you can imagine, and the game will keep up.

It's also ridiculously stable, in the ~thousand hours I've played I've never had a crash or a game breaking bug.

Meanwhile satisfactory puts you on a fixed-size map and will slow down to a crawl once you build too much. This limits a lot of the things you can do in the game.

The simple answer is scale. The recipes in satisfactory and factorio are pretty similar in complexity, but the production chains in factorio are quite a bit larger. Also a lot of complexity comes in with scaling production up in factorio, since you largely have to scale horizontally, but in satisfactory, most of the scaling is vertical. Later, you can do more horizontal scaling, but even into the mid-game, you're pretty constrained by the amount of resource locations you have access to.

I play both. Satisfactory has more game in the sense of doing things besides designing a factory. Factorio is close to pure factory building, whereas it's like 70% in saisfactory.

Satisfactory has more elements of a traditional open-world action game (that just happens to borrow factorio's factory system).

I think Satisfactory is “nicer” in that it’s prettier and more user-friendly, has higher production values, etc. Not to speak for the parent comment but I took “nicer” to be a very deliberate choice instead of “better.”

It is true that Factorio is currently a better game and I suspect that will be the case even when Satisfactory is complete. I don’t really think it’s intended to be a 3D Factorio (even if that’s the easiest way to describe it): it seems that the design is to be less intricate as a construction management game, but with more details in the world design and player exploration: there are elements of No Man’s Sky or Astroneer that are absent in Factorio. So I am still curious to see where the devs will take it.

Yes that's what I meant. I don't know factorio very well, "nicer" meant just that Satisfactory is more focused on being a pretty game with cool graphics and world exploration, and less "hardcore" on the automation stuff.

Now check out Dyson Sphere Program, it's like those games but in space ;)

I just got it recently and I enjoyed the heck out of it.

I've got to the "I probably need to leave my starter solar system in order to increase my DS production rate" stage, and honestly I think that's my limit. I find I can't recall where everything is (I think that can be fixed to some extent) and I'm not about to start writing documentation for a game!

>I'm not about to start writing documentation for a game!

Ha, I have a dsp.yml file for exactly that :D

I'll hand in my geek card at the next AGM. :D

The latest update in Experimental (in Early Access on April 13) adds Drones for point-to-point delivery without needing connecting belts or rails or whatever. They're pretty low-bandwidth so you can't use them to transport raw materials at any scale, but if your "microservices" are sufficiently focused at a higher tier, you can tie them all together pretty easily with Drones now.

At least, in theory. Drones are T8 tech, and I'm not _quite_ there yet to start putting that theory into practice. But that's the stated intent, at least.

Also dipped my toes into Satisfactory this winter - after spending over 2,000 hours in Factorio.

Games like these challenge your ability to manage complex systems. Remembering which parts of your system processes which data (aka, "materials"). Finding and addressing bottlenecks in production lines. Maintaining and upgrading components, while creating new production lines using the techniques you learn. Managing time spent refactoring old systems vs replacing them with new ones. Etc, etc, etc. Planning properly - as you mention - is extremely critical to building a good factory.

However, despite over a decade of software development experience, and some time in Factorio, my first several play throughs in Satisfactory where an efficiency disaster, and even my recent ones were an ugly mess of spaghetti for the first few days of gameplay.

It took several playthroughs for me to grok the mechanics well enough to build a somewhat efficient factory. If my first couple tries were reviewed in a job interview I highly doubt I'd get the job.

There are strong similarities between these games and the mechanics software development, but like any new system, it takes time to create a true intuitive understanding of the mechanics and demonstrate them in front of others. If you're going to interview someone for a job, you're better off testing their ability to play the game that they'll be playing on a daily basis if they get hired: "Software Development".

Software Development, the game!

Want to play a game where you will never run out of new content? Where you are constantly challenged by new issues that'll haunt your dreams and make you lose sleep? Software Development is the game for you! With an ever changing landscape, where components and frameworks are updated daily! That's right, DAILY! This MASSIVE-multiplayer-online game never turns off. There are actively hundreds of thousands of players right now!

And you know what the best part is? Software Development is not "Pay-to-Play" like all those other games that try to steal those valuable dollars out of your pocket.

No, in Software Development YOU get PAID to play. That's right! All you have to do is find a company, nail a job interview, and enjoy playing the game you love while they hand you buckets of REAL LIFE MONEY that you can spend on other games that you have to pay to play. And also, food and rent and stuff...

Download now at, the internet.

Factorio runs on Windows, Mac and Linux; Satisfactory only runs (currently) on Windows. Factorio is the better option, though Satisfactory is a great game non-the-less.

The major difference between Factorio and software is you can't refactor in Factorio. Your options are to spaghetti or start fresh. You can't really pick and choose good pieces of your layout because of physical restrictions and the game is a giant math equation, and you want inputs to match outputs at every step of the way - if you don't have the space to do this you don't have the space.

It's like if in the software world you could greenfield, waterfall, or spaghetti. There is no agile in Factorio, which is broadly accepted as most commonly the best way to develop a piece of software.

But you can refactor... you can clear away everything and you get back 100% of materials

Demolishing your base and building a new one is not anything like refactoring.

Why not? You clear out the part you don't like, and build anew

No, code refactoring does not fit perfectly in this analogy. And your ability to do so is somewhat dependent on how you build. But I think all those factors are simply part of the puzzle in refactoring a factory

It's exactly like refactoring. You can replace a component that does something with a different structure for accomplishing that. Of course, if the system was tangled, then you may need to refactor other components first (read: relocate them to give room). The structure of the base around something is going to place limits on the refactoring. If the base is so horrible that you decide to gut it entirely, it's just like a rewrite.

The efficiency of "refactoring" in Factorio is like refactoring code using notepad while typing with your toes and you're only allowed to compile the source once you're done. Which is to say: on a completely different level!

You can absolutely refactor in factorio...in fact you have to refactor as you get access to better technologies. Just like software, refactorability is largely dependent on how well you design your base at the beginning. If you leave plenty of room for expansion and compartmentalize different production areas, it makes it extremely easy to upgrade with minimal disruption to the factory. And also like software, if you don't put any thought into the design at the beginning, it will probably be easier to start from scratch than try to redesign on the fly.

Ultimately how well your base is designed is completely based upon what things you need to produce in the future, what their inputs are, what their outputs are, what those outputs will be inputs into, and in which quantities you need to produce all of this stuff.

These are all things a new player has no clue about. All you're really saying is an experienced player that has done enough researching and equation balancing (X miners feeds Y furnaces and a line of belt can support Z miners) ahead of time should be able to design a base that doesn't require too much demolishing. And presumably, not overdesign in unnecessary base components.

It assumes perfect knowledge.

The closest to refactoring in Factorio is the ability to use bots to tear down and build blueprints. But you get that ability long after it would first be useful - that would be around when you get plastics, which easily triples the size of your base.

Ultimately how well your program is designed is completely based upon what features you need to develop in the future, what their parameters are, what their return values are, what those return values will be arguments into, and in what scales you need to run all of this stuff.

These are all things a new programmer has no clue about. All you're really saying is an experienced programmer that has done enough researching and equation balancing (X algorithms feeds Y threads and a processor can support Z cores) ahead of time should be able to design a program that doesn't require too much changes in the future. And presumably, not overdesign in unnecessary micro optimizations.

It assumes perfect knowledge.

The closest to refactoring in programming is the ability to use IDEs to remove dead code and do code generation. But programers learn about IDEs long after they would first be useful - that would be around when you get to integrations, which easily triples the code base of your program.

Everything you mentioned is also true about programming

I think that's why I dont want to get into these games. They feel like work!

This is the reason why I don’t like playing factorio. I’d rather make real software and get paid or build something usable vs a game version of it. I spend a lot of time already doing it as my job, and it stresses me out in a similar way.

I like factorio because I don't have to have 17 meetings to get alignment on changing 3 lines of code. "I objectively need more electricity, so I will refactor the nuclear reactor" is the only agreement needed

That applies to any project where you work alone too.

This is exactly why I don't play games like factorio or tis-100. It's too much like work for me.

I honestly stopped after the winter holidays for this reason. At one point, it felt a bit too similar to actually working... but I enjoyed it very much until the "spaghetti factory point of no return" :)

I seriously think that "personal projects" are very underrated when it comes to hiring. If I see that someone has a lot of personal projects and has lots of time in Factorio they probably know how to code. I'm not sure you can rank them as Junior/Senior too effectively as these two things don't require much leadership, but you can basically skip the trivial phone interview questions at this point.

often in multiplayer factorio playthroughs you get one player who becomes the defacto "lead dev". it takes some nontrivial organization to get subgoals completed in a way that people aren't blocking each other. a common mistake is two players building separate factories too close together, which leads to some very messy solutions and limits how much the throughput can be upgraded.

Looking at personal projects biases against people who don’t want to or can’t do them, either contractually or due to time/energy constraints.

Why wouldn't I want to take every possible factor into consideration? If I have two people, one of whom has a github with lots of personal projects and the other one does not, I'm naturally going to get a better sense of the github developer in terms of their overall capabilities.

That doesn't automatically make him the better candidate but it definitely gives me more information to act on.

If people don't want to or can't work on personal projects then that's fine, but I'm not going to ignore the people who are passionate about building their own stuff.

Not unreasonably. Like it or not, at else being equal someone with more current experience is probably the better employee.

There's this feeling that employers should just take the next person who fits some quota because to pick and choose is to be biased which has somehow become a bad phrase. My employer literally pays me to be biased. My job skills include having appropriate and helpful biases.

I doubt jobs for urban planners ask what cities you design in your free time. I doubt jobs for lawyers ask what cases you've argued in your free time. Why do we ask what software you've built in your free time for software engineering interviews?

Because in software personal projects are easy to do. It just takes drive and skill, which are both relevant skills. Passion as also valuable, it tends to show that people are willing to learn and have interest in the type of work that they will be doing.

Artist and even urban planners regularly have a portfolio, I don't see how this is any different.

> drive and skill

It also takes the privilege of time. For example, a single childless person likely has more time for personal projects than a single parent of 3. This does not mean one is a better candidate than the other.

> Passion as also valuable, it tends to show that people are willing to learn and have interest in the type of work that they will be doing.

You can have both of those things without "passion".

Why not ask? Programming is a craft, and having a showcase could demonstrate initiative, capabilities, and inventiveness. They're concrete samples of your work.

Aside, there seems to be a swath of people who seem bizarrely resentful of personal projects. Yes, we get it, not EVERYONE in the universe has sufficient free time. Bla bla bla, maybe cut down on Netflix.

> Aside, there seems to be a swath of people who seem bizarrely resentful of personal projects

I wouldn't say anyone is resentful, just rightfully annoyed. It's not bizarre. I just don't think we as an industry should require everyone dedicate every waking moment to working. It's not healthy. Surgeons don't do surgery in their free time and are not asked about it in interviews.

> Yes, we get it, not EVERYONE in the universe has sufficient free time. Bla bla bla, maybe cut down on Netflix.

I think the implication of not having enough free time is that a person does not have enough free time to do things like watch enough Netflix to free up enough time to work more.

I would argue that if you consider your personal project to be a form of work, then something is wrong. If the principal reason that you're working on your own personal projects is to be able to pad out your resume, then I agree with you.

I don't think surgery is a good analogy. Engineering is a field where you're actually producing something, such as a product, that you can share and be proud of.

It's a great tool for junior developers. If someone hasn't had a dev job before, then a personal project -- even something simple like a twitter bot -- is almost a requirement in my eyes. IME, it is a strong signal for capability at that skill level.

Most higher-level devs don't build personal projects. I've worked with plenty of ridiculously capable people who work 9-5 and go home to their family. If anything, diligently keeping set hours is probably a stronger signal for a good senior candidate than personal projects are. A good senior candidate should be doing the thing they want to do already (thus, no need for side projects), or be capable of getting everything they need done for their job in 8 hours or less.

I don't think there is any silver bullet way to do technical interviews. The idea that you can extrapolate a few hours of exposure to a person into a reliable predictor of future performance is somewhat silly to me. Any technique you use is going to be biased for people that are good at "X", where "X" is a tiny subset of what you need in an employee.

Obviously, you still need to do interviews, but keep some perspective when deciding. Don't overweight any one thing.

You can choose to make X (what's tested for in the interview) a representative cross section of the harder things that you need.

Or don't even try and make X not even intersect with anything you need.

It still puzzles me how many people opt "don't even try".

I see what you mean, but in my experience there's this tendency where you think you're testing for "X", but you aren't really.

A whiteboard FizzBuzz might be actually testing for memorization, deliberate interview prep, extrovert tendencies, etc. And not for any kind of actual technical skill. Failing the FizzBuzz could also just mean high social anxiety.

True. I find I have to iterate on my processes quite a few times before I squash these issues.

They're conceptually similar to software bugs - inevitable, often surprising and impossible to fix all at once.

Beyond Fizbuzz, the standard algo type interviews are just optimizing for time spent on Leetcode or length of time since finishing College.

I heard good advice for life and interviews (be they for work, marriage, etc).

For life, it's that you aren't your weaknesses but your strengths. Don't worry about filling the skill hole in juggling for instance, put that time into your strengths or hobbies. Be a better you, not a more complete someone else.

For hiring though, if you've assured basic competence you're now more interested in weaknesses than strengths. You don't need the world's best person in the role, you need someone who won't bring down the group - either in their incompetence, malaise, nasty behavior, or whatever.

So when being interviewing for a role don't worry about being the best, shoot for 60% and you're golden - but do your best to avoid showing even a single red flag.

For anyone interested in Factorio as well as the technical notes of optimization. I highly recommend another game: Daison Sphere Program.

https://store.steampowered.com/app/1366540/Dyson_Sphere_Prog...

The leading programmer uses some sophisticated ways of optimization, including:

- DOP instead of OOP

- Leveraging GPU for parellely calculating certain stuffs

- Using a GTX 660 Ti for development

Here is a technical post but in Chinese. Lemme know if you are interested and maybe I can do some translation.

https://www.zhihu.com/question/442555442/answer/1711890146

Dyson Sphere Program is so good. I haven't played Factorio yet, but I think it has all the same sorts of things that this article described.

When I started playing DSP at first, I was just putting stuff anywhere. Then I realized that there are advantages in trying to make things modular in a way roughly analogous with code. At the same time though, it is very different than code, as the design is spacial, not as abstract as coding. Refactoring code is very different as a result.

Yeah exactly, those games really bring out the "programmer" mindset out of everyone :D

I've only played factorio demo (too expensive for me) but the parts seem almost identical. factorio doesn't have the different planets, and different solar systems, etc, AFAIK, but the basic by building blocks seem near identical to me.

The curved build surface causes lots of interesting aspects to arise.

What is DOP?

data oriented programming

I've never played Factorio and have no particular opinion on its value as an interview method, but I assume some of whatever value it has comes from forced team interaction rather than anything inherent in the game.

I have turned down an interview because of Diablo. I got recruited a couple times by a startup that would have been a very good fit for me. The first time it was quite small and I chatted with the CEO about possible roles and product directions and such, but I'd just quit a long term job and eventually decided that I wanted to take a break for a while.

The second time was years later after it had grown a lot and gotten tons of funding over multiple rounds. As it happens, the CTO was also a friend-of-a-friend. When the Diablo 3 beta came out, I played a bit with him and our mutual social circle, and he demonstrated enough personality traits during that brief experience that I decided I wasn't interested in socializing with him more, and just turned down the recruiter after checking to see that he was still the CTO. I would have at least discussed it further if he wasn't specifically in a leadership position.

It seems like Factorio itself doesn't matter here, and that any programming language can be used instead.

So really, this article is saying "live, self-directed coding is the best technical interview", which isn't very original.

There may be an anxiety factor which negatively impacts live coding interviews which wouldn't manifest as often in demonstrating similar skillsets in a game. It would be interesting to see how much of a difference something like this makes.

Yeah ... maybe...

There might also be anxiety around playing a game you don’t know well, which wouldn’t manifest in a language you were familiar with.

Honestly, this article is pretty superficial.

Performance anxiety (or the lack thereof) is a job skill.

Looking forward to this becoming the new standard with an ecosystem of bootcamps, an infinite stream of blogs, youtube videos, and personal tutors teaching you how to play optimal factorio while candidates sweat bullets and cry.

Also don't forget all the blog posts about how hiring is broken, particularly because high school dropouts are able to put hundreds of hours into the game while senior swe are busy with their jobs and families.

Yes, confidence to try something new and problem solving are important factors but once one solves for those then testing for the ability to communicate well with others might be as or even more important than having the required computer skills.

I used Zachtronic's TIS-100 [0] as a test variable and found the best person to work with. I believe it was a two way street because they saw an opportunity to work with someone who attacks problems a little differently. I am in real estate. Who would have thought knowing something about assembly would be helpful?

[0] https://www.zachtronics.com/tis-100/

I don't know that I agree. I don't really play games so I would be a bit lost here, but if you needed me to design and start coding a large distributed system I would happily oblige while he watched...

Isn't a factory a large distributed system? What's the difference between this and a visual programming language like LabVIEW?

In the abstract sure, but in the concrete and in the way I think through problems its just not the same.

Well, LabView doesn't have locals who get pissed off by your pollution, for one.

In Factorio you can turn off the locals.

But turning off the locals for your factory is probably not what you want to do.

I test my developers in Counter Strike to see if they have a backbone, teamwork and can control rage.

I dislike defining seniority by your technical knowledge or skills. I have a different, I think more useful definition that I use. It overlaps a little bit but excludes knowledge, intelligence and focuses on mindset and ownership.

More senior person is one that has more mature approach to their work. One that can get better results with less resources. Typically more senior people can also accept wider variety of tasks.

What is better results and what is less resources will depend on the project.

It has nothing to do with the knowledge or experience, in my opinion.

You can have good senior person get better results in a technology they have no previous experience with compared to a very experienced / intelligent / knowledgeable person that has track record of making bad decisions. Senior person will use their good judgement to warn the manager they are running outside of their knowledge and to know when they need to use some resources (like help) from somebody more knowledgeable.

A knowledgeable but less senior person may think they know anything but not be able to recognize they are trying to achieve something that is outside their area of expertise or not be able to recognize or accept they are bad at something.

For me senior developer is somebody I can entrust they will put honest, worthwhile attempt at making good decisions when it is called for and recognize when they need to come back for further direction.

Senior developers show ownership in that they seek to uncover problems and discuss those problems with the team, proactively and productively. Senior developers can smell problems even if they don't necessarily know the solution. They can act on the signs of the bad smell and maybe seek discussion (with architect? client? manager? team?) to figure out what is going on and what needs to be done further.

Senior developers understand there are many ways/levels to solve the problem and sometimes solution isn't more code but maybe procedural change or a shift in paradigm.

Senior developers can meaningfully help/direct more junior team members individually without taking over their projects.

Senior developers can keep working relationships even with difficult people or people they don't like.

And so on.

If you ever actually want to do this, be aware that it's not only unethical, but possibly illegal. Requiring candidates to play a game discriminates against people with visual impairments, who can't play games, but can code as well as anyone else.

isn't the best technical interview inherently just doing the job itself? find some aspects of the job that's representative of it and have the candidate do it - if they can, then you're hired.

the issue is with technical jobs it's difficult to surmise what exactly is "representative" to begin with. I imagine the only reason candidates aren't just asked to work in the actual job for say, 8 hours instead of doing say, 7 irrelevant 45min interviews is because of intellectual property.

I wonder if Google or another large company has ever tried to re-interview all employees with more than say, 4 years of experience and correlated their interview performance with past job performance.

On a related note, feature request for factorio - comments/labels/documentation.

For instance, I like to group chem plants together as a block, that I can move around easily, set up remote bases, or duplicate easily. That, of course, means a common input "format". I'd love for in-game blueprints to be able to label that (or am I missing it?).

The common meta, aka. what you get on https://www.factorio.school/, is to mark inputs using constant combinators. Alternately, at larger scales a couple of chunk-sized (and absolute-tiled) railroad blocks will do nicely; the railway stations work as labels.

If you dive even slightly into the (enormous) mod ecosystem, there's one that lets you make physical labels, at one tile per letter. (Text Plates, https://mods.factorio.com/mod/textplates)

Since blueprint tiling was added, it's become absolutely obligatory to make good use of them. Almost every blueprint should have at least relative tiling enabled. Note, the tile size shouldn't necessarily be the exact size of the blueprint -- it's often convenient to have tileable blueprints overlap slightly, and it supports that just fine by making the tiles slightly smaller.

The reason interviews suck is because FAANG pays multiples of most other career paths, so an awful lot of people want those jobs, therefore they need an ever higher bar for people to jump through. Then every other startup and mid-size company cargo cults the process because they want to hire "the best" even though their funnel looks nothing like FAANG. Basically at this stage whatever process the top company chooses is immediately subject to Goodhart's Law, and they end up hiring roughly the same number of muppets as every other big company, the only thing you get is maybe a slightly higher IQ floor.

Casually playing Factorio with someone might give you a strong hint of someone's potential as a software engineer, but the minute you bring it anywhere near a high paying job interview process you are asking for disaster.

I remember giving, for a period of time, candidate a simple kids block programming game. The task was to put blocks together to escape ever more complicated labyrinths.

I stopped when I noticed candidates do worse at this game than actual 7 year old kids.

For example, a simple labirynth. Kids just give simple instructions (forward 3 blocks, turn right, forward 2 blocks, turn left, etc.). Candidates... try to figure out a complicated algorithm with a result that they run out of time producing nothing.

I wonder how much of this is effect of knowing you are being watched and your program analyzed and how much of this can be related to real world projects that developers just overcomplicate for no reason leading to worse results than just simple code you could throw in a fraction of the if you knew you are only judged for the effect and not for the code.

90 of it is probably that they're thinking there must be a more involved solution than something a 7yr old could solve, otherwise no reasonable person would use it to differentiate candidates with a university degree for a professional position.

As an option, you could try prefacing the game with saying "the solution to this might be very easy", and see how many more solve it.

Then you'd know you're likely not making a decision based on any signal of skill or ability at all, and just on them trying to make a prediction on what you're looking for.

Edit: I had to come back to this, because the more I think about it the more it's mind boggling to me. You interview is very likely filtering out the smartest candidates and selecting for the rest.

A typical strong candidate, that's prepped on interview questions, has maybe had other interviews at companies with challenging problems is going to look at it and think "This is a problem this interviewer uses to differentiate between top tier candidates and those who don't get offers. There is no way they're asking me a question that's so easy that any 7 year old child can answer it. There's gotta be something I'm missing." And keeps digging into it.

And a very weak candidate will say think "This is my 10th interview, and sweet, finally a question I can answer!"

These absurd backwards mindgames are exactly (one of the many things) thats wrong with software interviewing.

First of all, this was many, many years ago. Nowadays my interview is half an hour of (hopefully) nice chat, followed by half an hour of technical questions followed by half an hour to an hour of coding exercise with real IDE.

We all learn on mistakes, my interviewing process evolves as I learn which things do and which don't work.

Second, one of the goals of the interview is to see whether you can deal with the problem analytically. I am purposefully setting up problems that do not require (much) prior knowledge to observe the process of how the candidate deals with the problems.

While our knowledge of particular topics changes, our general approach to solving problems is almost set in stone for similar/related categories of problems.

If you are developer who does not predict how the code is going to work but rather rely on running it umpteen number of times and reshuffling instructions until you happen to make it work, this isn't just a sign of your lack of knowledge but rather your general approach to dealing with them. I bet, and I have observed it many times, that the person will do roughly the same when faced with a number of varied problems.

I found refactoring demo... GildedRose to work well. It is super fun for people and they can talk through it. It helps surface how they are thinking about code, definitely not the something that can answer all the questions but reduces tension a lot with candidates.

I've got 1,300 hours into Factorio... the factory must grow!

It is analogous to programming, the programming I learned in the 1980s, not modern software development.

On the plus side, once you level up a bit and get constructor robots there is a bit of copy/paste and undo capacity. You can load and save your work, and use more than 8.3 file names. 8)

On the down side, there is no source control, no commenting, very limited macro capability (blueprints). The debugging tools are limited.

On the analogy side, pollution as technical debt that causes bugs which will bite you, is spot on.

My favorite game style is to do Rocket Rush on an island... and see if I can get rid of the enemy and get self-sustaining nuclear power with Kovarex going... the it rapidly becomes boring.

Would be cool if Mineraft went this far. At least I can play that with my kids and I’m not haunted by that feeling that I’m building something virtual and useless. I can actually enjoy it, through my kids. Yeah it’s weird.

When I was responsible for recruiting interns, I always wanted to use TIS-100 as an interview tool, or more likely a modified version of TIS-100 with different instructions/architecture to not give an advantage to people that have "played" it before.

It would test their ability to read documentation and apply what they've read to problem. It also doesn't rely on them memorizing frameworks/API's etc. And it even in the early "levels" you can ask them to optimize for program size or cycle-count. Unfortunately no one would let me do it.

An amusing thought experiment but obviously it would have catastrophic consequences in real life... Pretty much anyone can get good at the game by watching Let’s Plays or reading guides.

I agree with the sentiment, but the ability to study (read guides) and demonstrate understanding/retention of the material by actually doing something (playing Factorio) is a good signal.

Sounds like copying and pasting jQuery snippets from Stack Overflow.

Honest question: is that fundamentally any different than building skills by imitating StackOverflow solutions or studying principles and patterns to build knowledge? I would view those as respective software development analogues, though there's definitely room for disagreement, particularly for very deep problem and solution spaces.

The difference is that what you can learn on StackOverflow might actually be useful on the job.

I don’t know, but I wouldn’t take the risk to find out in a professional environment.

I have a similar pet theory that Baba is You has incredibly high predictive power for skilled application security people. It would be pretty heinous to hire based on that, of course.

> or, if the feature is sufficiently common, google it

I do that all the time. I haven't yet, today, but I'm sure I will soon. I'm working on some stuff I haven't done in a while. Googling has the extra benefit of helping me to do something that I already know how to do, but more effectively. I can always learn new stuff.

They also mention that people "don't have the time" for open source.

But I do have the time, and the result is a pretty massive portfolio.

It's been kind of amazing, watching it get ignored.

One thing that has worked for me is to just give easier coding interviews. Not fizz-buzz easy, but not totally unnecessarily hard either.

The trick from there is to be very well prepared yourself as an interviewer, and to make sure you’ve chosen a good problem. Good in this case can be anything, it should just be so thing that you know really well.

I like this approach a lot because if they’re great, then they should code the problem flawlessly in, say, 10-15 minutes. If it’s just an off day for them then there’s plenty of time to talk and see how they go about figuring out how to fix their first problem, ie they don’t know how to start. This also closely emulates how work is too. Most of what you have to do at work is actually quite easy if you really understand why you’re there everyday. If it’s not, then how do you go about fixing that? So this approach accounts for a lot of different styles.

Some people are just plain smart, and they’ll get it right away. Other people are very unassuming and just ask questions until they feel like they have the confidence to propose a plan.

Also you have a way of screening people who aren’t ready for the job. If they rush into a solution that’s bad and are defensive, then they’re not ready. Remember, this is an easy problem.

One thing I think is often left out when discussing interviewing is that it in fact is often a game, because work is often a game. You don’t get perfectly written asígnennos, delivered via email, where you just have to implement a function according to an interface. Instead, your job is to create business value for the company, while enjoying your life. I think a good interview format should actually lean into that. No, it’s not perfect, they don’t always perfectly test your coding skills, but real coding skills aren’t easy to test and are only part of what makes someone suitable. Ultimately, what matters is a.) can you recognize this is a game and b.) what trade offs do you make to win at the game, given your ability and character?

I think sort of easy, well chosen problems, based off personal experience are really the best for getting these answers from a candidate. Then again, it’s still not a silver bullet because it requires a really well prepared, interviewer who’s committed to giving the candidate a good experience and a fair shot at demonstrating what they can do.

I’ve had interviews where halfway I felt like the candidate was genuinely having fun, and others where we both knew this was the end for them. Like, the interview was so good at what it was designed for in either direction, yay or nay, both of us pretty much left agreeing with one another about how it went.

How well does Factorio performance correlate with job performance over different periods of time and different employee backgrounds?

I don't see this mentioned in the article.

Your comment assumes that both 'Factorio performance' and 'job performance' are measurable. That is questionable, in my opinion.

If someone is going to argue that Factorio is a useful tool for technical interviews, it's only reasonable to expect that they will present a visible correlation between in-game performance and real-life performance. Otherwise, how is this anything other than a novel just-so story?

How do you show correlation of two concepts that are inherently not measurable?

Do only statements that are measurable have value in your opinion?

Even if something isn't measurable, you can provide anecdotal evidence that a correlation exists.

If you're going to say "Methods X and Y pale in comparison to Method Z for evaluating W" and expect anyone to draw conclusions from it, one would hope for even just a little bit of evidence.

Ok, that's fair, I agree.

If you think about it, that's not that far off of a professional licensing exam like the ones that lawyers and doctors take: it's something that's peripherally, but not directly, related to the actual work, takes a long time to complete, can be improved somewhat through careful study and can be objectively examined for success.

software engineering and factorio are cybernetics and systems theory. That's the overlap - dealing with causal loops, buffers, queue theory, feedback, all that.

It really pisses me off that we as an industry don't broadly know our roots and the actual theory that we are putting into practice when we work. The "coding is engineering or art" debate? Easy; it's systems engineering.

The closest we get as an industry to teaching what's under the hood is to deal with computer science, which is great, but data structures and complexity heuristics are only a small piece of the puzzle.

It's no great surprise that there's such a big problem with cargo culting when most of us don't explicitly know what it is that we're doing.

TIL I am a senior developer. I'm taking this thread to my manager right now.

Clickbait title with very long content that people can’t be expected to read in full before the conclusion finally makes it clear that factorio is being used as an analogy, not a serious suggestion for interviewing.

I bet the author of his article will change their tune if someone makes them do an interview with a game they don't have any experience in, like chess, kerbal space program, OpenTTD etc.

Jesus wept, what a horrifying abomination of an idea. The way I play computer games is so utterly unlike how I write code or design/manage computing systems, I can only surmise this is the product of confirmation bias run amuck, a nightmarish echo chamber of bad ideas reinforcing someone’s ghastly monoculture.

Cramming these tortured metaphors into your hiring process is worse than whiteboard programming, something I didn’t believe possible until reading this steaming pile of numberwang.

I'm looking forward to "10 years experience with Factorio" appearing on job postings. Does 891 hours count?

One downside is that Factorio is almost immediately rewarding whereas programming generally takes a lot more effort for mediocre payoffs; hiring good factorio players will bias you toward people with enough focus for high reward behavior but maybe not so much for standard software engineering.

The fact that all of the game's functionality mirrors real-world situations when building system shows that you could just ask someone to build a scaled down system rather than asking them to play this game.

Any game sufficiently complicated as this will be akin to asking someone to write some example code in a language they've never used before, so you wouldn't be able to tell much anyway.

Factorio is fascinating because it also simulates critical issues in productions: bugs will come and eat your beautiful factory.

I also I threw together some thoughts about factorio and programming a few months ago:

https://dev.to/samborick/what-factorio-taught-me-about-work-...

Job candidates could die to biters during an interview! This needs to be tried once (and posted to youtube)! Preferably on max difficulty.

I'm pretty sure the only thing that playing Factorio really tests is the player's ability to learn and play Factorio.

If you think this is a good hiring method, please do it so I can avoid your company. I specifically hate games like Factorio because they remind me of tech busywork, with very little intelligence or creative thought required. If you want creative thinkers, try another method.

The problem with this is that a lot of it is actually learned skills. So someone with experience in the game or other games that are similar has a big advantage. Including the people judging noobs who already know strategies for layout etc.

4 hours would be a long job interview.

Has anybody ever played less than 4hr of factorio?

Yeah, Nefrums would pass the interview in less than 2 hours..

Just drop them into a highest difficulty game and see how they are at crisis management. And at prioritizing. You could even see what they do at peace times.. Using it for personality analysis sounds like a bad idea.

I actually bounced off from factorio becuase I'd rather solve it as a programming / optimization problem than run around im the game. I find this with a lot of management sim games.

Did you bounce before construction and logistic robots? That is basically the shift from traditional rts-style game to programming/optimization game.

yes... takes too long to get there. Also I played way before those existed.

I have 100 hours in Factorio. I’m looking for a job. Where do I apply?

Please no, no. We are talking about fixing the interview process. We already have Leetcode game and people complained already. Please don't make me play another game.

I guess I'm the only person here who doesn't like playing Factorio because my fingers just hurt too much from the constant clicking and dragging and dropping.

Twenty hours is too long?

Considering the average number of per-month applicants for software developer positions is probably less than one, you ought to have that much time.

Oh boy.. I can already see the bootcamp-style websites popping up to help you excel at Factorio so you can land a lucrative sw dev job

interesting, at one point I wanted to use sokoban puzzles to judge how candidates handle complex tasks. did not do it for variety of reasons but might not be such a bad idea to build a mock interview out of such non-whiteboard code schemes.

I do not want to live in a timeline where Factorio becomes the new Leetcode, please God

Or you could just pair program with them on a real-world problem...

I'm going to assume this is satire

$30 to practice infinitely.

Click-Bait title, and judging by comments here everyone went for the click-bait.

The author doesn't actually use factorio for interviewing, and thinks it world be a bad idea to do so.

A more accurate title would be "Comparison between Factorio and Software Engineering Concepts" - but that wouldn't have gotten it to the top of HN so fast, nor gotten so many comments so quickly.

I believe we may be at a point where it is understood that headlines are created with the goal of attracting readers to the article, and it may no longer be necessary to point this out for each article we come across.

That's not even the case. The point of headlines is to attract readers to the comment section on facebook, reddit, twitter, HN, etc. Some small percentage of people who have become engaged in the discussion seeded by the title then click through and provide a bit of ad revenue.

I'll stop when I'm dead.

What’s the limit? What if I write an article with a title like “Breaking: Google to deprecate its search engine, will get into selling beanie babies instead” but in reality the article is about the many different types of mayonnaise one can buy? I think it’s reasonable to expect the title to somewhat reflect the content of the article.

There's a difference between editorializing a headline vs having an article titled "X is the best", where the content is literally "Well X obviously isn't the best, that would be ridiculous. Also we have no idea what the best is."

When the headline is so obviously clickbait, it's probably more interesting than the actual article... And the comments here are likely more interesting as well, just based on the title.

So I no longer worry about people who haven't read the article. I just enjoy the discussions.

It's their job to sex up the headline and ours to deflate it to size.

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

I've replaced the title with representative language from the article body.

https://hn.algolia.com/?dateRange=all&page=5&prefix=true&sor...

I have a question about whiteboard interviews.

I have used and continued to use whiteboard coding in interviews. However I emphasize to the person that I am interviewing that there is no expectation that what they write on the whiteboard compile, or even be a real language. I'll usually do a quick demo and joke that my own white boarding looks like the bastard child of Python and FORTRAN. The point of the exercise is to have a design conversation and see if the person can think about algorithms or solving problems, not write valid code with a marker.

Do people still have the same level of disdain for this technique or am I misunderstanding the objections people have to whiteboard coding.

I don't think anyone is expecting syntactically valid code (or at least that would be huge red flag for me if I interviewed). You do get bonus points the more "real" and consistent your whiteboard code is though.

I think people dislike whiteboard interviews because there is a significant subset of people who do not perform well under pressure, combined with the fact that a lot of software development is not terrifically algorithm heavy. This means that it's a (by nature) uncomfortable task and does not adequately represent the value the developer can bring to the business. In my current role, I would much rather have someone who can show strong knowledge of dependency injection and unit testing on my team over the person who understands how to balance a binary tree from memory.

I really only use algo/code/sql whiteboarding for junior devs now, and most mid/senior devs get asked about some hard code bug they've solved, what their preferred software patterns are, and we whiteboard some architecture.

The problem is that algorithms were discovered by mathematicians/computer scientists doing proofs for 40 hrs / week. It isn't something that can be reasoned about. It must be learned and memorized. This forces people to study that which is already embedded in the tools they use to perform their job.

If you ask people to whiteboard out a system at a high level that accomplishes some task, then that can be reasoned about and better tests their understanding.

See that's what I do mostly. Most often it's "here's a table of data I have, how would you compute this report." Not things like inverting binary trees or implementing hash maps.

Pro tip: Don't take advice or listen to people who use furry or anime profile pics.

Even worse: the combination of the two.

I think Pyanodon's modpack might actually count as good interview material, if it didn't take 800 hours to play.

why there is no standardized Leetcode-style tests like SAT or GRE that can be administered once and applied to all software job applications.

The current situation is like every company coming up with its own version of IQ test, the randomness factor is huge. if current employees of any company were to simply retake their own tests most will fail.

I suspect that a lot western companies don't want to go down the "chitty" route.

And those tests are gameable as are IQ tests and discriminate.

Even though FANG and startups are giving similar whiteboard interviews, they are evaluating on different dimensions to select candidates that fit their respective companies and the specific position. Reducing the Leetcode skill to a single number would not work well for this reason.

Reply on news.ycombinator.com