Guess we can throw out "look at a job candidate's github repo" as a screening tool.
I have quite a few non trivial and complete OSS projects on my GH, and in my 10+ year career as a SDE no one has ever really looked at any of it.
But they do frequently ask me to take their code challenge to prove I can program.
I have a huge GH portfolio[0]. Dozens of complete, soup-to-nuts, documented, tested, and in-use projects. I have complete source for shipping apps, backend frameworks, APIs, SDKs, communication libraries, UI widgets, teaching material (complete course modules), etc. I also have many personal blog entries, and online documentation for stuff; not to mention years of tags and commit history.
No one ever looked at them. In fact, when I asked one interviewer, why they didn't bother looking at them, I was told "You probably faked it."
Needless to say, that interview did not proceed any farther.
[0] https://github.com/ChrisMarshallNY#browse-away (NOTE: I just archived and retired a bunch of stuff, because I maintain what I publish; even if I am the only consumer. I dogfood most of my work).
I hired someone on the back of their GitHub repo once. He talked me into it by saying he struggled to control nerves during coding interviews etc. Regretted it immediately - the code he wrote once he joined inexplicably bore no resemblance to what was in his repo. He also turned out to have a major attitude problem with the language we were actually working in, suggesting at one point that maybe he should be able to write his code in Common Lisp and then we should hire cheap Indians to rewrite it into our actual language.
This sort of thing seems hard to detect unless you actually watch someone code. The stuff on his repo could have come from anywhere or anyone, really.
This was not my experience. I never gave coding tests, and would have killed for a GH repo.
I was a manager for 25 years, and we did pretty hairy C++ programming.
What a code repo gives me, is a fulcrum for a discussion. I would never hire, based on a code repo, or a test.
What I did hire on, was fairly long, casual conversations, where I drew the applicant out, and got them to talk about their work, problems they solved, and design decisions they made. A GH repo would give me an "in" for that. "I see that you wrote the firmware for an AI-powered cheese straightener. Tell me about how it came to be. ... Really? How did you solve that problem?" etc.
A lot of the folks that I interviewed were not "people persons." I found that once we got into one of these conversations, they opened right up.
Also, and this actually doesn't have much to do with my tech experience, I'm a pretty hard person to BS. One reason, is that I come across as a naive, credulous, person, and bulshitters can't seem to help themselves. They just have to try yanking my chain. It's kind of fun, actually.
It's hard to scale. People are very easily BSd in interviews without training. If you need to define a repeatable process that's fair (for a fast growing team), it's easier for people to objectively evaluate coding skills they've watched than to handle an open ended conversation. If you don't need that framework, super, that's a rare and useful skill.
At the same company where I made aforementioned interviewing mistake, when I first joined they were in the process of interviewing a guy who billed himself as a high-flying consultant type. They were impressed and intended to offer. They asked me to meet him and ask some questions too, just as a last minute double check. He turned out to not actually have some of the most impressive sounding skills on his CV, in particular, he claimed to be an expert on the internals of HotSpot but when probed couldn't answer even basic questions of the sort you'd learn by reading the user manual. Several other claimed hard skills fell apart under direct questioning too. I got the sense that he had previously been talking about work and projects done whilst he was around, but not by himself.
After that I put in place standardized coding tests. My managers continued to do the open ended interviews they'd been doing before. After six months or so we noticed that whilst the coding tests were selective i.e. we sometimes said no, the open ended interviews had never yielded a reject. They were pointless for both us and the candidate because the answer was always yes. When I sat down with the second stage interviewers, we went through some of the questions being asked and I posed a meta-question: what answers to this question would cause you to reject the candidate? They couldn't come up with any.
I never made a bad technical choice, but I did make a couple of personality mistakes; people that had the tech chops, but couldn't work in the team. That is a lot more difficult to determine than tech chops.
I think one thing that helped, was that my company paid "competitive" (i.e. "low") salaries, although we were a marquée brand. I suspect that kept the grifters away. It made my job a challenge, though.
One thing that software companies, in particular, are obsessed with, is hiring "cookie cutter" people. We seem to have an allergy to hiring competent, skilled, experienced people. We want to hire lots of people with mediocre skills, pay them ridiculous amounts of money, and force them to write top-shelf software, using structure and process.
I don't think it works well, and is attractive to grifters.
My group was a small, high-functioning team of experienced C++ engineers, writing cross-platform, image processing pipeline algorithms. Not work for the faint of heart, and each one of my employees had a specific personality, strong point, and weak point. It was my job to understand each of them, as an individual, and help them to integrate into a pretty massive team, that stretched across three continents, and included some of the best engineers and scientists in the world. We were always under a spotlight, and we had to be constantly delivering.
Companies have relied on "key players" for hundreds of years. If they keep their "key players" happy, and productive, they do well. If they don't, they fall down. These companies aren't always able to get very large, though. With scale, comes mediocrity. Sort of can't be helped.
But it also brings bags of cash, which is nice. It's just that I see small companies, all the time, trying to act like big companies, and I don't think that works very well.
The proof of the pudding is in the eating.
Many people have these repos with a fork of some opensource project with like 2 commits in it, or something not parting too far from a template project. I am not saying this does not qualify - because those commits might be useful or insightful, or merely sufficient to solve a problem - but they are way too easy to fake.
Most first line screens/recruiters aren't going to do much more than skim code and aren't going to run it. Best case scenario now your recruiting pipeline is inflated with more poor candidates. Worst case scenario they're not one step closer to bamboozling their way to a job they're not qualified for.
Tools like this are going to become more and more ubiquitous, and I’m questioning myself whether it makes sense to avoid it almost entirely as I have been or to embrace using it to accomplish tasks.
Unfortunately (IMO) we are through the looking glass, as they say.
I’m using it in my workflow for when I’m writing really repeatable test cases or boilerplate. For anything more complicated, very quickly the limitations will start to appear which has brought me comfort that it’ll be a long time before this is going to replace me entirely.
I imagine it’ll be a bit similar to self driving cars, in that the technology behind the base 80% of cases is incredibly impressive, but the human element to solve the other 20% remains out of reach for longer than we hope.
I have, and I'm pretty convinced coding challenges are going to be a thing of the past soon. Either that or there will need to be extreme anti-cheating measures in place.
I doubt we would hire it, but it would definitely pass the first tech screens.
This has a strong feel of "why should I learn multiplication tables when I have a calculator with me at all times?" But instead of multiplication tables it's the entire curriculum.
I can see this becoming a standard tool, which shifts the burden on the student from writing code to critically reading code, writing tests, and learning how to judge correctness.
But the counter-argument is the same: you don't want to be reliant on a calculator at all times.
And that's assuming the tech can even get to the point of replacing the curriculum. It only works as well as it does because it's been trained on text that is effectively many past curricula. What if you add something new? The model will be useless.
I'd be shocked if ChatGPT knew as much about, say, Coq as it did about Python.
Asking for an AI to generate code is one thing. Explaining the code you have contributed to large open-source projects and why you have done so is another and ChatGPT is quite bad at transparently explaining its own decisions and why it got to that solution in the first place.
- offline testing
- asking candidates to create ridiculous and hardcore apps that even ChatGPT is not sufficient. For example, creating a web browser with Rust.