Compare to the invention of the perceptron, which took a joint effort between a polymathic neurophysiologist and a logician.
sounds similar to the problem with tech coding interviews. ive refactored the backend orchestration software of a SaaS company's primary app and saved 24tb of RAM, while getting 300% faster spinup times for the key part of the customer app, but i bomb interviews because i panic and mix up O(n) for algorithms and forget to add obvious recursion base cases. i know i can practice that stuff and pass, its just frustrating to see folks that have zero concept of distributed systems getting hired because they succeed at this hazing ritual.
but with that said, i suppose no industry or job will ever be free from "no true scottsman" gate-keeping from tenured professionals. hiring someone that potentially knows more than you puts your own job security at risk.
In other fields, it is expected that if you can "talk the talk" you can "walk the walk." Mostly because it is really hard to talk in the right way if you don't have actual experience. Tbh, I think this is true about expertise in any domain. I don't think it is too hard to talk to a programmer about how they'd solve a problem and see the differences between a novice and a veteran.
A traditional engineering interview will have a phone screen and an in person interview. Both of which they'll ask you about a problem similar to one they are working on or recently solved. They'll also typically ask you to explain a recent project of yours. The point is to see how you think and how you overcome challenges, not what you memorize. Memorization comes with repetition, so it's less important. I remember in one phone interview I was asked about something and gave a high level answer and asked if it was okay for me to grab one of the books I had sitting next to me because I earmarked that equation suspecting it would be asked. I was commended for doing so, grabbed my book, and once I reminded myself of the equation (all <<1m?) gave a much more detailed response.
In a PhD level interview, you're probably going to do this and give a talk on your work. Where people ask questions about your work.
IMO the tech interviews are wasteful. They aren't great at achieving their goals and are quite time consuming. General proficiency can be determined in other ways, especially with how prolific GitHub is these days. It's been explained to me that the reason for all this is due to the cost of bad hires. But all this is expensive too, since you are paying for the time of your high cost engineers all throughout this process. If the concern is that firing is so difficult, then I don't think it'd be hard to set policy where new employees are hired in under a "probationary" or "trial" status. It shouldn't take months to hire someone...
What part is too time consuming? What you describe in the engineering interview sounds like a software engineer interview process as well.
The stereotypical software engineering interview is heavily leetcode dependent. It's why leetcode exists and they can charget $150/yr for people to just study it (time that could be spent on learning other things). I mean somewhere like Google you can have 3-6 rounds in the interviewing process.
[0] Maybe you'll use a board or paper to draw illustrations and help in your explanations, but you're not going to work out problems. No one is going to give you a physics textbook problem and say "Go".
I suppose that you will reply "talk to them" or "look at their experience". But, we have learned through squillions of posts here on HN, it just isn't enough. There are many charlatans that will slip through that type of interview process -- "great talker / good looking". This is the reason for the three-decade-long "arms race" in the technical interview process where programming problems become harder and harder over time. (Side comment: Does anyone think that people who can solve harder leetcode problems have a higher IQ? Controversially, on balance, I believe it to be true, and, thus, I think programming tests are a great way to filter for higher IQ candidates.)
> But, we have learned through squillions of posts here on HN, it just isn't enough.
> There are many charlatans that will slip through that type of interview process
The real question is the "optimization" question. Considering the time and cost of the interview process, along with the difficulty to remove bad employees, how effective is the interview process. You are optimizing for the best candidate but it's a constrained optimization problem. Otherwise you need infinite resources. So don't ignore the constraints. There's more that I haven't mentioned.
> Does anyone think that people who can solve harder leetcode problems have a higher IQ?
LeetCode specifically is famously dissimilar to day to day programming, and therefore (probably) a bad measure for Quality
It's like saying that having a degree is better because it proves that you can follow trough
It's not necessarily wrong, but it's a tertiary situation that doesn't neccesarily correlate (Even if it might, mostly)
I've been asked about my former projects, my roles, what I liked or didn't like about them, how do I approach a new project, what did I find most interesting, etc.
I gather there are a lot of fakers in the software dev world. So maybe that's why more places try to make you prove you can actually write code.
Reaching for a book to answer, makes sense to me. That's what you'd do on the job, and nobody would think less of you for it.
I've seen that at BigCo, but that's the exception. Every other place strongly prefers a start date of ASAP, with O(weeks) from initial contact as a next-best option. If you state that you aren't available for months you probably won't be hired.
> concern is that firing is so difficult
There are lots of concerns.
Keep in mind, 95% of resumes are some sort of bot/scam, and 0.5% of the rest are actually at the skill level I'm looking for. There are lots of potential explanations, and I don't think it's that only 0.5% of developers are who I'm looking for (there are sampling biases, survivorship bias, and all sorts of things at play in that data), but from my position doing the screening and interviewing those are the stats I see.
1. Suppose you actually did hire everyone who passed a 1hr screen. You'd still have 20+ failed candidates before you found the right person. Even with a 2-week trial period, that's 3/4 of the year not having your projects properly staffed, a demoralizing experience for all their coworkers, and 3/4 of a SWE-year in wages and benefits lost.
2. Is it really fair to hire somebody if I know there's a 95% chance I intend to fire them? What if they have to move? What if they hadn't quit their old job till I accepted them? I suppose if somebody said they were confident in themselves and were willing to risk a trial period I might allow it, but the current set of social expectations is that once you're hired your employer will spend months at a bare minimum trying to make you successful, I'd want to be cautious with that sort of arrangement out of respect for the candidates.
3. Onboarding is even more expensive than it might seem since it sucks away your more senior talent for the training. If the cost of a bad engineer were just the normal day-to-day post-onboarding it wouldn't be _that_ terrible (you still have attrition and other knock-on effects to worry about), but having multiple onboarding sessions for a single hire (because of multiple trial periods) is the most expensive part of the process.
etc
> General proficiency can be determined in other ways, especially with how prolific GitHub is these days
I agree. Walking through a project with a candidate is one of my favorite interview sessions. They tend to be more comfortable, I tend to learn more, and I get to learn something about their technical communication on top of any coding knowledge.
Not everyone has a GH with anything interesting, so I make other interviews available for everyone, but my life is a little easier if public "proof" (till you talk to the candidate you really have no idea how much they know or who wrote what, but I thankfully haven't seen that problem yet in an interview) exists.
> Suppose you actually did hire everyone who passed a 1hr screen.
What I talked about is something people already do in other domains. We're not talking about something theoretical here. There's even other commenters saying that what I said was similar to how they were hired. So again, we're not talking about something theoretical.
And this all ignores that the authors are PhD scientists. So I'm confused how this is categorized as "medical field" in the first place. I found that the ability to memorize is essentially useless in PhD level biological science (I studied immunology, so I can't necessarily speak to other fields), and it is all systems level conceptualizing.
I think this is a team with many talented people who came together to do their best. But I'm sure I'm naive. There seems to have been a lot of new interest and debate about what is happening in the glymphatics sphere.
Others can be guilty of similar sins, of course, and since the early 20th century, when philosophy and the classical liberal arts in general evaporated from school curricula, scientists have generally been quite poor at this, despite unwittingly treading into subject matters they are ill-prepared to discuss. Compare how a Schroedinger or a Heisenberg[2] talk about philosophical stuff, and then look at someone like Krauss [3]. The former may not have been great philosophical thinkers, but there is a huge difference in basic philosophical education and awareness, and these are not just isolated cases.
[0] https://edwardfeser.blogspot.com/2011/01/against-neurobabble...
To really answer your question, I think I need to talk about the books modern day neuroscientists are writing and I have to say I simply agree. I think these self-help kind of books are not good! Too bad they are so easily propagated in the media.
The classic meme is that MDs love organic chemistry, but they hate biochemistry [1], because one is about memorization and the other is...less so, anyway.
But then again, neuroscientists do tend to love their big books of disjointed facts, so maybe it's more like medicine than I realize. I remember the one class I took on neuroscience was incredibly frustrating because of the wild extrapolations they were making from limited, low-quality data [2], that made it almost impossible to form a coherent theory of anything.
[1] ...except for the Krebs cycle! Gotta memorize that thing or we'll never be able to fix broken legs!
[2] "ooh, the fMRI on two people turned slightly pink! significant result!"
It's not impossible for people who are good in memorization to also be good in understanding systems.
Those people, in turn, are the ones doing this research.
Although common, it's not quite so that only people with a pure medical background do neuroscience.
All in all, having met quite some people in the field, the things you're hinting at never occurred to mee as an actual problem. My guess is because the people who actually have issues get weeded out very soon. Like: before even finishing their PhD. It's not an easy field.
and it might not be "good at memorization" that's being selected, it might be "conscientiousness", one of the Big Five, and a relatively important parameter.
That being said, I think the rise of "evidence-based" medicine is also causing issues. It gets used as a cop-out to avoid thinking about the mechanics of what is actually happening in an injury. While this is certainly a good things for treatments where A or B superiority is uncertain, there's a lot of cases where I think an RCT just doesn't really make sense.
A pet example:
I broke my ankle recently, and this dug into the literature and common practice. A significant number of people will get end-stage arthritis a few years after "simple" ankle fractures and often the doctors have no idea why. At the same time, an important part of ankle anatomy is often left unfixed (the deltoid ligament) because a few studies back in the 80s found it wasn't necessary to fix it. The bone that serves an equivalent purpose IS fixed (if broken) though. Mechanically, they restrict the ankle joint and prevent it moving in certain directions.
When presented with biomechanical reasons for fixing it, and concurrent common poor outcomes for some patients, I've seen the response from surgeons thusly - "it's not supported by evidence" presumably because there isn't an RCT demonstrating definitive superiority.
So much of medicine and treatment is literally just hearsay and whatever your surgeon happened to read last week. As a whole the standard is rising, but so much research is so disjoint, disorganised and inconsistent that doctors often have no definitive guidance. It's probably more of a problem in some fields (like ortho) than others, but its still surprising when you see it yourself.
> So much of medicine and treatment is literally just hearsay and whatever your surgeon happened to read last week.
> Also note that the medical field selects hard for people who can memorize information, to the exclusion of people who can understand systems.
I understand testing knowledge and intelligence is an intractable problem, but I my main wish is that this would simply be acknowledged. That things like tests are _guidelines_ rather than _answers_. I believe that if we don't acknowledge the fuzziness of our measurements we become overconfident in them and simply perpetuate Goodhart's Law. There's an irony in that to be more accurate, you need to embrace the noise of the system. Noise being due to either limitations in measurements (i.e. not perfectly aligned. All measurements are proxies. This is "measurement uncertainty") or due to the stochastic nature of what you're testing. Rejecting the noise only makes you less accurate, not more.
> if something happens comprehensively across fields, it's likely to be a good idea.
This is related to "trust, but verify". If a big player is doing something it is worth looking at to see if it's a good idea for you. Same with when something is popular. But you have to be careful too. It's easy to miss context or small details that make a big difference. As an example, Google has a surplus of high quality candidates. Any arbitrary filter is helpful to them as they just need to down select. So a highly noisy process (i.e. random with a small bias) will yield good results for them. You'll even MEASURE positive results! This isn't necessarily (might be, might not be) true for anyone who isn't big tech or highly bureaucratic. (Same is true for college admissions)
It's important to remember that big players don't maintain their status simply because no other can out innovate. Rather momentum is a bitch. It can make up for a lack of innovation and still out compete.
I refer to them as "fuzzy databases" (this is a bit more general than transformers too), because they are good at curve fitting. There's a big problem with benchmarks in that most of the models are not falsifiable in their testing. Since it is not open of what they have trained on, you cannot verify that tasks are "zero-shot"[0]. When you can, they usually don't actually look like it. Another example is looking at the HumanEval dataset[1]. Look at those problems and before searching, ask yourself if you really think they will not be on GitHub prior to May 2020. Then go search. You'll find identical solutions (with comments!) as well as similar ones (solution is accepted as long as it works).
IME there's a strong correlation between performance and number of samples. You'll also see strong overfitting to things very common.
That said, I wouldn't say LLMs aren't able to perform novel synthesis. Just that it is highly limited. Needing to be quite similar to the data it was trained on, but they __can__ extrapolate and generate things not in the dataset. After all, it is modeling a continuous function. But they are trained to reflect the dataset and then trained to output according to human preference (which obfuscates evaluation).
Additionally, I wouldn't call LLMs useless nor impressive. Even if they're 'just' "a fuzzy database with a built in human language interface", that is still some Sci-Fi shit right there. I find that wildly impressive despite not believing it is a path to AGI. But it is easy to undervalue something when it is highly overvalued or misrepresented by others. But let's not forget how incredible of a feat of engineering this accomplishment is even if we don't consider it intelligent.
(I am an ML researcher and have developed novel transformer variants)
[0] A zero-shot task is one that it was not trained on AND is "out of distribution." The original introduction used an example of classification where the algorithm was trained to do classification of animals and then they looked to see if it could _cluster_ images of animals that were of distinct classes to those in the training set (e.g. train on cats and dogs. Will it recognize that bears and rabbits are different?). Certainly it can't classify them, as there was no label (but classification is discrimination). Current zero-shot tasks include things like training on LAION and then testing on ImageNet. The problem here is that LAION is text + images and that the class of images are a superset (or has significant overlap) with the classes of images in ImageNet (label + image). So the task might be a bit different, but it should not be surprising that a model trained on "Trying for Tench" paired with an image of a man holding a Tench (fish) works when you try to get it to classify a tench (first label in ImageNet). Same goes for "Goldfish Yellow Comet Goldfish For The Pond Pinterest Goldfish Fish And Comet Goldfish" and "Goldfish" (second label in ImageNet).
(view subset of LAION dataset. Default search for tench) https://huggingface.co/datasets/drhead/laion_hd_21M_deduped/...
(View ImageNet-1k images) https://huggingface.co/datasets/evanarlian/imagenet_1k_resiz...
(ImageNet-1k labels) https://gist.github.com/marodev/7b3ac5f63b0fc5ace84fa723e72e...
But in my experience neuroscientists have to have a solid level of systems thinking to succeed in the field. There are too many factors, related disciplines (from physics to sociology), and levels of analysis to be closed off.
Honestly 'our knowledge of [X] is largely mechanistic and without a sense of the larger picture' is weirdly applicable to most scientific fields once they escaped the 'natural philosophy' designation.
This sounds like one of those complete bullshit memes that certain groups of people like to repeat. Very similar to tech people being "creatives" while other groups like sales are somehow not. Utter bullshit.
> Compare to the invention of the perceptron, which took a joint effort between a polymathic neurophysiologist and a logician.
While cross-field collaboration often yields the best insights, I hope you're not implying that computer scientists are somehow better at "understanding systems" compared to biologists. Not only are computer scientists hugely guilty of pretending that various neural networks are anything at all like the brain (they are not), its also the case that biological systems are fantastically more complicated than any computing system.
Easy to agree with
> to the exclusion of people who can understand systems
On what basis do you draw this conclusion? I'm not saying the field is full of systems thinkers, but I have no evidence that they are at higher or lower concentration than many other skilled disciplines. Many of the specialties within medicine require systems thinking to be effective physicians.