Ask HN: What would the Dunning-Kruger “test” be for your field?
What question (or problem) separates the wheat from the chaff?
A good sign of maturity is when they can notice "hey, that didn't go great" and use it to inform their future strategy - "technique X applies when these conditions are true. If one of them are false, I'm not sure what to do, but I should conduct more research (or talk to more people) before starting that project"
Or language (but I suspect that you might count language as included in "technique or technology").
when I've interviewed people, I made sure to ask questions like "what do you like/dislike about language/technology/technique X" and a lot of people really seem to struggle with that kind of question
Wait really? That's shocking to me.
You see who really thinks of marketing in terms of competitive positioning, differentiation, and how to create different messages and content for different parts of the funnel (and how to measure it all), and who just throws a random bunch of tactics at it.
Unfortunately in my experience this often takes more than a couple days to shake out, so it doesn't help in interview / recruiting.
It'd probably be similar to beefield's. Present a problem with a clear success condition and see how resourceful they are in solving it. A failure would be solving the wrong problem, making excuses (or "the problem is not solvable"), confidently talking in abstractions that don't concretely pertain to the problem, or failure to ask a question/say "I don't know".
I don't know that there's a technical question or problem I could ask to make a call on whether or not they know their stuff.
From what I've seen, a lot of people can rattle off reasons they've heard for why a particular thing is done, but have not deeply considered the opposite side of that opinion and what its benefits may be. Ymmv.
The problem I've run into is that it's a pretty easy question to game, if you know the purpose behind asking it; nobody is going to come out and say, e.g., "I think ORMs are dumb because that's what other folks seem to say, and I see those people having high-status/well-compensated jobs, and I'd like some of that too, plz." So maybe it's not a great "weeder" question. /shrug
Why wish your tool changed instead of focusing on using it, or something better suited, to produce things?
If somebody has a new model that gives the exact same predictions as a current model, lauds it as being obviously the truth, and dismisses all other models as falsehood, then it can be dismissed. This applies to all various interpretations of quantum physics, since they all yield the same measurable predictions.
1. Yes, I understand. (with no further elaboration or clarifying questions) 2. It's okay/good/going well (again, with no further elaboration when asking how something is going/set up etc)
First one typically translates to something between "I have a completely unfounded illusion of understanding" and "I have no clue but don't dare to say it"
Second one translates typically to "I have no clue whatsoever"
Or "I'm tired of this droning conference call/meeting where I can barely understand some people's accents and I'd rather just clarify some points in writing later where I don't have to rely on my memory."
>"I have no clue whatsoever"
Or it really is going fine and I don't want to spend forever going into technical details the person asking isn't even going to understand and get interrogated on anything that doesn't sound like it's going perfectly, when everything is under control on my end.
I do also ask questions without hesitation when needed (to the point where it starts to irritate non-developers), but saying "Yes, I understand" or "it's going well" when I do understand and when it is going well is not a red flag.
Whenever I have casual conversations with individuals that have the same skillset as me, outside of the environment where I wear my professional "all knowing" persona - Admitting that you don't understand something sparks incredible conversations.
What's even more incredible, is that every time I've stood up and said "I actually have no idea how X works. It is magic to me." Every single time somebody else in the room that had me completely convinced they had a strong grasp on those concepts will also admit they don't know either.
These answers are also my red flags. But I place more blame on culture than the individual sometimes.
The scenario is even more complex when working for govt. Your role is often that of a sponge: Absorbing all the information presented and volunteering very little, if any, of your own.
As I get older, I try to keep my sentences short and to the point. Often they are dense, layered with meaning. In my experience, folks often wax poetic and use all sorts if inaccurate analogies when they understand very little about the subject.
You didn't mentions the industry so maybe you're in an environment where you spoon feed everyone, but for me it translates "yes I understand the goal and the vague outline, let me go away and dig into the details, then I can ask good questions".
You can't know everything. What's important is that you know how to learn.
Otherwise, asking people who their favourite players are is a quick test, when suspicious. I've come across a few "jazz musicians" who can't even name a single jazz musician. Also, in chess, asking who someone's favourite players are is a quick test of whether they're any good.
The people I consider to be great a this will read a variety of content on a topic, think about it, synthesize it with their own experience, go try new things, and decide for themselves what is correct for their own project.
The less skilled will say, "See, this blog post said to do it this way, so that is what we shall do."
An analogy to software would be to pair program with a person. You both get a feel for each other's strengths and weaknesses while working on a particular problem. I feel that coding interviews try to replicate this, but fall short given time constraints.
Genuine question. It would be fascinating if there were documented examples.
Note that this is distinct from having a company culture where pair programming is common. I am skeptical that just because you occasionally look over each other’s shoulder and exchange ideas, that the software ends up being fundamentally different from the ground up. It seems unlikely that two people would both decide to rewrite an entire subcomponent of the project from scratch, whereas individuals do that often. And the results turn out better (but only when the person is an effective programmer, otherwise it’s just a mess).
I don’t think you’d learn much from pairing with me. You’d see me sitting, staring at the screen for long periods of time, saying nothing. Or experimenting with pasting various code tweaks into a repl. The final result that makes it into a text editor is the outcome of both processes, and these have long time horizons. Sometimes far longer than someone else could reasonably tolerate.
"We sat down one morning," recalls Steele. "I was at the keyboard, and he was at my elbow," says Steele. "He was perfectly willing to let me type, but he was also telling me what to type.
The programming session lasted 10 hours. Throughout that entire time, Steele says, neither he nor Stallman took a break or made any small talk. By the end of the session, they had managed to hack the pretty print source code to just under 100 lines. "My fingers were on the keyboard the whole time," Steele recalls, "but it felt like both of our ideas were flowing onto the screen. He told me what to type, and I typed it."
The length of the session revealed itself when Steele finally left the AI Lab. Standing outside the building at 545 Tech Square, he was surprised to find himself surrounded by nighttime darkness. As a programmer, Steele was used to marathon coding sessions. Still, something about this session was different. Working with Stallman had forced Steele to block out all external stimuli and focus his entire mental energies on the task at hand. Looking back, Steele says he found the Stallman mind-meld both exhilarating and scary at the same time. "My first thought afterward was: it was a great experience, very intense, and that I never wanted to do it again in my life."
https://www.newyorker.com/magazine/2018/12/10/the-friendship...
In most professions, you have a ton of feedback indicating where you are, often formalized. In martial arts, you wear a belt showing roughly what level you're at, which you earned through tests done by people far senior to you. In the military, you wear your rank, maybe skill badges, and do constant training. In most other jobs, you have a title and get feedback from supervisors and such.
The hypothesis of DK seems to require that the person is evaluating their performance entirely on their own, possibly because the authors were evaluating comedic skill, which is a very subjective task and something everyone knows at least a little about. Moreover, it's one where you get bad feedback: good comedy among friends is not the same as comedy that pleases a crowd.
But in the professional world, you practice a craft towards a result and are exposed to your peers, the information on how you stack up is pretty objective and coming to you fairly consistently.
And I think the cases people attribute to DK are better explained by other mechanisms. In particular, there's a lot of confusion between self-knowledge and knowledge of the complexity of a task.
I have done phone screens and asked people stuff like, "how many numbers can be represented with 8 bits." But that's to screen out people who were applying to a job without having a clue what the job entailed. Not understanding a job description is different from not understanding your level of skill.
And I see plenty of self knowledge in interviews; if I ask about algorithms and data structures, a common answer is "I haven't looked at that since school." You learn: 1. the person doesn't think that skill is relevant to the job and 2. the person is willing to admit they aren't good at it.
I may disagree with the person that the skill is relevant, after all, I asked the question for a reason, but it's not a lack of self-knowledge on the part of the candidate.
One of the reasons I like BJJ is because it's hard. There is no substitute for time on the mats, in the same way that you cannot become a good software engineer by simply graduating from a top-tier university or by reading every book on the subject.
I don't want to take anything away from your comment, because it's absolutely right. It's just a nice lesson to consider that you can perceive something, and it not being the case.
If you don't know, entity resolution is the process of matching unique rows in two or more databases. Are these the same movies? Are these the same person?
Novice DE: Oh easy, just merge on the name.
Intermediate DE: OH GOD NO. <michael_scott_no.jpg>
Expert DE: That's complicated, but I have a plan.
(This including things such as Unicode normalization and looking at other fields to determine if it's the same thing.)
And you get to handle duplicates too.
That is just the start, problem gets even more interesting in a real sharded scenario because eventual consistency is hard.
When people call 5G "safe" or "unsafe", it's clear they don't know about radiation effects on human health. The correct answer is "we don't have data".
Edit: The fact that people would rather make fun of me and imagine me as an "non-ionizing radiation tin foil hatter" (I am not, non-ionizing radiation has no negative long-term health effects) I hope is a demonstration of the power of one's own biases and forcing it onto a stranger to fit one's own convenient worldview.
As material science closes the tetrahertz gap, the longitudinal studies, by their very definition, take time to complete (longitudinal studies = exposing animals or individuals at lower doses for much longer periods of times).
It's natural for an undisciplined mind to assume they're also non-ionizing. But it is a mark of a disciplined scientific mind to be cautious and instead not extrapolate outside the existing dataset.
If we take the frequency of 5G and compare it with X-rays (the lowest energy ionizing photons give or take) we're only off by a factor of 600000
At higher frequencies, the effects get more and more surface level though.
The tetrahertz range is especially interesting precisely because it is an unknown between known non-ionizing and known ionizing effects. Extrapolating down from ionizing radiation biological effect data and panicking makes as little sense as extrapolating up from non-ionizing radiation biological effect data and making fun of the panic-ers.
I think it's a fantastic opportunity to prove or modify our existing models. But a mistake to assume the models hold true, given the historical intensity of debate around the (discredited) Linear No Threshold Model and subsequent alternative proposed models.
Edit: I would personally say 5G is more likely to be safe (no acute radiation-caused long term health effects) than unsafe. But I'm not one who likes to gamble, especially with the lives of the public.
[0] DOI: 10.1080/10643389.2011.574206
Current evidence suggests similar impact for similar power high frequency transmitters.)
Policy questions are not somewhere we can afford ignorance and black and white thinking. Precautionary principle is only good if you're not stopping a better and safer solution with it for sake of conservatism.
See, 5G is not just washing everyone with radiation, it comes with widespread beamforming, which is likely to send lower energy through anything obstructing it. Skin effect prevents getting penetrated with the waves at low energy used in communications.
Those are extremely strong arguments and the onus is really in anyone that says it's unsafe to actually show it is, properly and without doubt.
Otherwise it's plain scaremongering to suggest "we don't know" when the consensus is "it's safe". Exactly the same tactic as used by some climate denialists.
Sample study: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4997488/ (Read all the links to see the whole story. And these scientists used a rather high energy level compared to expected. The current analyses suggest the studies - which mostly show effects - inadequate. There's a strong bias towards publishing an evidence of effect when there's none. See: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6765906/ )
JUNIOR DEV: My code is simple and easy to understand.
MID-LEVEL DEV: My code is subtle, clever, innovative, expressive, hyper-optimized, and ingenious.
SENIOR DEV: My code is simple and easy to understand.
--
Source: John Arundel, https://twitter.com/bitfield/status/1219174978748370945
But take that for what it's worth: https://twitter.com/bitfield/status/1184741088067833856
It hypothesizes a meta-cognitive bias whereby a person who is incompetent is cognitively incapable of determining their level of competence.
Everyone has biases, and the reason citing DK may come across the wrong way is it tends to suggest that it's just Other People who have those biases.
From this piece[1] arguing that we routinely misinterpret DK:
> I suspect we find this sort of explanation compelling because it appeals to our implicit just-world theories: we’d like to believe that people who obnoxiously proclaim their excellence at X, Y, and Z must really not be so very good at X, Y, and Z at all, and must be (over)compensating for some actual deficiency.
But, generally, if the question of competence comes up, don't reflexively cite DK. It's been around for ages, it's not novel, and it's usually being miscited.
[1]: https://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-...
I don't see how that conflicts with the title. It is asking for a test that can assess one's rough ability.
First, it's adding a false veneer of scientism. All it's asking for is a rough test of competence; citing a wholly irrelevant science paper won't make that test grounded in any kind of methodology.
Second, there's this implied moral condemnation of people who are incompetent. Obviously, we're all incompetent in most things, so that's a strange thing to moralize. But the way DK is used in popular culture, there's an added dimension of, "this arrogant idiot thinks he's competent, but my test really took him down a peg."
Everyone thinks they can but almost nobody actually will. I'm not a particularly good writer, but I can do what most people can't/won't, actually follow through on writing something month after month after month.
My "long term" goal in terms of scientific and mathematical maturity is to git gud at calculating path integrals in quantum mechanics.
If you don't get it physically first then the feynman lectures are very good - physics is more than postulates and proof, at very least there's less to memorise if you start from the bottom.
Chaff: Understands computer science. Knows how to scale. Mentor of junior engineers.
Wheat: Understands people. Knows when to scale. Mentee of junior engineers.
More generally, the right answer to almost any question in software is: It depends. (That is, it depends on the usually large set of explicit and implicit, technical and non-technical requirements.)
Some people will dig in to the various tradeoffs involved with an ORM, and pick apart the structure of an ORM. And that should result in an "it depends" kind of answer.
Others are more interested in how a team is going to use an ORM; does it help junior devs write "not awful" code, how does it help me structure my application logic.
Others are stats nerds who will tell you which ones do best on benchmarks, are most popular, etc.
And many people will tell you a story about some ORMs they used, and why they liked them or not.
These answers can be useful for getting a sense of how a person thinks, but they won't tell you if the person is competent.
If they start saying something negative about someone without self-reflection, they are exhibiting the Dunning-Kruger effect.
Here is how I would screen developers for a general developer position. I would issue a 1 hour limit and ensure the candidate knows the test is graded by a computer only. The goal is can they read instructions and write simple code. I don't care how they write the code. I only care that they can.
Any question could be as complex or challenging in requirements as necessary, but the answer would always be a small function of few parts. The idea being to test reading comprehension, the ability to follow instructions, and write a simple function. An example of output format and data type would be explicitly stated with each question.
An example question:
A customer is spending cash to purchase a drink. Write a function that receives cash as the first argument and the cost of the drink as the second argument and outputs an object indicating the change in coins with preference to the largest denominations first.
1. Specify the grading criteria of the test. 10 points for each question correctly answered within the given time period. There is no penalty for answering a question incorrectly. The cumulative total execution time of all answered questions will be multiplied by 100 if in milliseconds or by 100,000 if in nanoseconds and be deducted from the final score. The idea is to solve as many questions as possible, but in the event that there are a limited number of positions and tied scores slow code is the tie-breaker that disqualifies a candidate.
2. Stress that code style is irrelevant. The test will be graded by a computer compiling the code and executing it again several scenarios in an answer bank. A human will not review the answers submitted.
3. Tell candidates that once the 1 hour test period has exhausted they will have an optional 15 extra minutes to review all prior attempted questions.
4. Force the candidate to perform 3 practice questions before the test timer starts to familiarize the candidate the expectations of the test platform and the appropriateness of answers.
5. Randomly pull questions from a pool of 200+ questions 1 at a time to ensure a developer is focused on the question presented instead of selectively gaming the question list.
6. Ask the candidate to write a function in a designated space that addresses the problem question/statement.
7. Allow the user to test their answer to review the output before submitting it for the next question.
8. Also allow the user to skip to the next random question with a 2 minute time penalty.
That is for a developer. For an architect I would have them write an essay on a prompt provided by the business. Architects review business needs, distill requirements, and communicate goals to provide a platform that executes the business needs while minimizing complexity as much as possible. If they can do that in writing they can do it in software. The most important skill is their ability to communicate clearly against competing factors. Have 3 separate non-developers read and grade the essay.