Relating natural language aptitude to differences in learning programming
nature.com
nature.com
Can anyone explain why did they do this? Was it just to prove their hypothesis? I bet this guy had really high numeracy scores as well.
Edit: They also botched some of the numbers:
> (Fig. 1A: mean learning rate = 1.25, range = 0.81–2.0, sd = 0.24)
The 2.0 point is not included in Fig. 1A, the highest learning rate is 15 lessons over 10 sessions or 1.5, so they included the outlier in that range but removed it in the plots.
Also I don't find it unreasonable that a person completed 20 lessons compared to the others 15, so I don't see why this was so extreme that he had to be removed. You could argue that the later lessons are harder so it was unreasonable, but then you just confessed that the learning rate metric is not linear so your regression models doesn't work since it assumes that going from .5 to .6 is equal to going from 1.5 to 1.6! Even though as we can see from the plot many of the best learners got stuck at lesson 16 for an entire session without making progress, while the slowest learners took 10 sessions to reach roughly the level these fast learners reached after the first. That is almost a 10x difference, but this plot shows it as just 2.5x.
In this specific case, I suppose they could have added some motivation for their decision, but I don't think they did it in bad faith. They only had 36 samples, and a 3sd event occurs once in ~300 cases: this hints to the fact that that data point might have been erroneous, or that there could have been some other factor (e.g. earlier exposure to programming concepts - even disguised as something else).
If you are interested in the study, you can find the data and the script in the linked repository: https://github.com/UWCCDL/ComputerWhisperers The range for the learning rate seems wrong. If you check the data, there is no data point with 2.0: https://github.com/UWCCDL/ComputerWhisperers/blob/master/Com...
Or it hints that the distribution of learning rates is not gaussian. When there's an "n-sigma" event, it's usually much more likely that the model is wrong than that the event is that rare.
Removing outliers is common, but extremely sketchy. The existence of outliers in the dataset at all is most likely because they are assuming a probability distribution that does not hold. If they are using the wrong distribution, none of their results mean anything.
It's likely a predetermined exclusion criterion (although they should explicitly state that). An extreme outlier on the primary measure may suggest that the subject was not doing the task correctly (resulting in an extremely low score) or it may suggest that the subject misreported their lack of programming knowledge (resulting in an extremely high score).
Depending on what you're looking at, it can be standard procedure to exclude extreme outliers, but it should be announced as predetermined and be principled.
The idea behind IQ is that lots of different cognitive tasks are correlated.
The short answer is that it's difficult to say from their results, and they don't explicitly test that, but it looks like it.
Also mastering a single language is harder than learning to get by in multiple languages, as even though a majority of Europeans can speak multiple languages a majority of them are not really all that good even in their mother tongue as we can see from standardized testing. They aren't worse than Americans, but they aren't better either, they get roughly the same results.
You haven't met many Europeans have you. Not all of them are Dutch, Scandinavian, urban youth or immigrants.
>a majority of them are not really all that good even in their mother tongue as we can see from standardized testing
This makes no sense. You can't be not good in your mother tongue - it's your mother tongue. Standardized testing has no say about how a language is structured, that's an absurdly prescriptivist point of view. As for foreign language ability, in real life it's not assessed by making them sit down and pass test questions, it's assessed by having them speak and write close to a native. Someone who casually uses slang, "bad forms" and occasionally makes native-sounding mistakes is more proficient at a language than someone who speaks an academic version of it and got good grades at a test.
Speaking three languages is not a big deal when you live in a village where everyone else speaks them.
>You haven't met many Europeans have you. Not all of them are Dutch, Scandinavian, urban youth, immigrants or throwawaybbb's great-grand-mother.
Please edit "white people" to "American and Chinese white people"... people from smaller countries are more prone to knowing more languages :P Also, people who migrate or travel a lot. Also, since race discrimination is still a thing in the US and not only, smart people of color do their best to appear at least as smart as they really are, and knowing foreign languages probably helps a lot.
Being white and of the right nationality in a developed country simply allows you to be lazier, and you can afford to appear "dumb" in casual life, or to focus on only what you care about, hence you'd probably not bother learning any foreign languages, the ROI would be very bad for you, others will learn your native language instead... We all have limited amounts of brainpower and we try to invest it as well as we can.
https://en.wikipedia.org/wiki/African_immigration_to_the_Uni...
I wonder if numeracy is actually a correlated factor to the propensity for modeling systems, which is very much relevant to learning to program in any programming language. In my anecdotal experience, being fluid with numbers often comes from having a bias towards thinking about the world in terms of numerical relationships.
If you're doing 3D graphics projections you're going to want linear algebra. If you're doing financial forecasts you'll probably want to know calculus. If you're modeling throughput of large menssage-passing systems with multiple cooperating processes you need to know ratios and proportions. If you're the one maintaining the query planner for a database or maintaining a large, high-performance data store you're going to want to understand relational algebra.
For a CRUD application or some light server automation understanding data flows, boolean logic, and some really light set theory helps but is not necessary to get started. There's an awful lot of code out there that is not actuarial, simulation, or engineering code.
People understand things numerically, sure, but also spatially, linguistically, mechanically, or logically. Mathematicians and physicists can give you more precise descriptions of many things that people don't do much math about in their daily lives. Just like people don't need to be linguists to communicate, people don't need to do a lot of heavy math just because the underlying tools are based in math and mathematical logic.
Math can definitely help one understand the system better and excel at optimizing code and solving certain programming issues. And as I said, a lot of the problem domains people write code for are themselves very heavy with math. There's a reason CS grew largely out of mathematics and physics departments at universities, but this study was about understanding the concepts of programming in a high-level language.
From the Wikipedia article on "numeracy":
Fundamental (or rudimentary) numeracy skills include understanding of the real number line, time, measurement, and estimation.[3] Fundamental skills include basic skills (the ability to identify and understand numbers) and computational skills (the ability to perform simple arithmetical operations and compare numerical magnitudes).
More sophisticated numeracy skills include understanding of ratio concepts (notably fractions, proportions, percentages, and probabilities), and knowing when and how to perform multistep operations.[3] Two categories of skills are included at the higher levels: the analytical skills (the ability to understand numerical information, such as required to interpret graphs and charts) and the statistical skills (the ability to apply higher probabilistic and statistical computation, such as conditional probabilities).
A variety of tests have been developed for assessing numeracy and health numeracy.
That's just basic arithmetic, not any of the heavy math that you described. I'd argue that for a lot of CRUD apps, numeracy as a skill is quite important - though that's not relevant to the topic of picking up programming languages/concepts.
From what I've seen online this is done with an if/else tree which is very far from a good solution.
The way I would model this is with a group [0]. Something you can't possibly see without a very substantial background in mathematics.
Programming has as much to do with learning a programming language as novel writing has to do with learning punctuation.
[0] in Python3.6+:
import random
class hand:
wins = {'r': {'r': 'r', 'p': 'p', 's': 'r'},
'p': {'r': 'p', 'p': 'p', 's': 's'},
's': {'r': 'r', 'p': 's', 's': 's'}}
def __init__(self):
self.value = random.choice(list(hand.wins.keys()))
def assign(self):
value = input("enter r, p or s: ")
assert value in hand.wins.keys()
self.value = value
def __mul__(self, other):
return hand.wins[self.value][other.value]
if __name__ == "__main__":
for i in range(10):
h1 = hand()
h2 = hand()
print(f"{h1.value} * {h2.value} = {h1 * h2}")Much of natural language aptitude is about fuzzy logic with lots of exceptions (in English at least) and contextual intuition (especially in Chinese). Whereas programming computers requires more mathematical oriented thinking and raw working memory for abstract symbolic logic.
I recall Joel Spolsky (I think) wrote in one of his essays a long time ago, that one of the obstacles to learning programming for some people is that they kept trying to find some kind of meaning in the symbols. Eg they remember things by associating meaning with them. Whereas people who pick up programming more easily have no need to find meaning, they just remember things directly without any kind of association aids.
Natural language is all about meaning on multiple levels - literal, contextual and implicit. Someone naturally good at that may not necessarily be naturally good at programming.
For anyone who read the study, was this addressed in any way?
Seems fluid intelligence is twice as important.
Not saying you're wrong, but do you have a source to back up this claim?
Language is a social construct. What you see on paper or a screen, the vocal sounds you hear as someone speaks, those are just physical representations. They don't carry any inherent meaning.
All human interaction is based on a shared understanding of the world. When I state that "one plus one equals two", I assume that my audience and me agree on what each of those terms in that sentence means. That we share the same frame of reference to assert that statement as true or false.
I could easily invent an entirely new language where the only difference is that the words "two" and "three" swapped meaning. At that point, the sentence "one plus one equals three" carries just as much truth, as long as we share a common understanding of what "three" means.
The difficulty then is that a common understanding of shared meaning isn't always clear cut. Culture, education level, aptitude, personality,... all impact how we perceive and interpret the world, and how we'll use language to build an abstract mental model from which we can can assert our own individual identity.
What many programmers tend to forget is that a "high-level programming language" is exactly that: a vocabulary and a syntax that mimics a natural language which allows you to describe the world. It's NOT merely an abstraction of low level internals of a computer as is often assumed.
Implementing a feature request then consists of interpreting whatever you've read in a spec, user story, brief,... transform that into your own mental model and then express that using the limited formal set of symbols provided by the language. Functional testing is basically verifying if your own unaware assumptions and biases didn't break a shared framework of understanding or created a disconnect between you and the client or the user.
Developers have a penchant for creating new frameworks and languages based on a fallacy: that theirs will be able to somehow "fix" the "fuzzy logic of language", whereas that's inherently impossible. Unless, you're the last person alive with no-one to challenge you on how to interpret of things, that is.
I wonder if there is any formalized way in statistical hypothesis testing to quantify how much the conclusions could be changed by removing at most k (e.g. k=1) number of datapoints.
> If the chance of getting a disease is 10%, how many people would be expected to get the disease? Out of 1000?
> If the chance of getting a disease is 20 out of 100, this would be the same as having a _____% chance of getting the disease.
> Imagine that we roll a fair, six‐sided die 1000 times. Out of 1000 rolls, how many times do you think the die would come up as an even number?
> In the BIG BUCKS LOTTERY, the chances of winning a $10.00 prize are 1%. What is your best guess about how many people would win a $10.00 prize if 1000 people each buy a single ticket from BIG BUCKS?
https://onlinelibrary.wiley.com/doi/10.1002/bdm.1751
I guess a lot of people never learn what % mean, and since basically all questions are related to % they would fail just from that. So I'm not sure if this is a good test of numeracy, feels like it relies too much on a few key items.
Edit: Also that test was designed so it gave a normal distribution for a typical population, in this study we see a lot of 0's and then no 1's and 2's. So it feels like the population tested wasn't representative, or some of the students asked just didn't bother with the math questions. Since students are forced to participate I wouldn't be surprised if a few of them just skipped trying at all on the math parts.
A classic example of a question you can only answer if you don't know what you're talking about. I bet they think the answer is 500.
500 is the most likely result, but the odds of actually getting 500 heads on 1000 flips of a fair coin are 2.5%, 1 in 40. A little ways out, at 505 (or 495) heads, the odds have fallen all the way to... 2.4%.
This is kind of like asking "Imagine that we roll a fair, six-sided die one time. Out of that one roll, which number do you think would come up?"
Except, assuming you make the best possible guess both times, you're more than six times as likely to be right for my revised question.
If you are at an exploratory phase and your plots look a sky map, then you have a bad data representation. On that case you can't even extrapolate a positive correlation into a theory that one value will grow when the other grows on any specific case.
I think one could reasonably expect to see a wide range of outcomes in this circumstance, but it does not seem newsworthy.
"Relating natural language aptitude to differences in learning basic PYTHON programming"
as it doesn't represent other programming discipline (e.g. low-level-programming, concurrency)