The social side of science seen in the research on programming language quality

That said, I think the article mentions science being "social" in the context of how we weight scientific research depending on the authors standing and expression, not the absolute truth behind it. Which actually is probably quite a shame.

Also, just to be clear, there can be social science fields which inquire into aspects of the engineering field of Computer Science, but that doesn’t effect the underlying character of the field.

Also, also, I would strenuously argue against the proposition that mathematics is in any way ‘human independent.’ But that argument is not entirely relevant.

I am half with you on this but thinking to SQL, its probably harder to find bugs in SQL than in an imperative language language, but its also a lot harder to create bugs - its a declarative language so doesn't allow you to create whole classes of bugs that you will see in an imperative language.

Personally I try to push logic towards the database for the very reason that it's less buggy in general in my experience.

Studies can have interesting conclusions even in the presence of confounding variables. Sometimes the state of the art in a certain hypothesis space hasn't progressed past of of these yet, but that's still better than gut feeling.

In some areas like public policy research you will never completely eliminate all of these factors, but that's no reason not to attempt to create knowledge.

One day I encountered a phenomenon that is as real and physically dramatic as, say, the electric field of a Van de Graff generator. Being a fan of science I went looking for the math/science/engineering for this phenomenon. Long story short, after a few years of research, I discovered that: 1) this phenomenon has been known since antiquity in a pre-scientific way; 2) several researchers had independently (re-)discovered it multiple times going back about 2.5 centuries; 3) each time it had achieved a certain notoriety but then been debunked and subsequently ignored or forgotten. The first was Mesmer & "animal magnetism", but there are several others including Baron Von Reichenbach & the "Odic force", and Wilhelm Reich & "Orgone". This is repeating today with several kinds of "energy healing" including Reiki (which is distinguished by being Japanese and unabashedly theistic (Rei means Divine)) where practice is widespread, but lacking scientific validation. As I say, I've felt Reiki, so I know it's real, but so far I've personally never been able to interest any scientists in studying it, despite offering to demonstrate it. And the studies that have been done haven't helped. The Wikipedia article for Reiki just straight up calls it pseudoscience.

Philosophy had almost the opposite problem at one time ironically, the concern that theory was gaining precision but becoming unhelpful for formulating comprehensible statements (in the non-mathematical sense) about the human experience.

The greater problem is the politicalization of science for which I'll use Gender Studies, though this is not to say Gender Studies is the only politicized field nor that it is the most egregious offender in any way. Having a cohesive theory of gender is useful, especially for comparing and contrasting different cultures even if it's at a level that's superficial compared to other fields like physics. At the same time, some social movements have realized that they can use aspects of gender theory to grant themselves legitimacy while leveraging the exclusionary nature of academia to silence criticism and largely control the public dialogue.

A good example of the utility gained having a theory of gender is the various African refugee groups who have come to the United States over the years. I had the luck to interview a psychologist who did work helping to evaluate and integrate some of The Lost Boys of Sudan. Many of the Lost Boys exhibited signs of mental trauma and her role in helping them integrate into American life was trying to provide relief for this trauma. She noted that a major challenge initially was that there was common mindset, stemming from a mix of age and a group outlook rooted in the need to survive, where personal problems weren't viewed or even necessarily understood the way they are in the U.S. The result was that many of the refugees she was working with not only didn't see the point in many of her questions (and by extension how to answer them) they also had difficulty engaging with her at all in a manner condusive to therapy as it's practiced in the U.S.

Handwaving a lot for brevity, her initial approach was to explore a trait she had observed and viewed as a quiet but severe sort of masculinity that was extremely common amongst the boys she was working with. Making early progress was largely about coming to understand this quality better and speaking to it in ways that introduced new concepts by breaking them down into smaller ideas which could then be explored by discussing them as contrasts. Is 'quiet masculinity' an especially scientific descriptor? Probably not, but it was nonetheless useful to her as an ad-hoc compass concept for breaking down barriers to providing effective therapy and it was made possible by synthesizing her understanding of gender theory with her broader knowledge of human psychology.

Contrasting the previous anecdote is my experience with undergraduate Gender Studies courses which are often a mix of course work on theoretical fundamentals and lecture presentations. The presentations are rarely outright superfluous from what I've observed but it'd be dishonest to say that they aren't also laced heavily with normative political statements masquerading as sensibility and which offer nothing extra to the lecture in terms of information or utility. This sort of soapbox sophistry is unhelpful because when it's eventually brought to the spotlight it erodes the public's trust in academia which is often too compartmentalized and arcane for a layperson to see where one subfield ends and another begins.

This is, in my opinion, why the social sciences in general make for such an easy punching bag. Because it quickly becomes unclear why in one situation an idea is treated as rediculous while in another it might be presented as a key component of some broader framework.

The penultimate point to all of this being, it's because science deals with running into hard problems that it's important to recognize when we're mostly paving familiar territory and when we're operating mainly with ideas that are 'the best we've got'. The ladder is no less helpful in the face of total ignorance but it can be much more vulnerable to biases of perspective and culture. Vigilant introspection is therefore a constant necessity.

Is this even a good idea?

We’ve just spent several thousand words on methodology to show how the original FSE paper was flawed. But was that all really necessary? Most people point to a much bigger issue: the entire idea of measuring a language’s defect rate via GitHub commits just doesn’t make any sense. How the heck do you go from “these people have more DFCs” to “this language is more buggy”? I saw this complaint raised a lot, and if we just stop there I could have skipped rereading all the papers a dozen times over.

Jan Vitek emphasizes this in his talk: if they just said “the premise is nonsense”, nobody would have listened to them. It’s only by showing that the FSE authors messed up methodologically that TOPLAS could establish “sufficient cred” to attack the premise itself. The FSE authors put in a lot of effort into their paper. Responding to that all is “nuh-uh your premise is nonsense” is bad form, even if the premise is nonsense. Who is more trustworthy: the people who spent years getting their paper through peer review, or a random internet commenter who probably stopped at the abstract?

It all comes back to science as a social enterprise. We use markers of trust and integrity as heuristics on whether or not to devote time to understanding the claim itself. This is the same reason people are more likely to listen to a claim coming from a respected scientist than from a crackpot.12 Pointing out a potential threat to validity isn’t nearly as powerful as showing how that threat actually undermines the paper, and that’s what FSE had to do to be taken seriously.

The dataset can be used to say how defect rates compare across programming languages, that's important given that in CS and PL research these things are usually claimed left and right without any evidence. What the dataset can't say is how that translates to quality and productivity, as that would require a lot more than just commit messages. But at the same time it's something people already have some rough ideas about, like that people writing in low level rigid inexpressive languages are less productive than in high level flexible expressive languages.

Of course that's difficult to evaluate. But if their gains were in proportion, I think you'd see these crazy spikes like what you're saying about these researchers becoming gazillionaires. And it just doesn't seem to happen.

Which is not to say there's no difference—but 1) the improvements seem to be by smaller factors than what engineers believe 2) they don't seem to generalize as much as is often claimed: afaict, a significant aspect of the gains has to do with reducing impedance mismatch between problem domain and language (but advocates will typically adopt a stance that the language is just intrinsically superior).

I would be very interested to hear counter-examples if people have them (I know the PG/Viaweb/Lisp story).

Difference in means can meet the definition of effect size, and are listed as an example right away in the wikipedia article on it [0]! The key is that it capture the phenomena of interest (and generally, the magnitude shouldn't be a function of number of observations). In psychology, where scales often have arbitrarily defined ranges (eg IQ scales), means usually are not useful effect sizes.

The case you list is a separate issue from effect size (probably the violation of distributional assumptions in some implicit or explicit model). Even using difference in means / variance (say, cohen's d), two observations that extreme would make the interpretation of both mean and variance calculations pretty dubious (a problem not solved by dividing them).

https://en.wikipedia.org/wiki/Effect_size