> At the end of the day, is all this measuring anything other than trends in filling up questionnaires?
I get that sense that this makes up a bulk of social science research. This would be a criticism of the field as a whole.
I get that sense that this makes up a bulk of social science research. This would be a criticism of the field as a whole.
I think the social sciences are useful and in fact indispensible. I disagree with their methodology and with the practice of copying methods from other fields that are really not suitable to the subject matter of the social sciences.
For instance, if you define a set of answers to a questionnaire as "agreeableness" and assign it a score, you can do maths with it, just like physics can define the measurement on a thermometer as "temperature" and do maths with that. But there are no thermometers in the social sciences and the maths seem to only be measuring the researchers' intuitions (and of course, their cultural biases). [Edit: don't ask me what a thermometer is actually measuring- but I know that if a thermometer shows the water in the kettle is 100°C then the water is boling. If my agreeableness is 1, what does that do? Does it have a consistent effect? Can I measure the effect? With what? Another questionnaire? So it's questionnaires all the way down? Well, I know for sure that whatever thermometers are measuring- it's not thermometers all the way down.]
It would be a lot more informative to hear what the researchers think, their intuitions and conclusions from their careful observations of human behaviour _without_ any attemt to quantify the unquantifiable. We would learn a lot more about the human mind by listening to the _opinions_ of people who have spent their life studying it if it wasn't for all the maths that (to me anyway) are measuring meade-up quantities. If nothing else, there would be more space left in their papers to explain their intuition.
Obviously, there is an entire literature that measures what it does in terms of life outcomes other than questionnaires.
If you'd like to learn something very elementary about a field you're completely unfamiliar with, you might be better off picking up a textbook, rather than borderline trolling of the "this entire field is nonsense, prove me wrong" variety.
It means you are more likely to answer other questions in a certain way. Other studies might even show that you might be likely to behave in a certain way.
> Does it have a consistent effect?
Yes, but like thermometers only work on groups of molecules, the questionnaires are consistent on groups of humans.
> Can I measure the effect? With what? Another questionnaire? So it's questionnaires all the way down?
No, you could easily do a follow up study by finding groups of people that answered the questionnaires in a certain way, and then have them participate in behavioral experiments.
> Well, I know for sure that whatever thermometers are measuring- it's not thermometers all the way down.
The manner in which molecules bump into eachother randomly. The harder they do it the more space they take up. The more people in your group with an agreability score of 1, the more space they might take up ;)
Opinions are not science, maths is. We don't improve the social sciences with more vague intuitions. If you want to read intuitions then maybe read a glossy instead. Science is about making quantifiable statements, and maths is the way you turn samples into those. We certainly don't need any more room for so-called experts to tell us how we should behave in their papers. We tried that, and it was awful.
That's a very well structured passage, thanks for the comment.
However, what I see is that psychology is trying very hard to make quantifiable statements about things that it can't do quantifiable statements about. Yes, maths can be used to turn observations into quantifiable statements. But just because someone is using maths, it doesn't mean they're turning observations into quantifiable statments. You can use maths to quantify non-existent quantities that you have never observed and the maths themselves won't stop you. I will mention Daryl Bem and his measurements of ESP now, but I don't mean that psychology is like parapsychology, only that you can misuse maths if you're not very, very careful. And just because you have maths it doesn't mean you're being careful.
And I think intuitions and personal expertise with a subject are the basis of scientific knowledge. The maths are there as a common language to communicate the intutions gained in a manner that makes them accessible to others who do not have the same expertise. Maths is the language of science, because it's used to communicate scientific knowledge, not because it's a set of magickal formulae that transform eveything to solid science.
That's what nobody ever addresses in these studies.
Even assuming all this linguistic questionnaire stuff passes for a measure of something (certainly not biology, it really falls apart on indigenous populations), the further mathematics gives the joke away. Factor analysis is done just wrong. Questionnaires are mostly positively correlated and no thought is spared to how Frobenius-Perron theorem produces spurious factors, that also are dimensionally invalid to boot (which, one imagines, is not unwelcome, as scaling the data may give a stronger result). Then the methodology manages to fail confirmatory factor analysis on its own terms anyways. https://sci-hub.tw/10.1007/s11336-006-1447-6 Clustering validation is not even attempted beyond trying different number of clusters (anywhere from 4 to 13 results in fits only marginally worse than 5).
Denunciations of Big Five (and friends) go far and wide decades back. Then there's a flood of reassertions as if nothing happened, and again some refutations of that new wave. Ascent of data science made things comical. One year they do a metastudy with one million respondents, some dude asks some basic questions, the next year they do it with two million as if this answers anything. It is an endless war of attrition and not worth anyone's time.
I cite a large passage from the paper you linked to because it's an excellent example ofthe kind of "misuse of maths" I meant:
Consider, for instance, the personality literature, where people have discovered that executing a PCA of large numbers of personality subtest scores, and selecting components by the usual selection criteria, often returns five principal components. What is the interpretation of these components? They are “biologically based psychological tendencies,” and as such are endowed with causal forces (McCrae et al., 2000, p. 173). This interpretation cannot be justified solely on the basis of a PCA, if only because PCA is a formative model and not a reflective one (Bollen& Lennox, 1991; Borsboom, Mellenbergh, & Van Heerden, 2003). As such, it conceptualizes constructs as causally determined by the observations, rather than the other way around (Edwards& Bagozzi, 2000). In the case of PCA, the causal relation is moreover rather uninteresting; principal component scores are “caused” by their indicators in much the same way that sumscores are “caused” by item scores. Clearly, there is no conceivable way in which the Big Five could cause subtest scores on personality tests (or anything else, for that matter), unless they were in fact not principal components, but belonged to a more interesting species of theoretical entities; for instance, latent variables. Testing the hypothesis that the personality traits in question are causal determinants of personality test scores thus, at a minimum, requires the specification of a reflective latent variable model (Edwards & Bagozzi, 2000). A good example would be a Confirmatory Factor Analysis (CFA) model.
Now it turns out that, with respect to the Big Five, CFA gives Big Problems. For instance,McCrae, Zonderman, Costa, Bond, & Paunonen (1996) found that a five factor model is not supported by the data, even though the tests involved in the analysis were specifically designed on the basis of the PCA solution. What does one conclude from this? Well, obviously, because the Big Five exist, but CFA cannot find them, CFA is wrong. “In actual analyses of personality data [...] structures that are known to be reliable [from principal components analyses] showed poor fits when evaluated by CFA techniques. We believe this points to serious problems with CFA itself when used to examine personality structure” (McCrae et al., 1996, p. 563).
If I'm not prying too much- what is your relation with the field?
To cope with that problem, science needs to detach the term from everyday use, and put an artificial definition in its place which allows to make repeatable statements. In its wake, the term loses a lot of its meaning. That is the price for preciseness.
Imagine there was a unit for agreeableness, so you could say 'John has an agreeableness of 4.4 Ag'. Then it would be clear that this statement refers to a formal definition (based on a standardized questionnaire), instead of our common vague understanding.
Now you could still argue that this new definition is so detached from what we usually mean with agreeableness that it becomes useless. However, you can't simply dismiss the method, you need to bring concrete arguments why the proposed definition does not capture what it is supposed to capture. For example, you could show that people with agreeableness below 2 Ag are married happily more often than people with agreeableness above 4 Ag. Do you have such concrete objections?
You have something specifically against psychology research, I haven't seen a single argument from you that could not be applied against any other scientific field. They applied a methodology that you didn't initially understand, and now you're refusing to understand it because you're committed to arguing against it. Weird thing is it's not even a controversial finding, just a confirmation of something everyone knows to be true.
It is different in the sense that particle physics quantifies concepts that are not correlated to how someone feels about them, or how someone answers questions in a questionnaire.
And I don't see how I misunderstood the studies we're discussing. They handed people questionnaires asking them how they think or feel about things. I don't see how any concrete evidence about anything can be found in this way, other than how people fill questionnaires.
But the claim is never "X people fill this questionnaire in this way". It's always along the lines of "X people are more agreeable" etc. This is misleading.
- Is it a valid criticism?
- If so, are the social sciences really sciences at all?
And one philosophical question:
- What happens if an entire field of "research" is dissolved as wholly subjective and not repeatable?
This would be much bigger than the debunking of phrenology or astrology as those don't have university departments, journals, or attempt to set social policy.