number of samples = 92
hard for me believe that you would infer anything statistically meaningful on population scale from that small of a sample, or am i missing something? is it common to work with so small sample sizes in psychology?
hard for me believe that you would infer anything statistically meaningful on population scale from that small of a sample, or am i missing something? is it common to work with so small sample sizes in psychology?
As sibling posts mentioned, a much bigger issue than sample size is representativeness: college psychology study samples tend to be drawn disproportionately from college students. They might not generalize well to the general population even if you increase the sample size ten fold or a hundred fold.
While that statement is technically correct, it is useless in practice. In order to determine whether a sample is representative (of the trait you are trying to measure), you need to know what the true mean value of the trait is – but if you knew that, you wouldn't need to conduct the experiment in the first place.
That's why it is important to have a large sample size, because under some very general assumptions, the distortions caused by outliers become smaller the more samples you look at. In the limit (that is, when the sample size equals the population size) those distortions disappear entirely, and the sample mean becomes the true population mean.
1. We assume that people have stable measurable independent properties f1(x1.., t), f2(x1.., t), ...
2. We assume these are modelled by a linear model, f1(t) = ax + b, etc.
3. We assume the parameters (a, b) of this model causally determine `f1` st. each corresponds to an independent cause of `f1`
4. We assume that we have some reliable measurement process, say m, which measures each property producing samples of that a,b: m(f1) = random samples of a,b
5. Then we assume that these samples are representative of the true values, ie., that mean(a samples) = a + random_error
The problem with the field is that (1 - 4) are false -- arguing about 5 is a little like arguing what time a broken clock should be set to.
The physicist replies, "At the atomic level I did not, technically, touch it."
Without a causal model of a human mind (and its relationship to the world) -- what exactly is there a "statistics" of?
You cannot really have a scientific psychology; the necessary experiments are impossible or unethical.
> While that statement is technically correct, it is useless in practice.
I don't really follow your argument for small sample sizes being useless in practice.
Isn't the whole point of things like confidence/credible intervals, the bootstrap, and other common statistical methods, to quantify the uncertainty of estimates due to the sampling procedure, including sample size and the possibility of outliers? And, while we may not know that our sample is representative, isn't random sampling from the population of interest a method that will generate a sample that is representative with high probability?
That these methods are well known in statistical practice suggests that samples are generally useful, even when they are not large. The existence of pathological distributions where we might get misleading results (e.g., the classical "wealth over the people in a room if you throw Bill Gates in there") doesn't make these methods useless, does it?
That depends on the sample size. The comment I replied to claimed that "you could estimate population statistics of millions (or even infinite) with a sample of just 30 or so".
Which is certainly not true if the sample is drawn randomly, except under very specific circumstances (such as population variance being extremely small). You don't need a billionaire in order for income distribution samples to be wildly misleading. It's enough for the variance to be high, so that very small samples are almost guaranteed to have a mean that differs substantially from that of the whole population.
How did I just produce a reliable estimate using a sample size of only six?
By quantifying the uncertainty of the estimate. There's a trade off between sample size and uncertainty, and that trade off allows small sample sizes to be useful for many practical applications, most of which can accommodate uncertainty.
Again, small samples are informative. Whether they are informative enough for a particular application doesn't determine whether they are informative enough for all applications in general. I certainly wouldn't call them "useless in practice."
Doesnt sound representative for me at all
Yes. Even much smaller sample sizes are common. I've seen "results" published in supposedly reputable journals with sample sizes as small as 9. Not 9 other studies that were meta-analyzed, mind you. 9 individual people.
This is just one of several reasons why most of psychology is junk science, if not outright fraudulent. Statistical methods is usually a required course for students in such disciplines, so I do not buy the "they don't know better" excuse. Not that you need to know statistics in order to understand that you can't conclude anything from a sample size of 9.
There are good ideas in the field, to be sure. But on the other hand...
This is without a doubt one of the worst and most normalized human rights violations happening in the world today, and the scale and impact of it is hard to imagine.
The police then take that person to a place of safety, usually a hospital.
I can speak from a lot of experience when I say I would trust a clinical psychologist over a psychiatrist any day of the week. Psychiatrists are just doctors who learn to prescribe medication based on a simple set of rules and psychometry. There are psychiatrists who take their field very seriously, but that's the exception rather than the rule.
It's basically impossible to separate the benefits of talk (social interaction, validation, etc) from the talk therapy. Lots of people whould probably be served just as well if they had someone much less trained, or not even trained at all to talk to in a similar manner.
Furthermore as the social bonds disintegrate studies suggest that individuals have less friends and less interpersonal interaction generally (social media use excluded) at the same time the need for psychologists has increased. In some demographics over half the members have been diagnosed with a mental illness. How can we be sure that all of the sudden lots of people have become mentally ill vs lots of people need someone to listen to them?
This goes especially for actually severe mental disorder, because the thing there is, those patients have things going on in their head that most people can't relate to or understand(like irrational fears for instance). Psychologists however have a lot of experience talking with these people, reading and interpreting abnormal body language. And they know the sort of things you definitely don't want to say to a patient.
As for your point about social media, my personal view is that you're mostly right except I do think real mental illness is increasing, because loneliness is such a strong cause of depression especially, but also social phobia.
Throughout my childhood most of my interactions with people outside my family consisted or bullying by other kids and neglect by teachers. Because of this, it's deeply instilled in me that in any social interaction people are catalogueing my every mistake or negative attribute, or just ignoring me. No amount of conversation with other people(or medication btw) has helped me overcome this, but the only progress I have made was in therapy. What I really needed was help to understand why my brain works this way, and how I can be more aware of my own irrational thinking. This has helped me to obsess a lot less about social situations. Though I still have a deep mistrust of people at my core, I've at least been able to let some people in and learn to trust them. And accept that they really do like me for who I am.
While a bit rambling, I hope that provides some context on what a good therapist can do compared to speaking to friends and loved ones.
For young men without strong male influences in particular I think a kind person in some sort of big brother program is probably worth a team of psychological professionals in many cases.
But we need to recognise that actually, there are people who can't be helped this way, who need more than just any talk at all, and that clinical psychologists have a lot of useful expertise on how to help these people, that can't simply be written off as unscientific or just a friendly conversation. There's a lot more to it than that.
I'm guessing that if 9 is enough to call it junk, then 20 would not sway your opinion. What is a good number then? 100? 1000?. Guess could do identical study on groups of 10,100,1000, and show that the results replicate across the size. Of course, this has been done, hence why they get away with smaller sizes.
That's precisely why psychologists take statistics classes as part of their education. I get that those aren't everyone's cup of tea, but people who don't even understand the basics have no business conducting and publishing studies. In fact, well-designed scientific studies typically start with a discussion of p-values, sample sizes, possible sources of biases, etc. If you don't see any mention of those in a study, you know you're looking at junk.
Psychology is still a newcomer, and study of the mind and the human animal is difficult because it is complex and hard to measure. That doesn’t mean you can’t start somewhere.
These are structural and methodical issues which are not at all related to it’s age. In many cases it is simply bias, negligence or pure fraud.
These are examples of a 'new' science. Humans are complex and hard to measure, so yes, there are problems with study design, and re-producibility. Because they are working it out. That is because of the field of study, not evil intent, some dark cabal of researchers putting out false data on purpose. If there is some obvious way to solve these problems, then those people should do it.
I was going to throw out some sarcastic example from another field. But guess, when I think about it, maybe it does happen. String Theory in Physics is an example of "group think" leading an entire field down the wrong rabbit hole of stuff that is faulty and can't be measured. So guess it can happen anywhere.
It's okay to not know. It's not okay to claim results when there aren't any to be drawn from the available data.
And astrology did not "turn into" astronomy. Astronomy developed (in part) as a result of astrologers trying to predict the motion of celestial objects. But astrology is not at all the forerunner of scientific astronomy, since its main concern (how celestial movements influence human lives) lacks any factual underpinnings. It's not that astrology produced incorrect results; the entire endeavor is fundamentally wrong.
Might need to do some history readying. Of course the study of celestial movements eventually became astronomy, who do you think was looking at the stars before astronomy existed. All of history isn't some nice linear line of analytic reasoning. I wonder if HN also has a general bias against history, since it is a non-technical field with a lot of interpretations.
Astrology is not "the study of celestial movements". It's the utterly baseless claim that those movements have a high-level influence on human lives. That's not proto-science, it's pseudoscience. And it's not at all comparable to incorrect theories in other fields (like phlogiston or the luminiferous aether). There is no reason to expect that astrology works, and its claims are made without any supporting evidence. That has nothing to do with science.
I am NOT saying Astrology is true. I'm saying that a lot of people over history, long before astronomy exited, observed the stars, and traced their movement, and yes, some of them attributed their movements to 'gods', or did what today we called 'Astrology'. Then eventually, someone was like, man, these celestial bodies seem to follow some patterns, and drifted away from thinking it was 'the gods', and started putting math to it, and boom, we look back on it and call it 'Astronomy'.
But Astronomy did not pop into existence fully formed as a science.
Electrical engineering, aerospace engineering, nuclear physics, and many other difficult fields of study are far, far newer than psychology. And yet have no problems with reliably producing significant, real-world results.
By that metric, Psychology seems profoundly lacking.
But, to do so, I would need to cite something in Psychology, and you can just deny it again. What evidence can be cited if all evidence is dismissed.
You will probably say something along the lines of 'what can be verified or repeated'. Ok, just go look. Why do I need to re-cite an entire discipline, just go read it.
From my personal experience, the problem is that people who study psychology are not interested or skilled in statistics. Unskilled professors teaching uninterested students perpetuates the cycle.
The field is slowly improving but still has a long way to go.
Unless those small sample sizes are justified by rigorous statistical modeling, showing that the result is still significant even with the small sample size, such studies absolutely are junk.
The burden is on the researcher to demonstrate that their experimental design is suitable for obtaining meaningful results. Failing to do so is the definition of junk science.
And needless to say, lack of statistical knowledge is not an excuse for not doing this. If a medical scientist is unable to perform an accurate statistical analysis of their data, it means that they cannot publish – not that they can skip that part because it's too difficult for them. The entire study is meaningless without it.
Pretending that these papers of “wow, we found a substance that when given to ten people in a cancer group with a 99% six month lethality and three survived more than two years” contribute nothing is a weird belief. Penicillin was what n = 1.
But you shouldn’t judge a group by its average. Also, it’s misogynistic in this case, obviously.
Combina that with the fact that they often use psychology students as experiment group and the fact that most research doesn’t go further than assume a normal distribution, and you might start to understand why more than 2/3rds of all psychology research cannot be reproduced.
Something can be statistically meaningful with a small sample size if the effect size is large for instance.