Comment by peatmoss | Hacker News Reader

Comment by peatmoss | Hacker News Reader

Parent Full thread

Sorry, I missed your comment / didn't realize it was directed specifically at me. So, if you go back and read what I wrote:

1. I definitely did not say that economics does not strive for generalizability. Quite the contrary. I think the vast majority of economists strive for generalizable conclusions (i.e. go to great care to find study samples that are representative of the population of interest and use statistical methods that might allow them to plausibly conclude things about that group)

2. I never said "all" work in other social science disciplines rejects generalizability as an aim. However, I do believe, that more "empirical" scientific method based studies in a number of disciplines would be good. That is predicated on 1. a belief about the prevalence of qualitative/theory based scholarship. 2. a normative preference that the ratio is undesirable. I could be mistaken on my perception of the first point, but I don't think I am. On the second point, you're welcome to disagree. These are, like, opinions, man!

3. "reject the goal of generalizable knowledge because they question or reject 'the scientific method'" makes me think you've either missed my point, or you don't understand the nature of other forms of academic scholarship.

I'm not leveling a diss that e.g. a participant observation case study isn't a method intended to generalize from. That's just an intrinsic feature of that kind of study method. Of the qualitative researchers I've known, I can't imagine any thinking there is anything controversial about what I've said on that point. Though, some would definitely disagree with me on my opinions about what ratio of scholarship should be of this kind.

And lastly you've selectively quoted me there at the end, and attached a conclusion to it that is not mine. Consider this a bit of original qualitative research on my part: a sampling of non-empirical research that I've read in recent years has suggested to me a lapse in rigor in certain disciplines. But this conclusion is grounded in my "situated knowledge" of the space, and thus shouldn't be used to generalize without a suitably operationalized quantitative study ;-)

I'm sorry, but the thrust of your earlier post was absolutely that the social sciences "have taken a pretty hard turn" and "might benefit from a renewed emphasis on methods that can result in generalizable findings", and that "a correction is warranted".

Pragmatically speaking, I'm unsure why you would choose this language if you wished to convey the nuance that qualitative methods are actually great, that you're merely wishing that more "empirical" studies would also be undertaken, where "empirical" I guess means "quantitative" and "rigorous" though you don't make that explicit.

For what it's worth, you absolutely can generalize from an observational case study. RCTs are not the only way of drawing generalizable conclusions—it depends a lot on what your epistemic goals are.

It kind of sounds like you don't like that some social sciences rely more on non-quantitative methods because you don't think those are definitive. That's fine, you're welcome to hold that belief, but let's not pretend like you're a fan of all methods and just wish there were a few more quantitative studies in sociology (or whichever discipline).

No need to be sorry, I think you're letting your priors overwhelm your understanding of what I'm writing. What if I don't think that these methods are irredeemable or useless or epistemologically bankrupt, and STILL think that a bunch of disciplines in the social sciences over use them and could benefit from a course correction?

Does that resolve what I think you are perceiving to be a contradiction, but that I do not see as a contradiction?

Where I think you and I have a fundamental disagreement is in the nature of e.g. qualitative research methods such as case studies and whether a case study is generalizable. This feature was taught to me by... qualitative researchers. If you truly believe a case study is a generalizable research method, then either you are defining generalizability in a different way than is typical, or you hold a minority viewpoint not shared by mainline qualitative researchers.

Not all research methods need to be generalizable to have some scholarship value, and I'm not using the concept of generalizability as a colloquial stand-in for ~"bad"

BUT, I still think social sciences could use fewer qualitative studies and pure theory grounded scholarship (i.e. "a correction is warranted"). It's what I said from the start, it's what I mean now, and unless I someone provides me a compelling rationale to change my mind or the world changes, it'll probably be what I believe going forward.

It seems important to you to attribute to me a total rejection / disparagement of these methods. I am just not saying this. I do not reject these methods, just as I do not reject apple pie. I just don't think that a steady diet of apple pie is good. And I don't think the research methodology diet of many social sciences is healthy either. A correction is warranted.

Hmm, alright, sure: Pick a field. What would the shift you are proposing look like in that field? What’s the ratio now between qualitative work and quantitative work, and what ratio would you like it to be, post-correction?

What will be the outcome of that shift? Some kind of better research?

On what basis do you think that field should agree with your perception?

On what basis do you think your perception is correct?

Note that I’m using “qualitative” and “quantitative” as stand-ins for whatever you think there’s too much of and too little of, respectively—please feel free to clarify if these words don’t effectively capture what you are trying to say.

It’s not at all important to me to attribute to you a “total rejection” of these methods, your writing implies to me that you have a preference for quant methods and think they’re better. You’re not, for example, complaining about too much quant methodology in economics, and too little qualitative. Sure, you don’t think that qual methods are bad per se, but you do think that the social sciences could use fewer of them, and that they’re overused.

I imagine you would agree that these methods can tell us different things, and that they’re not interchangeable for any given research question. You’d probably also agree that some fields bias towards certain types of questions, and that maybe the ratio of methods of work in a given field reflects the bias towards questions that are best answered by those methods. So, are you suggesting that entire fields should focus more on different questions, specifically those that can be answered quantitatively?

If not, I’m not sure I understand what the implications of your argument are.

For what it’s worth, here’s my bias: I am both a quantitative and a qualitative researcher, and I actually think the underlying issues holding back the production of generalizable knowledge have little to do with choice of methodology, and to the extent that they do, it’s in part due to a fetishization of quantitative methods that tell us something generalizable—but not necessarily something useful or even something true.

Reply on news.ycombinator.com