Hacker News Reader

Comment by jrochkind1 | Hacker News Reader

> To find out, we ran another experiment in which participants imagined buying a bottle of wine. We had half of the participants imagine considering opening it one night, but deciding not to.

This is the second study like this on HN in a couple weeks.

When the heck did asking participants to imagine something and then asking them how they felt, and assuming this tells you something valid about things other than imagining things... when did this become a popular form of research?

It seems crazy to me.

View on HN

Seems like you could probably do some effective and efficient research using games. I always have to force myself to use the one shot super weapon, or even a few grenades, in order to not end the game without ever using them.

Video games seem like the perfect arena in which to do and apply this research.

In a lifetime of regret and missed opportunities, this is one area at least where I've learned my lesson. I can now go through a whole game and use all the "special" stuff that I'm given, and usually even at the appropriate times! Kinda proud. It sounds silly but it took a lot of letting go.

It's not silly, you found a dojo for practicing "letting go"! Hope your can transfer this to life.

Will try your method if i ever find the appropriate time to play the "special" games on the steam backlog...

RPG players do this all the time - they save those health and mana potions for a time they really need it, and invariably never do.

Games that streamline these mechanics by making recovery items more fungible lose a little personality but are probably easier to balance.

And then it removes any challenge in the end-game when you can just drip-feed yourself from the huge pool of potions you've been hoarding.

I've identified this problem in my gameplay in roguelikes. It's hard to predict precisely what will be needed in later levels, so I find myself being quite a bit too conservative. I'd love to see research on this behavior.

Not a research on such behaviour but more an application to macroeconomics : https://en.wikipedia.org/wiki/Capitol_Hill_Babysitting_Co-op

It illustrates :

> the paradox states that an increase in autonomous saving leads to a decrease in aggregate demand and thus a decrease in gross output which will in turn lower total saving. The paradox is, narrowly speaking, that total saving may fall because of individuals' attempts to increase their saving, and, broadly speaking, that increase in saving may be harmful to an economy.

This is the fallacy of composition applied to aggregate demand. That's important to know about, but what others were talking about is the psychology of individual choices. There isn't any aggregation or composition happening there as far as I can see.

It's got to a point where in games like Skyrim I generally just sell the majority of potions and whatnot, because I know when I get down to it I'll always be thinking there's a better time to use it.

For whatever reason, I still can't let go mid-combat, but when selling it I have no issue.

Yeah: Imagine you had won the powerball lotto but decided not to cash it in. I can definitely "imagine" tossing it and burning it, etc. If it happened in real life, it would be very different.

It's a form of operationalization and that's a valid critique. My hope would be that there are some studies on the validity of the technique.

That being said, there's probably better ways. Additionally, I'd find it fun to riff on ways to make it better that have the same time/budget constraints as the original.

I'm inclined to agree with you, but I think the answer is actually more nuanced.

At a minimum, it seems like a good way to run a cheap and quick way to try to validate an idea before dumping more time and money into it. If it seems like there is something there, then a longer and more expensive study is easier to justify. And when that study does happen, you'll already have data on what people expect to happen which is nice.

It's not a perfect method for sifting through ideas, but at the end of the day you need some way of deciding what to pursue now, and without time travel there is no perfect method.

And because methods are included in publications, from a scientific standpoint there's nothing wrong with publishing it. Other researchers will read the paper and clearly see what was tested and be able to assign an appropriate amount of significance.

That significance may also vary through time. For example, somebody might conduct research about the accuracy of these imaginings in various ways. Maybe we find out we're actually quite good at them in some ways, and not so good in others. Maybe this is already done, and is why you've seen several recently? I don't know.

The problems mainly come from the popsci aspect. (Some) Writers and (some of) the general public see 'scientific paper' and more or less base their own assessment of significant entirely on that. You can also get a bit of the 'telephone game' going on, and the significance can get all jacked up. I don't know how to solve this problem, but I don't think the correct solution is to alter the scientific process itself.

> At a minimum, it seems like a good way to run a cheap and quick way to try to validate an idea before dumping more time and money into it. If it seems like there is something there, then a longer and more expensive study is easier to justify. And when that study does happen, you'll already have data on what people expect to happen which is nice.

That seems quite reasonable.

But lots of popular press coverage on "look what the scientists figured out" based on your initial validation justifying a longer and more expensive study... maybe less so.

Like, if this really is understood as that... should pop press be covering it at all? This article definitely is written with teh tone "look what the scientists have learned about humans" not "they got enough validation to do more research", right? Hard to say what the researchers themselves think about the strength of what they found out, if they have your reasonableness about it.

But the researchers are clearly happy to give interviews and be quoted; I mean, it's a lot to expect a researcher to give up good press, sure.

> And because methods are included in publications, from a scientific standpoint there's nothing wrong with publishing it. Other researchers will read the paper and clearly see what was tested and be able to assign an appropriate amount of significance.

I wish I had your generosity of spirit about contemporary practice and motivations of academic science.

> I wish I had your generosity of spirit about contemporary practice and motivations of academic science.

Ha! I seem to have given you a false impression. There's... so many problems. However, I don't think most (any?) researchers are actually ignorant of those problems. They are actually doing research somehow, so they've figured out the game well enough.

I apparently managed to disguise this well when I said 'assign the appropriate amount of significance.' The unsaid implication here was that it may well be virtually none, particularly if it doesn't fit with a larger body of work.

In this article for example, it's paired up with another similar finding from an actual physical experiment, which lends it more weight.

> But the researchers are clearly happy to give interviews and be quoted

I always wonder about this. When I've seen longer form interviews with researchers, they almost always seem appropriately uncertain. At least they do sometimes, I might just not be knowledgeable enough about the field to know if/when they aren't.

Meanwhile, you don't get much of that in popsci work, beyond maybe a small disclaimer at the beginning.

I think Adam Ragusea (a cooking youtuber of all people, though admittedly he's an ex-journalism professor as well, and also dealt being the interviewee and having things go poorly) actually covered the intricacies of this really well in a video[0].

I've tried too many times to write "Long story short, ..." here, and have ended up with way too much meandering. Just watch the video. It's worth it, and does a great job of covering the perspective of all parties involved.

I think it's appropriate to assume good faith in individual instances - there are reasonable justifications for misrepresenting the certainty of some work in this kind of context. It's unlikely to cause any actual harm, and probably gets more people intellectually stimulated (which is probably good in both objective and subjective ways). In this article it even lets them make a concrete suggestion that has no cost to try and can probably help some people.

I don't know that there is an objective way to weigh those benefits against presenting a more accurate representation of the certainty of some information, when a misunderstanding of it is unlikely to ever harm anybody. I assign quite a bit of value to that accuracy inherently, but that's coming from the core of my value system. I can't present a logical argument for it's correctness any more than I can the exact degree to which I think human suffering is bad. As a result I think there's room here for reasonable disagreements about where to draw the line.

[0]https://www.youtube.com/watch?v=fxUnwsttr_8 (I think this is very illuminating if you're interested in this topic. I highly suggest finding the time for it, even if you understandably don't find the time to read the rest of this over-long comment.)

> However, I don't think most (any?) researchers are actually ignorant of those problems. They are actually doing research somehow, so they've figured out the game well enough

The problem is that "winning the game" is about publications, tenure, and (less important but increasingly also) popular recognition, rather than about the size of your contribution to valid research results (one hopes that publications and tenure correlate, and yet...).

So, getting results of dubious validity published, and covered in popular press, can be exactly "figuring out the game well enough", an end in itself, rather than a minor step toward a research program.

You are right that I don't want to assume bad faith in any individual instances, and yet... here we are, in aggregate.

What are the costs? I dunno, mis-education of the popular audience, for one, I guess. Maybe that doesn't matter, but, it kind of does? And the opportunity cost of all the scientific labor being spent on "easiest to publish the most times" instead of what might be highest priority to discover. And the aggregate collective effect of a scientific community enabling each other on that.

> The problem is that "winning the game" is about publications, tenure, and (less important but increasingly also) popular recognition, rather than about the size of your contribution to valid research results (one hopes that publications and tenure correlate, and yet...).

> So, getting results of dubious validity published, and covered in popular press, can be exactly "figuring out the game well enough", an end in itself, rather than a minor step toward a research program.

You're describing Science by Press Conference/Press Release.

https://en.wikipedia.org/wiki/Science_by_press_conference

> The problem is that "winning the game" is about publications, tenure, and ...

Totally. 100% with you there. I didn't mean researchers aren't doing those things, but rather that papers built for those ends are probably not polluting 'science' as a whole. The body of work product is getting increasingly swampy, but the researchers are also the best positioned people in the world to sieve through it.

A large part of their job is to sort out what a paper actually tested and what it actually observed to begin with. These 'game playing' papers are quite blatant in comparison to the kinds of problems researchers already need to look for.

> What are the costs? I dunno, mis-education of the popular audience, for one, I guess. Maybe that doesn't matter, but, it kind of does?

Totally agree. I personally find it kind of abhorrent. I also can't honestly tell you why beyond "because" and am pretty certain most people don't feel as strongly about it as I do. I try to be mindful of that, but I think it mostly just gets used when I end up playing devil's advocate.

> And the opportunity cost of all the scientific labor being spent on "easiest to publish the most times" instead of what might be highest priority to discover.

Sure thing. It also increases time spent on literature searches for all that, and will for quite some time even if things changed tomorrow.

On the other hand, maybe it provides an easier on-ramp for teaching prospective researchers to be skeptical even if it's in a Paper, and even how to analyze them? At least it's nice to try to hope there's some benefit to all this.

This is where gaming-based simulations (usually based on videogames) are apparently catching on as a very interesting realm for psychological experiments.

It's a game and a simulation, so the set-up is low (once the game iself exists), but participants actually seem to get highly invested in the game mechanics. (This might be a stretch for people on HN to believe, but stay with me ;-) And it's possible to run some very-large-scale studies by tapping into existing gaming titles and platforms, particularly MMOGs.

EVE Online is among the instances I'm aware of specifically:

https://www.eurekalert.org/news-releases/765393

https://cs.stanford.edu/people/eroberts/cs201/projects/virtu...

I would guess that a lot of the validation for studies like this comes from marketing research. Marketing often does "study groups" to try to figure out if, say, one product name is better than another. Would you buy "ZimZam Fluid" or "Superlicious"? The study participants are not confronted with the actual product (it may not exist yet) but have to imagine which they prefer. If this approach shows success in marketing (which is all about peoples behavior) then it seems likely it is useful for figuring out other things about how people work and what they will think in specific situations. What people imagine is a clue to how they think, you could think of it as a simulation. People often imagine future outcomes as a way to make decisions in the real world.

What you are also saying is that the way research is funded determines how research is done. You need a reputation to get big money. To get a reputation you have to start small with the little money/time you have. Hence lots of studies are done that are too small, not well controlled, and are essentially useless because no one will believe they have conclusively proved anything. However, once you get 10 papers based on these poor studies you start to look like an expert and it becomes easier to attract bigger grants. By this time you may have actually formed some ideas about what it is you want to research as well and abandon your original ideas as castings in the dark. So it's good in that it helps focus research and helps researchers gain experience, but it is bad in that it results in lots of studies that are not very useful to anyone (like all the medical studies you read about where, when the topic becomes important like in a pandemic, you will find scientists calling out studies as underpowered, too small, not an RCT, and useless.)

It seems possible that making the basis of getting a PhD be making a unique discovery ("your contribution" as it is called) is misguided. It results in the above useless studies to gain reputation, plus it means researchers won't touch each others ideas for fear of being labeled derivative instead of unique. It may be much better to have all scientists in a field brainstorm ideas and contribute them to a common list and then individual grad students and researchers could choose one. The list could be ranked by the same crowd (and hopefully by outsiders as well) and used to assign grant amounts ahead of the selection of an actual researcher applying for the grant. That might result in better and more important topics being studied, an easier path to reputation for new scientists, and more useful studies being done that will actually advance science. Or maybe it would mean that a cabal of scientists and politicians take over the list and misdirect research for decades, which would not be much different from what we have now.

This is exactly the kind of weak external validity that has caused much of the replication crisis.

Reply on news.ycombinator.com