Open-Source Psychometrics Project | Hacker News Reader

Open-Source Psychometrics Project

openpsychometrics.org

67 pointsFr0styMatt8815 comments

I couldn't find on the website the "source" (i.e, tem questions, their scores and the what the scores means).

Nope.

Psychometrics is one step away from horoscopes and E-meter readings.

Back to the 19th century where you belong!

Except it isn't. There is clear evidence for efficacy of various psychometrical inventories in clinical settings. There is statistical evidence for the validity and reliability of the instruments themselves. Certainly there are limitations for what things you can use this type of methodology especially in cases where social desirability plays a large role.

[citation needed]

Psychological Testing, yes. Psychometric, no. For example screening for autism is effectively a Psychological test (https://www.cdc.gov/ncbddd/autism/screening.html) However its still easy to abuse/confuse/project into wrong diagnosis. In autisim you don't really get a number to determine to tell you that you are 55% autistic.

In some cases, "psychometric" tests used to determine specific issues work in clinical settings. When executed by a trained person, in a way that is not obvious to the patient. However even then, its not actually clear cut. Its very difficult to put a metric on something that inherently requires a subjective judgment.

They can be used in the emergency room to asses patients for self harm risks. However they are only really a stepping stone to more indepth testing.

Psychometric tests as used outside of clinical settings are deeply flawed, easily manipulated by both candidate and setter. They are a brilliant way to discriminate against any type of person you don't like.

It depends on your definition of "works". When averaged and aggregated, they show repeatable results and correlations looking at a population level (don't they?)

I agree they are probably too "noisy" to be of much use in a clinical setting to draw definitive conclusions about an individual (other than probabilistic risk estimation type things)

Here's a way to eliminate the subjectivity: When a person takes a particular test, you print out their report along with 9 other reports chosen at random, each with a code number but no name. Then you try to pick the one that most closely resembles you or the subject. The percentage of people who can pick their own report is referred to as the "accuracy" of the test. Also, you do this after securing robust legal protections for privacy.

By way of analogy, I'm thinking of a study where people try to pick a picture of themselves out of a collection of pictures.

Being able to tell which report is made from your answers doesn't tell you anything meaningful. It's like saying "describe yourself in ten words. OK now which short description describes you?"

That's fair, but does it tell anybody anything meaningful? How do we know that the reports are accurate?

I've seen some of the reports, and they're fairly elaborate, like multiple pages.

You are being critical of certain practices that in no way reflect the general state of psychometrics. I'm wondering what definition you're using.

"Big Five Personality Test"

I mean look at it. Its just fluff. I know which personality I want to get, I can easily affect the outcome without trying too hard.

And yet people put huge amounts of value on these tests because they have the veneer of "science" to them. its basically phrenology, but with slightly less overt classism.

What do you mean "which personality I want to get"? You don't choose between five personalities, you find where you are on the scale of different personality traits that are present to some extent in everyone.

And yes, you can affect the outcome if you want. You can affect the results of any test if you want. I can make my IQ test look like I'm very stupid. I can cram for some certification exam and forget all the material the next day. The idea is that if people actually want an accurate reading then they will answer truthfully.

I feel like you're conflating two arguments that are unrelated. The fact that the results of psychometric tests can be misused and manipulated by bad actors is a separate issue to whether or not they are scientifically valid when administered properly and taken in good faith. You've admitting that various tests can provide useful --though not 100% accurate-- information. That's the whole point: To provide useful information, which can be further explored.

Psychologists creating and using a test to evaluate five personality traits does is not them saying that there are only five personality traits or that everyone with a given result is the same. The tests are basic, they are primitive in our understanding of how the mind and personalities work, and they have just been shown to be statistically valid enough to be useful in informing other things.

On that note, I completely agree that psych tests shouldn't be used to determine anything of importance outside of clinical settings. As I said, none of this is anything but a primitive understanding and shouldn't be used to determine career tracks or anything about our place in society. I hear about companies using the MBTI (which I personally loathe) to place people and I cringe so bad.

psychometrics is basically the only area of psychology that actually replicates well

Would you have a decent source on that?

The best source I know of is this:

https://journals.sagepub.com/doi/full/10.1177/09567976198316...

Soto CJ. How Replicable Are Links Between Personality Traits and Consequential Life Outcomes? The Life Outcomes of Personality Replication Project. Psychological Science. 2019;30(5):711-727. doi:10.1177/0956797619831612

Not exactly the same, and I wouldn't say exactly what the OP said but yes, it suggests these kinds of things at least replicate fairly well.

FWIW, I'm very familiar with this open-source psychometrics project website (I think I used data from it once) and it's sort of a mish-mash of pop-fun-entertainment fluff and legitimate scientific stuff.

Your comment is poorly informed. There are many instruments in circulation that are poorly validated or misused for applications in which they have not validity evidence, but psychometrics as an approach or collection of approaches to developing measures of individual differences in cognition, personality, and so on is in no way on the level of horoscopes.

Reply on news.ycombinator.com