Comment by ACow_Adonis | Hacker News Reader

Comment by ACow_Adonis | Hacker News Reader

Speaking informally, but as a professional analyst here who is unfortunately at a computer with no analysis tools setup :( but who has worked with his fair share of social, survey and demographic information.

Check your data. My first gut-instinct on seeing data like that is not "Eureka - i found a huge first-birth-order effect", but rather "huh...looks wrong, I wonder why that is?".

What do I mean "looks wrong"? Keep in mind I have no real "pony in the race" on first-born effects (although I admit I am one), but intuitively it "doesn't look like real social data looks like".

The effect is too big. Its too loud. Its too constant. Its too nice. Something else is driving it and its your job to find out what...

The first place I'd look is data quality. Have you proven your data got into your final histogram data set right? I had problems with your csv opening it in libre office...looks like it interpreted it as having some value migration between the variable in whatever you did. Could be my computer, but in my experience, if there's a problem with the csv, these kind of things correlate with other mistakes.

The second place i'd look is survey design: was there a default option, are people clicking through by default, are the response frequencies of other variables lining up as would be expected (gender, birthdays, that kind of thing). If not, why not?

If you've crossed and ticked all those, only after that would I start the analysis...but again, a pattern and effect of that size and consistency is something that you should default to needing explained in the data. Reach the conclusion that its real only after you've eliminated everything else.

Just a quick follow up, the reason I suggest looking at things like gender or birthday frequencies is because they're two bits of social data that have brilliant properties for getting to the bottom of this sort of things.

In practice, gender is immaculately reported and recorded compared to most other variables.

Both gender and birth days have nice reasonableness bounds and statistical properties that can be transparently reasoned about. Deviation from those nice statistical properties not only confirms for you that something is going on in your data, often the way it can deviate gives hints as to the next thread you should start chasing.

Obviously it's not perfect, not all data problems will show up there, but in practice it's an extremely low hanging fruit on which to get started, and seeing strong deviations from 50/50 statistical gender splits and predictable day/month/year type distributions should start analytical klaxons going off in your demographic noggin...

Lots of groups don't have 50/50 gender splits.

It's odd to have to point this out in this forum; the tech community has female participation rates around 25%. That doesn't somehow make statistics about this community suspicious.

Sure, if you didn't know that the base rate was skewed, you'd want to check your numbers the first time you made that measurement. But it's well-known (I suppose only by regular readers of SSC) that the SSC community is significantly above the population base rate on a whole host of metrics (self-reported IQ, transgender, % of males, autistic spectrum, etc).

Edit: to clarify which point I was referring to

My point is that you shouldn't expect gender ratios to be 50/50 unless you're choosing at random, and therefore it's not suspicious

Sorry, should have made that more clear -- I agree with your point, your post's parent was the one I thought was strange.

Professional (American) football also has participation rates varying greatly from 50%.

In this case, I'd still be reluctant to make conclusions about birth-order effects if the gender split is not about 50/50.

The gender split is not expected to be 50/50 for this group.

It may not be 50/50, but if you didn't select for gender in your survey design, it would be a "brave assumption" that gender isn't connected to your outcome/suggested mechanism in some way...

which is why you also need to take age into account. certain age groups will have different ratios, but the standard deviation for those age groups should cover for those variances.

This. Not taking into account actual age is completely wrong. Age of the respondent is going to show a skewed result (I bet none of your readers are 10 years old...). Does that mean the older sibling is "smarter"? No, it means young people are not your audience...

Yes, absolutely. That's why you also look at things like the distribution of birth day digits, which is something you expect to be standard in almost every data collection that didn't directly use that as an explicit selection factor.

But regardless, back to gender.

Lots of groups don't have 50/50 splits. But where they don't, it's elementary to discover that they don't. Plus it's a peice of information that is traditionally very accurately recorded: recording mistakes and misreporting compared to other variables in practice is almost traditionally the lowest of any variable amongst most populations of social data.

How much they differ from 50/50, and why they differ is usually a big peice of the analytical puzzle.

Someone else pointed out "well, professional footballers all men". But that's exactly my point, if you were generally interested in discovering things analytically, and you got a set of data sources from professional footballers, you'd discover all men, and quite rightly you'd investigate and find out "yeah, gender plays a real role in professional football". There's actual mechanisms and predictions you'd eventually find.

But in this situation, where you're explaining birth order effects, we need to start hypothesising and looking for tests, falsifiable claims, and mechanisms of operation.

Now, I don't know what mechanism is at work here, but you can bet that an almost 100 make presentation rate amongst a survey that didn't explicitly select for it is highly relevant to the puzzle.

Why is it attracting these people? Are these representative males themselves? Are the females representative? Does the birth order effect only express itself in this community and these makes in these parts of the country/world/industry? What is its suggested mechanism of action that can be stated in a falsifiable way? If it was real, where else would you find it, and given our known distribution of makes in the population, how often and strong might you expect those effects to be?

One possibility that came to mind when I read his post that harmonizes both your point and his data is that the studies that have been done would almost certainly have been looking at very broad effects, on very broad populations, which results in a very averaged, fuzzy view of things. That is not a criticism, because it is a priori both plausible that there will be some effect at that scale, and that there won't be, so it's a worthy test.

But it is also possible that while broadly speaking birth order doesn't have any significant effects on things, when you go out to a population as niche as "people who read SSC and like it enough to fill out a form for 15-30 minutes just because Scott asked them too" and you slice the n-dimensional space in question with that much precision, that you will get a very, very particular slice of the world in which there are birth order effects.

This would imply that there are birth order effects, but that in general they are too weak to be picked up by a general-purpose study. However, perhaps if you magnify them in such a precise way you can see them, even though they are small. "Member of the SSC community" is a very information-rich cut of the general population and it would not be a surprise to see that poking out of the stats in many other ways.

Statistics being what they are, I believe it is completely possible for both "Birth order has no significant impact on whether you will show up in SSC's community" and "Given membership within SSC's community, birth order effects can be witnessed" to be simultaneously statistically true. Previous studies showing birth order to not matter would be testing a hypothesis much closer to the first one, Scott is obviously testing the latter.

This is merely a hypothesis that is not very complicated (in Occam's Razor terms, given the fact that "everything sorta correlates to everything" is a basic fact of the world we live in and thus using that does not really count as "multiplying entities") and fits both the facts, but not one I am claiming is certain to be true. A lot more analysis would have to be done, to a degree of rigor I doubt anyone is going to perform since nobody has the full picture and the effort to obtain the full picture is likely larger than the payoff. So let it be clear that I'm just spitballing, and aware of the fact that I'm just spitballing.

This hypothesis could well be true, but it's somewhat difficult to falsify; you can always posit an effect that only appears in the small statistically-underpowered cohort that Scott sampled, and which would be swamped by the average of the population as you start to increase the sample size to verify the effect.

As a counterpoint, Scott went the other direction in his post, arguing that the true effect was more general, but the metrics previous academics used was broken, and therefore undersold the strength of the effect they were detecting:

> I think the most likely explanation is that tests for Openness have limited validity, which makes the correlation look smaller than it really is. If being an eldest sibling increases true underlying Openness by a lot, but your score on psychometric tests for Openness only correlates modestly with true underlying Openness, that would look like being an eldest sibling only increasing test-measured-Openness a little bit.

It's simple to falsify this (though it requires some work); just do more experiments on the general population using the metric that Scott likes [1], and see if that detects a stronger birth-order effect.

[1]: https://www.researchgate.net/publication/230787941_Construct...

It wasn't a default option (you were asked to enter into a text field your number of older brothers, older sisters, younger brothers, younger sisters).

He can't cross-reference by gender very well because his audience is overwhelmingly male, to the point where there's probably not a lot of statistical significance in the female audience.

But yes, I think that these data are very open to an "impossibly hungry judges" critique. If birth order effects were so very visible and so very strong, why haven't we noticed them before?

Anecdote time/thought experiment: and forgive me, I'm deliberately vague here.

Pretend there's a business somewhere.

Pretend they implemented a rule that said people had to enter a number/estimate into a box for a certain peice of information before they could proceed to the next box.

Pretend as an analyst you're looking at the values resulting some way down the pipe from that data entry point.

Do you think it reasonable to assume the distribution of numbers entered into that box that were invalid would be uniformally random?

If the answer is no, what numbers do you think people would enter disproportionately?

It seems tough to me to suggest that people are putting a default 0 in for older siblings but a default non-zero in for younger siblings. Why wouldn't they be consistent?

That said, I still don't believe these data. I don't know where the flaw is, but I none the less think there's a flaw. If it were a much smaller effect, it would be much more believable.

> If it were a much smaller effect, it would be much more believable.

Why do you think that?

We know that the SSC readership is in many ways quite distinct from the general population, so why do you find it surprising that it has correlations with birth order? Is it because you find it surprising that anything has correlations with birth order?

The effect would have to be smaller because most effects in psychology and sociology are pretty small. Else they would be obvious (common sense) even without field experiments.

I agree with the other comments here that it is a rather common sense effect, but it has eluded psychological research until now (which is what makes me question it). I know plenty of people who have a common sense view that birth order is important. So that part isn't missing. I can't even grasp how many times I've heard about the middle child being so different. Or being a typical little brother, etc.

For me the question is, if it is so obvious and measurable, why is this the first study that finds this obvious effect.

Most effects are small if you sample general population randomly. However, if you make a survey among basketball players, asking them for their height, you're going to find an extremely large effect. And in fact, birth order effect seem to be common wisdom too.

I like this way of approaching the problem, but I don't think it convincingly contradicts the GP's point; if participants were just entering invalid data to skip the question, then I think they would enter a similar type of number in both "# older siblings" and "# younger siblings", e.g. 0 and 0, or 100000 and 100000, or `rand(10)`.

The key observation is that I'd expect the same RNG to be used for each of the four fields.

I think you'd see much more `0`s in the fake data as well (it's a Schelling point). But if the fake-data population was disproportionately recorded as only children, that would have reduced the first-child effect, not attenuated it.

>If birth order effects were so very visible and so very strong, why haven't we noticed them before?

I'm not sure where you're from but everyone I know (including a psychologist) sees birth order effects as very real with noticable effect. Have you never seen the stats of first borns being overrepresented in management, for example? Am I misunderstanding what you mean by "haven't noticed them before"?

Another example, having older brothers is one of the most consistent correlations with homosexuality for males. Birth order effects are absolutely visible.

As the article mentions in like the first couple of paragraphs, those results haven't generally withstood replication.

But more so, the effects in the Slate Star Codex survey are extremely strong. You'd expect that an effect that strong would be part of the common wisdom, not something that has to be teased out in a psychological study.

Well, to what you just mentioned, it -seems- to be a "common wisdom" that is disregarded by scientific study. So, maybe this is a case where the study was ill-designed, but everyone took it for granted?

I don't think it is common wisdom. Like, look: which of your coworkers are first-born? How many of them did you intuit that without explicitly asking?

That depends on the field. The guesses can be very accurate based on ancillary data like profession.

> why haven't we noticed them before?

LMGTFY

https://scholar.google.com/scholar?q=birth+order+effects

Was it optional though? He said some people didn't give him enough data, which makes me suspect first-borns were more likely to go "oooo this question is about me being awesome - better fill it in!" and other people to go "eh wtf I'll skip it".

All questions in the survey were optional. He had 2965 respondents for that question, out of 8077 total respondents, so that's a plausible source of bias.

> It wasn't a default option (you were asked to enter into a text field your number of older brothers, older sisters, younger brothers, younger sisters).

Is blank coerced into 0?

That would seem to explain the size of the effect.

No.

Off-topic, but is this a common practice? How do analysts ensure that studies aren’t incorrectly invalidated when bad data ends up ‘looking good’? (BTW: I know little about data, so forgive my ignorance.)

A few ways come to mind:

Randomized double-blind studies

peer reviewed question sets

longitudinal cohorts

The goal is to control for known biases. It’s hard, however, to discern conclusions from a dataset that is already biased. There’s very little in this to conclude effectively that “intellectual curiosity” was caused by birth order. They might be independent variables from each other on this survey born out by another variable. Or are completely spurious in nature.

Reply on news.ycombinator.com