Hacker News Reader

Comment by gpt5 | Hacker News Reader

This is an area where hacker news shows its weakness. We have:

1. A chart showing a very low increase (1-2 percent)

2. Nothing to control scores rising in every school in America in the last school year (due to reduction of COVID effects).

3. Scores not moving immediately after the ban, but only after the start of a new school year, which means a new cohort of students muddying the data.

Yet the data fits people's biases here (regardless whether it's right or wrong), so the celebrate it and add anecdotes and explanations why it's true.

View on HN

There's no study that's good enough for HN.

I don't think I've ever seen a science or research article posted here that didn't immediately get picked apart for this or that in the comment section. The methodology is flawed. The data is flawed. The conclusions cannot be drawn. There are confounding variables not accounted for. The sources are questionable. It's become a trope at this point. Either our commenters' standards are way too high, or all of science reporting is deeply flawed.

Why not both? :)

No study is perfect – research is and has always been expensive, and playing devil's advocate while seeing the arc of promising research is one of the fundamental skills of reading and doing research.

Playing a devils advocate in topics you're not versed in or know the context just makes you a timewasting arsehole most of the time.

Maybe most studies actually are junk.

At least the ones with attention grabbing headlines

No study should be good enough for HN. If a single non-obviously-flawed study is enough to convince you to something, then you can be convinced of anything and everything under the sun.

One study can find any effect it's looking for.

A study shouldn't move consensus. A study finding an effect is a signal that more studies should be done.

Once they are done, and people who know their stuff pour through them and reach some consensus is the sort of bar that needs to be crossed for a reasonable non-expert to 'follow the science'.

And sometimes those experts get it wrong, and accepting that degree of uncertainty is part of it.

And on top of that, I expect most of the complaints about all these studies are made by people who have never designed or run a study or research program of any kind in their entire life.

It's that common phenomenon where people think they can use general logic (which they generally are good at) to draw strong conclusions about something that isn't in their wheelhouse. I'm certainly guilty of it myself, sometimes.

It makes much more sense if you think of these threads as nerdsniping to support a preconcieved personal biases and addictions. Not very related to finding the truth.

And they do all this by reading the abstract.

I'd go further and say its a global weakness and unbelievably destructive. The bulk of current discourse today is:

1. Read a headline/tweet/instagram.

2. Decide whether or not it fits in your worldview.

3. Move forward with the confidence that you are better informed than everyone else who agrees/disagrees with it.

You see it everywhere on all sides of all beliefs.

It didn't use to be like this. We used to read articles, we used to read common news sources, we use to not have media overrun with bad actors who know exactly what to say to get the most engagement and solidify people in their own world views.

It's all over HN and I could have hoped there'd be more willingness to say "let me consider the contents and the source before deciding if I accept it". That attitude is just lost and I don't think it will be regained and I think it's the reason we are all in a death spiral.

When was it not like this, though? I think people are rosey about the past here. A small educated set was different in the past but probably the bulk of the population has always done something like this - now you can hear them online easier.

We didn't always have bad actors directly injecting rage-bait into our blood streams.

Not as tuned for engagement as now, but we had to have yellow journalism laws for a reason too. There's always been lots of propaganda and manipulation and bad actors in journalism.

Sure we did. Or is that also just us conforming to our preferred worldview of the past?

Ok here is the crucial part of the paper:

It's a difference in differences design, using individual-level test scores and de-seasonalized data (p. 13). Their wording is:

> Y_igst is the outcome of interest for student i in grade g in school s in time period t, HighAct_s is an indicator for high pre-ban smartphone activity schools, D_t is a series of time period dummies (t = 0 indicates the first period after the ban took effect), δ_s is school fixed effects, and θ_g is grade fixed effects. In this setting, β_t are the parameters of interest, reflecting the difference in the outcome of interest between treatment and comparison schools for each period, with the period before the ban serving as the omitted category, holding grade level constant.

To me some modeling choices seem a bit heavy-handed, but I'm not an economist and could not do better.

what it means is that this paper shows probable causality and models a lot of interesting features. it is most definitely not flawed.

i think the tough thing is that 0.6 percentage points gain for the average student is quite small. it's actually less than you gain by studying for 1h for the SAT, which is probably about 0.9 percentage points, depending on how you interpret college board's research (it recommends 20h of studying). that is to say, if students studied one fucking hour for the FAST, they would probably get a bigger benefit on it than all the time they get back not looking at their phones throughout two years of school.

so whatever cell phone use (1) in school (2) causes, it causes a small effect on test scores.

you would have to pick some other objective criteria, for example mental health assessment, for maybe a larger effect, or seek a larger treatment, perhaps a complete ban of cell phones period, to observe a larger effect.

Thanks for the context!

To me this was the most informative comment in the thread because it offers some effect size comparison.

To be fair, all those details are in the paper. And a 1-2 percent increase does not seem low to me for such a measure.

I don't think you are being totally fair to the paper, but you do point out something that drives me crazy: local freakouts about year-to-year changes in math tests scores. It's like people don't realize that these 8th grade students are not the same as those.

The lack of control and cohort following are legitimate criticisms. The effect size is not. Even a single-digit percentage increase over a single year from policy treatment is incredibly impressive when we open the door to cumulative effects.

> Yet the data fits people's biases

It does. But it also fits priors, particularly those we've seen documented when it comes to teens and social media.

> But it also fits priors

I know that it's important to look at data and not rely on our own assumptions and "common sense" about things (as reality can often be surprising).

But.

Based on how kids seem to actually use their phones in class (that is, not all that much for educational things related to the coursework at hand), and based on what we know (conclusive study after conclusive study) about how by-design addictive social media and smartphone games are, it's honestly hard to take seriously the idea that smartphone use in class hasn't hurt education and test scores.

Priors matter!

There is another major factor that could cause scores to rise globally: ChatGPT. It is now good enough that it can explain topics to students at home, like a private tutor.

Yea, it's strange that the line didn't move quickly. I would give grace for a couple weeks to a few months, but next year? The timing feels really disconnected.

The chart is showing percentiles not percent. Put your smartphone away before posting on HN.

Reply on news.ycombinator.com