This is very interesting because I see a lot of AI detractors point to the original study as proof that AI is overhyped and nothing to worry about. In this new study the findings are essentially reversed (20% slowdown to 20% speedup).
It’s hard to make reliable, directional assumptions about the kind of self-selection and refusal they saw, even without worrying about the reward dropping 66%.
(Notable to me was how few other studies they cited, which I think is because studies showing AI productivity loss are quite uncommon.)
A lot of them barely rise above the level of collected anecdote, nor explore long term or more elusive factors (such as cross-system entropy). They’re also targeting an area that is fairly difficult to measure and control for.