A well designed study would mix "bullshit" and "non-bullshit" statements of similar complexity and look at signals on both for comparison. Better still, the "bullshit" statements would be derived from mixing words from the "non-bullshit" statement pool.
PDF: http://journal.sjdm.org/15/15923a/jdm15923a.pdf
And there's some followups...
Dalton 2016 - http://journal.sjdm.org/15/15923ac/jdm15923ac.pdf
Reply from Pennycook et al. - http://journal.sjdm.org/15/15923ac/jdm15923acr.pdf
> participants were shown a list of activities (e.g., biking, reading) directly below the following instructions: “Below is a list of leisure activities. If you are reading this, please choose the “other” box below and type in ‘I read the instructions’”. This attention check proved rather difficult with 35.4% of the sample failing
The perils of using poorly compensated students for your studies!
Eventually, I suspect/hope that NLP and neural networks will enable more authentic data collection across many spectrums.
From the article itself