This is often required work in supervised learning / any machine learning with labels. Sometimes you get better result with human judgment on the labels, and that means you have to grind through a meaningful corpus of data.

If you believe what you're doing is important and beneficial, it doesn't feel like too much of a grind. Ultimately this was probably about a week of work, if every labeling participant independently reviewed ~3-5K articles.

Some people might have chosen to mechanical turk this, or to gather feedback from a subset of users and use those labels. Doing that might be a better approach as it might not only be less labor-intensive up-front but also allow the system to be regularly retrained easily (ie by gathering new label information as clickbaiters inevitably try to adapt their headlines). I imagine there were reasons for starting off this way, though.