Misleading headlines and clickbait make it difficult to efficiently browse news feeds. Natural language processing and recurrent neural networks could soon be leveraged to generate unbiased, accurate summaries of articles and may replace traditional hand-tailored headlines.
A 2015 Stanford paper[1] describes a process for generating headlines from the text of news articles using recurrent neural networks. The process described in the paper utilizes the first 50 words of an article, and concludes that most of the time, the summary is valid and grammatically correct.
In the years since the Stanford paper was published, several open source projects have emerged that simplify the process of getting started with automatic headline generation[2][3]. The software is readily available - the next step is for developers to include these open source tools in their own consumer-facing applications.
It’s unlikely that major news outlets are going to adopt NLP-generated headlines any time soon, and even if they did, it’s unlikely they’d be employed to benefit the reader. Even so, there’s a place for these programatically generated headlines: in the hands of consumers and software developers working in the realm of news aggregation.
NLP-generated headlines could still be optimized for clicks without compromising their accuracy. Several versions of these headlines could be AB tested without any manual effort required on the author’s part. It’s also worth noting that journalists at major media outlets generally don’t write their own headlines anyway4, it’s usually done by editors.
Headlines could even be tailored to each reader: a technical reader may appreciate a more sophisticated headline that contains jargon, but a layperson might appreciate something more simplified. News reader applications and content aggregators seem like natural fits for this type of technology.
With the amount of time people spend scanning through headlines every day, a more effective way to summarize content could drastically improve the reader’s experience. Google became the best search engine in part because it trusted the PageRank algorithm more than webmaster-supplied summaries. In a very similar sense, it’s become difficult to rely on the headlines provided content creators, so it's quite possible that the concept of traditional headlines is ready for disruption just like to concept of search was 20 years ago.