I imagine search engine could offer publishers an API to send content for indexing without having to publish it. Google and others could even charge publishers for that and then published could ask readers for subscriptions.
Publishers, if you want to have people pay for your content, make honest paid subscriptions and deal with it that you vanish from the openly accessible web.
You're imagining that it can't work because it can't work in a generic way with all search engines, including future ones that don't exist yet. But that doesn't have to be the case. Most content providers only care about Google.
"Cloaking refers to the practice of presenting different content or URLs to human users and search engines. Cloaking is considered a violation of Google’s Webmaster Guidelines because it provides our users with different results than they expected.
Some examples of cloaking include:
...
- Inserting text or keywords into a page only when the User-agent requesting the page is a search engine, not a human visitor
" [0]
Google is happily showing LinkedIn, FB, pinterest and news sites content. But when I, Joe User, go to the page, I see nothing but some login/register/pay now form. How is this not a violation of the cloaking guidelines? Clearly google is getting different content than what I am!
(Presumably this is how article's extension works... by masquerading as GoogleBot -- again proving that these sites are serving up different content)
[0] https://support.google.com/webmasters/answer/66355?hl=en
edit: formatting