Adding a feature for an end user to say restrict a search in some manner might be nice, but the point, at least in my opinion, of a web search engine is to find any and all matching content on the web. If some of that is paid or otherwise blocked content it’s still doing its job properly.
This was nondiscriminatory, and I think it was a great policy.
I'd expect this behavior to be a red flag in Google's ranking algorithm. Don't drop Quora, but just let them slide below the fold on results.
Maybe a flag for the pages that require accounts would be in order. But for the shills on Quora, well, why would Google show ads for businesses' services as top results for free - especially since that's the vast majority of their revenue?
What does "average" mean here, though? Do you rank restricted sites higher if more people have accounts?
They could probably do a better job if they knew what you're subscribed to. For example, people with a Quora account or Washington Post subscription should get that ranked higher. But this would mean telling Google more about you, which seems to be unpopular around here?
If a webpage requires login credentials, which the crawler obviously does not have, the crawler should have received a login page.
This is where the early search engines started out at. Turns out there is a LOT of content found for many searches, and the real value to most people is in finding _relevant_ content.
The devil is in the detail of how do you define relevance? This is why I think it is so important to have multiple search engine choices, each with their own algorithms and different sets of user options.
I feel at least a little weird about giving crawlers a carte blanche to paid content. Creating relevant pages, showing them to the crawler to get ranked higher, and then not showing them to the user but requiring something in return first... I don't know.
I do see your point: it would be good to have everything indexed, even paid content or content available only in certain regions or whatever, just so you know what your options are if you're willing to pay for it. But then I think there should be a button to hide those results (either opt-in or opt-out, to show paid content).
As it is, it just weirds me out to show the crawler different content than you show the user and not be seen as fraudulent and get delisted altogether.