All these "should" statements always seem to be baseless opinion masquerading as fact.
The unfortunate future is going to be all data hidden behind a login screen. Creators being able to sustain themselves is more important than consumers being able to get free and more convenient things. If consumers don't want to pay for something they should just create it themselves. See what I did there?
This isn't competition, Perplexity isn't doing their own reporting and seeing who gets more readers/revenue. This is wholesale theft.
CVS's moat isn't the security guard standing at the door. But like parent comment said, this is going to create the digital equivalent of those plastic lock covers over the decongestants.
But I think we can also both agree hoping that people will abide by your robots.txt file wont save us from that unfortunate future.
As long as that content is valuable it will be scraped and used.
Theres no solution to this. So the structure of content sites will have to adapt.
How? I dont know, likely will vary by publication and content type.
Is it fair? Probably not
Is it stoppable? No. Not without heavy, global legislation. (read impossible)
If a house is robbed, you don't give the burglars a pass and say "well their security system wasn't very good."
Yes, the defensive measures probably need improvement. But that doesn't mean that the act they are defending against is wrong.
Its a sign on the door that says "Please don't rob me" while the door is unlocked and no body is home.
In such a large ungoverned system as the internet morals don't dictate what people do / dont do.
So it doesnt matter whats wrong / right.
What does matter is what system participants will / wont do
And they will scrape your site if you don't put up any real defences.