I know HN (and probably SV at large) has an anti-journalist bent, but ultimately by draining the individuals and companies that do the reporting, they wont have new material to scrape in the future.
(Disclosure - I'm a former journalist building a way for the news industry to fight back. https://letsgo.forth.news)
Part of licensing is the implicit understanding that if the parties do not come to a deal, they cannot use the licensed content. It seems from this article that Perplexity is not operating in good faith, ignoring robots.txt, using unpublished IP addresses, and then lying about the amount of referral traffic generated.
Not to mention, whether or not its legal (and im not a lawyer) doesn't seem to be the issue. It's being so shortsighted that you build a business that steals a product from an unwitting third party, and when that third party is drained of resources, where will you pull from?
Robots.txt is not legally binding, it is a good will among web stakeholders to respect it. But if you are repeatedly breaking it, you will have an army of web masters against you.
And as far as I remember, big news outlets are already suing LLMs companies or are already licensing their content to them.
And yes, of course they are suing the LLM companies. Because they shouldn't be stealing their stuff, regardless of robots.txt.
- Mississippi Today spent more than 5 years investigating and reporting on the welfare agency and uncovered huge corruption going all the way to the governor's office: https://mississippitoday.org/2023/05/08/pulitzer-prize-winni...
- Lookout Santa Cruz was on the ground reporting during catastrophic flooding this year, including setting up a text message service to get information to people because the internet was unreliable: https://lookout.co/lookout-santa-cruz-wins-pulitzer-prize-fo...
This is the stuff you lose without journalism.
Go to the NY Times or LA Times and tell me it's based on Tweets. If you're talking about a place like The Verge which is a tech blog with journalism, they have articles based on tweets but also plenty that are legitimate, researched articles.
What part are you talking about? Journalists, and everyone else, has to worry that anything they publish then gets served up by the magic bot that billions of dollars are going towards making it an everyday part of life. And Perplexity with $165M in funding is so shady that they can't respect robots.txt and scrape without attribution.
> The truth is that the bulk of what is called "journalism" today isn't on-the-ground gumshoes reporting, it's rewording tweets/press releases/PR statements and maybe a couple of phone calls mixed in
I think that's kind of true but don't neccessarily think that's the majority of major newspapers. It's definitely true for blogs or something like Techcrunch that is 90% press release and 10% articles based on other articles from other outlets. Looking at the LA Times it seems to me like they are reporting the news: https://www.latimes.com
That's the point, though. It isn't replacing journalists, it's just stealing from them. The reporters are the ones doing the work, the AI is just plagiarizing. It's using their work, while not only removing credit, but more importantly, undermining their livelihoods.