Hacker News Reader

> Google can't index Facebook content or Twitter content even though it's public

They index plenty of their content [1][2]. What they don't index is content not explicitly marked as "Public". e.g. Facebook posts with visibility settings or protected tweets. FB, Twitter, and now LinkedIn have plenty of content that's not publicly accessible: that is, content which doesn't require a logged-in account that explicitly agreed to TOS/EULA, but they have tons of publicly-accessible content, too. The latter is fair-game.

That said, the Craigslist lawsuit is still bewildering to me. That content is explicitly public, does not require a login, and the only agreements are the automatic unforceable browsewrap ones. The LinkedIn v. HiQ case is very similar to the Craigslist v. 3Taps, however the decisions are in opposite directions.

[1] https://www.google.com/search?q=site%3Afacebook.com

[2] https://www.google.com/search?q=site%3Atwitter.com

What's bewildering?

Published does not mean uncopyrightable. A radio station broadcasts a song; that doesn't remove the song's copyright protection. Receiving a copy is not making a copy.

3Taps settled put of court; there was no decision.

I'm not sure the radiostation analogy holds. Radiostations can't claim copyright to the song.

Yep - I specifcially remember using Padmapper (3Taps) around the time that Craigslist yanked their content.

Padmapper is just a much better interface for craigslist data. Even though craigslist pulled their data, craigslist didn't create a new interface to match or even bother to improve their experience, even though they have the user mindshare for posting this type of data.

Essentially it's as you said - innovation is being killed to preserve walled gardens like Craigslist, and users are the ones being hurt here.

> craigslist didn't create a new interface to match or even bother to improve their experience,

That's not completely true. After lots of bashing online [1], craigslist started adding some features such as searches on maps[2]. Even though it's not as nice and easy to use as padmapper was, it would still be considered an improvement.

[1] - https://www.inc.com/abigail-tracy/craigslist-quietly-makes-c...

[2] - https://sfbay.craigslist.org/d/apts-housing-for-rent/search/...

What part of the indexing isn’t allowed, is it the scraping or the publishing of the scraped material? Or something else such as a “use” of the material?

Example: Does it matter whether the data is ever stored? Say I make a listing notifier that pings people when a house is listed in their desired area. I’d need to crawl real estate listings and but I’d never store or republish any content from the site.

The hardline copyright view is that to "index" content from a website, you need to make a copy of that content in your indexing server, which is an action copyright law legally prohibits without an explicit exception. (The people who argue that somehow also seem to argue that publishing the webpage legally gives "explicit permission" for your web browser to make a copy, without also giving permission for your web scraping software to do so. Not sure how they get to _that_ position logically...)

Yeah but to e.g. make an app that notifies people when a house is published in area X really doesn't need to index anything. I can walk through all the listings once per week and all that is in my database is the table of what user wants what notification. If the data on the site matches my notification table I send an email (or whatever) and after the weekly run is done I still have zero data copied or "indexed" from the target site.

Makes sense from a technical standpoint, but it's the legal people you'll need to convince.

Actually, you still have a copy of the data, in memory.

yes. And I suspect it's not legal. Which is why I'm wondering what the actual definition is. Because it can't be that I "keep" or "index" the data, nor that I "republish" the data.

It's actually that I "read" the data, regardless of my intention and regardless of what I intend to do with it. But taken to the extreme there has to be something that makes it legal for me as a consumer to scrape ONE item from their data, while at the same time making it illegal for a company to scrape ALL their data in this way.

The worst situation to be in is one where it's "legal until it isn't" i.e., when you actually cause a problem (bandwidth, revenue) you will be sued - and until then you no idea.

It's that you "copy" it. That's the "right" that copyright law extends to creators.

Technically (and it's complicated these days because copyright law was mostly written well before computers and the internet existed) any time you create a copy of someone else's creative work, you need an explicitly granted right to do so. You can _read_ a book, but you cannot write down all or part of what you just read. You can listen to a song, but you cannot perform all or part of that song (An Australian band called Men At Work lost a copyright case for a flute solo that contained 2 bars/11 notes of a 1930's folk tune "Kookaburra" who's copyright had been sold/bought in 2000.)

There are some explicit legal exceptions to requiring a license, most commonly people talk about "fair use", but that's again a "pre internet" body of law, and is extremely open to interpretation as to how it applies to digital copies in computers.

https://www.copyright.gov/fair-use/more-info.html

https://fairuse.stanford.edu/overview/fair-use/what-is-fair-...

post 3taps, craigslist won't be able to stop scrapers.

https://www.eff.org/deeplinks/2018/04/dc-court-accessing-pub...

That is about the CFAA, not copyright.

how do you claim ownership of data you release publicly at no charge to the users? how do you enforce anyone who copies it by hand or programmatically, does the minimum work to derive some insight and then copyrighting that? You can't. Once it's on the internet, anything you release will simply be relinquishing control.

It's foolish to assume that the lawyers and the justice system will help you out. This is simply not a case of somebody pirating a movie and slapping on their own copyright.

CFAA was how craigslist won but that's been thrown out thanks to the EFF lawyers.

Going forward, there will be little to no recourse for "web scraping public data".

There are a few forums where I don’t expect nor would prefer indexing. They may be niche forums, etc. Forums like reddit, of course I expect to be indexed. So, basically, it depends.

Isn't this what robots.txt was created to convey?

Yes. But not all scrapers and indexers respect that? So it’s still a problem.

Require a login/invite then

I would like to know your opinion regarding the content problem that people prefer sharing their content on the established platform (simply for widest reach) which prevents new players from emerging (since they are intimidated from scraping said content, even if the original poster intended the widest reach).

Suppose there was an intermediary platform so that the user uploads his content/video once to the intermediary platform, and can then select which platforms may use the content under what conditions (i.e. perhaps for free, perhaps for a user-set minimal amount of remuneration, ...), or perhaps instead of selecting which platforms, selecting the rules to which a platform must adhere in order to distribute the content (so that new players can instantly start hosting a diverse set of content as long as they adhere to these rules). The user thus only needs to upload once.

Obviously it would require youtube etc, to provide API access for the intermediary so it can post the content without being able to hijack a user's account on youtube etc...

This would not be in their interest of course, so anyone attempting to create this intermediary upload service would face a never-ending cat and mouse game, somewhat reminescent of youtube-dl...

Perhaps instead of silly link tax rules, it would be better for governments to force platforms above a certain size/usage to provide API access such that these intermediary upload services can flourish.

They might effectively become some kind of consumer/producer protection organizations, by proposing multiple usage standards (like my video MUST NOT have ads, or perhaps the opposite for another user, my video MUST be remunerated at x cents per y views, otherwise the platform may no longer distribute my content, ...) from which their users can select...

apart from "pure format" content like music, videos, ... as soon as there is more compound structured content, like a blog post with images, it would be hard to standardize the format.

then there is the problem of authentication, especially if a user uploads a video through the intermediary, and 1 year later a new platform wishes to distribute the content on its network too, and after another year the user wants be active on this platform, there is already an account, and how does he get his login credentials?

In theory the intermediary could have a messagebox for each user, and the platform can send the credentials to the user in this messagebox, so that if interested a producer can decide to make use of his account on a new platform...

Edit: so while currently all the platforms act like publisher and printer separate the market into publishers (intermediary rights managers) and printers/distributors (big platforms like YT etc). Similar to how humans decided it was better to split up doctors and pharmacists to prevent de facto quackery of selling whatever you happen to have on hand, or cheaper deals for...