Hacker News Reader

Comment by jonathansampson | Hacker News Reader

You can use !g on Brave Search too, or turn on Fallback Mixing in Brave Search Settings, which will anonymously call out to Google and pull in results as needed. This helps to train the nascent engine more rapidly. I hope this helps!

View on HN

I am not able to locate this setting. I am using Brave Search on Firefox. Is that available only on Brave Search on Brave browser?

Yes, the Fallback Mixing requires the Brave browser, since it pipes the request through the participating user's machine (only if the user has first opted-in to the feature).

Is that legal?

Why wouldn't it be? Google scrapes the web to populate it's results, why wouldn't other search engines scrape the web as well? Google is a website.

This is naive. Web sites have various terms of service.

Violating a website's ToS is hardly illegal, though.

> Violating a website's ToS is hardly illegal, though.

Can you be more specific as to your claim? Not illegal in what sense(s)? And what is your basis for the claim?

I'm not a lawyer, but saying "violating a website's ToS is hardly illegal" is fraught advice.

While individuals may get some leeway when it comes to ToS violations (see [1] and [2]), I would expect companies scraping and/or extracting content would be treated differently.

[1]: https://www.eff.org/deeplinks/2010/07/court-violating-terms-...

[2]: https://arstechnica.com/tech-policy/2020/03/court-violating-...

[3]: https://www.octoparse.com/blog/10-myths-about-web-scraping

[4]: https://www.law.com/newyorklawjournal/almID/1202610687621/?s...

The only way for it to be illegal is if they're breaking a law, but then it would be illegal regardless of what the ToS says.

Because Google's robots.txt disallows it, and those websites allow it.

robots.txt is not a legal contract. It's just a convention to express the wishes of the site author, but there's no legal obligation to follow these wishes.

It does indicate that those other sites want Google to scrape them, while Google does not want others to scrape their results, which is an important distinction ocdtrekkie ignored for whether the scrapee will want to take legal action.

You may wish to review https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...

Google Search results are definitely "public data" so long as Google provides them to anyone who asks.

Then why does Startpage pay Google and DDG pay Microsoft?

While scraping search results isn't illegal, by any means, it's also not illegal for Google or Microsoft to block requests they believe are from competing search engines. Presumably the cost of paying them is less than the cost of hiring engineers to constantly try to find new ways to outwit Google and Microsoft engineers.

Again, if scraping data from websites without permission, Google simply wouldn't exist. Bear in mind, robots.txt is a feature that Google and Microsoft choose to respect, but the default assumption search engines have made from the beginning, is that they are free to grab whatever they want from the web, unless you ask them otherwise to please not.

> the default assumption search engines have made from the beginning, is that they are free to grab whatever they want from the web, unless you ask them otherwise to please not.

Which Google's robots.txt does.

> scraping search results isn't illegal, by any means

While scraping the results for yourself to look at might be OK, scraping results to display verbatim in another search engine without permission stretches fair use.

> While scraping the results for yourself to look at might be OK, scraping results to display verbatim in another search engine without permission stretches fair use.

No, it doesn't, because Google results aren't copyrightable, hence, there is no such thing as fair use. It's just information anyone is free to collect and use as they see fit.

Why would rankings not be copyrightable?

Why would they be? Again, if all things being copyrightable by default, Google could not even exist, they assume they have the right to consume any data they want.

If a monkey can't copyright a selfie because they're not a person, an algorithmically generated spew of stuff Google ripped from elsewhere certainly lacks merit for copyright.

All things are copyrighted by default. Once again, those websites grant a license to search engines to consume their content via robots.txt, and Google does not.

It's not scraping static text in order to point you to those sites, it's using the features of the site to perform a service better than you can do yourself. It's completely different.

If I made a site that claimed to help you with your math homework and simply sent the queries to WolframAlpha, that would also not just be "scraping."

This is basically what Google does to Wikipedia and rebrands as "Knowledge Graph".

Google has donated many millions to the Wikimedia Foundation, basically in payment for this.

(But, again, there's a difference between scraping and echoing a request on another site and waiting for its response. The latter is basically unauthorized use of its API, not scraping.)

It'll be very telling if Google switches to using Wikipedia Enterprise: https://www.wired.com/story/wikipedia-finally-asking-big-tec...

My guess is Google's "donation" is pennies on the dollar from what they benefit from Wikipedia, and more of a token gesture than anything else.

It's just a redirect to Google.

There seem to be two different features:

1. Redirection to Google.

2. Piping Google's results back to Brave Search's UI: https://news.ycombinator.com/item?id=27594754

The original commenter was asking about the latter.

Ah, I see, Thanks for the heads up.

Is scraping Google for your own commercial search engine legal?

Reply on news.ycombinator.com