Why we're taking legal action against SerpApi's unlawful scraping | Hacker News Reader

Why we're taking legal action against SerpApi's unlawful scraping

blog.google

66 pointsxnx102 comments

Google really doesn't have a leg to stand on here. They scrape the Internet. They replace content against the wishes of users multiple different times, such as with AMP. Their entire business model recently has been to provide you answers they learned from scraping your website and now they want to sue other people who are doing the same.

Data wants to be free. They knew that once.

EDIT: Also to be clear I am not saying they can't win legally. I'm sure they can do legal games and could shop around until they were successful. They are in the wrong conceptually.

Eh, and in 20 if SerpApi or whatever the fuck becomes the next google, they’ll have a blog post titled “Why we’re taking legal action against BlemFlamApi data collection”.

The biggest joke was all the “hackers” 25 years ago shouting “Don’t be evil like Oracle, Microsoft, Apple or Adobe and charge for your software, be good like Google and just put like a banner ad or something and give it away for free”

We need a legal precedent that enshrines adversarial interoperability as legal so that we can have a competitive market of BlemFlamApis with no risks of being sued.

Unfortunately they do have a couple of points that may prove salient (though I fully agree about them being scrapers also).

You can search Google _for free_ (with all the caveats of that statement), part of their grievance is that serpapi use the scraped data as a paid for service

Lots of Google bot blocking is also circumvented, which they seem to have made a lot of efforts towards in the past year

- robots.txt directives (fwiw)

- You need JS

- If you have no cookie you'll be given a set of JS fingerprints, apparently one set for mobile and one for desktop. You may have to tweak what fingerprints you give back in order to get results custom to user agent etc.

Google was never that bothered about scraping if it was done at a reasonable volume. With pools of millions of IPs and a handle on how to get around their blocking they're at the mercy of how polite the scraping is. They're maybe also worried about people reselling data en masse to competitors i.e. their usual all your data belongs to us and only us.

> You can search Google for free

I thought the ads counted as payment? That seems to be the logic used to take technical measures against adblockers on YouTube while pushing users towards a paid ad-free subscription, at least.

If viewing ads is payment, then Google isn't a free service. If viewing ads isn't payment, then Google should have no problem with people using adblockers.

I don't disagree with the logic and it definitely is/was their business model, scraping/crawling the web and subsidising the service with ads. But clicking on ads are optional.

No google's business model is showing you ads, not clicking on them. That's the job of the person who designs the ad.

Google would like you to click through as it looks better for their stats, but they don't actually care.

> You can search Google _for free_

Well not through their API which you do need to pay for and is a paid service.

As the post says, Google only scrapes the websites that want to be scraped. Sure, it's opt-out (via robots.txt) rather than opt-in, but they do give you a choice. You can even decide between no scraping at all and opting out on a per-scraper basis, and Google will absolutely honor your preferences in that regard.

SERP API just assumes everybody wants to be scraped, and doesn't give you a choice.

(whether websites should have such a choice is a different matter entirely).

This is a bad argument because Google is using its monopoly to effectively force websites to allow Google to scrape them.

requiring me to explicitly opt-out of something is NOT the same thing as getting my consent. So your argument breaks down there.

You know what getting my consent would look like? Google hosting a form where i can tell them PLEASE SCRAPE MY WEBSITE and include it in your search results. That is what consent looks like.

Google has never asked for my consent. Yet they expect others to behave by different rules.

Now where google may have a reasonable case is that google scrapes with the intention of offering the data “for free”. SerpAPI does not.

It's never been the case that if you put something into public, then you get to reserve your right to refuse public access. Either it's public and strangers can look at it. Or it's private and you need to implement a gate.

> You know what getting my consent would look like? Google hosting a form where i can tell them PLEASE SCRAPE MY WEBSITE and include it in your search results. That is what consent looks like.

Just for anybody wondering, they have always had such a form as well. Apart from their general crawling.

If this is about protecting third parties from being scraped, why does Google have an interest at all? Surely Google won't have the relevant third-party data itself because, as you say, Google respects robots.txt. So how can that data be scraped from Google?

I don't think this suit is actually about that, though. I think Google's complaint is that

> SerpApi deceptively takes content that Google licenses from others

In other words, this is just a good old-fashioned licence violation.

they have the Leg to stand on. It's called Money. Second one is called Position (on the market). Third is Lawyers. It's a stable tripod

I bet SerpApi is getting more business than ever due to the Streisand effect. I hadn't heard about them, but if I want an API for Google results I'm definitely going to choose the one that was so hard for Google to block that they had to sue them instead. I see on their website they even advertise a "legal shield" where they assume scraping liability for their customers.

Can confirm, I just signed up /because/ of this announcement.

And its API seems really easy to use.

"Google follows industry-standard crawling protocols, and honors websites’ directives over crawling of their content."

Is that true with how they trained Gemini? Doesn't everyone with a foundational model scrape the web relentlessly without regard for robots.txt?

They're in a unique position where many people allow googlebot but try to block most other bots

Allow for the purpose of indexing, not training models.

Like if you give a friend a key to your house so they can check on your plants when you're out of town but they throw a rager and trash the place.

> throw a rager

That was not a phrase I expected to read on Hacker News! Haven't heard it since I was about 13. I always assumed it was a Scottish phrase.

I've never been to Scotland and I don't know any Scottish people.

But yes, I am dating myself with that phrase...

No, but AFAIK they pulled some shenanigans with "bundling" Gemini scraping and search engine scraping.

Almost everybody wants to appear in search, so disallowing the entirety of Google is far more costly than E.G. disallowing Openai, who even differentiates between content scraped for training and content accessed to respond to a user request.

While there isn't a way to differentiate between scraping for training data and content accessed in response to a user request, I think you can block Googlebot-extended to block training access.

Google honors robots.txt. They're not "everyone with. a foundational model" though.

Reminds me of (the ironic AI summary) https://www.google.com/search?channel=entpr&q=celebritynetwo...

Testimony https://medium.com/@brianwarner/celebritynetworths-statement...

CNW ended up putting up content for fake celebrity's after declining Google's request for API usage to prove that Google was scraping them.

I had an idea - take SerpAPI and save top-10 or 20 links for many queries (millions), and put that in a RAG database. Then it can power a local LLM do web search without ever touching Google.

The index would just point a local crawler towards hubs of resources, links, feeds, and specialized search engines. Then fresh information would come from the crawler itself. My thinking is that reputable sites don't appear every day, if you update your local index once every few months it is sufficient.

The index could host 1..10 or even 100M stubs, each one touching on a different topic, and concentrating the best entry points on the web for that topic. A local LLM can RAG-search it, and use an agent to crawl from there on. If you solve search this way, without Google, and you also have local code execution sandbox, and local model, you can cut the cord. Search was the missing ingredient.

You can still call regular search engines for discovery. You can build your personalized cache of search stubs using regular LLMs that have search integration, like ChatGPT and Gemini, you only need to do it once per topic.

Fetching web pages at the kind of volume needed to keep the index fresh is a problem, unless you're Googlebot. It requires manual intervention with whitelisting yourself with the likes of Cloudflare, cutting deals with the likes of Reddit and getting a good reputation with any other kind of potential bot blocking software that's unfamiliar with your user agent. Even then, you may still find yourself blocked from critical pieces of information.

No, I think we can get by with using CommonCrawl, pulling every few months the fresh content and updating the search stubs. The idea is you don't change the entry points often, you open them up when you need to get the fresh content.

Imagine this stack: local LLM, local search stub index, and local code execution sandbox - a sovereign stack. You can get some privacy and independence back.

CC is not on the same scale as Google and not nearly as fresh. It's around 100th of the size and not much chance of having recent versions of a page.

I imagine you'd get on just fine for short tail queries but the other cases (longer tail, recent queries, things that haven't been crawled) begin to add up.

Pre the AI era at least morally they would have a stronger case, but now this is kind of hypocritical.

They also started caring about this, probably because they don't want their competitors to get the same data as they have.

google will lose, and I'm surprised they are even trying. hiQ v. LinkedIn already settled this: scraping public web pages isn’t “unauthorized access,” even if the site says no via robots.txt or ToS. Those aren’t locks.

In Germany, this was also already ruled lawful by the highest court (in the context of plane ticket prices scraping).

HiQ lost on appeal, Microsoft won

Incorrect. OP's view is present day 9th Circuit precedent.

No, it's more complicated than that: https://www.morganlewis.com/blogs/sourcingatmorganlewis/2022...

The short answer is that scraping isn't a CFAA offence but might be a terms and conditions violation, depending on the specifics of the access.

Isn't search engine results a product that Google offers? [1] I find the argument quite strange that website owners agreed to Google being able to do anything with that data beyond displaying them in their search results when they wrote that robots.txt maybe ten years ago, but others shall not access those results programatically.

I certainly did not and find using the content google scraped from my website for money or AI (which they also sell on a token basis) more questionable than some third party offering API access to it.

[1] https://docs.cloud.google.com/generative-ai-app-builder/docs...

I bet the core of the problem for Google is that more folks use programmatic access to search which is not great on their side. Naturally you end up using Serp or other similar search APIs as they are great for the job. I believe this is also an issue especially in cases where search is performed on behalf of the user (scripts, ai tools). Google is just losing ground here, why would they bother otherwise, think what will happen to their stock if the search usage will drop? Another thing is that this builds pressure to whoever is integrating such a search tool in their products, clearly Google wants to grab that market as well.

well well well, how the turntables.

I'm not sure of the legality but I definitely appreciate their product. This lawsuit seems odd because google themselves scrape content for their indexes. From what I see SerpApi is really just providing a machine interface that Google themselves refuses to provide users and visibility into SERPs which is also something that users should have available to them.

I'm probably just being naive though...

Google publishes how to control their bot - with robots.txt. They then obey those instructions. Google also takes some effort to not use all your bandwidth. Google isn't perfect, but they are at least making a "good faith" effort to be nice and this does count in court. Overall most will agree that in general what google does to allow people to find their website is worth the things that google is doing.

You can of course argue a lot of edge cases if you really want. For the most part I want to say "it isn't worth the argument". In some cases I will take your side if I really have to think about it, but in general the system google has been using mostly works and is mostly an acceptable compromise.

What's nice about scraping all the content for their own good while killing off websites left and right? Google needs to be sued also.

Along with all the other AI companies out there, the've committed the biggest theft in human history.

But their robots are enabled by default. So it is a form of unsolicited scraping. If I spam millions of email addresses without asking for permission but provide a link to opt-out form, am I the good guy?

At this point everyone knows about robots.txt, so if you didn't opt-out that is your own fault. Opting out of everyone at once is easy, and you get fine grained control if you want it.

Also most people would agree they are fine with being indexed in general. That is different from email spam where people don't want it.

Looking at SerpApi clients, looks like most companies would agree they are fine with scraping Google. That is different from having your website content stolen and summarized by AI on Google search, which people don't want.

The claim is SerApi is not honoring robots.txt, and they are getting far more data from google/more often than needed for an index operation. Or at least that is the best I can make out of the claim in court from the article - I have not read the actual complaint.

People are generally fine with indexing operations so long as you don't use too much bandwidth.

Using AI to summarize content is still and open question - I wouldn't be surprised if this develops to some form of "you can index but not summarize", but only time will tell.

Or by Google codewiki, which is morally the equivalent to making a business out of ersatz travel guides by ripping off the authors of real ones

Who says robots.txt is legally binding? Where's the Sherman Antitrust analysis?I'm more confused than before.

The courts say. With this as a long standing tradition they are likely to agree.

> The courts say.

Do you have an example of a court saying that violating robots.txt violates an existing law?

In Ziff Davis v. OpenAI [1], the District Court for the Southern District of New York found that violating robots.txt does not violate DMCA section 1201(a) (formally 17 U.S. Code § 1201(a), which prohibits circumvention of technological protection measures of copyrighted content [2]).

It's my understanding that robots.txt started as a socially-enforced rule and that it remains legally voluntary.

[1] https://blog.ericgoldman.org/archives/2025/12/are-robots-txt...

[2] https://www.law.cornell.edu/uscode/text/17/1201

In other news, Tailwind had to layoff their team due to lack of spending on new web sites as Google Search AI is answering search requests from scraped data without sending visitors to origin sites.

Very rich coming from an entity which scanned your neighborhood without asking for permission

is this post written by gemini? Cmon google, it wasnt even 10 mins long to write this.

This is why I stopped using google wherever possible - they pushed the frontier of useful fair use and copyright precedents and established that things on the public internet displayed to the public without a login mechanism are fair game for scraping. The US supreme court ruled that you have to incorporate authentication and not simply serve your content to the public internet if you want to restrict usage.

Then they bend over backwards and do the "but not like that!" crap with their legal team and swing their wealth and influence around to screw over other companies and people, and a vast majority of it just vanishes, gets memory holed, with NDAs and out of court settlements, so you never get to see the full scope of harm they inflict unless you're watching like a hawk and catch the headlines before they get disappeared.

Google needs to be broken up and we need to legislate the dismantling of the current adtech regime, with a privacy and sovereignty respecting digital bill of rights that puts the interests of individual citizens above that of giant corporate blobs and the mass surveillance data industry.

Move fast and break things unless they are the things owned by billionaires and gigacorps instead of stuff owned by normal people

Pot calling the kettle black

Now that the rocks are out in the glass houses, can someone please pelt one at Perplexity?

https://blog.cloudflare.com/perplexity-is-using-stealth-unde...

Google scrapes so what even is this? Beyond that I think it is unreasonable and monopolistic that Google can use all this data (like YouTube) to bolster their AI products but no one else can. It just means the megacorp will keep being megacorp and smaller players are doomed to have to work much harder and get very lucky. It’s not fair competition. So I view scraping Google as necessary for our society.

Reddit Accuses 'Data Scraper' Companies of Stealing Its Information

https://news.ycombinator.com/item?id=45695433

Our Response to Reddit, Inc. vs. SerpApi, LLC: Defending the First Amendment

https://news.ycombinator.com/item?id=45739889

From the filing

> SerpApi’s answer to SearchGuard is to mask the hundreds of millions of automated queries it is sending to Google each day to make them appear as if they are coming from human users. SerpApi’s founder recently described the process as “creating fake browsers using a multitude of IP addresses that Google sees as normal users.”

> Defendant SerpApi, LLC (“SerpApi”) offers services that “scrape” this copyrighted content and more from Google, using deceptive means to automatically access and take it for free at an astonishing scale and then offering it to various customers for a fee. In doing so, SerpApi acquires for itself the valuable product of Google’s labors and investment in the content, and denies Google’s partners compensation for their works

this has to be satire. Is Google not the #1 entity guilty of exactly this?

No, Google doesn't use deceptive means. They identify their crawler as GoogleBot, and obey robots.txt.

Google doesn't have to do that now after already having established its own monopoly... just like SerpApi wouldn't have to act deceptively if they had a monopoly on search.

What about for their LLM products? We know that OpenAi does not respect the robots.txt file

Google uses the same crawler and robots.txt file for training data.

It's actually a different crawler for training data: Googlebot-extended so you can exclude yourself from the training data though not the search summaries.

Because they've forced everyone to allow them. They're the internet traffic mafia. Block them and you disappear from the internet

They abuse this power to scrape your work, summarize it and cut you out as much as possible. Pure value extraction of others' work without equal return. Now intensified with AI

But yeah, you're right. They're not deceptive

> Because they've forced everyone to allow them.

nobody is forcing anyone. This is the same argument that people said about google search. Nobody is forcing anyone to use google search, google chrome, or even allow googlebot for scraping.

Thousands of poeple have switched over to chatgpt, brave/firefox ..

Your argument sounds like "I dont like Apple's practices, and I'm forced to buy iPhones. No buddy, if you dont like Apple, dont buy their products"

> Your argument sounds like "I dont like Apple's practices, and I'm forced to buy iPhones. No buddy, if you dont like Apple, dont buy their products"

No, not really. There are alternatives to Apple. Whereas here Google controls the gate to the majority of internet traffic

For many it's "block Google and your business dies"

> Thousands of poeple have switched over to chatgpt, brave/firefox ..

If you want people to visit your website, limiting yourself to the "thousands" of people who don't use google isn't really an option.

> Your argument sounds like "I dont like Apple's practices, and I'm forced to buy iPhones. No buddy, if you dont like Apple, dont buy their products"

Well, I don't like Apple's or Google's practices, but I basically [1] have to use either iOS or Android.

[1]: yes there are things like GrapheneOS and librem, but those aren't really practical for most people.

> SerpApi deceptively takes content that Google licenses from others

They have a different definition of "licensing" than most people I guess. Aren't site operators complaining about Google using this "licensed" content in AI overviews... not to mention the scraping for AI model training.

The pot is calling the kettle black.

As far as I know, Google respects robots.txt and doesn't obfuscate their crawlers, so you can easily block them if you want. It seems like an important distinction?

There's no law that says you have to do that. It used to be a sensible thing to do, in the early internet. In the current internet, obeying robots.txt is a self-handicap and you shouldn't do it.

DDoS remains illegal regardless of robots.txt.

It's rather odd to use words like "should" when you're advocating for disrespecting other people's wishes. There are sometimes reasons not to cooperate, but it seems like a good default.

The web is now hostile. If you're starting a search engine, everyone else has written a robots.txt that bans you from starting a search engine. You either ignore that, or you abandon your plan to make a search engine.

Maybe only ethical choice is not to play? Or to do it the hard way. Scrape what people allow and try to make deals to get more data.

"Making a search engine is unethical" is certainly one of the takes of all time. I'm sure Google is glad you believe it.

Google can afford to respect robots.txt because it has a monopoly on search and nobody would consider actually blocking them in said robots.txt anyway.

SerpApi doesn't have that privilege.

but SerpApi is not scraping websites, it is sending malicoius requests to google.com.

SerpApi is scraping Google. The "maliciousness" if the requests is a matter of perspective. Of course Google considers it malicious; that doesn't necessarily make it true.

Google has respected robots.txt from the start.

Some domains do block Google, often partially. There are some statistics here:

https://radar.cloudflare.com/ai-insights#ai-user-agents-foun...

robots.txt is not a legally binding document, nobody needs to actually respect it

SerpApi wouldn't even be a thing if Google offered an equivalent API...

why does google need to offer it?

Because Google scrapes other site's data to build its AI market dominance in Gemini. The promise of web 2.0 was APIs, Google aims to cement its position in web 4.0 while suing others for doing what it does on a mass scale.

Adversarial Interoperability is Digital Human Right. Either companies can provide it reasonably or the people will assert their rights through other means.

Why would Google offer an API? This is similar to saying when Apple sues an employee stealing IP "Nobody would steal the IP if they gave it away for free". The question is - why?

What’s the difference between scraping and malicious scraping? Does google engage in scraping or malicious scraping? Do the AI companies engage in scraping or malicious scraping?

The size of your legal team.

Malicious scraping is when people other than them do it. When they scrape the internet to train their AI, it's "lawful" because they said so.

Note that I am not defending the merits of Google's lawsuit, but they did describe in this very post what they believe distinguishes their scraping versus SerpApi.

> Stealthy scrapers like SerpApi override those directives and give sites no choice at all. SerpApi uses shady back doors — like cloaking themselves, bombarding websites with massive networks of bots and giving their crawlers fake and constantly changing names — circumventing our security measures to take websites’ content wholesale. [...] SerpApi deceptively takes content that Google licenses from others (like images that appear in Knowledge Panels, real-time data in Search features and much more), and then resells it for a fee. In doing so, it willfully disregards the rights and directives of websites and providers whose content appears in Search.

To me this seems... interesting, for sure. I think that Google already set a bad precedent by pulling content from the web directly into its results, and an even worse one by paying websites with user-generated content for said content (while those sites didn't pay the users that actually made the user-generated content, as an additional bitchslap.)

But it seems like at the very least Google is suggesting that SerpApi is effectively trying to "steal" the work Google did, rather than do the same work themselves. Though I wonder if this is really Google pulling up the ladder behind them a bit, given how privileged of a position they are in with regards to web scraping.

It's a tough case. I think that something does need to ultimately be done about "malicious" web scraping that ignores robots.txt, but traditionally that sort of thing did not violate any laws, and I feel somewhat skeptical that it will be found to violate the law today. I mean, didn't LinkedIn try this same thing?

>bombarding websites with massive networks of bots

Like GoogleBot?

And yeah, robots.txt is not enforced by any law.

I think this is just about dragging SerpApi through a lengthy legal procedure and fees.

Whether you obey robots.txt (Google does, SerpApi doesn't) seems like an important distinction.

Permission

yoink

* that's the sound of a ladder being yanked up

If google provided a useable reasonably priced API, then SerpApi wouldn't exist.

Disgusting behavior by google. Scraping is google's whole business.

And then pretending that they're fighting for other people's copyright is just the cherry on top of the pile of hypocrisy.

Google can eat a bag of dicks.

Their entire ai model was scraped.

Reply on news.ycombinator.com