Comment by sharkweek | Hacker News Reader

Comment by sharkweek | Hacker News Reader

Many moons ago, I put a ton of work into a (now defunct) content site that provided specific information in a small vertical. Each post took hours, but I enjoyed writing the articles, and the revenue from the ads was enough to enjoy a little extra cash every month (we're talking just about enough to cover a car payment, for context).

Google decided they liked a lot of the content and started providing it in their knowledge graph. It was initially great to be validated and the link to my content at the top of the search result was pretty cool to see. But it did tank traffic, as Google scraping my content and giving it away at the top of their search result page meant people didn't need to navigate to my site.

Felt pretty terrible, if I'm being honest, but I also have always subscribed to understanding the risks for relying on other networks for my own benefit.

BUT!!!

As a regular user of search engines, I love getting the answer super fast without necessarily having to guess what sort of ad-trap I'll have to navigate to get the answer I'm looking for on some random content site.

As a user, when I'm asking some non-critical question, it's nice to just get a snappy answer. I appreciate it.

The problem of course arises when content creators stop making the content for Google to scrape and index, what then?

In my experience, "Google Instant Answers" are more often than not factually incorrect. It makes me queasy every time I see that... it speaks volumes that's so much more important to them that they keep you on the page that they'd rather give you wrong information than let you go.

A classic example is the instant answer to "who invented running?"

What I find entertaining is that while answers may be wrong, you can see from the question statement and the returned sample text that it is working perfectly in a mechanical sense. The parsing of the question is correct, the searching works, and so does the ranking. But at the end of the day, it is just applied NLP at a large scale, not AGI, so it doesn't know that it is returning a wrong answer. It doesn't know anything at all!

For those that are curious it displays a Quora answer that states:

>Thomas Running

>Running was invented in 1612 by Thomas Running when he tried to walk twice at the same time.

It's actually quite and interesting post if you follow the link. Obviously it's not correct in the purest form but there is justification for this answer.

On the other hand "who discovered running?" quotes a tweet telling a variant of the same joke.

I wonder if GPT-3 has learned about Mr Running yet?

At what point do the layers of abstraction and sarcasm become indistinguishable from truth?

I don’t know, but I feel like our societal perfect storm + AI + time will get us there.

That wrong info came from the pages which are listed in the search results. So much objectively, factually incorrect info is written in to blog posts which show up in google search results which then get copied to instant answers. Particularly historical info is filled with mistakes, people misunderstanding the context of some old news article and then taking it out of context to drop on wikipedia where it is taken as fact.

The error usually arises from Google misinterpreting and decontextualizing the page’s content, for example reading a table incorrectly then giving one stat as another stat. The page is correct, it’s Google that’s wrong.

> The problem of course arises when content creators stop making the content for Google to scrape and index, what then?

This has already happened. People put content on their sites only long enough to figure out what works, then turn that content into a book that can be sold. Every Cal Newport book came out of his blog posts.

Others are turning content into paid newsletters, courses and Patreon-only walled content.

Wrote something 15 years ago? Just set the "Date Modified" to last week and Google will think it's been updated.

The 'content' you and I are seeing are just second and third-hand summarizations of popular topics written by content marketers.

>Wrote something 15 years ago? Just set the "Date Modified" to last week and Google will think it's been updated.

...which is awful! Try searching for historic information on a viral topic and the results are littered with incorrectly dated listings that may satisfy the search criteria but are from the wrong time span.

“Show me results from the last 24 hours”

“Ok: here are results dated 4 years ago, but I gotchu. The ‘other articles like this’ changed last time I crawled it!”

It’s hard to see how this isn’t anticompetitive. Using their market power in search to show their own ads but prevent anyone navigating to view any others seems like something that should end up in court.

Google has always respected robots.txt and has well documented how to restrict crawling.

I don’t see it as anticompetitive so much as copyright violations. The fact that Google makes advert revenue using this strategy just contributes to damages.

Anticompetitive would likely require Alphabet (and potentially other companies) from preventing users from accessing the original web page (which might be possible but more difficult to prove).

IIRC you can't copyright facts

You can absolutely copyright the packaging of facts (see any textbook ever printed).

Google often has entire paragraphs ripped from the underlying page in their knowledge graph. Would it be copyright infringement if Google displayed paragraphs of text from copyrighted textbook? If so, is there a difference between doing that and doing the same with, say, "facts" from a blog post?

IANAL, but my understanding is that an individual fact can’t be copyrighted, but an organized collection absolutely can be.

The example I remember is that A baseball game score can’t be copyrighted, but if you run a website that lists all of the baseball games throughout history and you gathered that data and did work to organize it (other than to scrape it from another collection), you have a claim to a copyright.

Now I’m curious: what would happen if someone had no idea that Google existed (insert wild reason here. I’m leaning towards lived in the woods for 40 years and self-hosts content without reading anything, like a hermited author)?

Does Google still get to say “everyone knows we scrape your content” and “We have instructions about opting out”?

My take is that if you serve web content to unauthenticated clients, you must expect humans and robot users will look at, remember, and catalogue your content.

To the extent that you want to control your content, you need to take measures to do so. Reject unauthenticated, assert copyright/trademark controls, or do some basic searching to implement open standards which most/all legal agents adhere to.

The thought exercise is moot if you consider how many laws people are governed by in the modern world that they don’t know about but are still expected to learn and abide by. Also worth thinking about what recourse you expect this naive website owner to be able to effect given they have done the absolute minimum to protect their content.

I feel like it might be dangerous to equate Google with societal laws, but you have some compelling arguments.

If I may counter to flesh this out:

Take books. Authors have copyright laws to protect the content but they also have to accept that humans will look at, remember, and catalog the content of their books.

And we ended up with libraries which catalog on a large scale, as well as laws that dictate how people can use the content of those books (especially when they do borrow them).

But Google could be considered as a company that borrows all the books, has people transcribe and file them, returns the books, and then advertises that they are “better than the library” because they have hired the best and fastest librarians in the world. However, they charge a cover fee. (In reality this fee is paid by advertisers, but it’s still there and Google is still a for-profit company)

I think that it would be hard to argue that that type of business is reasonable or expected, and to bring it back to the example of a content creator: I think many of us would push back if a reclusive author brought a manuscript back to the city after 20 years of being a hermit, had it turned in to a book, and then found out that every single person who read it did so without paying him, without him being able to connect with those readers, and without him even being able to know how many people because this for-profit library chose not to tell him.

Did they scrape your content and put it into their own database? That is, if you had deleted all of your content, would it still have appeared in the Google results?

I'm pretty sure Google is rather quick to delete dead listings, but their is a twilight period where pages persist in results. This is especially annoying with forums hosting dead and un-archived links.

I’m pretty sure Google actually deletes nothing.

The original goal was to have all data digitized in 200 years.

Of course, they remove things from results and from public view, but delete? I suspect no.

Will content creators stop, or will a new payment model that isn't "pay for eyeballs" arise?

The financial sources for information-collection-and-creation undertakings throughout human history have been manifold (wealthy patron and collective financing, to name two). It is entirely possible, if Google finds its data sources drying up, that Google itself will pay for data collection (they already do this with several categories of data they host).

Knowing the way they are, you might want to consider applying for a job and telling that story during the interviews.

"The problem of course arises when content creators stop making the content for Google to scrape and index disappear, what then?"

communist revolution?

Sounds like the plot of Atlas Shrugged.

Reply on news.ycombinator.com