Hacker News Reader

Comment by jrmg | Hacker News Reader

I think that part of this is that small web sites simply are not being indexed by Google at all any more.

My ~15 year old blog has, according to Google Search Console, 15 indexed pages, and 174 ‘Discovered - currently not indexed’ pages. The number of indexed pages is going _down_ over time, despite occasional new posts.

The Search Console page says “Examine the issues […] to decide whether you need to fix these URLs.” But about the only suggestion I can find is just to wait for them to be indexed - which doesn’t seem to ever happen.

I wouldn’t argue my blog is the best or most exciting content in the world or anything, but I can’t believe anyone would say it’s worse than the often-incorrect SEO-informed duplicative nonsense that fills the first few pages of Google search results for anything technical nowadays.

View on HN

It's not just you - this is widespread.

I run a SaaS to help site owners get their content indexed. We're seeing an influx of users, I think a lot of the issue is simply because of AI.

New web page additions were pretty linear over time, and then AI copywriting tools came out. Suddenly page additions basically went "hockey stick"/vertical.

Now, you can publish thousands of pages in a few minutes, and it's created a huge backlog in Googles crawl queue, thus increasing overall time to get indexation, disproportionally affecting smaller sites.

> I run a SaaS to help site owners get their content indexed. We're seeing an influx of users, I think a lot of the issue is simply because of AI.

I think that google just isn't interested in putting resources into their search engine anymore. They used to need it to gather data on people and what they were doing online, but chrome gives them people's internet histories now and android lets them collect endless amounts of data on people's lives offline. Google doesn't need search to spy on us anymore. It's only natural that they'd let it stagnate.

Sites are actively being deindexed. My entire website, probably 20 years old now, used to be indexed. Now only a few pages are.

Same here - most of my personal site has been stuck in Google's "Discovered - currently not indexed" limbo for nearly 2 years, despite regularly submitting pages for indexing, and doing all the usual things like optimising Lighthouse score and so on. I went on the Google Search support forums and it was quite sad - just vast numbers of people crying into the void. It is almost like Google have given up on search.

If they actually wanted to improve it, the key would be to move away from the advertising funded model - as Larry Page and Sergey Brin warned in 1998 "advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers"[0]. Personally I think they could at least pay running costs with a paid support model - it might not earn as much money as advertising, but as a gateway to the internet and other Google services it could still be very valuable to both Google and their users.

[0] "The Anatomy of a Large-Scale Hypertextual Web Search Engine", Computer Networks, vol. 30 (1998), pp. 107-117, (noting that the quote is in Appendix A which seems to be missing from some more recent online versions).

40 years ago news organizations (news papers, radio, and TV) had strict policies that the ad department (which was always in house - never outsourced to doubleclick/google) was not allowed to talk to the news department. You bought an ad because you wanted to reach people who wanted news from orginizations would "bite the hand that fed them". Internet ads don't seem to have that. They could, the culture existed before in other forms and wouldn't be hard.

I am somewhat in the same boat - my site isn't as old as yours (eight years old maybe) and I do put new content on it semi-regularly, but occasionally old pages mysteriously become unindexed, and a handful of pages seem to be stuck in the discovered but not indexed bucket too. I periodically try and update them or expand on them a bit but usually it doesn't help get them indexed. And like you, my site isn't the most exciting content but it does get regular traffic.

It's even worse on brand new website, google will index your homepage and ignore everything else unless it has some external links. Even with a good sitemap and good usability score.

This is a well known phenomenon of "Google Jail" where to combat spammers just setting up hordes of new sites, getting dinged and then migrating to a new domain, Google penalizes new sites until they've been "aged" for some number of months.

Dunno if it's about age so much as it is context. Like my search engine does something similar. The actual text on the pages are only a fraction of the signals used by search engines to put a website to put the putative search result into context.

It's really hard to rank a website if there are no links or traffic to it.

you don't serve Google ads like the SEO crap. hopefully the antitrust action breaks up that particular conflict of interest.

nothing is indexed by gogle anymore. it doesnt work at all as its supposed to for a long time. forget about it.

Reply on news.ycombinator.com