Comment by inglor | Hacker News Reader

Comment by inglor | Hacker News Reader

Well, to be fair the website looks like it does the most it can possibly do to avoid good SEO and search engine visibility.

It uses `<meta keywords` and other techniques that apparently search engines like Google do not take into account.

On the other hand the HTML of the page (view-source:https://antipolygraph.org/ ) does not contain any guidance as to what's the page's focus. No header tags are present, everything is `<table>` based and contains deprecated markup and almost all of the links it contains are internal. The markup is HTML4 which might be an indication to Google the page has not been updated in a while.

The website can easily be found in Google (https://www.google.com/search?q=site%3Aantipolygraph.org&oq=...) - it just appears to not be ranked very high as it aged.

Don't forget Google tends to favour newer and updated content to older content. The website also doesn't have a sitemap as far as I know which doesn't help with the problematic markup.

Unfortunately all these changes mean that it's now tougher to find old websites with specific information that you're looking for on Google nowadays, in general. It feels like Google used to be much more useful for that ten years ago than it is now.

Google's purpose seems to have changed from helping people find the most useful information, to funelling people to information that Google wants them to find on a particular topic, whether its the most useful or not.

Are we really still pretending like Goog/FB/etc. aren't effectively mass information control mechanisms first and foremost?

What we can do in this day and age to find relevant information fast? Use another search engine? If yes which one?

As far as search engines go, DuckDuckGo is pretty neat, have been using it for 3 years now and can count with my fingers and toes how many times I've used Google since then. And half of those times it was because I typed the search term directly into another person's browser's address bar which defaulted to Google.

Second DDG. It wasn't very good a few years ago, but it's improved significantly since then and is now my daily driver.

I haven't used ddg as it used to be fairly terrible, but google may have slipped enough and ddg may have improved enough now, I'm going to give it another shot.

i've also used DDG for the last few years. it has indeed improved a lot. i don't really ever need to turn to google to get what i'm looking for these days.

> Google's purpose seems to have changed from helping people find the most useful information, to funelling people to information that Google wants them to find on a particular topic

This statement needs examples to stand.

Normally I only use Google, but I've also experienced this. In contrast with a few years ago, more recently Google gave me such bullshit that I was like, "Just give me the pages that have these keywords, this is so obvious" and I went to duckduckgo.com. There, with literally the exact same query, every one of the top links was relevant and had my search terms. On Google none of the first page of links had what I searched for. This certainly supports "they show me what they want me to see", rather than what I asked for.

On the flip side, I still find Google useful and I must say that I often am happy to read what they decide to give me: even if it's not what I searched for.

I've done searches on Google with 3-4 words and the results given back had excluded all but one word. The "not on page" results are usually pretty infuriating since that's what I am looking for

Those aren't "examples". Can you be specific?

If you haven’t run into this I submit you don’t use google much.

It’s infuriating

It is impossible for me to give you a search term that will be garunteed to show you this effect. Google tailors every search to each person too much.

It ends up feeling like you've been gas lit by Google when you report one situation and no one else can see it

Add double quotes around the words, works like the old +word syntax.

(I don't work on Search, but use it a lot :))

Quotes or pluses stopped working a while ago.

Even "Verbatim" isn't actually verbatim.

I am amused that people are only now realizing just how awful Google has got compared to 2000s.

That's simply not true. Quotes superseded pluses and they definitely still work and it's trivial to demonstrate.

"mainframe" "kubernetes" vs mainframe kubernetes

https://www.google.com/advanced_search?hl=en&fg=1

agree on ‘not on page’ then why are you showing this!? facepalm

Stratechery has a great article about how Google chooses to show their local results over more useful results from its competitors:

https://stratechery.com/2018/the-bill-gates-line/

The relevant explainer video: https://youtu.be/k9UqqmIJW4g

> Are we really still pretending like Goog/FB/etc. aren't effectively mass information control mechanisms first and foremost?

No. They are a search engine and social network first and foremost.

Well, no. They are massive databases of detailed, specific information about every individual, first and foremost.

One of Google's cofounders once said that if there is more than one search result then that is a bug. That sort of thinking inevitably leads to this sort of thing. There are always shades of grey.

I'm looking for a link, will update when I find it.

Here is an article citing a Charlie Rose interview from 2005.

https://techcrunch.com/2013/01/04/an-old-eric-schmidt-interv...

yikes, i really hope they are not so shortsighted these days.

a lot of the time people are searching for something, they do not know exactly what they are looking for enough to articulate the perfect string of search terms and need a plate of more general results to get to what they want... this seems painfully obvious.

Page and Brin are saying that they want to deliver that specific final result even when someone starts off not knowing what they’re looking for by using context and their user data to understand what the search is really about. Not that they want to deliver one general result when someone makes a general query.

I’m not necessarily agreeing; just explaining their thinking.

We have a blog on agrifood tech associated with our site and we’ve published over 1,000 articles often interviewing some of the most successful entrepreneurs, VCs, and senior execs at food and ag companies. We’re by far the most authoritative thought leader in the industry and if you go to any industry you’d hear our name a few dozen times but we get massively outranked by Forbes articles, TechCrunch, CB Insights (which is a poor authority in our space). It’s quite sad just how bad Google can still be.

What do those out-ranking publishers do differently?

higher domain name authority

But not domain expertise. Google has a rich get richer ranking that favors broad generalist publications. If you’re seen as an authority in anything then it seems like you can also rank high for something tertiary.

yea VOX ranks high for everything even though they are generalists than specialists. .

They have bigger and more general-purpose audiences and thus a higher probability that someone going to google is looking for them. Self reinforcing.

That sounds to me like Google is optimizing retrieval for deeper searches within a narrower set of "authoritative" websites. The ideal solution would involve putting in the logistical and theoretical engineering work to classify and validate a huge number of websites good for fewer, specialized answers.

But for whatever reason, the reality seems to be a relative few websites are selected to provide "good enough" answers on shallow queries for a huge number of topics or objects, Wolfram Alpha style.

Maybe this is due to developer laziness, gradual dilution of cutting edge engineering, explicit desire for simpler solutions or real practical limitations, even at Google's scale. In any case, the result appears to be that sites like yours are poorly surfaced because they're not very generalizable. Whereas CB Insights claims it knows about your query already, there's an internal API setup for exposing CB Insights already, and CB Insights passes a basic sanity check on the answers.

This is really unforunate and I sympathize with you. I think there is a lot of opportunity for a search engine that can optimize for very specialized queries across many domains, including arbitrarily historical resources. But I think such a search engine would have very different goals and incentives than Google from the outset.

I think you’re onto something. Google’s “fake news” algorithm change probably didn’t want to be seen as politically or viewpoint driven, so an easier approach was to just de-prioritise large “authoritative” websites even for deep searches where more relevant results exist.

Actually, that’s almost exactly how Google described the algorithm update...

People still seem to find us depending on the exact search terms you’re using but it’s Google’s user experience that really suffers. If someone randomly publishes “the top 25 Agtech companies” on Forbes it’ll jump to the top of the list when searching for Agtech and stay there for half a year. Google is embarrassing itself with these kind of shallow and superficial results.

If your industry doesn't have a large web community, then it's difficult for Google to know that you're authoritative. From Google's outsider perspective, all the links are going to Forbes.

You are thinking in terms of the old world, the PageRank algorithm.

Everything you say is true in that old world.

In today's world, Google is optimizing for ad revenue.

I could see how optimizing for PageRank could go out the window in such a world.

Even if PageRank still has influence, it's not clear to me the site doesn't have a large web community with a lot of links. Maybe it does, and the ad revenue considerations are overriding that.

doesn't google "rank" journalistic sources by perceived credibility? Could be that nobody at google has bothered to give your website a rank?

No, Google ranks journalistic sources by revenue and distribution, and pretends that it is ranking by credibility.

What is an "authoritative thought leader"?

Semantic markup : Internet conspiracy theory :: fluoridated water : real world conspiracy theory.

Here's your monthly reminder that "relevance" is fundamentally subjective, and therefore all search and discovery services, both on the internet and offline, give you their opinion of what you'll find useful.

How useful you actually find a given service to be is entirely a function of how closely you agree with that site operator's opinion.

I definitely think that's an incredibly legitimate concern and wouldn't want to dismiss it because it's such an important issue. It is also important to remember, though, that a lot of these changes are made in ways that (seemingly, at least) attempt to prevent sites from gaming the search algorithms. I have seen the quality of results I'd expect from a Google search go down, and I attribute that to quite a bit of factors, including Google's self interest. I also try and consider just the massive rise in general content available through the web, the further and further growth of it as a tool for manipulative information or propaganda, as well as its growth as a marketplace. I only mention these things because I think a lot of people, including myself, will tend to ignore a lot of the external factors that forced as well as allowed Google's search and results algorithms to be in the position they are in.

It's much more adversarial than it was at the beginning.

This may be a good thread to let people know about a hidden setting I found out about recently called Google Verbatim. I thought certain sites I used to search for had fallen off the Internet, but the change was how Google ranked its search results.

Try the two searches below to see the difference:

https://www.google.com/search?q=java+vm+language

https://www.google.com/search?q=java+vm+language&tbs=li:1

Enjoy!

How do we use it? Just append &tbs=li:1 ?

In any search result page, click Tools > All results > Verbatim. That is also how you restrict e.g. the temporal range of results (Tools > Any Time > ...)

Yes! It is time for someone to disrupt Google. Please, we need a better search.

We really need good, alternative search. I miss Lycos, Hotbot, Yahoo and the days where if you didn't find someone in one place, you checked another. Searching for things took real work and you could find a lot of different rabbit holes.

Today, rarely do we go past the 2nd page of results. About 1/3 of the time, I get frustrated with Duck Duck Go and add !g.

All of what you said the website doesn't do is exactly what a decent website shouldn't need to. Google should index it according to the user-visible content, not hidden metadata.

Metadata defines what is the content, the sections and the titles. A blob of text is worthless without context.

HTML pages (written properly) don't need metadata for that. It's part of the document.

Metadata is good for things like thumbnails, authorship, or summaries, none of which are necessarily distinguishable in valid HTML.

You're conflating metadata with the HTML meta tag. Please consider that everything is metadata except the bare text.

The point of the message above us was that the content on this site is not formatted properly as HTML.

Tags aren't just meta. For example, a sentence has a different meaning when it's alone vs. part of a list vs. a caption of an image. All of those things have their own tags.

We are saying the same thing.

Google does crawl "the way you see it" and not via raw html parsing. They interpret JavaScript and CSS.

Try putting display:none on a div with some text in it, or putting the font-size to 1px. It will not rank for that and you will probably trigger a penalty of some sort if it's egregious.

Invisible or near invisible text can be discerned without rendering the page.

Google does execute JavaScript but as best people can tell it doesn't happen all the time, just periodically. It would be computationally expensive to do this for every request on every spider view for every web site.

Less than encoding millions of hours of video per day? Libchromium is quite fast and Google has lots of resources for this.

They had rendered screenshots of every result with the matching section highlighted... almost 5 years ago.

I'm not saying they don't do it - everyone knows they do. I'm saying for many, many cases there's no reason for them to do it for every page and every site.

Then again it's the third result on DuckDuckGo when you search for 'polygraph' so it's not like it's that hard for search engines to find it.

None of these are particularly red flag SEO issues. The page is clean, mobile friendly, served over HTTPS. Google doesn't need to assume content update frequency based on the style of markup used, it literally spiders sites and reads their content, likely multiple times a day.

Nonsense, seo doesn't matter (as per Google) when people come or seek your site. The more likely explanation is that Google and its 3/4 quarter-of-a-trillion valuation depends of having good relations with the Feds, anti-trust and all. See At&T how it manages to screw everyone when it comes to protecting their own interests. They do what the Feds and NSA asks them to do.

Your 20 year old site doesn't age in months to go from #1 to page #3 (dead!). The fact that the site has gotten press is even better, it should help your rankings. But by now Google is not longer a fair engine. Their rankings are manipulated to boost their bottom line, directly or indirectly.

Nobody has any illusions about Google being fair but if the first thing people do to "solve" this is to bring out all the hairy conspiracy theories instead of fixing the stupid site, no wonder the ranking still sucks. facepalm

Only once that is fixed and the rankings are still bad is complaining about "feds" and what not maybe warranted.

">> instead of fixing the stupid site" Fix what? If Google wants to demote you, doesn't matter what you do. See https://www.mcclatchydc.com/news/special-reports/insider-thr...

What happened that it went from #2 to page 3? Nothing. Page 3 is a black hole, used to be called a page 3 penalty (get zero traffic, might as well be banned.) The site is accessible and readable. A lot of sites are old fashioned...see Drudge.

This tendency to characterize the recognition of basic aligning interests as "conspiracy theory" is really odd.

I personally don't think GP's theory passes Occam's Razor, but that does not refute the underlying contours of the situation.

Reply on news.ycombinator.com