How Google uses blacklists, algorithm tweaks and contractors for search results
msn.com
msn.com
Blacklisted domains are blacklisted forever, because they crash the crawler. This happens 1x - 3x times per year. Changes here are all tracked and only owners of the first stage of search (who have no connection to the ranking algorithm) have change rights.
Bullet #5: search quality is assessed by thousands of contractors worldwide and you can become a part of the crawl quality team, although it doesn't pay that well in the USA. The book they follow is 180 pages and available on the Google website for many years. It has guidelines for how to determine sexual, offensive, or illegal (or child porn) content in ALL countries. It has guidelines on how to rank news source reputation and credibility.
>These moves are separate from those that block sites as required by U.S. or foreign law, such as those featuring child abuse or with copyright infringement, and from changes designed to demote spam sites, which attempt to game the system to appear higher in results.
The article directly refutes what you are claiming here.
The article has very few specific examples, a few mentions of some draft and "suspicious" discrepancies between bing and google.
None of which really mean much, except that it sounds like the typical liberal bias meme.
Funny factoid for you: Google hires those guys in Russia through recruiters and innocuously sounding shell companies in Saint Petersburg.
The guy can work for the shell for years without realising that he works for Google.
> The guy can work for the shell for years without realising that he works for Google.
Your comment makes this practice sound shady. I don't see how this practice is even probably shady, if the general thrust of you say is true.
First, you have to set up a different company for each country where you're doing business. Second, it's not a shell company if it hires employees or contractors and has clients. Finally, if the subsidiary doesn't engage in your primary business activity you don't call the subsidiary by a name similar to the primary company's name.
It's a very common corporation structure.
I asked "Google?" It raised their eyebrows, but they said they can't answer that.
>More than 100 interviews and the Journal’s own testing of Google’s search results reveal:
• Google made algorithmic changes to its search results that favor big businesses over smaller ones, and in at least one case made changes on behalf of a major advertiser, eBay Inc., contrary to its public position that it never takes that type of action. The company also boosts some major websites, such as Amazon.com Inc. and Facebook Inc., according to people familiar with the matter.
• Google engineers regularly make behind-the-scenes adjustments to other information the company is increasingly layering on top of its basic search results. These features include auto-complete suggestions, boxes called “knowledge panels” and “featured snippets,” and news results, which aren’t subject to the same company policies limiting what engineers can remove or change.
• Despite publicly denying doing so, Google keeps blacklists to remove certain sites or prevent others from surfacing in certain types of results. These moves are separate from those that block sites as required by U.S. or foreign law, such as those featuring child abuse or with copyright infringement, and from changes designed to demote spam sites, which attempt to game the system to appear higher in results.
• In auto-complete, the feature that predicts search terms as the user types a query, Google’s engineers have created algorithms and blacklists to weed out more-incendiary suggestions for controversial subjects, such as abortion or immigration, in effect filtering out inflammatory results on high-profile topics.
• Google employees and executives, including co-founders Larry Page and Sergey Brin, have disagreed on how much to intervene on search results and to what extent. Employees can push for revisions in specific search results, including on topics such as vaccinations and autism.
• To evaluate its search results, Google employs thousands of low-paid contractors whose purpose the company says is to assess the quality of the algorithms’ rankings. Even so, contractors said Google gave feedback to these workers to convey what it considered to be the correct ranking of results, and they revised their assessments accordingly, according to contractors interviewed by the Journal. The contractors’ collective evaluations are then used to adjust algorithms
>Mr. Brin still opposed making large-scale efforts to fight spam, because it involved more human intervention. Mr. Brin, whose parents were Jewish émigrés from the former Soviet Union, even personally decided to allow anti-Semitic sites that were in the results for the query “Jew,” according to people familiar with the decision. Google posted a disclaimer with results for that query saying, “Our search results are generated completely objectively and are independent of the beliefs and preferences of those who work at Google.”
>Finally, in 2004, in the bathroom one day at Google’s headquarters in Mountain View, Calif., Mr. Page approached Ben Gomes, one of Google’s early search executives, to express support for his efforts fighting spam. “Just do what you need to do,” said Mr. Page, according to a person familiar with the conversation. “Sergey is going to ruin this f—ing company.”
I guess money caused it. When you have a market-cap of $900+ billion you’re not going to keep up with promises made back when you were “only” a $90 billion company or a $9 billion company. I remember the pre/IPO days when each index update was followed almost religiously on webmasterworld.com and where an user called Googleguy (supposedly Matt Cutts) was implicitly promising all the website owners in there that Google was our friend, that they were never going to become a portal and that they would never steal anyone’s content.
Microsoft figured out that this was going to be a problem first -- you may recall that Bing was marketed as providing answers instead of results, which was a nascent threat. Google moved towards providing answers, and found that embedding ads / product placement in those answers was profitable.
Also remember that the users of Google in 2002 or whenever were a different audience. The "win-win" for you is confusion to the average punter.
I don't want to come off as a defender of Google -- but search is an incredibly complex business that means something different to many cohorts of users. I think the surveillance stuff is approaching or crossing a line nowadays, but I think it's fair to say they have done a pretty good job considering how the online and offline world has evolved.
I don't think so, and that s thanks to people who keep writing blogs, tutorials, make thoughtful videos, posting thoughtful comments etc. The "Old web" had a high technical barrier to entry and that acted as a selection filter. The "new web" is the former TV audience, which moved to facebook/instagram and mobile apps. It's big audience, and marketers are after it, but that didnt make the "old web" disappear. I ve never read anything interesting inside facebook except from Yann Lecun's posts (which he keeps there out of courtesy to his employer).
I just hope people stop thinking the old web should also move inside those because that's where their crowd is. It's not true, that's a different crowd.
Google has a permanent demotion applied to a site I've run since 1996: http://onlineslangdictionary.com/ . I estimate that my traffic would be 2.5x - 3x what it is now, were the demotion not in place.
These demotions are hidden, permanent, and cannot be appealed. Moreover, these demotions can be performed by hand internally within Google - for whatever reason they choose, or for no reason at all. That is to say, some demotions are manual and not automated.
I have never been officially notified that the demotion exists, in any of Google's available tools or any other way. However, a Google employee checked the internal status of my website and there is, indeed, a permanent demotion in place.
There is no reason for my site to be demoted. This demotion was put in place while Matt Cutts was the head of the web spam team. I asked him about it here on HN, and he lied about it. (I know he lied because of my communication with the Google employee.) You can read my thread with Matt here: https://news.ycombinator.com/item?id=5408087 .
I'd like to make the email chain between the Google employee and I public. But I don't want to ruin someone's career / life, just because they did the right thing and told me about the penalty.
So... I don't know what to do. Thoughts?
I don't expect special treatment. All I want is for the demotion to be removed so that I can compete with the rest of the web on a level playing field.
This sort of thing is why we have courts.
I wish I had the money to pursue that.
A Google demotion looks different. What do you expect? Ranking on the first page for every slang word?
More details follow. But whether my metaphor is apt or not doesn't eliminate the fact that a Google employee informed me about the demotion.
The following is a FAQ taken from the page on my website about the demotion. It's written for a general audience.
Q: I just did a Google search and your site appeared in the first few results. Does that mean that the penalty has been removed?
A: No.
"Google Juice" is an informal term for how favorably Google views a website and pages on that website. There are a lot of factors that go into how much juice websites and pages earn.
Every time you do a search, Google's algorithms evaluate every page on the web to decide how relevant each page is to your query - how much Google Juice each of the pages has for your query. Then they show the search results, which are ordered by the amount of Google Juice each page has.
What the penalty does is subtract an amount of Google Juice from every page on this site. But whether that means we appear 1st, 2nd, or 307th in the search results depends on how much juice each of the other pages on the web have for your query.
You could think of it as a foot race. The penalty doesn't work as in: however you finish in the race, Google will drop you down by 9 places. It works as in: Google attaches a 20 pound weight to your foot, and whether that means you finish 1st or 307th depends on how good the other runners are.
So sometimes pages from this site appear towards the top of search results. That just means that those specific pages have enough Google Juice - and other pages on the web have so little Google Juice - that even with the penalty, we can appear towards the top. But overall, the penalty drags our pages down far enough that we would get about three times as many visitors if the penalty weren't in place.
That said, I agree that penalties/demotions seem to have a somewhat dampening effect, e.g. resulting in the previously unmodified rank but times 0.2 or whatever.
However, penalties should not last 13 years if the causes have been fixed. What did said Google employee tell you about your case?
edit: also, there is a way to submit a reconsideration request. Have you tried that?
True. I wanted the text to be immediately understandable by non-technical people. I'm on like revision 3,194 of the text. :) Any suggestions for improvement are welcome.
Also, there is a way to submit a reinclusion request. Have you tried that?
Yes, several times.
However, penalties should not last 13 years if the causes have been fixed.
They shouldn't. But in this case, it has lasted that long. It's a manual penalty turned on by a Google employee. The only way to get rid of the penalty is for them to (manually) turn it off.
What did said Google employee tell you about your case?
That there's nothing I can do.
If you're a Google employee, I'd love to discuss this with you! My email address is waltergr@gmail.com .
I'm a little unclear on that. Matt specifically mentioned two active penalties.
Matt said that there was an automated penalty due to advertising on my site. Following this, I removed all advertising from the site. My site's ranking did not change, nor did traffic referred by Google searches. Why is that? If there's nothing I need to do for the site, why have those metrics not changed?
Matt also said that there was an automated Panda penalty against my site. Following this, I removed all citations from the site. My site's ranking did not change, nor did traffic referred by Google searches. Same questions as above: Why is that? If there's nothing I need to do for the site, why have those metrics not changed?
What did the prior manual penalty against my site that Matt mentioned have to do with Web Build Pages / Jim Boykin? I had never even heard of them. What specifically was that manual penalty? I have seen no evidence in ranking and in traffic referred by Google searches to ever indicate that this penalty existed. Furthermore, I was never informed by Google via any mechanism that there was a manual penalty against my site. Why is that?
Have Google employees ever been able to apply demotions, penalties, or any mechanism whatsoever to drop a website's positions in Google SERPs, in a way that the website owner is never made aware of it?
Has that ever been done to my site, The Online Slang Dictionary?
Thanks very much for your input.
> Have Google employees ever been able to apply demotions, penalties, or any mechanism whatsoever to drop a website's positions in Google SERPs, in a way that the website owner is never made aware of it?
should have read, in part,
> Have Google employees ever been able to manually apply...
In it he admits that the site is being penalized for having prominent ads above the fold.
90% or more of Google’s revenue comes from presenting prominent ads above the fold.
How is demoting the organic results of sites that use the same business model that it does not the epitome of anti-competitive behavior?
Yep. And so I removed all advertising from the site for months.
There was absolutely no change in ranking.
Matt Cutts also wrote, "Your site is also affected by our Panda algorithm..."
My website had citations of slang use. By definition citations are 'duplicate content' since they also exist somewhere else.
So I removed all citations from the site for months.
There was absolutely no change in ranking.
The 3 claims Matt Cutts made were:
1. "...the only manual webspam action I see regarding onlineslangdictionary.com is from several years ago (are you familiar with a company called Web Build Pages or someone named Jim Boykin?)..." Wrong. I've never heard of him or his company, I've never worked with anyone involved with SEO, and there was never a change in ranking / site traffic that would suggest that my site had a penalty. Given the very very dim view of SEO practitioners among technically savvy people, I can only assume that his implication ("are you familiar with") was designed to discredit me here.
2. "You're affected by a couple algorithms in our general web ranking. The first is our page layout algorithm... your site has much more prominent ads above the fold compared to Urban Dictionary." Wrong. (As above, I removed all advertising from the site.)
3. "Your site is also affected by our Panda algorithm." Wrong. (As above, I removed all citations from the site.)
I can't say that anything Matt Cutts told me was truthful.
However, Google search is acting on behalf of the users, trying to find them the result that brings them the most value. And everything else equal, that is the one without ads above the fold.
Google absolutely does this. Around 6-9 months ago bluelight.org (a somewhat controversial drug harm reduction forum) disappeared from search results overnight.
It will show results if you explicitly use `site:bluelight.org`, otherwise nothing
* I feel the need to add a disclaimer here: I don't abuse drugs, but find blue light an interesting source of information regarding side effects and bioavailability of various prescription medications
The whole goddamn point of a search engine is to privledge certain results over another. The claims of armies of contractors reek of the zombie lies of their persecution complex.
http://ilpubs.stanford.edu:8090/361/
> Since it is very difficult even for experts to evaluate search engines, search engine bias is particularly insidious. A good example was OpenText, which was reported to be selling companies the right to be listed at the top of the search results for particular queries [Marchiori 97]. This type of bias is much more insidious than advertising, because it is not clear who "deserves" to be there, and who is willing to pay money to be listed. This business model resulted in an uproar, and OpenText has ceased to be a viable search engine. But less blatant bias are likely to be tolerated by the market. For example, a search engine could add a small factor to search results from "friendly" companies, and subtract a factor from results from competitors. This type of bias is very difficult to detect but could still have a significant effect on the market.
Google does both nowadays. Shame
Why is this distinction important? From the article itself:
> THE JOURNAL’S FINDINGS undercut one of Google’s core defenses against global regulators worried about how it wields its immense power—that the company doesn’t exert editorial control over what it shows users.
Surely there is some foss project that has been promising, somewhere?
The internet badly needs a big alternative search engine that isn’t beholden to advertisers or dependent on a single corporate owner.
The benefit of such a search engine (whose main incentive is to just be a good search engine) is obvious for the public, but would also give companies who rely on their own OS leverage against Google.
Paid by user search? Discourage curiosity or just using the alternative.
Deep pocketed sponsor? They have the control now.
User donations are the biggest "maybe" I can see which would be not worse but depends upon charity and campaigning to some degree.
If companies like Apple and Microsoft care about providing a great user experience, Google search is risky. I think users would prefer not to see ads when they search, or worry about Google harvesting their data. If this is so, it might be worth it to fund some sort of independent “search foundation.”
I reckon a simple text-only search engine – like Google before it jumped the shark – would actually be quite cheap to develop and operate.
Besides the question of the funding of this hypothetical search engine. It being „fair and objective“ or even completely transparent about its ranking, would mean it‘d be SEO‘d into oblivion by everything from click/content farms to trolls to more nefarious actors. As long as many on the internet want to make money or manipulate people somehow it‘s not really doable in my opinion.
- Ads do not adversely affect customer satisfaction.
- The combined forces of Microsoft, Apple and others could not create a serviceable search engine (despite Microsoft alone having already made one).
If not, let’s just agree to disagree :)
I‘m sure they can. And, as you say, MS does already. supported with ads as well unless I‘m mistaken. Are you suggesting they offer a search engine and subsidize it. That they offer it as part of their ecosystem benefits, sort of?
I can’t disagree that users tolerate ads, since advertising is the model of plenty successful websites. It’s just that, like bundled OS crapware, the user experience is better without it.
The last concept I hacked together was a custom search plugin for Grav and a command line util to use for querying.
It goes like this.
Use the command line util to search a term. The command line util run that term against the search engine _inside_ of the websites CMS itself. You essentially have a list of sites related to a topic that you chose to execute the query against.
I got this working against some sites and the proof is there. But it’s obviously highly inefficient and I haven’t figured that out yet. :-/
>The practice of creating blacklists for certain types of sites or searches has fueled cries of political bias from some Google engineers and right-wing publications that said they have viewed portions of the blacklists. Some of the websites Google appears to have targeted in Google News were conservative sites and blogs, according to documents reviewed by the Journal. In one partial blacklist reviewed by the Journal, some conservative and right-wing websites, including The Gateway Pundit and The United West, were included on a list of hundreds of websites that wouldn’t appear in news or featured products, although they could appear in organic search results.
Gateway is trash and I'm not sure what United West is but they can't say they aren't blacklisting political sites at this point. Pretty big challenge for them ahead of the bipartisan AG investigations and 2020 elections.
Blacklists in the search results themselves would be problematic.
It's as much "news" as Gateway Pundit is, which is to say "not at all".
Your jews example is a positive use case but not all conspiracy theories are created equal. Before Epstein killed himself, google's autocomplete steered you away from any negative searches involving Prince Andrew, despite the fact that it was public knowledge at that point that one of Epstein's victims had named the prince as an abuser
Should they suppress queries about area51, Scientology, The_Donald, The_Müller, r/politics, Trump is a Russian stooge, etc?
But others turn out to be true. How do you deal with that?
“Check out what they won’t let you see or talk about!” is one of their main marketing draws.
And if Google was just a cold representation of human behavior (not bots though) then it might help people develop their own editorial voice.
I don’t know. It’s their call at the end of the day.
A better option would be to remove search suggestions and only match proper nouns at most.
The user isn't going to supply the very detailed information necessary to objectively filter down (and rank) everything on the web to a set of relevant results, without the search engine making any judgment calls. That would be a ton of work, more than almost any user would want to do. (And many users wouldn't have the technical skills to do it.) It would be like going into a restaurant and handing the chef a recipe for everything in your meal, down to the level of detail saying the vanilla extract in your dessert should use this type of vanilla bean, infuse it in this type of alcohol, and for this long.
The corollary is that any useful search engine will have to guide you a bit. It will take the incomplete specification you gave about what you want, and it will fill in reasonable guesses about everything you didn't say about what you want (but that it needs to know to find it), and then it will give you an answer that's useful.
Obviously, it's ideal to make this guidance objective or unbiased, but how do you even ensure that you're doing that? The whole point here is that you are making guesses about what the user prefers. It's not useful to guess randomly. Objectivity is a good goal, and you should avoid any unnecessary subjectivity, but the idea that you're going to be totally objective seems like a fantasy.
i think the wording of this is insufficiently nuanced. consider that many issues involve more than two clear dominant points of view, and that for many issues most people would not consider all points of view to be equally credible.
> Why is Google deciding why that specific Hillary query is taboo?
because, as another poster pointed out in another subthread, any search engine that is usable (at the level of time and technical skill that most people have) will necessarily have to make essentially editorial judgements. after almost a lifetime of being the sort of nerd that likes to make lists, categorize things, geek out over philosophical classifications, etc, and after a few years of working in the library world, i'm convinced that coming up with any system of abstraction or classification necessarily implies making editorial judgements and value judgements. i think objectivity is a great and important thing to strive for (in reporting and in information classification), but i think achieving it perfectly is definitely not possible, especially on divisive issues, and especially where lots of people disagree (or claim to disagree) on the basic facts.
> And who gave them that right?
i don't know about right, but effectively they have the ability because: 1) they built a really good search engine, 2) they built a really successful ad business on top of that to monetize it, 3) through ignorance and laziness we let them hoover up our data and use it to greatly improve their ad business, which let them provide us even more free services that everyone got hooked on, 4) everyone seems too apathetic to make the effort to move away and no one seems interested in competing with them as a search or email provider for most people. and here we are.
> hillary clinton emails
to
> is hillary clinton still controlled by the jews
Nice try.
Its autocompletes:
is hillary clinton running for president
is hillary clinton running in 2020
is hillary clinton going to jail
is hillary clinton an alcoholic
is hillary clinton democrat or republican
is hillary clinton sick
is hillary clinton under investigation
FYI, Google's "is donald trump" includes "is donald trump the antichrist" so they're certainly not consistently selective.
Second, I think the big chunk of the problem here is lack of transparency. Google has traditionally been very secretive about its algorithms to avoid tipping off spammers. So if you ask them directly they will hem and haw when in fact they ban spammers and also, as the article reports, they moderate inflammatory content and manually boost rankings of specific websites. The question is - what is the exact scope of these activities? Where is the red line that they will not cross? I think the public deserves to know.
Of course the exact nature of changes and boosts remains unknown but that's just underlines the need for transparency.
Would it be a "flaw of Google" in your opinion if Google blacklisted such an autocomplete if that is actually what a lot of (real non-troll) people were actually typing in and interested in finding results for?
I've watched the changes since there was a 'googleguy' and the 'big update was charted with moon phases'.
Since about the time Page started pushing things around it's gone downhill slowly with more and more censoring and less and less transparency.
Google is still benefiting from public trust that was earned years ago, when it was truly doing lots of good around the world.
The lack of transparency about the censoring is a terrible thing for the knowledge of the planet - an entire generation of people are learning truth about life (and the afterlife) trusting google, and being filled with info from youtube. Even the censoring spam filters in gmail are affecting people's lives in the real world today.
I mentioned some of the issues with users not getting transparent info about their searches being censored in a comment here recently: https://news.ycombinator.com/item?id=21487318 (and how I think more web sites need to put notices about the increased censorship of big G)
It's fraud for the all webmasters as well.
"Make a good site, we'll find it and display it. Don't do any SEO, that's evil - just make good content" - well many of us have spent hundreds of hours making good content and watching others who have less content rank higher.
Is there a blacklist from the time period Matt Cutts was on the spam team and around the time he left?
I heard a rumor that if your site was on one of these 'we caught you' blacklists with a certain googlers name on it - you can't get out of the de-rank jail unless that specific person lets you out.. a shadowban, and public notice to do more work to fix it, and whole while knowing nothing will fix the ranking for that site.
maybe not regardless - but telling webmasters to make disavow lists and spent that stupid amount of time putting them together - and still not putting their sites back into the top 10 (knowing you've crafted the blacklists / shadowbans and tweaked the algo to push them back as well) - that's fraudulent isn't it.
You've made people spend tons of time trying to fix things for google - knowing they were wasting their time and losing money.
I'm guessing the goal was to destroy lives and knock the spirit out of those people who would 'game the google system' - those evil SEO people should be destroyed.
I'm sure there's a valid excuse - the algo changed - we added more manual reviews - we have this stay in your lane thing, this your life your money thing - you have to have all this other info to be legit -
Is that really what the end user needs when they are looking for entertainment sites? No - it's a sneaky way to put down a bunch of sites to raise up the others.
Then a video comes out - some googlers say, well it is okay to hire an SEO company now - don't hire a bad one or it'll penalize you - you should only hire one that says it will take 6 months for you to rank.
So google keeps saying one thing and doing another - then saying another thing and not doing that.
Webmasters should be able to get details about anyone who has 'manually scored' their site.
In some cases it's not just are they looking for what's in the manual - their location in the world, their religion, and other factors, could influence how they feel about a site and it being downranked by someone in the Philippines could have drastic consequences for a webmaster in the states, and for all the users who might enjoy that site from Europe.
So the one big G statement said that NOT being transparent is the best thing so bad actors don't take advantage of knowledge of the system - well I believe you are hurting more good actors and more users by hiding everything.
There's even some recent evidence that telling users why things have been moderated actually leads to less problems: https://news.ycombinator.com/item?id=21513871
[1] InMyBiasedOpinion - and not a lawyer, doctor ymmv yada yada
I put in my comment that my opinion is biased, and I think it is obvious what side of the issue that bias is from. I will add that for a couple of years I had a site in the number one position or top 3 of some cool search results and it was a site that gave the searchers what they were looking for.
For a long time google was good. I even took my love of google and content creation to other businesses in town, got them to make better web sites and even partnered with them to spend more than $100k on adwords over a couple years.
When things are good, they are great, but when cracks start to show - the algorithm changes to enhance national publishers, there is little help when you discover fraud clicks, customer support is a 'volunteer top poster not a google employee' kind of thing.
as was said at a hearing recently; "“Small businesses cannot survive on the internet if they cannot be found.”" - https://www.marketwatch.com/story/tech-giants-google-amazon-...
knocking people out of business with hand wavey 'make good stuff, don't do seo' - knowing that they will be screwed forever and knowing they most likely will never know about the secret manual that some know about and how it actually plays out, it's worse than mean.
imho
http://weblog.globaladvancedmedia.com/2010/research-before-b...
If I were judging by the first few paragraphs of this entry from your site, I would lean toward blacklisting it. Reasons: typos, grammatical errors, and a general lack of polish and punch in the writing. It looks at least superficially similar to thousands of keyword-stuffed, semantically-impoverished blogs that I have encountered before. The page source contains another red flag:
<!-- This site is optimized with the Yoast SEO plugin v9.2.1 - https://yoast.com/wordpress/plugins/seo/ -->
The co-occurrence of the terms "SEO" and "optimized" is almost a good enough signal to blacklist it on that basis alone.
I am not saying that you are personally trying to exploit search or recommendation systems to trick people into visiting your page. There are also two big counter-signals that show me this entry isn't part of a content farm:
- You don't link to commercial sites.
- You don't show ads on the page.
The problem is that there are armies of people churning out "SEO blogging make money fast" content, incorporating ads or commercial links, and spreading it across a multitude of domains. For every blog entry like this one -- unpolished but harmless -- there are many that look textually similar and are purely mercenary.
So the one big G statement said that NOT being transparent is the best thing so bad actors don't take advantage of knowledge of the system - well I believe you are hurting more good actors and more users by hiding everything.
This is where I disagree most strongly. Google already struggles to keep junk out of search results. Process transparency would enable content farmers to evolve more quickly. Thousands of brilliant engineers are not a match for millions of people who pollute the web as a full time job. Some good actors will be hurt, granted. I think that you are badly underestimating the number of bad actors when you say that opacity hurts "more good actors" and users than bad actors.
may things have changed since then, updates and hacks and thankfully less spyware infected systems friends call me about.
It's not made to take the #1 spot for anti-virus, and the experiment with MT and WP has led to some interesting growth in other areas.
So it's not written to win any writing award or be referenced in a phd or anything, just for the average joe I meet on the street and don't want to spend 20 minutes telling them about viruses and know that sending to sofos or something is not going to help them learn or do anything different in the future.
The yoast seo plugin I have used on other sites as well - the main reason for using that plugin is that wordpress does not handle meta title and descriptions very well out of the box - for the most part they put that into the chosen theme to handle - and most do not handle it well.
When you check how your wordpress site looks in google's eyes, many people with a WP site will get warnings about 'duplicate meta descriptions' which cause a penalty in the results I believe. So the yoast plugin can create meta descriptions on pages and posts that do not have them - and it can set some standard robots.txt type rules to block 'duplicate content indexing' as WP out of the box often puts the same text and such on the homepage, and a category page, and an archive page, and others..
So it's a quick an easy fix a few problems with WP that google will alert you about if you login into webmaster tools or maybe when you run a pagespeed check, some of those issues.
So included the yoast seo plugin is to fix some things, it does not create comment spam for seo or try to insert hidden text for britney spears disney and shady redirect to a porn site or anything like that.
you could rename the yoast, the all-in-seo and similar plugins as "remove the negative self created search engine penalties caused by WP and your lousy theme with one click" - They are even more important on sites that run buddypress on top of wordpress for the same issue - and sometimes cause more - but that's for another post.
I think it's possible that we could both be right (although I certainly could be wrong) - in that I still think more good actors are hurt by the non-transparency, and you could be right in that there would be millions more junk sites / pages, although I think there are many fewer people making the bad stuff, I do know they make a lot of it... I still think that by trying to fool a few thousand bad actors big G is actually seriously hurting hundreds of thousands of web site owners.
Just think how many thousands of people have bought shady seo because they have no way to know what works or what doesn't and what's good or what's bad - I definitely think more good people and businesses are being hurt and the playing field is less even for the average business owner because of the lack of transparency.
A time when I read google was ranking edu type sites higher, and blogs lower... I noticed more news sites in top results and mayo clinic types..
It dawned on me that it would be a convenient truth to point to a bunch of 'high brow signals' to justify sanitizing results a bit - and that this would be a slippery slop slide into censoring lots of adult stories and other entertainment, while also playing into the bigger companies that can afford to spend the adwords money.
Could be good reasons for this (less public pressure to remove the porn and such) - could be nefarious, censor the web for users and cater to those who can afford to pay the big bucks, less companies to contend with content questions - while not be transparent to the users and content creators.
This allows big money to influence the results via ads easier, and limits choice - those publishers who spend a lot of time creating content are cast aside, even though one side of the big G keeps saying 'create good content you will soar to the top'.
I think we, er they, big G especially, crossed a threshold of being able to determine intent more often than not and so the need to filter search results for 'how to have sex' and 'watch free sex' for example are different and can be, and should be handled differently.
I know they are handled differently to a degree, I feel it's important to point out that these two different intentional searches should show results even much more different than they are.
The first one would likely benefit from ranking higher sites that meet a lot of the points on the pdf manual checkers document and other factors for trust rank and what have you. However I think the other kind of sexual entertainment searches would actually benefit from not using many of those factors in the ranking process.
I believe you will find many professional sex people do not advertise their address on every page of their site, and many do not use real names in order to make it harder for bad things to happen to these people. For example.
I also think the need for perfect grammar and such is much less when people are looking for erotic entertainment. Millions of penthouse stories magazines have sold over the (pre-internet, get that stuff free via searches from content indexers, years) - and I am pretty sure that if every story had perfect grammar like it was written for a college thesis, that they would not have sold as well month after month for years.
If you combine this with the type of grammar and spelling you see a majority of people using via textual communique - look at Insta, fbk, snap... people expect, engage with, react with, and continue to pursue content that is not grammar and spelling perfect.
I'd go so far as to say that a majority of people in the US at least (?) are actually mostly trying to find more crude discussions and writing styles, and it's a much smaller amount of people searching daily for phd level high brow perfection.
Of course this is different when looking for electrical engineering searches, and even searches for putting together prefab furniture - they are definitely searches where you want things to be accurate and no fluff, no extra personality needed.
Given that I believe this to be obvious to most, and that we do not have the computer systems of 1991 running the search giants, I believe that they know they could provide tons more content that browser reporting behavior would show that people enjoy and are looking for - yet they choose to use some of these trust rank things to censor bigger portions of the net for various reasons.
Hey, I'm a big believer in private companies doing what they want - I just think transparency is seriously lacking with big G - why not be honest about how many semi-good sites are not being shown because google is employing new content filters?
We used to see this chilling effects notices regularly, and some results show that X number of pages or sites are not being shown due to dmca requests... but being honest about how many sex chat sites google use to show in the results, and how they have pushed many good ones down and many more straight out of the index.. we don't see any posts about that.
Sadly, for many people the internet is what google shows it is. I understand there are many in the world who think whatever if on fcbook is the entire internet. Well if things are being removed from these platforms and it's not being understood - then it's a huge disservice to humanity, imho. It's closer to people learning with today's tools may never find Mark Twain and others for they are not perfect in the eyes of the elite.
I must admit that choice of words is partially to be inflammatory - but we are talking about a very inflammatory subject (google blacklists, shadowbans, and non transparency about censorship with users and web site publishers) -
If I could edit it, remove the word fraud and change it to 'intentional deception to secure unfair or unlawful gain' - would that take the sting out of it?
I mean it could be said that big G telling people to make a good site and it will rank high if the content is good - and if you make a mistake do a disavow list.. and don't try to do seo - but the other people in the top obviously are - I mean, you could call it a hoax instead of fraud, but (according to wikipedia); 'A hoax is a distinct concept that involves deliberate deception without the intention of gain or of materially damaging or depriving a victim.'
and I am suggesting that I think it's obvious that some teams at google have indeed tried to damage and deprive victims of time and money by telling them to do this or that - and knowing that it's not going to work to bring their sites back to the top - so it's worse and does not qualify for the hoax definition in some situations
I don't mean fraud like some AG is going to put google in jail for criminal fraud.
I do mean that it is obvious that big G and some people there have been purposefully deceitful, and that google has profited from telling people and companies to keep publishing - keep doing things google will like, (schema data!) - and they use that data to profit from it - and they purposefully do not rank and send traffic to many different sites for different reasons.
So no, I don't expect the whole model number switcharoo defense to be made - and even though at least one state's AG has looked into fraud with google, I don't think they know enough about these issues to bring a case, and google has enough money to pay off all 50 states and the EU and sorry not sorry, not putting the algo on trial kind of go away money.
So I'm not suggesting that's going down. What I am saying that is that some there have been intentionally hurtful, both taking people's time and money and it's publicly shown they are / have been deceitful on purpose.
If you don't see that, perhaps you are not familiar with the timeline of all these events as I described in the downvoted comment above. I am sure there are plenty of webmasters and seo people that have witnessed this timeline - and many people have scratched their heads wondering wtf google (and many people pulling their hair out!) over the years - there are plenty of public posts on non-shady forums showing this.
This has been done on purpose.
Lots of people have lost their jobs, their money and time, in some cases their homes. Partially because google changed things with the algo; but they have not been transparent about that and actually have suggested things to do that they knew would not work - giving people false hope and getting them to spend excess time and money, all the while knowing there was no way out of the downrank hole for most and it would all be spent in vain.
funny thing, it's not their money, their life - so they care? I bet the spam team and algo team celebrated some of these changes, laughed at seeing people try to change things - and watched as ad sales increased and their stock did - not caring about the little people out there - and not even notifying the users that they were censoring the shit out of the results now - which funnily, makes the sites in the ads more appealing.
there are plenty of synonyms; con, scam, shell game, double dealing - they could be used in place of fraud in my original statement. I am sorry (truly), and not sorry, that it is inflammatory in this context, as I think the issue of censorship by itself is a serious subject, yet this story goes well beyond that.
I agree it can seem non-nonsensical, but if you look at the events I describe over the time line - I can't actually think of a more sensible term to give it. I guess from the other side of the glass it could be called a funny and profitable business move, as I would guess some did.
Give me back the Google of 5-10 years ago, and the rest of the internet from that time, not this ad and blogspam dominated AOL 2.0 joke of a net that we're quickly centralizing into. It's sad to see where ad based economics are driving the net.
But right now I am still getting decent results.
I don't use Facebook anymore, yet we had some hysteria about that being manipulated.
How can you know?
On the rare occasion that I search for some general concept like a current event, I'm quite happy that nutjobs like Daily Stormer get filtered out.
Even in those cases, I'm usually just trying to get straight to a relevant article on a publication that already know of. e.g. "wapo impeachment"
This isn't really down to any huge improvement on DDG, but rather a decline in Google's results. I'd just been putting it down as a consequence of Google having less data on me as I made a conscious effort this year to diversify my usage of other sites.
However although DDG is not going to surpass Google in that field, it is indeed getting better and better every year, and there's one thing it could seriously spank Google's ass: implementing a working discussion filter. The discussion filter was one of the most useful filters Google once had: using it in a search meant one would get only results from blogs, forums, Usenet etc, that is, comments from users of X rather than sellers or advertisers promoting that X. It wasn't perfect but helped a lot to filter out shills, astroturfers, fake forums and similar trash, therefore it didn't suprise me much when they removed it probably because their sponsors didn't like that function. So why not implementing it back at DuckDuckGo?
Instead, I get (in order):
1. GeeksforGeeks
2. Tutorialspoint
3. W3Schools
4. Programiz
5. Stack overflow
6. And then, finally, the official documentation... For python 2.7.
How does this happen? Are these sites just paying Google a bunch for the rankings?
The documentation for str.replace is located halfway down an enormous page that describes every single built-in type in the language [1].
And then, once you manage to find the entry for str.replace, what does it tell you?
Return a copy of the string with all occurrences of substring old replaced by new. If the optional argument count is given, only the first count occurrences are replaced.
That's it. No examples, no link to re.sub or other functions you might want to use for replacement. Stack Overflow or even W3Schools (gasp!) is much better results for this.
Paid articles = ???
Nobody's ever happy..
Regardless, I realize we shouldn't be complaining about paywalls in comments. I just wanted to address your question.
Then, the publication suffers and the quality falls.
???
Why would the publication just give away the product of their labour for free? [Especially the WSJ. That's against their cornerstone beliefs AFAIU.]
You just made me realize I'm a bit of a hypocrite. I don't want ads, but I will never pay for this either. Then again, I don't really want to read it anyway hehe
That being said, if ads actual ads and not targeted surveillance crap, I'd be ok with that. I might have to change my adblocker settings to reflect that...
I know here in Toronto, our library subscribes via Pressreader which gives access to a large number of periodicals and newspapers.
https://www.torontopubliclibrary.ca/detail.jsp?Entt=RDMEDB01...
Or try an aggregate service like Apple News where you pay a little more total but get wide-reaching access to a large number of pubs.
edit: Sorry, it doesn't look like Pressreader has WSJ, but they do have a large number of other American pubs. Apple News+ has WSJ, I believe.
The point is that it's an interesting article. It's not hard to search for the keywords in the title and find some other news outlet covering it, albeit in less comprehensive detail, or look for comments that reiterate the salient details.
This is in the FAQ at https://news.ycombinator.com/newsfaq.html and there's more explanation here:
https://news.ycombinator.com/item?id=10178989
https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
https://duckduckgo.com/?ratb=c&q=autism+vaccine+link&t=h_&ia...
https://www.google.com/search?q=autism+vaccine+link
The 3rd result on duckduckgo.com is learntherisk.org. That result is not presented in the first 10 pages of results on Google.
It seems quite likely to me that one of these sites is manipulating search results. Because the organic search results for these two different search engines should not be that far apart.
Edit: Why the downvotes? Here is an even more egregious example:
Try searching for "learn the risk autism".
https://duckduckgo.com/?ratb=c&q=learn+the+risk+autism&t=h_&...
https://www.google.com/search?q=learn+the+risk+autism
Note that I am not promoting an agenda here. This seems like an example of manual manipulation. The article sites "vaccinations and autism" as an example.
I have no idea if Google is specifically pushing vaccine denier sites down the list, though I can say that the entire concept has far more "mind share" than the notion merits scientifically. That much, at least, is objectively true.
Is this an authoritative source of information? That website screams confirmation bias all-around.
https://www.scientificamerican.com/article/how-robert-f-kenn...
Advertise ? How it's any profitable for advertiser to have multiple same adverts on page ?
Thruth about Bill gates and Epstein is filtered away..just like hn filter and delete comments :)
Are we elevating Google's search engine to public utility status?
Yes. And we might have too I'd say. Given the defacto monopoly they have, I think it is reasonable to agree that they fulfill a public utility function and thus need to be held to a higher standard. This should however be done through regulation, cause that is why we have governments.
Inferior compared to what, exactly? I have yet to find a good replacement.
I tend to disable moderation settings because I'm an adult right?
Google does a pretty good job distinguishing intent. For example a search for "<actor|actress>" and "<actor|actress> nude" with moderation settings off yield very different results. On Bing? Not so much.
Agreed.
I have switched to duckduckgo as my default search engine a couple of years now, and I constantly find myself having to re-enter the query with a !g in front.
As much as I dislike what Google has become, their search engine quality is still the best, and by a very, very large margin.
Inferior to what Google was 5+ years ago.
and bing is not that far behind for web.
Right now search engines are not a hot topic. But if people smell that google is starting to lose it, you bet they 'll go after his market share like rabid dogs