Googling for Software Development: What Developers Search for and What They Find | Hacker News Reader

Googling for Software Development: What Developers Search for and What They Find

neverworkintheory.org

71 pointszdw48 comments

I feel like Google is dropping the ball in this area. Lately when I search for something I find a lot of sites in the top results that have simply scraped Stack Overflow and reposted their content verbatim.

I don’t know if these copy-paste sites are a new thing, or they have always been there but I never noticed because Google de-prioritized their score because it’s low value content and recent algorithm changes now promote then higher.

This is basically the case for every search I make now. Recipes, how to guides, DIY, programming, you name it. The SEO spammers have taken over the world.

Perhaps controversial but I feel that we reached a saturation point with google's insistence on AI/natural language queries, where ddg is now giving me better search results than google for technical queries.

I think the only area where google is still able to outcompete other engines is searching for regional results in Google Shopping, apart from that SEO's are effectively eating google for breakfast.

The only way Google could claw me back from DDG at this point is finding a way of eliminating the endless SEO spam and useless content mills from their results. It plagues both of them but at least DDG makes a point of being less insufferably nosy.

I bet they make more money by not fixing the problem.

Google could pretty easily drop scraping sites hard to the bottom of the list and prioritize fresh content. But it probably pays not to. So sadly you're not worth the effort.

I think DDG, being somewhat privacy focused has a more traditional index, whereas I strongly suspect google would be profiling the devices and people making queries based around time of day and device used to determine what type of results to favor. I actually get good results with google. But my technical queries are all done from my work laptop, so i expect there's a profiling, fingerprinitng going on on the google side that's shaping the index I use. or its magic.

Could you ever see yourself paying for search? How bad is google, what would you pay to change it?

I’d pay a subscription for searches that effectively removed cruft and heavily SEO’d listings. The classic example is looking for a recipe and getting an essay of SEO bullshit about how these cabbage leaves were grown by the Mad Monks of Machynlleth and remind the author of the their childhood frolicking around the depths of Mordor or whatever, but also things like stackoverflow or GitHub cut-and-pastes, blog spam, empty posts with nothing but trigger words to get the ranking higher, literally any link to things like Pinterest, and other annoyances which push actually useful results down the page.

Essentially I want a search engine that can differentiate results that exist to solve a problem I have and results which offer nothing and just want to divert my eyes from a problem to show me ads. That’s the difference, I don’t mind seeing ads next to things that are actually useful but many of Google’s results exist for no other reason but to show me ads and offer nothing useful. I’d happily pay £20 a month for a search engine that removes these pervasive annoyances reliably.

This is about my experience as well. Google has just about become useless for a ton of searches I had no problem with a few years ago.

The results take so much longer to find relevant results for just about anything

A little off topic but I've recently started seeing duplicate search results on the same page, one after the other. Not sure if this is a bug or if it's yet another SEO loophole that's being actively exploited.

I've also noticed a lot of sites that now scrape GitHub Issues and the discussions on there that are increasingly spamming search results.

GithubMemory, GitMemory and Giters are the domains that come to mind immediately, but there's loads of other non-obvious ones as well.

Domain denylisting can't come soon enough as a feature from search results, urgh.

> Domain denylisting can't come soon enough as a feature from search results, urgh.

Crude solution: At least in Firefox you can assign keyword shortcuts to search engines. Since you control the requested url you can -site:w3schools.com the offending domains.

Also there's an addon for hiding results from specific sites: https://addons.mozilla.org/en-US/firefox/addon/hohser/

The uBlacklist browser extension lets you blacklist domains from Google Search. See if that helps you.

It came and went, sadly.

https://searchengineland.com/google-finally-discontinues-the...

I agree. As a German I'm even getting some odd results of a auto-translated Stack Overflow Q/A. There's one site which is doing this automatic copying+translating and when I land on it it drives me absolutely mad, I get angry that this is even a thing and even more so that Google is indexing and offering me their pages in the results.

Other than that, I prefer to search on google because I know that I will get a couple of good links from which I can then first open a couple in background tabs (mmb-click), then close the search tab and start looking at the pages.

Diese Seiten gehen mir auch extrem auf die Nerven!

Wütendes hochwähl.

Allein deshalb klicke ich auf deutschsprachige Suchergebnisse schon gar nicht mehr drauf.

As far as I am concerned google already dropped the ball. For 2 years I am getting useless results immediately when I am not searching for something very generic. I am leaving shadow of a doubt that there might be a reason that I am never logged in and I am cleaning all the tracking cookies. I have skipped to self-hosted searx instance and I get much better results when they are aggregated.

I believe that the whole problem comes from their AI. Since most of people on planet are searching for things that are highly non-technical, the AI is learned to give them priority over the technical documentation etc. which almost nobody uses. I am also experiencing that the sites that I know are there, are not returned at all (or somewhere on 100th page).

I would really love to see the google search as it was 20 years back. At that time it was far more useful, quite frankly I didn't see any improvement in precision of search results from those times, it only went to the worse.

As a counterpoint, I feel like Google's results are far better than they used to be. What's more, I find Google ridiculously good for technical queries compared to the competition, and I'm often surprised at how Google will know the intent behind my search even if my query is vague. For example "cat pipe" gives me a bunch of answers related to the shell (as well as some pictures of cats dressed like Sherlock Holmes). They still aren't perfect, but most of the time the result I want is in the first page of the first query.

Twenty years ago, I had an actual advantage over others due to knowing how to write a good query and find the appropriate result (Google-fu). Nowadays that skill is largely obsolete.

This site cloning has always been an issue. Before the current iteration of SO/GH SEO clone farms, you had sites doing the same thing with expertsexchange and cloned forum pages. I do find it odd that Google won't just block the big ones, but perhaps they internally don't like doing one off hacks and prefer to find a cleaner algorithmic solution?

The counterpoint is that it's now actually useless to craft a query that tries to match exact terms, because there's this extra layer on top of it.

So you might be lucky if the inferred intent of the engine was correct, but good luck steering it away. It doesn't help pretty much all query logic operators are merely hints nowadays, more often ignored than not.

I very much preferred a dumber engine for this reason, since it was way easier to search precisely and avoid SEO, even as the SEO game changed.

I'm also using a local searx instance now. I'm not terribly happy about it as bing/ddg also have very similar issues, so searching for exact terms still doesn't work the way it should. But it's much easier to blackhole SEO silos, pre-filter queries NOT matching my exact queries, as well as yielding more obscure content.

I want pure keywords-based search engine back [1]. If the keywords verbatim are not there, give me zero results back. It will be less time consuming to refine my query, or conclude that the stuff I am looking for is just not there. It's much better than sifting through low-precision results.

Google has 'verbatim' mode as such, which used to work well. I can't figure what they are doing to it, and why.

[1] https://news.ycombinator.com/item?id=24609881

For me (both logged in and not), for many short & simple tech queries Google fares better than competition (for example, googling some package name or API method shows GitHub repo or official docs as first link, while in DDG it's often not the case).

On the other hand, for slightly more complex queries, in last months there's been an enormous flood of low quality GitHub / StackOverflow / blog post copy-pasta spam pages.

And overall Google quality is really getting worse for discoverability of internet gems or interesting pages with good writing; unless you use specific unique keywords (site's unique name etc.) and/or heavily use double-quotes, it will find SEO-optimized pages with low quality.

Maybe it's just me, but I feel that there's almost a counter-Google-fu these days. Finding something that isn't on the front page, or even just trying to use an older search style often leads to Google showing the weirdest results.

I use Searx or DuckDuckGo for these searches. While their results are often worse at the top, they provide a wealth of understanding and documentation that Google seems to just.. skip? To me, finding things is more difficult when there are two opposing sides trying to outsmart one another.

I have just tried. For `cat pipe` I get zero results related to `cat |`. On first 5 pages only cats, I didnt try to go beyond.

I dare you, remove all google cookies, logout (like - dont use chrome*) and try again. I think you will get to the same results as I did.

And regarding google spying on me - I will rather not use it at all. Even without the searx, there are viable alternatives like qwant.com.

As I have said, I believe the results are based on your queries from past (AI based) and if you are not using the google spyware ecosystem, the results are horrible.

Remember, the e commerce industry is $500bn a year in the US, and a huge proportion of those purchases started ultimately at Google searches. That isn't even including other online revenue like banner ads which aren't aiming to have you buy something, or online purchasing decisions that aren't e commerce (like when you choose a restaurant or real life activity based on Google and the websites it sends you to).

That $500bn is what makes the bad parts of the internet so resourceful. It funds all the blogspam and content mills and malware when you look for recipes, cleaning/DIY/repair content, advice on purchasing products, lyrics and guitar tabs, tourist advice, investment advice, health advice, error messages. It funds the scam "local" locksmiths and repairmen in Google Maps search too. By comparison, Google's revenue is $170bn and that's global. So could anybody "fix search" on Google's budget?

Yep.

There's also lots of sites that copy/paste GitHub issues.

Just bring be directly to the GitHub issue, not to these clones.

> I find a lot of sites in the top results that have simply scraped Stack Overflow and reposted their content verbatim.

Since last year I get very often results from websites that scraped GitHub issues and just reposted them, often with no easy-to-find reference to the source. For some reasons they seem to have extremely good SEO.

Edit: commented too quickly, that's already mentioned by 2 other comments...

For the SEO, the content in the issues is still good.

The real question is why GitHub itself isn't higher, and the most obvious answer is that it's owned by Microsoft. User generated content is only prioritized if it's from a Google run community

I’ve been using Kagi for the last month. I’ve made a point of trying out every upstart search engine I can get access too and so far this one has been great. A good sign is that I think very little about it. It tends to hit canonical sources more often than scraper sites like gitmemory dot com or the stackoverflow scrapers you mentioned. I’m a little annoyed that I have to be logged in because my default search engine doesn’t work in private windows, other Firefox containers, etc.

Something I noticed with Brave (my last experiment) was that it was awesome at first and then started to slide. It feels like restaurants that are great when their first open and become meh after a year or so. I wonder if popularity hurts search quality so I’m almost reluctant to share Kagi. I hope they can continue to delivery this quality.

One problem with google is its ridiculous, and absurd word substitution.

This started more than a decade ago, with things such as 'Bob" being aliased to "Robert". Back then, before the annoying Google Plus came onto the scene, the + could be used to insist that a word is never aliased, always present in search results.

This was deprecated due to conflicts with Google Plus searches for name+ or whatever, and half-assed replaced with 'use quotes they're the same', which of course they are NOT the same. Google just gives slightly higher precedence to quoted text, and will chose to ignore its existence and alias if required.

Thus, 'verbatim' search was created, under a hard to find menu item, doesn't work with things such as date matching, isn't available until after you search, and is only a partial replacement for the now missing + (which only did one word).

These days, google auto-aliases everything. Debian is aliased to ubuntu, and others. dhclient is aliased to dhcp (due to dhcpd/dhcpcd), and on and on, so searching for the precise thing is only even remotely possible with verbatim.

Of course, verbatim itself often breaks now, randomly still aliases things, and is just a PITA to use.

The reason behind all of this, is voice search, plain and simple.

How can you search for 'here' 'hear' via voice, without aliasing? Even searching for 'tanya' via voice, means people might want 'tania' 'tanyah' or whatever other close sounding word is the case.

So, google continues to destroy search results for those wanting precision, because google thinks the code should be the same.

One of the richest companies on the planet, can't manage a few extra coder cycles, pulled away from whatever advertising scheme they're working on, to slap in a few if statements for 'text' or 'voice' differentiation.

I used to love google. Used to.

Quoting still seems to work? As I remember + was always for emphasis/importance and quoting was for verbatim, but maybe I remember wrong.

I get highlights for Robert here: https://www.google.com/search?q=bob+downey

No highlights for Robert here: https://www.google.com/search?q=%22bob%22+downey

Otherwise the found pages are similar here, but that's maybe because the search is very specific.

+1. I just wrote a similar thing here: https://news.ycombinator.com/item?id=29351147

I hope this is not a sign that someone here will disrupt today's web search using, for a change, older technology. :-)

Yesterday, I wrote a short blog post titled "Don't mistake the internet's intelligence for your own" [0] which is very closely related to this blog post here. I wrote it it to shine some light on the fact that knowledge on the internet (Google, SO) is not ~your~ knowledge. This fact often gets painfully exposed during interviews. Funny timing.

[0] https://www.janmeppe.com/blog/dont-confuse-intelligence/

It makes me think of how much complexity we agree to live/work with, being actually unable to manage it without searching the internet. It’s really funny that interviewers try to expose that, cause regular developers (think fullstack configuration vs software algorithms) do just that: convert the internet to a source code of mediocre value but high complexity, and then implement a crucial business logic on top of that. Which cannot be learned beforehand, unless you’re reimplementing something for legal reasons.

Is that really a bad thing?

I think most of this complexity is a failure in scope limitation. No exaggeration I wouldn't be surprised if the average piece of software today is 1000 times bigger and slower than it needs to be.

You need to spell check a text? In the 1980s, you could do that relatively quickly on a computer with no network access. The dictionary data was pretty big back then but today it will fit in an L3 cache.

Today, each checked word likely gets converted to JSON and is then sent around the world through load balancers and proxies to some cloud instance where it is deserialized and checked and a new JSON message is encoded saying "{\"message\:\"yup, it's a word\"}", which gets passed back through a bunch of servers until it ends up in a mobile app that is really a souped up HTML widget. Spell checking isn't by any means trivial programming, but it doesn't require planet scale computing either. There is no benefit to that. It's just a waste of resources.

So why do we build things that way? Because of Parkinson's law. If we had put a constraint on resources, we would have found a faster and better way of doing it. This is in part why I advocate targeting low power hardware when designing a new system. If it's fast on a Raspberry PI, then it will be fast on any hardware. You will identify scaling problems early while the code is still malleable and easy to redesign.

I consider the internet an external brain I reference (mainly through Google search). I've done this for decades. At some point systems like linux got so complicted and changed so rapidly it didn't make any sense to store information in my head if it was already stored and easily fetched externally.

Much of my ability in using internet information comes from being able to rapidly prune what I can tell are incorrect answers ("do this to systemd to fix that problem" when systemd isn't related to the problem because on OS whatever, they disabled that feature) while discovering the one random answer that has the deep, non-obvious fix.

Now, in terms of interviews. I'm not looking for people to spit out information from memory in an interview, beyond some fairly basic reference information. I'm looking to see if they can apply the skills they have to solve useful problems better than other candidates.

I failed my first interviews at Google because I didn't know quicksort (the details of the implementation) off the top of my head; anybody who had memorized the code could have answered the question instantly. After later joining and spending 12 years as a software engineer, I can assure you: knowing how quicksort is implemented isn;'t helpful for the vast majority of SWE roles at Google. Knowing that quicksort can be faster than heapsort or bubblesort, but sometimes slower, is useful, and how to select the right library implementation and benchmark it with a good set of data is far more important.

One thing it can't do well is search for negatives. In some cases it will latch on the opposite of what you want because that's what most people search for.

yeah, when I did, it felt like Google just ignored the negation. the results didn't materially differ with or without it.

coffee shop -starbucks -dunkin

used to work in Maps. not anymore.

:eyeroll:

pro-tip: For finding good coffee in strange places, search for "espresso roaster". I have had great success with this approach.

I use Google less and less. Their results have dropped in quality dramatically. Even the first page is littered with spam and often dangerous sites disguised as legitimate. I wouldn't dare using it on Windows - it's pretty much guaranteed now to get malware. Apart from spam there is also lot less results in terms of quantity or no results at all. Year or two ago I could find a lot of interesting content, today it's just spam mostly.

https://homepages.dcc.ufmg.br/~andrehora/pub/2021-msr-googli...

My theory is that developers are spending more time on the Google and wandering around to find the answers. Half of the time simple stackoverflow search will reveal the correct answer. And other half is simply a reference to particular documentation.

This is why I started a hacker assist project but not started it yet. I was planning to implement the documentation search and the stackoverflow search in the common place. Also little bit of the blog search.

If I provide these curated results in the single place then the chances are that the developer get the results quickly rather than search through the Google.

I'd tend to think that at this point people happy with Stack Overflow will go directly there. Going through generic search would be more to get either the official documentation link (Google can be faster than searching the language site ...), or trying to see what option there are on the subject we're looking for.

In particular, random errors can be easier to search when including blog posts and other docs that just SO.

Also not every search needs to be quick or optimized, finding alternatives along the way is nice sometimes.

W3Schools presumably comes up because Google favours them. I don't trust it.

There are still some hidden gem features of search that are still there, like prefixing your search terms with site:example.com, or sometimes even better: -site:example.com .

It seems to be skewed towards web development, with W3schools as one of the top 3 sources. Selection bias or natural sample bias?

I’ve been getting more and more w3schools for non-web python development. Even ranked higher than python’s standard library docs.

And thats on DDG as well. Top-tier SEO there.

Same with geeksforgeeks.

Reply on news.ycombinator.com