Hacker News Reader

Comment by ambrosite | Hacker News Reader

I do remember those websites. For me, the difference is that now the Web is much more useful, but back then it was a lot more fun. True, you could sometimes waste hours following random links hoping to find something good, but the thrill of discovery when you stumbled across a gold mine of information was a huge part of the appeal.

Nowadays, anyone with a basic understanding of search engines can find almost anything they want within seconds. That makes the Web on the whole much more useful, but the thrill of the hunt is gone -- that's what Jakob Nielsen was referring to all those years ago when he talked about "information scent".

https://www.nngroup.com/articles/information-scent/

View on HN

Nowadays, anyone with a basic understanding of search engines can find almost anything they want within seconds.

Almost anything popular and rather insipid, yes. Try to find more detailed information on very specific topics, however, and you'll discover that search engines like Google have been optimising more and more for the former content, making it harder to find the latter. I don't think that's a good thing at all.

To add insult to injury, if you do try very hard to seek the latter by carefully repeating similar search queries with slight word tweaks, quotes, and trying to dig through all the pages of results to see if you've found what you're looking for, Google will quickly decide that you're a bot and either give you a CAPTCHA or just ban you for a little while entirely.

>> Try to find more detailed information on very specific topics, however, and you'll discover that search engines like Google have been optimising more and more for the former content, making it harder to find the latter. I don't think that's a good thing at all.

This. A thousand times, this.

I really feel is the unfortunate consequences of our short attention span and everyone marketing towards an insanely short attention span is you get the firehose of information, but zero depth to any of it.

If I want to dig; really dig into a subject, 99% of what Google pukes at me is not helpful at all. A few times lately, I've ran into something I've found through Google, but trying to dig any deeper is futile. I've had to go down to the local library and start digging through news clippings, books and magazines. Actual, real old fashioned leg work.

I've come to feel like the internet has no depth anymore, everything is just enough to satisfy a .001 second attention span. Sad when you really stop and think about it.

As someone who grew up with Google and is now trying to get into niche interests, I have basically no idea how to do the real, old fashioned leg work. I still don't know exactly how much of this is Google's fault, because it remains great for providing information on a lot of more broadly accessible subjects.

in the old times there were _a lot_ of search engines, and they even had directories, you could find lots of interesting websites and communities linked between them, no optimization or ads, you'd also use usenet to talk about topics (like today's reddit and digg)

plenty of creators

Agreed 1000x, today I needed to do a real search on a subject and all I got on the first pages was total crap...

Google is not great for finding stuff. I’ve complained at length about how bad it is for doing research, but recently I’ve also found it’s totally useless for finding things like reviews of products (unless it’s something like the new iPhone). It’s like legitimate reviews are a casualty in the war between Google and clickbait SEO crap.

It's amazing how between reviews, sellers, features, and availability, when I make a google (or DDG) search of some product none of the links are of the kind that I want.

I've gave up googling for products I intend to buy (what has probably the largest revenue potential).

I swear the last five years or more of searching for products lead to the first two pages being fake reviews/ads. Google must make money from this otherwise they would fix it.

Do you have an example query handy?

Google for any game, and then check which of the articles doesn't just copy its enture text, images, and score from the stuff on the press server.

If you have access to, say, the bethesda press ftp, and compare the material there to any review of the games, you'll see that over 90% just copy-pasted without writing much themselves.

It's all fake.

Oh, and those that don't copy-paste often still review everything positively, because otherwise they won't get games before release anymore.

That explains why every BioShock review I read back in the day seemed like it had been written by the same person. Suddenly, everyone were talking about protagonists and antagonists in their game-reviews.

I just did

washing machine quiet

and 75% of the matches are ads.

Consumer reports was first but it's an ad - I should have stated I stopped using Google five years ago because it was so full of spam but now at least #3/#7/#12 are relevant and the rest are still ads and spam sites for me.

Google is excellent at giving the appearance of finding everything. The fact that there is an enormous world of which Google remains completely ignorant is a fact of which we seem increasingly unapprised.

Care to expantiate on this ?

Have you never noticed that all the really good sites you 'find' have been because someone posted the link? For example in a comment on HN, or reddit. Or a related link on Wikipedia or the sidebar of a subreddit?

You have a point there.The digital equivalent of 'word of mouth' being the best form of marketing ?

That to an extent, and also the fact that _real_ artificial intelligence - the kind that you could share exactly where you're up to with programming and it could tell you exactly what site to read next - that doesn't exist yet.

I once read that Google indexes 0.5% of everything publicly known.

That figure (0.5%) would suggest to me like theres a lot more room for disruption.

Yup. Google are doing everything they can to ensure their future. They know this.

Once I found very nice forum about linux servers. I found it thanks to my friend saying I have to check it out. After spending some time there this forum started to showing up on first page of google every time I looked some solution for my server problem. Google is closing your in your own bubble. It will present to you websites that you already know or that are related to sites that you know.

> Google is closing you in your own bubble. It will present to you websites that you already know or that are related to sites that you know.

This is a good explanation for why my friends search results are very different from mine. Google would spin this as individualization. Is there any way to get around this bubble and unearth fresh results each search ??

Have you tried incognito mode? I also recall there was option to turn off customizing search results (or something similar) in google's privacy settings.

EDIT: I did not notice any differences so I don't really know how to test it. I search highly technical stuff for work or very mundane "popular" things outside of it. Seems like I'm bad target for the customization.

As far as I know, Google isn't very accurate with Chinese content.

Actually, I don't think that is the case. I think people unwilling to pay for reviews killed reviews. Before, you had magazines and newspapers to do reviews. Since those are dying, people tried to do reviews online. Now it's hard to tell who is reviewing for real, who's reviewing for ads, who's reviewing because they're paid, etc.

This is true. The only way I can find product buying advice is:

1. To find a community of enthusiasts and see what they have to say about products in that space. This works pretty well for products that actually have enthusiasts.

2. Go to Amazon and read a product's bad reviews. The good reviews are nearly useless. But, with bad reviews, you have to weigh the number of bad reviews against the product's popularity. If something has twelve negative reviews, that means very different things for a product that sells 10 units a month versus a product that sells 50K units a month.

3. Read the tiny handful of reliable product recommendation sites. Cool Tools, Wirecutter, annnnnnnd.... I'll let you know when I find another one.

I bought a subscription to Consumer Reports and have been happy with the two purchases I made on their recommendation (headphones in the $100 range, and a new food processor). Do you consider them compromised? It's actually just about time to renew my subscription and i've been wondering whether I should.

I've also used your strategy of reading the bad reviews on Amazon!

Consumer Reports is top nouch and very thorough.

It can be a bit behind on the far changing stuff like electronics, and it can be pricey.

But I've never found them lacking in quality and detail.

Good trick

The appliances I bought using their advice has been hit/miss - ranges are decent but the fridges and dishwasher I bought are noisy and required service barely outside warranty period. These are big name brands too.

I don't necessarily fault CR but I with the hundreds of models released all slightly similar but different (I hear partly to prevent price-matching across stores) it must be as impossible to review for CR as it is for consumers to keep track of.

I wouldn't count the appliance reliability against CR as this great article shows appliance industry is doing it on purpose:

https://news.ycombinator.com/item?id=13909365

Typical cartel-like, profit-boosting activity of oligopolies which appliance market seems to be with a handful of companies owning most brands.

I used to be a subscriber. Theoretically I like CR's approach better than Wirecutters; CR purchases their products anonymous and therefore should be quite impartial and totally independent, receiving no special treatment or assistance from manufacturers.

But I found that they simply didn't update their guides nearly often enough to be useful.

Wirecutter's modern twist on CR's formula -- maintaining a frequently updated leaderboard in popular product categories, plus information on deals -- just winds up being more useful to me.

It also seems like Wirecutter has more relevant experts writing their articles. Maybe that's because they're actually better experts; maybe it just seems that way because WC articles begin with the author/authors stating their credentials and experience.

Good trick on the #2, reading the negative reviews. Of course, now that you've mentioned it here - soon enough that'll be gamed, too.

Not OP, but tbh, I thought that was more or less common practice. Many people I know do that already.

I tend to only look at 1-star reviews and 5-star reviews in passing. The first ones are obviously pissed with the product and that will colour their opinions. The latter are either enthusiasts that will love everything about it or people that heve just received it and/or are still on the honeymoon phase. I think ignoring them outright is as bad as focusing only on them, but I don't pay them too much attention.

I find that focusing on the reviews around the middle (and, of course, the number of those compared to the total number of reviews) usually gives me a better idea of the pros and cons of a product.

> annnnnnnd.... I'll let you know when I find another one.

Recommend the BIFL (buy it for life) community on Reddit.

That's also good heuristic for 1. - if you're looking for a community about $obscure_topic, there's a good chance there exists reddit.com/r/$obscure_topic. Personally, I always search for a related subreddit as a first order of business when researching an unfamiliar area.

Yeaaaaaaaaahhhhhh hmmmm.

I have very mixed feelings about that. I have some serious experience there; I moderate a subreddit dedicated to audio hardware in which people often ask for and receive purchasing advice.

So on the plus side, you have some fairly knowledgeable people offering advice.

However, even your average "enthusiast" is pretty lacking. No enthusiast has experience with enough relevant speakers to really give informed recommendations. There are thousands of speakers on the market. I've used... dozens? And owned... half a dozen? I mean, nobody has direct experience with even one percent of what's on the market. We try to supplement that direct experience by staying on top of reviews and things, so we have a sense for what's popular and seems to have garnered a lot of happy owners, but it's still very conjectural.

And speakers are a product category where it is somewhat possible to own multiple products, and there is a semblance of objective measurement possible. So we have it easy.

What if you want to buy a fridge? Or silverware? Or socks? Or blankets? There probably aren't fridge enthusiasts and if there are, it's doubtful they own many fridges each. So, you wind up with users just giving anecdotal stories, tainted with heavy doses of confirmation bias, because of course people want to like the fridge they spent their cash on.

Or, worse, you get the opposite effect: brands that sell lots of volume will appear to have the highest failure rate, even if their failure rate is low. If you knew nothing about cars, or the sales volumes for various brands, combing the internets would give the impression that Corollas fail much more often than Lamborghinis. Because tales of mechanical problems for Corollas probably outnumber those for Lambos by a factor of 1,000:1. (My wife owns a Corolla! It's had a problem or two) Which would seem really damning if you didn't know that Corollas outsell Lambos probably 100,000:1 and in reality are probably an order of magnitude more reliable.

You get this effect with Apple products: any Apple-related forum has countless horror stories of failures from Apple owners. But, Apple products are actually generally reliable compared to the competition. It just so happens that they're the Corolla in this analogy, not the Lambo.

Do you mean this [Cool Tools](http://kk.org/cooltools/) (http://kk.org/cooltools/)?

Yeah! It's pretty good IMO, are you a fan? The links are Amazon referral links (same with Wirecutter) so I still try to be heathily skeptical but I've found both to be reliable...

I just googled it. Can't say I'm a fan, I didn't even know about it. It seems very focused on home-maintenance and tools, which I know nothing about, but I'll certainly check it out in greater depth ;)

Also, with bad reviews, you can keep in mind if the bad reviews seem to be legitimate complaints, or whether they're just written by either someone with an axe to grind or someone who doesn't understand the purpose of the product. For example, some negative reviews of textbooks on Amazon seem to be people who are reviewing a course they took using the textbook. And some negative reviews of restaurants on Yelp seem to have wanted the restaurant to serve a different kind of food than they do.

Also also, regarding restaurant reviews: Always consider that a negative review that only calls out bad service is quite possibly a review of the reviewer more than the restaurant. Anecdotally, I've seen my share of notes of bad service for restaurants that I frequent and have never had issues. I suspect that the reviewer had at least some part to play in such issues, but conveniently leave out or drastically underplay their own contributions.

I do research, what are the best resources that you use (if you were referring to academic research)

I do historical research in the context of legal work, and for that Hein Online is pretty good. It’ll have old copies of speeches given in Congress, contemporaneous articles, and that sort of thing. It also has tons of old articles and research papers in the social sciences. ProQuest is indispensible for more modern stuff. If you’ve got a Library of Congress card, many of these resources are free.

A lot of stuff just isn’t online. For one case I needed to look through a bunch of 19th-early 20th century rail and telegraph tariffs. These were generally published in booklets, but they were commercial materials so the Library of Congress doesn’t have many examples. But EBay did!

As a millennial I fall into the trap of thinking if it’s not Google it doesn’t exist. But the traditional methods your librarian taught you are still highly relevant even today.

I highly, highly recommend Thomas Mann -- The Oxford Guide to Library Research.

There are all kinds of cool reference works out there. And Mann has a striking criticism of the impoverished, dumbed-down search interface that's ubiquitous now: the search box.

People don't search the same. For example, if I'm looking for something, I rarely use Google. Always Reddit, which will take me to a specific community with a wiki, and discussion. My Girlfriend will type something into Google, without using an Adblocker, and click on the first thing (which is always an ad), and filled with SEO garbage. I'm assuming most people search this way as well, which is why it's so lucrative.

ah yes, normal users

I’m still surprised the resistance-to/wailing-about Google’s placing ads inline with search results died down as quickly as it did. It’s a total shitbag move on their part. Bottom-feeding scum behavior, right there in your face at the top of search results screens. Maybe the worst, because so effective, dark pattern on the entire web. That anyone from the company gets away with speaking in public without being shouted down by hecklers is a bad sign for tech and for society in general.

DuckDuckGo does the same. I think you're exaggerating.

It’s the Internet equivalent of guys who go door to door selling old people driveway treatments and crappy brick façades they don’t need, at obscene markup. Preying on the least able to defend themselves. If you have any friends or relatives who aren’t “good at computers” Google tricks them for money, as a core part of their business model.

And yeah, it’s bottom-feeding scum behavior any time you see it, fit for failed used car salesmen and other low-ability scam artists. That’s the company Google workers keep, ethically speaking. Could be worse, but still bad enough you don’t want them near your kids or grandparents.

You didn't address the DDG part. I know you're an advocate.

> it’s bottom-feeding scum behavior any time you see it

So, if you see it, then...

Oh, this is so true. Just the other day I was thinking how hard it is to find reviews of non-tech products. I was lloking for information on a new fridge and what I came up with was useless marketing crap and what seemed like affiliate-link collections dressed up as review sites.

The Wirecutter fills this role with in-depth reviews, but their selection is limited and I prefer to have multiple sources when comparing products.

User reviews aren't helpful to me. They are very subjective, may value different attributes of a product and disappointed customers seem to have a greater incentive to broadcast their negative experience. The fact that your fridge broke down isn't necessarily representative of the product.

Another task Google is really bad at, is finding results by lyrics. As a music enthusiast and crate digger I often search for obscure, rare music I hear in sets, mixes and bootleg recordings. If there are vocals it's an obvious attempt to search for the lyrics - mostly without results. On the other hand the song I'm looking for is very likely up on Youtube, which has a surprisingly large selection of rare music uploaded by other collectors. Yet it's impossible to find those uploads by searching for the lyrics featured in them. Google should use its powers to auto-transcribe music on YouTube to lyrics. Sure, there will be a lot of mistakes, but it should be enough to give some results.

It's also useless for finding specific product information.

I was trying to find troubleshooting advice for a particular model of printer and no matter what keywords or modifiers I tried all the results were for low-quality shopping or review sites. Impossible to find actual information over the flood of SEO 'content'.

I agree, totally. I used to wonder why I'd find reviews that feel like they are trying to "sell" me something instead of providing an honest review. It's extremely frustrating and seems to be more and more of an issue (thinking back to the HN thread a month ago - The war to sell you a mattress).

This is an interesting series on fake reviews: http://www.sitesell.com/blog/2017/07/fake-reviews.html

The company became interested in the topic because a competitor was training their affiliates to write fake reviews of the company's product as a bait-and-switch tactic. (They later did a study with methodology that showed their product does a 33X better job at enabling individuals to create high-traffic sites.)

Some of the worst niches, I’ve noticed, are those that have the potential to prey on people due to the nature of needs - health, make money online, pain relief, weight loss. If you can find a fair review when searching on product reviews, it’s pure luck.

I imagine that Google’s AI component of the algorithm should eventually be able to recognize and penalize reviews that actually do the customer a disservice. Until then, some of the suggestions in this thread are excellent.

Hope you find the link useful.

Alta Vista allowed LexisNexis style querying against the web. It was magical. I only reluctantly switched off of that search engine after they quit indexing the web and their database became uselessly stale.

I always hoped that Google would add that functionality. Perhaps as a premium, paid service. But that ended up not being the business model they went with.

What is 'LexisNexis style querying'?

I think he's referring to the ability to use boolean expressions and other more sophisticated operators. See, for reference: http://www.public.iastate.edu/~hschmidt/srch.html

This was very powerful and enabled you to do very powerful keyword searches. In some ways a very different way to search. I was using this for some truly deep searches of internet sites with the content being very much from the original web. The original web being what this article is about.

Nowadays if you 'deep stalk' someone or some thing then you will find their social media posts and profile on linked in.

Back then you were far more likely to be finding them on forums or sites they created, even on those 'guest book' pages - remember them?

Alta Vista and the advanced search functionality was useful up until 2003 or so, then they became part of Yahoo! and we all know what happens when that happens. I stopped using it then but also my particular use case had gone, I was doing different work and no longer needed the search results beyond the search results, i.e. where the whole internets is bisected for the match of two or more particular search terms and their negations. Happy times, but you did have to have the skill to search like that, it was more like using the index cards in a library than browsing the shelves, a mini superuser trick we no longer have even in Google 'verbatim' searches.

This brought to mind this site which got attention several years ago:

https://millionshort.com/about

Maybe 5 or 10 years ago. It's a search engine that filters out the top N websites.

However I just tried it and it seems to be poorly maintained -- it's crashing on certain queries, and giving me captchas after only a few searches.

Some people noticed this problem many years ago, and here is a solution which didn't quite work. But it might be something to learn from in attacking this problem.

Wow, that's really neat! Thanks for sharing that.

Is that information even there anymore? I've sort of taken for granted that like good API documentation, whole subjects simply don't exist on the web in any depth. Maybe it is strictly a google problem, I definitely noticed a decline in quality when they started to combat link spamming. I guess at this point gaming google (sorry SEO) means that only people with commercial interests show up in the first 20 pages of any given search result. The result of most searches these days is either some reddit or stackoverflow like discussion board after the obligatory first page of links to pages trying to sell you shit.

Assuming that the information even exists, in a way, I think actual human curated links (aka like early yahoo.com) might be a better way to search if you could hire a few dozen people to start building them. I've even went down the thought process of building my own webcrawlers for the explicit purpose of building a personal search engine so that I could kill-file the few dozen sites that tend to show up as the default answers to 99% of google queries.

I find the same problem with ebooks and other things.

Nowadays, I just use blogs and ebooks to better guide my searches to find the (print) books or academic papers that I need.

I'm sure that information is still there, but Google just refuses to index or let you know it is there.

It's almost coincidental how the quality of Google's results fell dramatically, shortly after the death of https://en.wikipedia.org/wiki/Fravia .

does anyone remember web rings?

i remember web rings. i remember a full circle of sites providing 99% of the same content because it was easier to keep a website updated reusing the same popular content. and since the webring is the #1 source of popular content, it becomes a death torus into total homogeneity.

They still exist, they're just not called that anymore. The modern term for them is PBNs (private blog networks) and they fulfill the same purpose.

> search engines like Google have been optimising more and more for the former content, making it harder to find the latter

To emphasize this point, just the other day my 4-year-old asked me "how do seeds inside tomatoes grow?" The answer to such a simple question cannot be found in either DDG or Google because there are literally thousands of sites in this search that helpfully tell you how to grow tomatoes from seeds - the obvious search engine optimization to look for anything with "tomatoes", "seeds" and "grow" being definitely wrong in this case.

What Google also loves to do is to show you results with a one term striked-out. The striked-out term happens to filter results the most so it's the most important one. Google decides to serve you more popular (and useless) search results.

They (Google) have a feature built-in to the search engine so that if you put all the terms you want to absolutely see in quotes, for example: "tomato seeds grow", it won't return results without any of those terms.

Note that it will only return results with pages that have those terms in that order.

if you wrap each word in quotes it works the intended way.

eg. "tomato" "seeds" "grow"

Another way to do this is start your search with "allintext:"

    allintext: tomato seeds grow

or +tomato +seeds +grow

There is the Verbatim search option for Google text search, which keeps me sane, but frustratingly there is no equivalent for Google Image search.

Quite often I'll search for an image of, say, a specific aircraft registration and be shown lots of other aeroplanes of the same type. Very clever, Google, but useless.

Unfortunately if you use verbatim you can't filter by date. That has been quite frustrating sometimes.

You might already be doing a lot of this, but using the special operators makes Google a lot more useful. Using quotes around text forces it to be included exactly as written and removes the "struck out words" issue the other comment on your post mentions. 'OR' logic is helpful as well, as is - (e.g. -Word) to exclude words.

The full list of operators is here: https://support.google.com/websearch/answer/2466433?visit_id...

Or there's also advanced search: https://support.google.com/websearch/answer/35890

DuckDuckGo has much the same set of operators.

In terms of how seeds end up inside fruits, Google found me this: http://scienceline.ucsb.edu/getkey.php?key=640

> the obvious search engine optimization to look for anything with "tomatoes", "seeds" and "grow" being definitely wrong in this case.

I'm going to side with google for once: you should be doing your search in multiple steps, acquiring more specialised vocabulary as you iterate on your searches. Starting from seed, you'll end up learning that they're fertilised eggs, which you'll use in your second query which'll start digging you into domain-specific academia where younll assimilate enough keywords to have a pretty good idea of what you're looking for in domain-appropriate terms

Just ask that exact question literally as if Google was a human being. That's the key to finding answers to more obscure questions. Use natural language.

To emphasize this point, just the other day my 4-year-old asked me "how do seeds inside tomatoes grow?"

Does the query "seeds growing inside tomatoes" (without the quotes) give you any relevant results?

The quoted string returns just one unhelpful result for me.

Contrary to the GP comment, the unquoted string returns pretty much the exact answer to the 'question' as the top result:

https://uconnladybug.wordpress.com/2014/04/11/seeds-sproutin...

> hormone controlling the seed dormancy is exhausted or runs out, letting the seed grow in the moist environment inside the fruit. This warm, moist environment is perfect for germinating seed to grow. If the tomato was left uncut in the warm conditions, the new plant sprout would eventually poke through skin of the now decomposing tomato.

That link has nothing to do with the original question.

It's the answer

You want in-depth information. Books are the go-to media for that.

Try Google Books.

The Internet encapsulates all media.

That is true, in any specialised field (apart from, for some reason, programming, which seems particularly great to search for) it does very badly. For example in Physics it can be impossible to find the answer to very reasonable questions. For example, try googling "Fabry-Perot cavity transfer function" which in my field is a very simple and fundamental concept, and you get nothing relevant at all

Oh god yes.

The other day I saw a marabou bird in a wildlife documentary and was surprised by the hair-like stuff on its gular sack (the pink bag hanging from its neck). Wikipedia doesn't mention it and if you search for "marabou feathers" you only find that the world apparently obsesses about the fuzzy white crest on its neck that inspired fashion some decades ago. "Marabou hair" and "marabou hair feathers" likewise only yield information about fashion accessories. Even if you explicitly include "gular sack" Google unhelpfully tries to lead you towards information about the crest feathers.

This is definitely a case where even if you think you know all the relevant words you're unable to phrase a search query in a way that convinces Google you're not interested in the generic results everybody else is likely looking for.

And this isn't even a case that matters much. I was just trying to find out whether those feathers are actually a different type or just look more hair-like for some reason. It reminded me of a video I once saw about primitive proto-feathers we now know some dinosaurs had (affectionally called "dino fuzz").

In case you didn't find the answer to your question, those feathers are called 'filoplumes' [1]. Relatively obscure anatomical detail like this is more easily discoverable in the scientific literature than on the web. I found the answer by searching Google Scholar for "marabou stork", browsing the results for a paper about the "gular sac" (although it has other names), then reading the most promising looking paper - the answer was in the second paragraph [2].

1. https://en.wikipedia.org/wiki/Feather#Classification

2. Akester, A. R., Pomeroy, D. E. and Purton, M. D. (1973), Subcutaneous air pouches in the Marabou stork (Leptoptios curmeniferus). Journal of Zoology, 170: 493–499. doi:10.1111/j.1469-7998.1973.tb05063.x https://sci-hub.io/http://onlinelibrary.wiley.com/doi/10.111...

Ah, interesting. Odd that even the article only describes them as "filoplume-like" not simply filoplumes. But based on what I find when I search for "filoplumes" it seems their function and evolutionary origin isn't entirely clear.

I found http://lygte-info.dk/ (website about batteries, chargers, and more stuff like that) through a search engine when looking for specific information about a LiIon cell. But then I started clicking through that maze of links and found so much useful information that I would never have thought of searching for. I spent hours "trapped" on that page.

The navigation is crappy (randomly interlinked handwritten HTML-pages) but the content makes up for it. The article by tttthis very much resonated with me. Many modern sites look great but don't have all that much actual content.

If you're interesting in electronics and software in general, and can read German (or can stand the surprisingly good Google Translate), another site I recommend: https://www-user.tu-chemnitz.de/~heha/

For Dutch, try tweakers.net.

Jesus, that website must have taken years to make. The reviews are so in-depth, fantastic. I've been looking for someone that reviews USB chargers, as they go into mains voltage and are dangerous if not shielded properly. Amazing resource, thanks for linking it!

I recently tried to search for certain Nintendo 64 ROMs because I wanted to try out an emulator. The first couple of results on Google all lead to horrible, ad-infested websites that claimed to offer a download but either kept on redirecting me to more ad-filled sites, or tried to get me to download some sort of "download helper", "media codec" or "PC cleaning software". Then there were the sites with dozens of fake "Download" buttons surrounding the real one. I'm still not sure whether any of them actually offer any ROM downloads (I found trustworthy sites on the second page of the results). It's no wonder that many users struggle to keep their PCs free of adware, toolbars and similar junk, when it is shoveled into your face every step of the way.

bittorrent would probably have been easiest since it's already ranked by popularity.

archive.org is the unofficial official source for ROMs on the Internet now, FYI.

Ah yes, I didn't even think of that. But now that I've checked it out, it seems like they only have ROMs up too the SNES, and none for the Nintendo 64 ("only" videos, soundtracks and manuals)

You wanna search by obscure-to-outsiders collection names. “No intro”, “tosec”. That kind of thing. Pretty sure they have n64, in with the rest of a large multi-system collection.

I find emuparadise to be decent for ROMs without anything fancy.

They are really good, but they don't have everything (maybe due to copyright complaints, specially for Nintendo games which might have been re-released for new platforms.)

This was the case back in 1997 as well.

The only thing that's changed about shady emulation sites in the last 20 years is that they no longer give you ill-conceived legal advice about ROM ownership being legal if you delete the ROM within 24 hours and/or are not affiliated with law enforcement.

>Nowadays, anyone with a basic understanding of search engines can find almost anything they want within seconds.

But now I'm less likely to find stuff that I didn't know existed and don't quite have a name for. This kind of stuff was/is usually discovered by following hyperlinks. But I'm not sure why I have this experience less often.

Browsing vs. searching. We don't seem to do much browsing anymore. We search and selectively pick what we consume.

It's faster but, as you say, I don't think it is as fun. I used to enjoy picking a subject and then clicking all the DMOZ or Yahoo! links. I'd dig through webrings and and link aggregators with wild abandon.

Now, I search a term and click on the first few results, at most. Sure, there are exceptions but they are rare. As you say, it's much more useful.

>> I used to enjoy picking a subject and then clicking all the DMOZ or Yahoo! links. I'd dig through webrings and and link aggregators with wild abandon.

Yep, that's exactly what I used to do. That's really what "surfing the web" meant back in those days -- that you would spend hours just clicking links, seeing where they would take you. And of course, they would quite often take you to completely unexpected but cool things that you never would have found or even thought to look for otherwise.

Is that any different from spending an hour finding links on Wikipedia or HN or Reddit or Facebook or whatever you're into?

Invested effort yields quality.

It was/is different: folks who are excited enough about something to build a website to share it have more interesting and deep content than you typically find on social media.

These days The same goes for people with personal blogs (or website) as opposed to medium and other one design blogging platforms.

People don't seem to build personal websites in the same way anymore - back then they weren't doing it to generate clicks, page views, passive income, build their profile etc, and everything wasn't all slickly styled and presented on a modern blogging platform. As a result websites were more amateurish, idiosyncratic, original, and sometimes charming, and I think the essence of the creator's motivation or interests often came across more strongly. The web doesn't have the same magic it used to for me. My god though some things were crap about it back in the day. If only we could have have kept the old web preserved alongside the new.

The thing is, people do build personal websites, and that old web still exists. Do you really believe that, with (according to Quora) ~140,000 new sites being created per day, no one is putting amateur, personal stuff on the web anymore?

It's just that the web has gotten bigger and more mainstream, and most of us now limit our browsing to a few silos and then act as if everything that no longer bubbles up the filter must have ceased to exist.

But so many personal websites are just self promotion of one kind or another, and that's fine, we all have to make a living, but I'd far rather read something where someone just wants to share what they know/love. And everything looks like the same set of off the shelf web design templates. I liked the weirdness of the old web. Kind of like games on the Amiga, every site was different, and that was part of the charm.

But perhaps you are right. Any tips on how to find the new gems?

>Any tips on how to find the new gems?

That's a good question - I don't know. Discovery of anything not SEO optimized is difficult, and a lot of that content was on Geocities and Tripod, so the modern equivalent might just as likely be on Wordpress or a service like Wix, making the design homogeneous even if the content isn't. Ironically, I've found most off the beaten path sites through references in Youtube videos and Reddit. When I was getting my CS degree, the web design class involved building a quick personal site, so URLs connected to colleges might also be a good place to look.

But what's more likely, that no one is writing their own custom HTML sites anymore, or that some people are, but we just can't see them because they exist in their own ecosystems beyond the Googlesphere (and possibly even the Anglosphere?)

...I just realized I'm vaguely arguing for (but not defining the parameters of) a Drake Equation for the web, where N is new personal content. Even if the parameters are low, the scale of the web means N should be significant.

My recollection is that for every interesting personal website with carefully compiled albeit idiosyncratic information, there were 10 sites along the lines of "Mikey's Ultimate Web Site!!!" with animated "under construction" GIFs, marquee tags, a guestbook script, and about ten webrings.

For better or worse, the closest thing today to the feeling of the old personal webpage era is Tumblr. The kids get up to some crazy stuff over there.

The stuff I submit to HN or comment on that wows people often came from me digging into websites you dont find with a quick Google. Esp academic web sites or just those made by smart folks with odd little organization schemes. Im guessing same was true for some of the more obscure, but valuable, stuff I see others submit or bring up.

Some websites used to have a "webring" where they would link to each other and you could just click on "next" and it would take you to another site on the same topic.

I used to like those.

I'd go further and say: browsing vs scrolling.

I remember 10+ years ago we designed websites so that important content was "above the fold". Today people are addicted to scrolling their Facebook feed.

Sites that cared about "above the fold" were part of the problem, not part of the solution. The primary reason one would be actively interested in putting content "above the fold" was if one wanted to "capture audience", which is generally not the first thing on the mind of a person posting deep information as a labour of love.

The Facebook feed is necessarily long (in fact infinite) and low-density, so scrolling it is a necessity.

But individual entries are definitely advanced examples of "above the fold" design that is good for advertisements, not for pages.

> We don't seem to do much browsing anymore. We search and selectively pick what we consume. It's faster but, as you say, I don't think it is as fun.

Dunno, how much of this is also just getting older and set in our life routine? When was the last time I explored my own city on a bike like I would as a kid? Our interests and patience for browsing change.

How old are you now vs in this "golden era of the web"?

Most of those stories just sound like nostalgia to me. Like suggesting that games these days aren't as good as they used to be because look how less time you spend gaming these days.

When I first used networked computers, you paid someone to do searches for you and got a reply in 3 to 7 days.

It pretty much sucked, honestly. Translation was obscenely expensive and not much better than Google can do today. You paid by the minute or, in my case, the university did.

It was pretty terrible.

Then, this fancy world wide web came about, only it had been world wide before that, albeit at a different scale. It was pretty great.

Now? It's much better in that it is much more useful. I can still make discoveries of novel content, but I seldom do. I search, rather than browse, when I seek information on a subject.

It's not nostalgia, I don't think. It's much more useful today. It was pretty terrible the first times I used it. In fact, it wouldn't improve for like a decade, maybe longer.

If it helps, I also hated computers back then. They were slow, unintuitive, and I had to learn to program to actually have a functional (well, useful to me) device. They clanged, beeped, and whirled. They made dreadful noises and did exactly what you told them to do, even if what you told them to do was stupid. They didn't have things like a recycle bin. Hell, they didn't even have internal storage.

One computer was so horrible that I had to buy 'memory chips' (RAM) just so I could use lowercase letters. All told, it cost me more than a brand new car. You know what I did with it? I played Zork.

Edit: that translation was done by a human, by the way. It could take a week, or more, to get something translated.

As for games, in the 80s, no one knew what worked or didn't work as far as computer games went, so everybody was pretty much throwing everything against the wall and seeing what stuck. And you ended up with some memorable games [1][2]. But over time, people learn, genres develop, budgets rise and you get less original and more homogeneous games.

It's the same with the web. When it went big in 1994, no one knew what they were doing, so everybody was pretty much throwing everything against the wall and seeing what stuck. But over time, people learn, genres develop, budgets rise and you get less original and more homogeneous pages, only this time with so many ads that it takes 500 requests to download your typical page these days.

What I think people are missing is this original, outrageous and anything goes type of experience. But I suspect the Cambrian explosion of web expression is dead for the time being.

[1] Qix (https://en.wikipedia.org/wiki/Qix). An abstract game that was mainstream and quite popular at the time. Haven't seen much like it sense.

[2] "The Earth Dies Screaming", a game I've never seen but the name alone is so evocative that I still remember it to this day. I think it was in the "alien invasion" genre popular back then.

i don't think browsing's that rare, actually, it's just different, having been centralized by facebook, tumblr, wikipedia, reddit, and the like. it's less like going out and exploring the web and more like just living there.

Living there and watching TV instead of going out.

> I used to enjoy picking a subject and then clicking all the DMOZ or Yahoo! links. I'd dig through webrings and and link aggregators with wild abandon.

I sometimes still do that with Wikipedia articles and their sources, it's a fun way to kill a couple of hours and can lead down some really interesting rabbit-holes.

Useful? I would argue that it's actually a lot less useful today.

Please do ! I'll gladly remove this comment if you give an outline to your argument.

This is kind of meta, but I think we'd be better served if you wouldn't remove the comment even if proven wrong. It serves everyone better to see both sides of the argument. It would be more appropriate to reply conceding that your initial argument was incorrect, or to add a note on the bottom of your original comment indicating the same.

Sorry, "this comment" referred to itself, I did not post the parent comment. The conversation should not be hidden, just this barely useful meta comment.

Hey, I'm putting some thought into a more thorough reply. I have a reply partially written at home but am now at the office lol. I see the comment thread has garnered a lot of replies in the intervening time though, so I'll just post a quick tl;dr:

okay yes the web is more "useful" to most people, and maybe only slightly more useful to me, but that usefulness is significantly offset by some majorly-serious concerns about privacy, performance, and reliability.

I'll have a better reply later :)

Okay, I'll clarify. Is it more useful? Yeah, depending on what you want to do. But it's also a hell of a lot more dangerous. It's especially more dangerous if you're not careful.

+ Google Search is about as good now as it was back in 2007. I like that its interface has (largely) changed.

+ Google Maps with smooth scroll is nice, and its integration to 3d street view is nice too.

+ 1990s MapQuest without AJAX was nice though, in its own way. Even with Google's smooth scrolling, the only real thing Google Maps has going for it is IMO a cleaner display.

+ Wikipedia's generally better today than 10 years ago, yes.

+ Airline websites are a nightmare to use. Try getting an optimal route that doesn't use a single airline; and try getting that optimal route without ended up getting spammed or surcharged.

+ Youtube having virtually every song is nice for some people. It's not so nice for me. The songs on there are usually pretty boring or low quality audio or video or both. I pay for a music service (di.fm) for a reason.

+ Reddit/HN are pretty neat.

Okay, that's about where we agree (along with your comment on Facebook).

Let's look at some of the downsides.

- Personally, I find Reddit's method of linking to articles vs loading comments to be counter intuitive and confusing.

- Ebay is still about as trustworthy as you can spit.

- Amazon.com is among the crappiest web interfaces I've seen, and that's ignoring the company's history of abusing compensation and livelihoods of their employees and contractors. It's slow, extremely noisy (which contributes it to being slow), difficult to navigate, and I definitely never intend to buy from the marketplace specifically because I might as well have bought stuff from ebay.

- Just about every website has tracking up the wazoo. That contributes to a slower web experience both in bandwidth cost as well as hardware performance. Not everyone has a wonderfully new computer to handle all that. And if you're in America, chances are pretty good that your bandwidth is sub-par as well.

- Email. Why do we still have to deal with phishing and tracking? This is something that has been around pretty much since HTML was introduced to email.

- Instant Messaging. Slack and Discord are the newcomers but they're also somehow the best. They're popular because they get it "right"; it's super easy to text, talk, video, share files, share screens, etc. But they also get it wrong. That Electron UI has got to be the one of the biggest memory behemoths I've ever seen. Both are yet more walled gardens.

- The newer versions of Skype are absolutely dysfunctional compared to the older versions, MSN was better, AIM is (was) better, even IRC does file transfer better more reliably (although optional and clunkier).

- Mobile Apps. JFC banks going to mobile apps when the marketplaces are full of fake apps. This is absolutely not user friendly. Build on top of that the fact that phone numbers are super easy to fake or steal and you've got a clusterf!@# personal security nightmare just waiting to happen.

- Phones, operating systems, applications which track absolutely everything you do. It eats up my bandwidth, my system resources, and provides no visible benefit to me whatsoever. What happens when someone figures out how to track what I say to whom and when, and use that to "guide" me to buy things?

I could go on and on about more stuff but I don't care to at the moment.

I'm just going to leave this here...

https://tttthreads.com/thread/927593460642615296

I grew up with 1980s dial-up modems (slow 300 baud!) to dial directly into BBS systems and USENET. I then continued dialup access into the early internet (GNN not AOL) in 1995. I now have Verizon fiber optic.

To me, the modern web today is better in virtually every way compared to 1990s web surfing.

+ Google search engine is better than Alta Vista and Excite.

+ Google Maps with smooth scroll, 3D streetview, and satellite view is better than the clunky 1990s MapQuest without AJAX. Google Maps also uses colors to highlight traffic congestion.

+ Wikipedia is better than the non-existent online encylopedias of 1990s

+ airline websites now bring up a graphical map of the airplane and let you click on the actual seat you want. When I booked etickets from airline websites in 1998, it was just rudimentary ticket purchase.

+ youtube obviously has videos but it also has virtually every song. If some music forum mentions an obscure song, I think to myself that someone else may have already uploaded it to youtube. Sure enough, 99% of the time, it's there. It's even better than Spotify in that respect.

+ discussion websites like reddit/HN are better than Usenet. The better replies are voted and sorted at the top. I learned a lot of C Language from USENET "comp.lang.c" in the 1980s and I'm jealous that today's programmers have StackOverflow which is better. Again, the superior answers are voted to the top.

+ Amazon.com also made the UI better. In 1990s, they didn't have "Look Inside" feature to see sample pages of books. I use that all the time. One thing they did make worse is you now have to manually click "see all 50 reviews" to read all reviews; I assume they did that to prevent web scrapers. In any case, Amazon is good enough that I stopped wasting gas driving to Barnes & Noble.

I'm having a hard time thinking of a 1990s web experience that's superior to now.

I can only assume that a big part of why some think that today's web is worse has to do with Facebook and its walled garden. To those people, the web feels "less open". Well, I don't have a Facebook account and none of my longtime friends do either. Therefore, it doesn't feel like the web is any less open to me.

The web is more useful now in 2017 than 2007. And back in 2007, it was more useful than 1997.

Respectfully disagree on more votes == better. Visible score systems often lead to groupthink, and commenters aiming for the lowest common denominator in the form of karma whoring.

While HN has some measures to reduce hivemind voting, it's quite rampant on the default subreddits.

While this can be true, Usenet was so much worse in every way.

It seems to me like HN still has a long way to go. I find my detailed, careful, well researched comments typically get an order of magnitude less votes than my one sentence inanities.

I doubt it's fixable - it's just humans being humans. If you write a short one-liner, anyone can quickly evaluate it and decide whether they like it or not. When you write a long essay in a comment, a lot of people won't upvote it because they won't even bother to read it, and others may find that they disagree with one of your many points but don't have time for a thorough response, hence they just won't upvote and move on.

Yes. Upvoting and downvoting are the brokenness; users are just interacting with the UI model they're presented with. Better low-effort, high-reward consensus systems are needed.

I've actually changed the topcolor to #222222 in my settings so I don't see what my account's karma is. It completely ruins the ability to see the "new" "threads" etc links without a truly irritating amount of eyestrain but it's worth it - I recently asked genuine question about something I wanted clarification on and the comment ended up being downvoted to -4. Now I can't see my karma score I don't care about this, whereas before when my karma was shown on almost every single page throughout the site I'd obsess over making my comments universally good.

One simple improvement that would apply both here and on reddit is to move the up/downvote arrows to the end of the comment rather than the start. At the moment it's disproportionately easy to upvote a short comment because you're still at the right place for the arrow.

Voting is gamed by corporations and governments -- easy with a few thousand IP addresses.

Comments are manipulated by teams of paid "trolls", including in English, and including Hacker News (when the article is of interest to a particular corp or gov -- like the Tesla mass firing article on Hacker News recently, just to give an example I know of).

The web is better today. And personal home pages or web pages are still around in the form of blogs, the only problem is that most people don't blog.

> discussion websites like reddit/HN are better than Usenet

I strongly disagree with this, as Usenet and even mailing lists offer many advantages over web forums. With web forums, I can't natively...

- Store, read, search content offline

- Read only new messages

- View thread context

- Access information redundantly (no SPOF)

- Killfile senders

- Integrate viewing and composing with local tools

While a particular site mitigates its disadvantages, perhaps via a better UI or via information not available elsewhere, the medium as a whole cannot. A website might be a fine view of information, but it should not be the only interface to that information. Doing so prevents all of the above uses and makes it much harder to reuse content in ways unforeseen by the original site. I suspect this is the root of the problem -- such reuse is not desirable for most sites, as it conflcts with their business model.

Reddit covers almost all of those and has as wide a range as Usenet did. In terms of the broader web, there's a tension between wanting customizability and wanting consistency; realistically the only thing most people do with any given content is read it, and so the web focused on that use case and things that helped with that at the expense of offering consistent structure.

>Usenet and even mailing lists offer many advantages over web forums.

It's interesting you didn't list any disadvantages of Usenet. NNTP protocol doesn't have identity authentication. Therefore, you get spam and forged headers. This isn't any fault of NNTP; it was built for a different era. From a content standpoint and not a technical one, USENET was at its peak, before 1990s, when it was primarily universities and governments. The smallness of the tech community and the inherent collegial atmosphere didn't exploit the vulnerabilities in NNTP. I'll agree with others that it all rapidly declined after AOL opened it up to the masses. To regain the "reading pleasure" I got from USENET in the 1980s, I have to use Reddit/HN/Stackoverflow.

You listed the following technical advantages:

>- Store, read, search content offline

>- Access information redundantly (no SPOF)

>- Integrate viewing and composing with local tools

I'll group those 3 bullet points together since they are related to each other. I agree that those technical advantages are true for USENET. Your comment spurred me to revisit my USENET usage history and attempt to articulate why it's no longer relevant to how I use the web.

I participated in USENET from ~1988 to 2006 -- so about 18 years worth activity. The last years of downloading and reading messages was a tedious chore. With Usenet messages having no voting system, I had to slog through a sea of bad replies to get to the good comments. It was a horrible waste of limited reading time.

Once I got hooked into Reddit in 2006, reading crowdsourced comments became more of a pleasure again. Yes, I 100% agree that voting leads to "groupthink" and "hivemind". But even with those problems, catching up on 500 unread messages ranked by votes instead of 500 messages in pure chronological order is easier. I already suffered through unvoted messages for 18 years. I don't want to go back to that. Experiencing Reddit made me realize that the "terrible voting system" is the unofficial way we combat the "Eternal September"[1]. (Also, the Google search engine was better than anything else because it weighed the "unoffical votes" of links in the PageRank algorithm.)

Forum voting is like that saying about Capitalism -- "it's the worst forum system, except for all the others".

Usenet was like doing homework. Reddit is like a quick dopamine rush.

It's good that both Reddit and Usenet can coexist. A subreddit like "reddit.com/r/cpp/"[2] can coexist with USENET "comp.lang.c++"[3]. Although I have almost 2 decades of training to compose offline messages for "comp.lang.c++", I have no urge to do so. The technical advantages for Usenet you listed doesn't offset its flaws in content. These days, I just go to the C++ subreddit to get my daily fix on what's new because the content is better presented for limited reading time.

[1] https://en.wikipedia.org/wiki/Eternal_September

[2] https://www.reddit.com/r/cpp/

[3] https://groups.google.com/forum/#!forum/comp.lang.c++

I won't deny that the current Web is much more useful, on average, than it was back in the nineties when I started using it.

On the other hand, during that time the technology became immeasurably better - my first dial-up connection was 9.600 kbps if I remember correctly, now I have 4 Mbps; I didn't have a math co-processor back then and now we have multicore monsters of CPUs - and so on.

The question (to me) is not if the current Web is more useful, but if the increase in usefulness is anywhere near the technological progress?

At some point, the technological limits - like no YouTube because both the necessary download speeds and compression algorithms were not there - were replaced by something else. Back then we could believe that, in time, seeing the rate of progress, we'll get everything we ever wanted. The tech advanced and now we could have all these things we dreamed about, but don't - because of political and economic factors.

Yes, the Web improved drastically, despite a couple missteps (like IE6) along the way. But I believe the improvement is much lesser than the current technology would enable were it not for non-technological factors.

On the brighter side, once developed, technology is here to stay, so if we deal with all the current problems we'll be free to advance the Web further. But that's a pretty big "if"...

i agree in total todays web is better, but today its so closed, and we lost a lot as well.

- excite chat was better than anything we have today, irc chat remains awesome.

- on social networks of yore, you could actually meet new people. you could search for new people and expand your circle to people completely outside it. I believe its against facebooks TOS to try and connect to someone you dont know. i know my account was banned for 3 days because someone declined with 'they dont know me'.

- youtube sucks for viewing songs. like really sucks. you have commercials MID SONG. one time I had a playlist play a 15 minute long commercial. FIFTEEN MINUTES. half the songs in my playlist end up getting removed, and It doesnt tell me the song it use to be. so if i find a song i really like, add it to a play list and it gets removed, i completely forget about it. song lost. if you are going to compare songs, youtube losses hands down. napster / audogalaxy / was far superior. youtube is still great for watching a cat jump in a box, after watching a minute long commercial of course.

- google search is great, its superior to alt vista, which is why i switched from alt vista to google when it came out. but google gets stuck, if it cant give you want you want in the first 5 hits, it basically cant do it ever. you have to change search strategies. alt vista was bad, but the content was diverse. you could still find gems 5 'nexts' in. google also gets locked in on what it knows about me. sometimes i have to search in incognito mode to find what i want.

the 1990s web experiences was not just about how well content was delivered to you. you could chat a person you never met with a simple ASL. maybe you continued talking after the first day, maybe you didnt, but it was fun and you met new people all the time. people dont want to meet new people anymore.

people dont email or contact eachother anymore. before people would put effort into staying in touch. i remember checking emails with excitement. now even though you can talk to your network at any day, because you are always connected people dont. and its just a 'hey whats up'. people dont write long responses saying how their day was, and how their cat got locked int a closet, etc.

edit: i forgot to say the original digg was awesome, and in may ways better. also, I think most of us are comparing the web from 1998 - 2006, not the late 90s.

I think the Internet's gotten worse in general (and, specifically, less useful) over the last decade except for one thing:

YouTube is insanely useful for learning how to do basic household stuff, minor car repairs, that kind of thing. Insane.

Mundane? Yeah, but life is like at least 50% handling mundane shit like that. Maybe 90%. Maybe 95%?

Want to know how to repair some brick stairs in front of your house? Sharpen a lawnmower blade? Mulch a garden? Seal a driveway? Change the cabin air filter on a 2008 Mazda 3? Make pancakes using campsite cooking tools? Multiple videos for each. Perhaps even dozens.

Exactly, we're in the golden age of vlogging.

YouTube is also incredible for educational stuff. There's very pop science, pop science, hard science, and then things like hundreds of lectures from Leonard Susskind on everything from classical mechanics to the latest happenings in string theory/m-theory.

Phenomenal stuff, but it seems like the things that most places (universities in particular) are still holding out on are things like problem sets. Hard to know how little you know until you have materials challenging your depth of understanding. Watching a lecture without doing associated problems is like trying to learn to play basketball without touching a basketball.

Yes. I think Youtube inherited the culture of the old web. Lots of amazing channels about obscure interests. Household repairs but also welding, machining, woodworking, electronics repair, obscure retro tech, distilling, etc.

I guess it forces the presenter's personality into the spotlight a bit more than pure text and images. Probably attracts a slightly different kind of person. But I think the same spirit is there.

It would be kind of cool to make a modern network of sites robots.txt'ed out of search engines.

It'd also be cool to make a search engine that catalogged all of the websites that other search engines politely did not.

Is there any proper enforcement of robots.txt or is it just polite? Because I don't see how one search engine could maintain an advantage if others start catalogging every site.

Well, most sites want to be discoverable, so the incentives are aligned for the time being.

It's polite, but the teeth behind it is that it's much easier to secure a CFAA conviction for scraping if you can show evidence that you specifically forbade a scraper from accessing your site (via robots.txt, authentication, or IP blocks) and they circumvented that restriction than if you gave them no notice that you dislike what they're doing.

Legit search engines and aggregators have a strong incentive not to kill the golden goose by getting sued, particularly when the vast majority of sites actually want to be indexed.

I remember the days pre-google when Yahoo was king. And most sites were filed under a topic/category. That was the default way to find a group of somethings. It's also a little suggestive and explorative, whereas keyword based search engines relied more on you being more specific. Crowd stoked search results and AI have bridged the divide these days. What the author failed to mention was web rings. Basically authors with like minded material would link to each other. So once you found something you could hop to something else and bypass the search engine. These days you are almost crippled without the search hand.

> but the thrill of discovery when you stumbled across a gold mine of information was a huge part of the appeal.

To be fair, this feeling is very similar to what you feel when finding something really great in your Twitter queue.

Stumbleupon ? :P

Do a lot of people actually use it? It seems like you could just as easily replace that site by simple tweets when you actually stumble upon something worthy. Which is what most people do IMO.

Based on some very perfunctory research: they certainly used to have a lot of users, but I get the feeling they have declined over the past ~5 years.

Anecdotally: I used to use StumbleUpon, and found it fun for a while. I'm not sure what caused me to drift away from it, but the reason I have no interest now is primarily that I am drowning in interesting content. (I have enough articles saved to Pocket to keep me going for years.) Getting recommendations tailored to your interests is nice, but unless the personalisation algorithm is really good, I think you can do better by cultivating your own set of sources and intermediaries.

I use it less then before, mostly because there are a lot of old articles

Rather, nowadays, the search engine has a basic understanding of you and obliges you with what you think you want.

Reply on news.ycombinator.com