Hacker News Reader

Comment by trevex | Hacker News Reader

I unfortunately share your experiences. However in Germany there is a foundation called “Stiftung Warentest” [1], which does independent testing of products. With reviews full of Amazon affiliate links dominating search results, I tend to purchase their tests for ~3 Euros more often than ever.

However this only works for popular product categories, but less so for specialized equipment.

My impression is also that affiliate links hurt consumers in the long run as they reduce the selection of products in reviews or blogs to those the authors can earn money with. This however leaves out potential alternatives. More often than not the winner of product categories (at least those I was researching) of independent tests were not available from sites running affiliate programs. For example a consumer-grade lawn mower from an otherwise professional gardening company or a tent from a Scandinavian brand.

[1] https://en.m.wikipedia.org/wiki/Stiftung_Warentest

View on HN

Sadly, Stiftung Warentest often doesn't have the expertise to properly test many products. I often notice the shortcomings in product tests for products that I know and use.

I was reminded of this with their last test of 3d printers. Their test results where far from what everyone with experience in the field would consider accurate.

For some products, I agree, but realistically its about as good as it gets unless you do thorough research on a given product and are actually able to find a somewhat unbiased review.

Take washing machines for example. How do you know which ones are good and which ones are not? Public reviews from any website tend to only be a good indicator if there a lots of bad ones. I have 0 faith in the average consumer to accurately rate a product. Overwhelmingly negative reviews will clearly show a deficiency but positive ones are unfortunately more and more a gamble.

Stiftung Warentest isn't perfect, but they do, on most occasions, put in a high amount of effort to test products to the best of their abilities without any personal opinions. I don't know of a single other person/organization/website where that is the case.

Most washing machines offer quit similar features. The key difference is reliability.

Testing for reliability is expensive, so most likely Stiftung Warentest and similar companies don't do that.

No, that's exactly what they do test and test well. Their model mostly breaks down for computer technology and peripherals in that space. When testing a mouse they would test how often it can click before breaking down, and that's only slight hyperbole. Might be a bit better now in that area than back then when I read their tests regularly.

I bought their test of basmati rice and these monsters had an intern court the broken vs. whole rice grains in a 500g package.

I assume they only counted a sample of the the 500g, but it's funnier to imagine otherwise.

Wonder if Count Things can do that https://countthings.com/

But then Basmati rice is judged on grain length too. Wonder if they report like "22 cm of broken pieces per km of grains".

s/court/count/.

It took me a while to understand what you meant.

I mean that would be even funnier. Sorry for the typo.

My brand new LG turbowasher model lasted 3 months before one morning sounded like a hammer smashing something. I ran to the machine and a support bracket inside had broken. It took 2 months for them to replace it I was not impressed. It has since lasted 3 more years no issues but I doubt I would go with LG in the future since it was a horrible customer service experience getting my first one repairs under warranty. Had they helped me better after it breaking I may have said it was just a fluke and still recommended them.

I base those sorts of purchases on duration of warranty. Recently bought a dryer, and apart from a brand that cost significantly more, they all came with 2 year warranties. One reasonably priced machine had 4, so I picked that one.

Counterexample: Korean auto manufacturers Hyundai and Kia offer longer warranties than Japanese manufacturers Honda and Toyota because that’s the only way people will buy Korean cars.

Wonder if they are taking advantage of bath tub curve and 8 years is slightly costlier than 5 years to a manufacturer. In India, Toyota is selling rebadged Suzuki cars offering lower cost and more warranty.

Warranty is ridiculous. Of course it's good to have if something breaks early. But normally I want my stuff to last much longer. Our fridge broke last summer: 30 years old. It had one repair 18 years ago. Our washing machine is 22, zero repairs. I hope it still makes it while. Well, none of them were the cheapest ones. From central European factories which might no longer exist...

Some would claim these old appliances waste energy. I am not convinced. We need to heat our buildings here from September to May. Whatever a very modern appliance warms the building less the heating has to substitute. It's all energy and losses produce heat.

There is also all the energy required to churn out newer appliances that break and get recycled.

Getting one great one that will last for a few decades isn't a bad call. That's what I do personally.

And there are still ways to save, reduce energy impact. Moderation is a big one. Just be frugal and prudent with the appliances. That has a major impact and everyone could do more and capture those gains right away.

They ran 3 machines of each model 1840 times and always exactly recorded the statistics (is the 60C program really using 60C water, etc) and results to test for reliability.

As long as their methodology is clear you can at least judge their results for yourself and whether you consider them meaningful, even if it wasn’t perhaps as good as an expert could have done.

Rtings seems a similar project; some of their test results are free, some are paid. They have a wide set of comparison tools, they test for a lot of features and they document their testing procedures well; it's a really nice change from SEO spam articles.

I recommend their tests of headphones [1], I especially like how they measure breatability [2]. They also have a page about printers [3].

[1]: https://www.rtings.com/headphones

[2]: https://www.rtings.com/headphones/tests/design/breathability

[3]: https://www.rtings.com/printer

Used Gsmarena.com to narrow down which Android-based phone would be the best choice.

Ended up with the OnePlus 8T due to good battery life, fast charging and reasonable size.

Still very happy with that purchase, which wouldn't have been as easy without that site.

Finding review of tech gadgets and hardware is quite easy. Even niche products usually have dedicated communities and sites with in-depth knowledge. It's the boring everyday products and large appliances like washing machines that are really hard to evaluate sicne their reviews basically amount to affiliate spam.

Thanks for the links, but especially the printer one is exactly what I was talking about: They are all marked as unavailable on the site.

The US has Consumer Reports, which is similar I think... but yeah, only works for major product categories.

For kitchen tools, there's America's Test Kitchen reviews (from Cooking magazine, IIRC). It's a limited segment but high quality.

On the other hand, it has the same problem as Consumer Reports: they only test and review a single model which will probably be out of production before you need one. On the third hand, if one manufacturer consistently gets good reviews (OXO Goodgrips, for example)...

Cooks Illustrated is the name of the magazine, and I recommend it to anyone who ever spends time in a kitchen.

Some of the purchases I have made with their guidance have been (trivially) life-changing.

You are absolutely right! Sorry about that --- I zoned out on the magazine's name.

There’s also the NYT WireCutter which tried to do a good job isolating it away as an independent review source that you can trust.

Some of the WireCutter’s picks were fairly terrible, which makes sense: testing diverse categories of products is too expensive for affiliate links to cover. I’ve gifted dashcams on their recommendation that shot beautiful QHD but had MTBF measured in single digit months. The TP-Link C7 Archer was kind of a turd and within a stone’s throw of a decent router price-wise.

To be fair, MTBF is not something you can measure in a reasonable time for these kinds of review sites. Far better for this kind of thing is niche-specific youtube channels.

But in any case, what you're asking for here is a prediction of your future satisfaction with a product. It's a non-trivial problem even for the most innocuous purchases.

Will I like Lysol or Clorox wipes more? Who knows, and the reviews aren't going to beat first-hand experience in any circumstances.

I’ve written extensively about this recently, but these days with $300+ “gaming” routers using crappy sweatshop software on whatever Atheros router SoC, many users would be much better served with legit SMB routing / switching / wifi systems that are available for around the same price.

I've always found anything tagged as "gaming" to be lower quality and more expensive than the business machines.

====

This message sent from a Thinkpad business machine that I use for gaming (and ML, theoretically).

I mean this is a known thing in general; the best kitchen supply stuff you can buy is often stuff intended for commercial use, because it quite literally goes through the wringer with near-constant usage over long periods of time.

The problem is finding a place that sells commercial things to individual buyers. That, and sometimes what a commercial kitchen needs is vastly oversized for a regular house; you're probably going to set off your residential fire alarm very often with a massive commercial range designed for woks, for example, unless you also upgrade the ventilation, etc.

I switched to Unifi access points and a wired router and switch and am much happier with the result.

The consolidation of router, switch and access point means you can't upgrade individual parts. It's the modern equivalent of the TV-VCR combo and most consumers don't realize they actually can be separated.

I didn’t know what good Wifi until I switched to using some TP-Link Omada equipment.

If you run your own controller, you can set up a small network (router, PoE switch, and AP) for less than $300. Hardware controller is ~$90. A controller isn’t strictly necessary, but I don’t recommend doing a standalone setup.

Downside? It’s business class equipment and you need some idea what you’re doing. It’s not plug-n-play. Also, it’s layer 2 only. If you want mDNS across vlans, you’ll need to run a reflector. (Not difficult. It’s built into avahi.)

I'd love to read what you've written. So far, my research into commercial Access points hasn't really been that fruitful. I refuse any cloud based management interface. For the router, I use mikrotik which has been great but I returned the access points from them that I tried. In the end, my access points are Asus home routers because they were the best I found.

The C7 is a good router if you put OpenWRT on it.

> The TP-Link C7 Archer was kind of a turd and within a stone’s throw of a decent router price-wise.

Can attest. I finally ditched mine, got tired of it falling over multiple times each week.

I've got the A7, which I believe is the same thing except it has some sort of ability to enable Alexa control for something or other.

Mine has worked great. The only issues I've seen are (1) the traffic stats don't count IPv6 traffic, and (2) there is something odd that sometimes goes on when a connection ends that can result in packets from the LAN side showing up on the WAN side without the LAN-side IP address being replaced with the WAN-side IP address.

The first is a bit of annoyance, and the second as far as I saw didn't actually cause any problems.

Maybe you got a lemon I've been using mine for a while now. Not as nice as my old one with customer firmware but still consistent.

wirecutter's picks are definitely bad, more like products to avoid.

My experience with NYT product reviews is pretty awful. I wish I could be more concrete. Tried to look in my history but I could have sworn at least one article was just effectively, "top 10 most popular on Amazon", with quotes from user reviews. Maybe I'm getting my sites mixed up.

I believe you're thinking of The Strategist, which is through New York Magazine. That's basically their thing, to summarize Amazon reviews for you and filter through a product category based on that. I also don't really see much of the value in it, but I suppose I can see how someone might.

Yes, you are definitely confusing something. And someone.

Bought a pair of audio-technica headphones based on their review. Sound quality was as described but it was so uncomfortable I returned it 5 minutes after picking it up. It was described as comfortable for long wear but it had a hard band with little padding and was uncomfortable for any duration.

Comfort is a pretty subjective thing. Your experience, while valid, is merely a single data point and it would be quite premature to disregard the review or even the whole outlet based on it.

That fell to complete shit a while back. They are not wrong but I would ignore their rankings and focus on the review aspects

I find Wirecutter and Consumer Reports are pretty reasonable for product categories where I just want a reasonable choice and don't necessarily have deep knowledge and preferences myself. And, yes, it's worth reading why they picked something. But if I were buying an interchangeable lens camera or a computer I might read their recommendation but I'd look elsewhere also. For a sprayer for a hose? I'm sure their recommendation is fine.

No, for hoses they recommend ones with the archaic brass threaded fittings common in the US instead of the far superior Gardena type.

Rtings is my go-to for most things. Been satisfied with several purchases researched there.

I’m a subscriber to consumer reports, but as it put Tesla Model 3, the most successful car in the past few years on the back as the least reliable, I feel that I can’t trust its results to be 100% independent reviews. I don’t have a Tesla, but if something grows so fast where people pay a significant amount for it, it can’t be that bad.

Are you thinking of the Model Y? CR gives the Model 3 average reliability, and gives it the highest overall score in the electric car $45-55k category. The only other car with average reliability in the category is the BMW i3. The Ford Mustang Mach-E and Polestar 2 get the next tier down for reliability, and then the Model Y brings up the rear getting the bottom tier reliability score.

In the electric cars over $75k category, everything except the Audi E-Tron gets the tier between worse and average, including the Tesla Models S and X. The Audi gets the bottom tier.

This article talks about why the Teslas other than the Model 3 get low reliability ratings [1]:

> Commonly reported issues from Model Y owners included defective sensors that had to be replaced, problems with heat pumps, air conditioning, body panels that didn’t line up and water leaks in the trunk due to missing seals, according to Fisher. Owners also reported a variety of electrical and hardware issues with the higher-priced, and less-popular, Model S sedan and Model X falcon-wing SUV.

> Older models typically fare better in reliability, as companies tend to make tweaks and redesigns to solve known problems, while sticking with the same parts and suppliers.

> But Tesla deviates from this approach, Fisher explained. “At almost random times during the year Tesla will switch major components, suppliers or sensors and other units. The more you change, the greater the chances you’re going to have some problems.”

[1] https://www.cnbc.com/2021/11/18/consumer-reports-2021-auto-r...

Ford recalled almost all their mach-e cars build until first half of 2021 with unglued roof, so that there was two different risks to roof fly away. As of december 2021 they still didn't fix the keys. Software glitch and a security recall of seatbelt issues. And yet, it is more reliable compares to tesla lol :)

I don't know anything about how reliable or not Model 3 is, but surely reliability != popularity?

It’s the growth that’s staggering. What I read is that older car companies changed the rating to include small software bugs in the entertainment system. As Tesla has much more non-essential features, these while these small bugs are not that important for the end user, can bring the ratings down vs other cars that don’t even offer the feature.

Tesla is well known to whip cars with sometimes dozens of minor issues. It’s a new company and these processes need time and experience.

The reliability data also is likely sourced from outside, such as the AAA, insurers, or any other place where car trouble data naturally accumulates.

I see...it's interesting that they can get away with it while growing so fast

Tesla got it's fair share of issues, but they fix them and detect them fast. They adamant about security. Only Tesla and Volvo got their own testing facilities. Last recall of backup camera is actually a positive call. On a very small group of cars they got this issue. They called it a defect and recalled an entire batch ( 500k cars) to fix a potential issue.

Well, there was just a recall of 500000 Tesla vehicles, including Model 3s, for safety issues:

https://www.cnet.com/roadshow/news/tesla-recall-model-3-s-fr...

Yeah, I was also frustrated. Tesla always been in the top. And boom, suddenly it stoped been there. I don't believe in reincarnation. All my peers who owns tesla are the most happy customers every, including me. For 2 years owning Model Y I asked for service twice. Once i damaged the car, they came to my backyard, I found it super convenient.

Second time I request a retrofit a speaker from a newer models and hardware + work cost me $100 usd. Engineers did all the job at my home while i was working. I only clicked from the App open the car . Oh well, consumer report doesn't count this things that makes consumers happy.

Consumer reports also doesn't count that after two years of ownership i received tons of new features and my car still feels fresh. I can put here a huge list. Since I bought the car :

- my range improved

- winter regenerative brakes improved

- automatic blindspot cameras after turn

- improved climate control, especially automatic seat heat

- more music/video sources

- dramatically improved autopilot

- view cameras from phone

- updated for free hardware for temperature measurements during one of the unrelated visits

- improved charging time ( faster)

- charging network doubled

- automatically synced profiles to my second tesla

- better charging scheduling, that works well with my local electricity provider incentives, saved me ~600 usd already

- far better navigation included way points

- better security when backing up ( sound)

- improved auto wipers, that become 100% reliable( more a fix)

- i don't include tons of fun stuff like games, easter eggs etc...

This are only improvements that are useful to me. The actual list is waay bigger.

Oh well... someone paid this journalists to portrait it in a bad way. But that basically shows how vulnerable is the system. You pay 10-20 journalists and boom, you got your marketshare of people who trust to some bs like consumer reports.

I'm in a strange position here. I fundamentally agree that reviews for the most part are absolute steaming pile of crap of a space for many niches. I work in one of the worst - web hosting reviews. It's plagued by fake affiliate reviews dominating basically every search result. I've been trying for 10 years to run a company that did reviews differently in the web hosting space.

Full disclosure, I have affiliate links on companies that have them too. But I also list companies without them and it has had zero bearing on any result in 10 years. In fact, when I launched I had to beg the CEO of the top rated company to create a special affiliate program for me. Why? Because he didn't believe in review sites and affiliates in the space. It took months, but I told him if he didn't create one, what I was trying to do would never have a chance because I'd never make a dollar - you're the top rated company. I want to do something different, but it needs to remain somewhat financially viable and if you don't have a program I'm dead before it starts.

So what happened in those 10 years?

Honestly, not a whole a lot. I have mediocre rankings (often page 2-5) on some of the most competitive terms on Google. I can't afford to buy the links my competitors do because they make 10x or more what I do pushing the highest paying affiliates and designing for conversion. The site has some traction within niche communities - especially the WordPress hosting space - because I also run annual performance benchmarks (https://wphostingbenchmarks.com) where I document and thoroughly test most of the meaningful players in that space.

It makes a couple grand a month, I've disclosed the revenue publicly on IndieHackers (https://www.indiehackers.com/interview/reviewsignal-e1ddcc26...) and it's gone down since then.

The data I'm providing is almost surely the most transparent data tracking the industry and maybe the least biased (the reviews work by analyzing Twitter sentiment at scale - everything publicly documented in terms of ranking algorithm and published comments).

But outside little bubbles in communities that care, nobody noticed. Google doesn't care. Google happily ranks affiliate sites spending six figures buying links off apache.org and other open source projects (look at those sponsor lists on a lot of open source projects - hosting/gambling is a bad sign).

I got excited when my work fighting against .ORG registry price increases and sale at ICANN (https://reviewsignal.com/blog/2019/06/24/the-case-for-regula...) got a lot of press, even getting cited by the California AG in his letter which effectively killed it. I got backlinks from a lot of large news sites and traffic. I honestly saw no meaningful improvement in rankings or traffic.

So I'm stuck, I keep the sites running - part time - mostly between other projects. I've moved back heavily into consulting and other projects because being an honest affiliate - I can't compete. Providing honest, transparent data and presenting it with the goal of informing versus pushing sales is a terrible business model. The majority of people simply don't care. A lot of 'in-the-know' folks read and get informed by my work. They advise their clients using it, and I never see any financial benefit from it. The broader world, especially Google, doesn't know or care.

Is the root problem affiliate links? It certainly skews incentives and pushes manipulation. If we removed them, what fills the void? Ads? Sponsored content? Something else? I don't think the problem goes away - there is so much money in some of these industries and the stakes are so high. Companies and people will take advantage of it one way or another.

How do we identify honest / good content from the garbage seems to be the bigger question. After 10 years, I'm don't have an answer and I'm certainly not being noticed.

This is a perfect example of how fundamentally flawed modern SEO is. @ohashi has been producing the best hosting reviews in the WordPress ecosystem since forever. It seems like at some point in the past decade on page quality signals have been completely drowned out by backlink signals that can be easily amplified by bigger content producers and a resurgence in cheap on page SEO tricks.

Sorry Kevin I don't know what the answer is, but I'd thought I'd just say a huge thanks for the great work you do year after year. It might be VERY niche but perhaps some sort of annual premium membership for professionals in the WP ecosytem might be something that might work? I'd certainly be happy to support your work on an ongoing basis with access to niche "members only" performance reports on things like WooCommerce benchmarking tests etc.

Thanks edbloom!

It really does help motivate me to hear people get value from my work, because it definitely isn't financially motivated at this point.

I appreciate the offer of premium members only payments. I want my data to be public, I like open sourcing my contributions when possible. I think transparency and openness are the key to better reviews. So I've not ever approached that route. I've also avoided taking any sponsored money from web hosting for any sort of advertising because of even the potential conflict of interest (many have offered to sponsor those benchmarks).

There might be something in the works to help it financially this year, nothing really concrete, but I'm exploring some new options. Ultimately though, I'd rather it wither and die than just doing what every shitty affiliate ruining the space has done.

Answer, if you want to make money is play Google's game. Go get some VC funding to buy ads. Google with put the ads on their front page which will boost your search results. Eventually you'll figure out you dont even need a good product, you win with a mediocre product and a lot of SEO/ad spend.

Break Google up. Course that will never happen as politicians like the power Google has over the internet in the hands of a US company.

The whole reason I built what I did was because after a decade in the web hosting world, I still couldn't name a review site that was worth referring anyone to.

I wanted to build something different and meaningful. I did. It just hasn't mattered in a broad context.

If making money was the goal, I'd put whomever wants to pay me $300/sale first and pimp them out as the best company in the universe. Fuck that. I hoped that I could make a reasonable amount (at least enough to keep it going and maybe scale it a bit) and create a more honest review space.

Winning in this case isn't defined by money earned, you're advocating a race to the bottom I wanted to avoid. Google needs a shakeup and so does search as a whole.

Why not go through the direction of having a patreon or other monthly donation? No need to necessarily give rewards, just asking for contributions

I am honestly having a hard time articulating why I don't like the idea of people donating their money to me for what I do. I will try, maybe I am irrational?

There is so much money being passed around in web hosting review space, by companies to affiliates. I still make thousands per month off these affiliate deals. If my site were to rank at the top of search engines, I would be plenty compensated. Having individuals sponsor the work and it continuing to languish is a position I'm not really a fan of. A single sale commission can range from a few dollars to high hundreds or occasionally a thousand+. I can't imagine enough people wanting to donate a meaningful amount that would change that equation. I'd personally feel bad for every Patreon sponsor putting in 5,10,20 dollars because I might feel like I'm not doing enough with it. If those people just sent one person who ended up purchasing via a link on my site, it would likely outweigh most donations, it would also spread the reach to places I likely can't get to.

That's just what is running through my head, I don't know if that makes any sense.

Is your business model capable of sustaining a paywall of the most useful value, by eliding the information? Your affiliate links still point to your review effort, but the actual top ranking information is behind the paywall.

Make it a variation on the "businesses pay for business-class stuff, amateurs get the free stuff" model.

For the top say, 20% ranks, it is behind a paywall. People will know which vendors are members of the set of the top 20%, but not their ordinal collection. The bottom 80% members of the set's ordinal collection is displayed. If I'm looking for a site to store Grandma's cat pictures, the bottom 80% is probably good enough.

You could also tilt the paywall so free only reveals who NOT to do business with, and who ranks at the bare minimum acceptability.

You can have the paywall only enforce an embargo, instead of all historical results. Say some period beyond which the businesses that can afford to pay you will not care for the results, like a year or whatever.

You can have a "free hour(s)" once a review cycle just before when you publish where the in-the-know crowd can get the results for the newest review cycle, tied to a YouTube livestream, where you do a monetized Q&A.

There are lots more variations to ponder, but I can't help but think there must be more useful ways to slice and dice the valuation of the information here, and monetize in tranches like finance and entertainment do, instead of all in one go.

I guess the problem is two fold, could I charge some people a premium for premium information? Absolutely, yes.

Does hiding some information behind a paywall fit with my goal of a better review landscape in the web hosting industry? Sadly, no.

Sure, I would come out financially ahead, but the industry basically wouldn't be any different. I'm not going to rank any better or have my information spread more effectively behind a paywall. The number of folks impacted would also surely go down as many wouldn't pay.

If the goal is simply profit maximization at this point, there are definitely avenues to explore like what you've suggested. That's boring to me honestly, unless it can have impact to along with it. I do appreciate your thoughts on the matter and suggestions.

Personally, if it gets to that point I'd rather sell it off and move on to working on something else. Most of us are software developers and entrepreneurs. I'd rather build something more interesting and with more potential. I can make money doing consulting work any day, after 10 years, the allure of simply trying to profit maximize on what I built by introducing barriers doesn't appeal. That might be part of the reason it hasn't been as successful as it maybe could have been. I suppose that's the hill I may die on - open, honest, transparent review data for everyone.

Aha, that makes it much more clear. You likely can achieve your aim with more marketing, along the lines of setting narrative in the industry.

You already have an objective measure to flog, build a "council" or "board" around it, with the "Platinum" members being the providers who were most consistently in the top three ranks in all the past reviews. Get some good writers who can whip up some clickbaity articles that would find placement for some fun.

"You're getting cheated out of bandwidth if you don't use these web hosting providers!"

"Most Web hosting providers don't want to let you know this secret council!"

Make it tongue in cheek, self-parodying maybe to the clickbait industry.

Intermix with and reference to some more serious PR to avenues where industry people are likely to read it, where you lay out your concerns of improving the industry and the benefits to them. Make the report red-orange-green-easy to at-a-glance interpret for non-technical management for good-enough outcomes, then analyze the results into a historical trend for the aggregate industry's progress.

Honey versus vinegar, basically.

Your consulting expertise on designing projects to make it easy to transition between providers, and a calculator to compute when to pull the trigger to perform the transition would probably more than pay the bills.

Maybe the marketing hasn't been great. But nobody seems to care. The status quo benefits the companies that get the most marketing, why invest in trying to change the rules that benefit them?

I tried some more expose type stuff, revealing spam networks operated by hosting companies, publishing pay to play attempts, exposing fake reviews and even getting one CEO on record supporting astroturfing. Despite how outrageous this behavior is, it never gets much attention. That last company I'm talking about tends to be on all those affiliate garbage sites you will see and has popped up in the past few years (https://reviewsignal.com/blog/2018/07/22/hostinger-review-0-...). I make sure to let as many people know as possible anytime I see the brand. But their marketing/astroturfing campaign certainly dwarfs anything I can do. And most people don't care. The publishers take the money, the company takes the customers and funnels more money into deceiving poor customers via affiliates and search. The only loser is customers as a whole, who have little voice or power in the equation at any level.

In little bubble communities, I do have some traction and recognition. People may know, but they tend to be echo small chambers. Sure, some people flow in and get recommendations. But the majority of people, it's just who ranks at the top of search results. They don't invest in doing deeper research. They want to see 'X is the best product' and move on.

That's another problem, nuance. Is there a best web hosting provider? Not even close. Even with a very specific use-case, it's extremely rare to say, yes, this is the best company. I've been tracking 10 years of data and no company comes close to perfect reputation. The extreme top just crack into 80% range in terms of positive sentiment. The generally pretty good are in the 70's. That means 20-30% of people are having bad experiences with the best companies.

I've publish the review data with ranks based simply on statistical numbers, but I don't do ranking for performance benchmarks. I also encourage that it's one data point, simply being in the fastest tier or having the highest consumer opinion doesn't mean it's the best. There could be other factors which matter.

For example, Pantheon is the perfect company for has a great reputation but I wouldn't recommend it to most people to consider. First, they specialize in WordPress and Drupal. So if you're not on those platforms, it's not relevant. Even if you do, they are very opinionated hosting. If you have a development team and want strict Git workflows where code moves up and data moves down, they are fabulous. If you're a semi technical person who mostly does marketing, it's probably a terrible fit because you don't want to learn Git, have to push every change through testing, staging and then production to make progress. But their hosting in my tests were fast. They have great consumer reviews. For the people who it makes sense for, they love it. And I would recommend it to probably less than 10% of people who meet the basic criteria of using WP/Drupal for the reason stated above.

As far as consulting experience goes, I actually end up consulting more for the people who build sites. After running the largest benchmarks for WordPress for ~8 years now, people have brought me on to help test new WordPress products or scale existing sites. That's more interesting and rewarding honestly seeing direct results, being compensated to help them achieve goals. My compensation overall is okay, it's just not coming from reviews these days.

I was always fascinated by the idea of "field testers" testing products in their daily lives and writing a short review about their experience. A cook would write about a chef knife, a post officer about comfortable all-weather boots, a craftsman about a tool etc.

The biggest problem would be how to incentivize such people, but gamification and some monetary rewards from the community could probably solve this.

A big part of the best wire cutter reviews are the “why you should trust me” introduction. Some of their reviewers really know their stuff, others obviously less so. You can calibrate your expectations accordingly.

They are OK, but I remember when they decided a Nokia phone was better than an iPhone.

Tests usually have subjective criteria that determine the ranking.

> which does independent testing of products

Can we have some independent testing of web search engines?

Which? https://www.which.co.uk/ is great, based in the UK, not very expensive to join for a month to look up one product.

Just one data point, but I've had a hard time trusting then since reading a headphones section they had in the physical magazine quite a few years back. The number one rated headphone was the Bose QC3. Great noise cancellation and comfort, sure... But certainly not better audio quality than so many other products in that price range. All rock / metal sounded so muddy I had to return my pair within a few weeks.

I have used their tests in the past and I’ve never regretted a purchase influenced by their tests.

Reply on news.ycombinator.com