Opening up search is an ethical imperative
opensourceconnections.com
opensourceconnections.com
When thousands or hundreds of thousands of companies live and die by your platform, its economic suicide to leave it in the hands of one single private entity.
A few things need to happen in order to guarantee fair play:
- Transparency: Your process can no longer be secret. People need to see whats going on to ensure no foul play is going on.
- The owners of the platform can't participate economically in it. Amazon is the obvious example. You don't want to be kicked out of your own platform? Don't open it to others. Its a simple matter of conflict of interest.
- DUE PROCESS: You can't have some minimum wage employee following a flowchart in call center making life of death decisions for companies living in your platform. There needs to be a proper and accessible appeals process with independent arbitrage if necessary. And above all, innocent till proven guilty. You can't fucking shut down my company and THEN I can go through the lengthy appeal. No company can survive that.
Too expensive? Tough shit, go wipe your tears with your trillion dollar market cap.
I'm sure there's a ton more that needs to happen. These are just the ones I can think about.
Maybe it gets exploited because its made exploitable by design. How do you know if you can't check? You worry about bots but you don't worry about outright massive cheating from the company itself?
If every black-hat SEO knows how to cheat and they grab the first ten pages of results then the useful stuff gets lost in the noise and the search engine is no longer useful to me (or anyone else).
Even evil-Google wouldn't do that because they do have incentive to keep people coming back.
Without transparency, you have no way of knowing.
Good old fashioned corruption is way more effective than modern hacker approaches.
Let’s take Otterbox and Amazon for example. If Amazon notices that Otterbox is selling exceptionally well, they can require proof that Otterbox is selling similar numbers on other platforms otherwise they will be throttled. Just like that, there’s no dependence on Amazon because you’re obligated to diversify your sales. Don’t like it? Tough shit, now you’re not allowed to sell on Amazon at all because you must maintain compliance with the law.
Also, while we’re labeling internet properties as utilities, can we label internet access a utility first?
People seem to think the first guess is upon the House: https://news.ycombinator.com/item?id=22079174
They just click a button and you're gone.
This needs to be regulated. A simple analogue of the civil process would suffice. Innocent till proven guilty would go A LONG way. These companies ban you at the first sign of trouble. They are very often wrong.
Yes, this puts a lot of burden on them. Like I said, tough titties. This is your prize for growing to such a behemoth size. You can't have your cake and eat it.
I care way more about engines of innovation than whingey content creator types.
You can have the best car in the world, but if all cars are bought in carmazon.com and they ban you, you're fucked. Sure, you can make your own outlet but you just lost billions in investment and you wont recover from it. All because someone clicked a checkbox in an amazon monkey moderation center.
Also, you're showing a ridiculous bias against content creators. They're a completely legitimate and respectable profession.
A lot of people, creators, merchants, etc; often get taken out of business by simple misunderstandings or arbitrary decisions from these platforms. Nobody is going to ruin anything, just have some form of due process where everyone's right to exist is respected.
People imagine Google without ads will be better or paid search makes it better - they are mistaken, since the moment Google (or any top search engines) completely stops serving any ads (or equivalent - everyone using ad block that blocks search ads), all of the companies spending money on ads will instead spend those money on SEO against those search engines. i.e. Selling ads is an effective way to limit the amount of money poured into SEO. As long as the cost of ads is lower than the amount you need to spend on SEO to get the similar amount of traffic through organic results, companies would rather buy ads. This adds more incentive for Google and other search companies to focus on fighting against SEO manipulations also, since a successful SEO effectively lowers the ad price companies are willing to pay. Paid search would make this problem actually worse - people who can afford to pay correlate strongly with the target demographic companies want to show ads, and that means once people starts signing up for paid search, companies will throw more money at SEO.
Now, the doomsday scenario (of SEO winning and all organic results becoming bad) may happen anyway because SEOs end up winning regardless despite Google and other search engines fighting against it. For humanity's sake, we better hope that's not how it ends up. It's less likely given some of the structural advantage search engines have, but SEOs have advantages too, so at this point it's not at all clear one side will necessarily win or lose.
That said I'm not sure I trust the current American political climate to step in and regulate, and I'm not sure they have a right to... The influence of these businesses extends far outside the borders of the United States.
US and EU have different payment process requirements. Same for data collection. There are numerous examples and all platforms find a way to implement them and comply.
Having some creator and merchant protections is not a technical issue. The US could legislate to protect US users and they could comply only in the US. No big deal.
Also note that companies are far from honest in this work - just look at what Google is doing to erode the web platform, and how it is leveraging it's search monopoly to drive businesses towards paying for ads.
I said regulation. You keep your company, but after a certain size, you're obliged to treat the OTHER hard working companies living on your platform with respect and not completely destroy them on a whim.
An example: A friend of mine worked for a battery selling business. They had most of their sales in Amazon. Everything was fine, they had a few years at this and around 100 employees.
One day, Amazon decided one of their papers or whatever wasn't good and closed them down. The appeal process too 3 weeks and bankrupted them.
Amazon was wrong.
Now, you think that's fair? All that hard work and all those people out on the street because some dumbass in amazon's underpaid customer service centers fucked up?
Like I'm saying, there needs to be due process. They shouldn't be able to that.
The problem here is that government doesn't understand technology well enough.
Which makes sense, the majority of people don't. Your post is an example, no disrespect meant.
You can't have transparency in search. It's an arms race between search engines and black hat SEO gaming. It's been that way for decades, likely always will be. That's why the algos absolutely have to be secret, and there has to be no barrier to rapid iteration and innovation. The minute that changes, black hat wins and search goes back to looking like Altavista before it died. In that scenario everyone loses.
It's also imperative that there's a financial incentive to continue to innovate and stay (sort of) ahead in the arms race. It's easy to imagine that a government (or a nonprofit) could run search well, but that's only without an understanding of the scope and depth of the technology involved.
I'm not sure what the best solution is, but there's no question transparency isn't it.
I do, by the way, appreciate your idea of mandating more reasonable due process. However I'd add that if your business relies on search, you're gambling. It's often a smart gamble, for as long as it lasts organic rankings are free money, but be honest with yourself about it.
If Google were to “split up”, I as a customer would like a paid-for, transparant, utility search on one hand. My money can go to dealing with SEO abuse. On the other hand as a business I would like an ad-serving product, to which customers can voluntarily subject themselves to.
The two organizations would have to become separate business units, because I don’t want what’s currently happening: I search for information and get disinformation that others paid for.
Have dataset/s and see where the different technologies, win, fail and IDK.
Solr's Phonetic Matching I always thought needed improvement, it would be interesting to get a report on those for instance.
What metrics would you use? I would be fascinated to know.
Color me interested! There’s so many layers to measure: from natural language understanding to pagerank-style relevance ordering, filtering duplicates to presenting the different permanences of content (breaking news vs books on the same topic).
Yep. It's interesting. How do you deal with versioning. The same document 10 times on a CMS might overpower the search results. Does uniqueness rank higher? There you need a human judge at which search engine / search engine defaults is better.
> natural language understanding
I'd like to know if any search engines do this. I feel like it's out of scope. If they use the words "natural language understanding" I think they are running a scam and mean statistics.
Then prescriptive.
The idea would be a framework so someone can make (or grab their current data) the test different engines for different search terms against the data and mark which is correct or better.
You don't have to 100% agree what search result is better. But to just get to the stage where you can discuss that would be a great start.
And I suspect not agreeing what is better is not as contentious as one thinks.
I also think Google does search real well. So use that for a set point of the truth.