Mandating a commonly accessible crawl, with cached versions of the pages, would help new entrants a lot.
Also, there're large network effects with ad networks. It seems unlikely that many marketing managers are going to take the time to do targeted keyword queries on your search engine with 1/1,000,000 the traffic of Google.
I'll also add plenty of sites don't block any engine but confer special privileges to google bot which depending on the site and their size are almost the same thing.
Edit: And I'll add to limit confusion Reddit hides the sitemap and denies access there's is not an outright ban -- it just makes it a lot harder.
https://support.google.com/webmasters/answer/80553?hl=en
So unless you have a google domain your sol, it's also just generally frowned upon. We have our own UA WhizeBot with an email contact so you can let us know if our crawler is doing anything you'd rather it not.
There have been a few legal cases that protect scraping publicly available information on the web but we'd rather follow robots.txt to avoid the potential for shenanigans in any case.
'Faking' the UA is much more likely to annoy site owners and end up with a permanent block.
They are part of the story of why Google is degrading, not why they're doing well.
You also have bots that seem to be credential stuffing, bots that seem to be content scrapping (stuff with same typos shows up elsewhere after their visits, really obvious on new / fresh articles, bots that seem to be exploring for copyright claims, rando bots (maybe comment sentiment analysis for stock trading) etc.
Google is much more welcome by comparison.
Generally I wouldn't qualify a lot of those as search engine web crawlers but more web scrapers looking to re-use data, not just surface it.
Like I said in another comment, if you’re someone that just wants to surface Amazon results they love you. It’s the people that want to take advantage of amazon data in some other way they’re trying to stop.
While I don't know if I'd say "hostility" I would say passive aggressive to other crawlers.
I've definitely seen my share of robots.txt that give special permission to Googlebot, but maybe my corner of the web was unusually aggressive toward crawlers.
We have an API, in case of interest.
Ask anybody in much of the Midwest where they will choose to buy groceries and they will answer Walmart.
This kind of reasoning is not an argument against the philosophy behind antitrust regulations. Rather, it is emblematic of the situation that monopolies create for themselves, by using their economies of scale to create pricing that cannot reasonably be expected to lose any competition over customers.
The same economies of scale benefit consumers and the economy as a whole as long as the business behaves in a manner consistent with the shared values of society. As soon as they decide to behave differently, though, society is at the mercy of a monopoly because they have consumed the entire market.
The case hinges on the no-compete, tying, and default placement agreements. If Google didn't pay anyone to be the default search engine, didn't require Android Phone makers to pre-install Google apps and make Google the default search in order to get access to Google services and the store, etc then there's no problem.
The complaint also says that one of the reasons Google's results are better is the network and scale effects that come from owning 80-90% of search. A competitor has a very difficult time no matter how good their algorithm because you need insights you can only get when people use your service at scale. I don't personally know if that is true but that's part of the complaint.
The proposed antitrust violation is likely the combination: using their dominant position to outspend any other players when paying for default placement to ensure they're the only one with the scale to have the best search results. That forms a self-reinforcing cycle: users who do choose pick you because you have the best results and you use that fact to fund no-compete and default placement agreements to ensure you have unmatched scale which further reinforces your ability to deliver the best results.
Again: I don't know if this is true or a valid argument, but it seems to be what the complaint claims.
For example, let's say I could be curious about DuckDuckGo if I had to choose a default search engine when I got an Apple device instead of Google being the default, then that could be revenue to DuckDuckGo for them to improve their search engine.
Yahoo has been Bing for years, and DuckDuckGo pulls from Bing and Yandex - the latter being mostly for searches in Russian.
Not really sure what the DOJ expect Apple or Google to do in this situation, it seems to me that 2 companies entered a mutually benefiting contract. You can't argue that Google "colluded" with Apple, there's a lot of throwing punches between each other (all the privacy oriented moves Apple is doing are hurting Google's business) and again Apple could be making its own search engine anytime they wanted, they already replaced Google Maps with their own thing.
If the only two players (Google and Apple) both use Google Search by default, then Google has effectively captured 100% of the mobile ad market. (And since Google is paying Apple et al for that default state, it is indeed a market.)
The court could require device makers (including Google itself) to prompt users to select their search engine provider, or potentially ban Google from buying search engine defaults from other companies (Apple, Mozilla, etc.) as an anticompetitive practice.
Sure, they could sell it to someone else, but without the biggest player bidding up the price, it'd probably sell for a fraction of what it does today -- which could be devastating to Mozilla, given that almost all of their funding comes from the search deal. What would that do to competition in the browser space?
Using Firefox as a reason not to break up some of Google's hold on search seems extremely short-sighted. Besides, Edge using Chrome changed the whole browser market dynamic as more and more people are using it. Once Microsoft reaches a somewhat decent percentage of install we will be back at a two (three?) player situation.
I completely disagree with this part, but you're totally right on people continuing to use it.
Search is highly dependent on the query. At first DDG's results seemed bad to me, but after a while I think I've changed how I write my searches. It's hard to explain, but I guess I'm putting more thought into understanding what it is that I hope to find.
Now, Google's results seem to just be a listing of whoever did the best SEO targeting on the subject, and ultimately that means worse results for me. It's less about what I'm looking for and more about what Google has to show me... and Google always has something relevant to show me. When DDG doesn't, I'm forced to re-consider my query and try again, ultimately reaching a better destination.
However, "change the way you search" is niche at best. Google is satisfying because it's so easy to use, that you almost don't even have to write a query. It's like an automatic "I'm feeling lucky" based on it's knowledge of you and your location and time of day, etc...
I was just searching for "quill", a javascript editor. I clicked the first result without really looking (after all, i m searching name, right? ) - and it took me to some competing product called fro-something. I wasted my time, their money , yet google still made money.
This seems reasonable. What's your suggestion on how to make this better ?
(the query must have been "quill js" or "quill editor" cant remember -- it's an example of the infamous 'google® tax' - an obviously keyword-targeted ad)
https://www.google.com/search?safe=active&client=firefox-b-m...
It is a search for "express-http-proxy" "logging" but at leat I got maybe one relevant result and a bunch of wildly unrelated ones, despite doublequotes.
As for most of the rest of the first 10, null would have been a valid, more correct and generally better result!
I actually wasted several minutes on one of them right away since I haven't still gotten into the habit of verifying every single Google result to se that it actually is what I asked for and not whatever filler Google decided to use this time. Not that it should be necessary, - they had this sorted back before 2010!
Seriously: why can't Google or DDG get this right anymore?
You didn't buy their product, but Google helped them get your gaze briefly.
What is search but a list of adverts associated with your keywords?
1. It allows alt search engine or alt browser to succeed. So what? The alt search engines and browsers are worse than the google ones. It doesn't benefit any actual people to have them rise up.
2. Google screwed my friend one time. Every single company (and person) on earth is guilty of this. One swallow does not a summer make.
3. Google is restricting this tech I like, or google is sometimes wrong about their bans. See (2)
And so on. I haven't seen anything compelling in this thread, just a whole lot of hyperventilating.
And to your point. No, search is not a wall of ads. But if you feel that way, literally change your search engine. It's easier than typing this comment you just posted.
The events of celebritynetworth.com are a good example of why they've not been behaving as a good citizen (there is a thread about that on this forum).
A good number of people have took up your suggestion and changed their search engine, DDG's traffic count continues to rise. The reality is that many users will use whatever is put in front of them, i.e. how many people use the URL bar of their browser for searches, or use a search toolbar for a specific URL? Google knows this well, hence their paying to be the default search engine choice and paying an amount that no other engine can match.
Google's knows they are the best in class when it comes to general search, but to be sure they have been aggressive in ensuring that no one else gets a look in.
The UK's CMA report estimates that an entrant to the market would require around £20 billion of capital to be a credible alternative, as in, a proper crawling search engine, not a meta search engine like DuckDuckGo. That's before paying the likes of Apple billions of dollars simply so that people will use "what is there".
Having such a dominant position comes with responsibility, which is not solely Google's. The social networks along with Google are essentially the gateways to the web and how they present that information is important- and essentially only one point of view. Having room for more competition, choices, algorithms surely is a good thing.