Hacker News Reader

Hacker News

Comment by TekMol | Hacker News Reader

Full thread

TekMol

In your about page, I see you are using Bing's API. I didn't even know Bing has a search API that everyone can use!

How much do you have to pay them for this?

View on HN

It's $4/1000 queries, but the rate is increasing in May to $18/1000 queries. The Bing API is available through Azure.

I hope the author knows this, and won't be surprised by a bill more than 4x larger.

That comment was written by the author.

Oh! I am scatterbrained.

quite a price hike. Bad for the sustainability of the other portals I use that are backed by the bing index. Will either increase their pricing or efforts on monetization.

Is it easy for you to rely on more search index providers, what are your options?

>Is it easy for you to rely on more search index providers, what are your options?

I have a few options:

* Switch index providers. For example, Mojeek has an index with 6bn pages and has a web search API; may be more sustainable to switch to their index in the long run.

* Build my own index. This is my preferred option and I've already started to work on this.

* Look for funding sources to offset the price hike.

own index: the wiby.me DIY writeup is fantastic. Each person will do this differently based on knowledge and experience, but the step by step guide was educational.

marginalia put out the news yesterday getting a NLnet grant with one of the sub-goals "to produce and offer portable data in order to bolster adjacent efforts in the search and discovery space", maybe talk?

As an end user I profit from the many licensees to the bing index, different interfaces and ideas can compete with less effort required.

> $4/1000

Is it just me or does this seem insanely expensive already?

Yeah for my search traffic alone that feels like it'd be a $4/mo service

Pretty sure that only Google and Microsoft have the money and resources to crawl the entire internet. Or perhaps the only that can AND are willing to.

Correct me if I’m wrong though, but I’m pretty certain that all other search engines in the same category use one of these as their backend. Eg I’m pretty certain that counts for duckduckgo as well.

I think Yandex have their own index and some others too like Marginalia, but the latter couldn't be called "in the same category" as the other three.

Yandex also suffered a security breach and their source code is available[0] although utilizing it in any way is ethically and legally dubious (at best).

[0] - https://arstechnica.com/information-technology/2023/01/massi...

Brave Search has its own independent index too - https://brave.com/brave-search-beta/

> Pretty sure that only Google and Microsoft have the money and resources to crawl the entire internet. Or perhaps the only that can AND are willing to.

Money and resources and a dominant-enough position so that your crawlers are not blocked by websites.

Unfortunately.

There's definitely gatekeeping on websites but having done a bunch of crawler work I can say you'd be surprised how rarely a site will outright block you if you just do the right things: have an identifiable user agent with a working URL that explains what your crawler does, respect robots.txt, implement polite crawling. As for actually being able to crawl the whole thing though, yeah it's stupidly expensive :-/

Mwmbl has its own index but it's orders of magnitude smaller than commercial search engines.

What about Common Crawl?

https://commoncrawl.org/

Reply on news.ycombinator.com