Hacker News Reader

Comment by SwellJoe | Hacker News Reader

What are the alternatives for a full web search API? Yahoo BOSS closed down a few weeks ago. Bing seems the last man standing. And, unfortunately, none of the ones I've tried are very good; one project I'm working on needs to prioritize newer results over older ones, and only Google had a sort by date option (which I think was also removed in the past).

View on HN

Google Custom Search is one option, but it has a limit of 10,000 queries per day and is very expensive compared to Bing ($5 per 1,000 queries).

I've migrated to Bing, myself -- no volume limits, generous free tier, and much cheaper than Google.

I'm not seeing how to make GCS perform a full web search. It won't allow creation of a search without at least one schema defined, and I need schemaless data (or rather, I need the whole web, including a variety of schemas). Am I missing something?

Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search.

Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous URL: https://datamarket.azure.com/dataset/5BA839F1-12CE-4CCE-BF57... . I think that's what you're talking about? Note the link on that page to "Bing API FAQ" -- is broken. If this wasn't MS, the poor docs would make me think it was surely a terrible product.

>Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search.

You can, refer to this SO answer: http://stackoverflow.com/a/11206266/3354209

Thanks. One of the comments on that SO answer from Aug 2015 says:

> WARNING: we did development using the free version, but to upgrade to the paid version (to do more than 100 searches), google forces you to turn off the "search the entire web but emphasize included sites"

I remain not very confident that you can really search the whole web with google custom search, or if you can that it's not some kind of a loophole that google might close without warning.

But if someone has actually done this succesfully, with a paid account for more than 100/queries a day, I would definitely be interested in hearing about it!

I am doing this successfully with 1,000 queries in a day, and the limit it 10k queries per day (pay per 1,000 queries)

The billing system is confusing. You just have to sign up for billing, and you can raise the limit.

The upgrade in the google system, is only if you want site search with no ads.

Yes, you can search the whole web. We have been using a pool of credit cards to make up to a million requests per month.

You can check out the new home of Bing Search API here: https://www.microsoft.com/cognitive-services/en-us/bing-web-...

You can get key right from there and an API console.

Thanks!

Should I chalk up to conspiracy that this page is nearly impossible to find on, um, Google, searching terms that you would reasonably use to find it?

But if you have an in with MS, it would probably help an awful lot if they removed the old pages with broken links, unhelpful content, and possibly out of date inaccurate content too.

The page you link to has "coming soon" for pricing for Search (which was kind of confusing to find) -- does that mean that the pricing info on the page I found is not accurate, or soon won't be, with no indication on the page I found that that will be the case?

MS is really really bad at documenting this service. Only because it's MS am I willing to entertain that the quality of docs may not reflect the quality of service, generally I'd give up on something that markets and explains itself so poorly and figure that if they can't get that right, they probably didn't get the service right either.

Here's the old home, with pricing laid out (missing from the new one): https://datamarket.azure.com/dataset/bing/searchweb#terms

Here's the older link to the Bing Web Search API -- while the new page doesn't list actual prices, this one does in nice detail:

https://datamarket.azure.com/dataset/bing/searchweb#terms

Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services. All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pricing plan if you want to extend the monthly quota.

Some folks mentioned this page: https://datamarket.azure.com/dataset/bing/search. I want to clarify that the page includes the old Bing Search APIs that are still in use, but will be deprecated in the future.

You can send feedback or questions here: https://cognitive.uservoice.com/

[Edit] Here is also a blog post with useful resources: https://blogs.msdn.microsoft.com/bingdevcenter/2016/04/13/re...

I keep checking out the Bing API every couple months, hoping for sort-by options (I need last changed date, most recent first). Any hope of ever seeing sort-by-date from Bing? Google has it, I don't think anyone else does.

Hi, just a quick bit of feedback -- 1k API calls / month is so low as to be a nonstarter. Of the half dozen hobby (non-commercial) projects I am currently working on, this amount is tragically low. Per day would give me pause.

I don't mean to rain on your efforts, but I would personally never consider using an API with such limits, even for trivial hobby projects.

Agreed -- 1k/mo is not worth the initial time investment.

Thanks for the feedback. We do not have any immediate plans to change the call limit for the preview offering. We've seen from our data that 1K calls / month are sufficient for most developers who want to get started using the APIs. We we will certainly consider revising this in the future if there's strong demand. I encourage you to file this as a request here: https://cognitive.uservoice.com. We review user ideas and feedback there on a regular basis to improve our services. Thanks again!

Is that 1000 calls per API key, or per IP address? IE if we make client-side calls can we do higher volume?

per API key

> https://datamarket.azure.com/dataset/bing/search. I want to clarify that the page includes the old Bing Search APIs > that are still in use, but will be deprecated in the future. >

I just migrated to the Bing API when Yahoo BOSS closed 31 March :-(

Will there be a new image search api similar to the one you offers now? Can you share any timeframe for the deprecation?

https://www.microsoft.com/cognitive-services/en-us/pricing does seem to indicate that there will be a Bing Image Search API under the new umbrella.

There's one https://www.microsoft.com/cognitive-services/en-us/bing-imag...

As others pointed out, you can find information about the new Bing Image API here: https://www.microsoft.com/cognitive-services/en-us/bing-imag.... I cannot provide an official deprecation date yet, but here's a rough timeline: we will first announce the pricing plan for the new APIs in early-to-mid summer. We will then provide a notice to existing users of the old APIs to switch to the newer version within 6-12 months - note again that the dates are not final.

This is very interesting functionality to me, that I would use if it works well. Thanks for trying to clarify what's going on with it.

But when between one fairly poorly documented API that you've told me will soon be deprecated, and another fairly poorly documented API that doesn't have pricing beyond 1K calls a month....

...my response is still "Okay, that could be good in the future, I guess I'll sit back some more and wait for pricing and better docs to show up before paying more attention to it, I hope it does soon!"

I need to know pricing before deciding to use something. And I need good (and google-able! Sorry, even if I'm using Bing API, I'm using Google to look for info on it) docs, along with of course a well-designed API that works well, but I need the first two things before I even get to evaluating the next.

Google Custom Search[1] is still available, albeit limited to 100 free queries a day (and 10k max, even when paying).

That being said, I'm still looking for alternatives as well. This broke my IRC bot and a few scripts rather unexpectedly.

[1]: http://www.google.com/cse/

Doesn't this only allow you to search your own page, as mentioned at the URL you linked?

"With Google Custom Search, add a search box to your homepage to help people find what they need on your website." (emphasis mine)

Their video also seems to indicate the same thing: https://youtu.be/Qd9z48Bo8ZA

They say you can search "one website or even a specific topic across multiple websites". This is specifying multiple domains, not a full-index search.

http://gigablast.com/searchfeed.html

Yandex Bing Gigablast

Those are the only web scale indexes that have an API I am aware of these days.

Hoolie's search API hasn't been returning any articles critical of Gavin Belson any more, but it sure is faster now that they're using that new ranking system!

There was someone on HN a few weeks back who was making a new search engine - I'm curious as to whether they offer a search API.

the first thing that came to mind was DuckDuckGo :) https://duckduckgo.com/api

DDG is not a web search API.

Its not a web index. They are unable to offer that as a lot of their results come from external services such as Yandex and Bing.

The results are their zeroclick info snippets and it is very useful for pulling an answer from wikipedia since they have already parsed it.

But do they consider spidering context problematic? What would stop someone building an API on top of their simple websearch?

It has no web search api.

Their search result HTML is pretty easy to scrape though

Reply on news.ycombinator.com