DuckDuckGo Hits 14M Searches in a Single Day
searchenginejournal.com
searchenginejournal.com
We wrote a post covering it in our newly launched blog: https://spreadprivacy.com/10-billion-fc7808c91343
It is always an amazing and humbling moment when we hit a new milestone in our metrics. Our team is so incredibly passionate about what we do. To have people from all over the world rally around this concept of raising the standard of trust online and build this product is one thing. But, for users to endorse that mission at such a growing pace and to share us with their friends and family -- it lets us know we're truly making a difference for people who want to be more private online, and that is incredibly rewarding.
So thank you, to all of you who search with us, all of you who know privacy matters, and all of you who work along side us to show that privacy is not a fringe interest but something we can all have!
GoDuck might be interesting. "GoDuck it."
And then all of a sudden we're talking about a duck instead of why private search is important.
* The above landing page doesn't work (typing a query and pressing enter returns me back to the same page).
* Even if I manually enter a search URL like this https://duckduckgo.com/html/?q=test it doesn't fully work (page navigation and requerying is broken).
From the browser console, the origin is that of the web page you're currently viewing, so you're free from CORS concerns.
So, from my browser console in Chrome 56, with duckduckgo.com/html/ open in the browser tab, I should be able to do this without any trouble:
fetch(window.location, {
method: 'POST',
body: 'q=testddg'
})
.then(result => {
result.text()
.then(text => {
document.documentElement.innerHTML =
new DOMParser()
.parseFromString(text, 'text/html')
.documentElement.innerHTML
})
})
But, surprise surprise, it doesn't work. There's no change in the document, where I'd obviously expect to see the search results page loaded, because the response was the home page -- as it is in your case, when you try to search for stuff.So I tried to make a POST request without a body instead, appending the query parameter to the URL as one would with a GET request:
fetch(window.location + '?q=testddg', {
method: 'POST'
})
.then(result => {
result.text()
.then(text => {
document.documentElement.innerHTML =
new DOMParser()
.parseFromString(text, 'text/html')
.documentElement.innerHTML
})
})
This worked perfectly, all the search results were loaded. Works as a plain GET request too. So what we seem to be able to surmise is that posting a request body causes the search to be rejected...But that's absurd! The native submit behaviour of a form with method 'POST' necessarily sends a POST request with a body, the body can be of multiple types, specified by the enctype attribute, default being x-www-form-urlencoded, which mimics a GET request.
In my browser's dev tools, I checked the request in the networking panel. The fetch requests were all there, but they differed from the native requests in an important way: the body was called 'Form data' in native requests, where in fetches it appeared as 'Request payload'.
The latter is obviously not going to be identified by DDG or any website as form data to parse, so I tried fetching with the correct content-type header explicitly set:
fetch(window.location, {
method: 'POST',
headers: {
'Content-Type': 'application/x-www-form-urlencoded'
},
body: 'q=testddg'
})
.then(result => {
result.text()
.then(text => {
document.documentElement.innerHTML =
new DOMParser()
.parseFromString(text, 'text/html')
.documentElement.innerHTML
})
})
Boom, worked perfectly, just as we'd expect.So what was the point of this? We haven't really learned much, because handling form submissions via AJAX always requires such considerations, and that's irrelevant here anyway because the web page in question does not use AJAX, it's a plain HTML version of the site!
What we have determined is that there's most probably nothing wrong with their web page and the problem is more likely to be originating from your side.
I get the feeling that your browser is, for some reason, not applying the correct default enctype for the DDG/html form or uMatrix is interrupting the request and re-sending the request as a generic POST without the explicit content-type.
I've been unable to reproduce the bug on my machine. Do you use any other browser extensions? Does the site work in other browsers you may have?
Here's the complete request/response data in HAR format: https://gist.github.com/hrj/f538ed3e67636c94934e221efe67b323
(Note that the UA string is being spoofed by uMatrix, but I don't think that's relevant. I tried disabling the UA spoofing with the same result.)
I wonder how an exit will play out though. A big factor with DDG is user trust - that seems to limit the list potential buyers to the few that could credibly maintain that trust.
Then there is the IPO route, but that has it's own challenges. You could argue that up until now DDG has had no competition because Google doesn't think it's worth raising its hand to swat a fly. But there is no flying under the radar after an IPO. What if Google promoted a new "no track" feature which matched DDGs benefits without having to do dumb things like append query arguments, modify location, and run plugins?
You might say Google would never give up this much tracking control, but there are two caveats. First, if it's just an option buried in settings there will be limited uptake yet DDG's ability to claim differentiation would be nullified. Secondly, as DDG itself says, search can still be a great business without having to track people.
If I were lucky enough to be in DDGs position I would push for a Microsoft acquisition. MS has kept enough promises in recent years that they just may be able provide DDG users enough guarantees to maintain the trust. People have mentioned Apple but that seems like a lower probability for a variety of reasons. An MS deal would avoid the massive hassle of an IPO and enough resources to chip away at Google indefinitely without being pressured into short term decisions.
For now I'm just glad DDG is here trying to do some good.
There is a financial sweet spot where the contributors to DuckDuckGo are fat and happy and there aren't greedy investors wanting to monetize all that trust.
Google has already occupied the greedy investor niche. DDG has carved out a niche where, I fear, the greatest threat is a lack of money for the people building and maintaining DDG. If they feel underappreciated, they may start to monetize that trust. The DDG team should be rewarded for their hard work.
Instead of sending me to example.com, DDG redirects to a weird constructed address [2]). If I go to duckduckgo.com/lite and search there, it works fine. Any ideas? What would be the best (eh, sorry) place to post this?
Of course, thanks for your great work, and bangs are amazing!
[1] http://lists.dillo.org/pipermail/dillo-dev/2017-January/0110...
[2] https://duckduckgo.com/l/?kh=-1&uddg=http%3A%2F%2Fexample.co...
When a search doesn't quite give me what I'm looking for, I prepend !sp on my query (to use startpage.com). I use this less and less and for most things DDG gets me where I need to go.
I'm a huge fan. I'm thankful services like this are available--that respect our privacy. I did have good intentions of contributing to duckduckhack.com but have regretfully not gotten around to it yet.
I like DDG a lot, but I feel that Google knowing so much about me prevents it from showing ruby gems over ruby lang search results. Must say, this has improved greatly last year, but I don't trust DDG just yet fully.
Really the only times !g actually turns out to be useful is when I'm searching for something that's well within my filter bubble on Google, but generic enough that I get the wrong results when outside of my filter bubble (e.g. DDG). But that's not really all that often.
DuckDuckGo employee here - Be sure to use the 'feedback' button that you find on the search results page whenever you feel a search isn't up to standards.
Our team has been working hard at increasing relevancy, but the best changes come when users let us know what they need.
Since we don't filter bubble results (https://www.ted.com/talks/eli_pariser_beware_online_filter_b...) results certainly feel different when making the leap, and that takes an adjustment period for most people. But, if you actually feel you have examples where you can't find what you're looking for, then we want to know so we can provide you a seamless search experience as well!
I use DDG as primary but any time the search results aren't relevant, I fall back to Google. I fall back a lot right now and I think this is the area for DDG to focus in.
Flights, times, and fares is deceptively hard to get right. There's a reason there are only a few providers.
Edit: Ahh. If you're saying DDG could get by with just flights and times, and no fares, I see what you mean. That seems to be reflected in the pricing for the FlightStats product vs Google's flight/times/fares API pricing, yes. Parity with Google search requires fares though.
If you know of a good data source for scheduled flights you can easily contribute an Instant Answer, or leave the source as a suggestion for our community!
http://www.flightstats.com/go/Mobile/flightStatusByRoute.do?...
It looks like a nice weekend project for someone.
I typed in sfo and jfk, but it looks like the java script submitted longer names, so there might be some non-trivial airport name handling to wire in.
Similarly, Google sells api access to their flight search api: https://developers.google.com/qpx-express/v1/pricing
I have no idea if DDG could leverage either and not wreck their business model.
Edge cases I have in mind are usually related to indexing of forum posts, where I search for a specific-term (say bookkeeping) and roughly know the Google results of the most important forums by heart, but DDG does not even show them.
Second case I just used yesterday was searching for an exact match for an @gmail address of a priest (for our marriage) where Google had a lot more results than DDG.
Keep up the great work!
Some people are put off by the fact that you don't substitute synonyms as aggressively as google. A concrete example was "foss [search term]". Google seems to rewrite this to "[open source] foobar", even with bubbling off.
I prefer the duck duck go behavior (and google's aggressive query rewriting drove me nuts back when I used google -- it makes it hard to run precise searches!). After all, if I bothered to type an obscure acronym, I'm looking for results that actually contain the acronym!
Maybe a "did you mean 'open source [search term]'" link at the top of the page, and even a preference toggle to enable aggressive query rewording by default would help.
Thanks for the suggestion. Part of the reason we don't do this is the returned search results on Google are Google's commercial property. To use them in such a capacity would require us to reach a commercial agreement with them. These agreements come with stipulations that we don't feel represent the privacy interests of our users, so we don't pursue such agreements.
The other reason, is because of the 'filter bubble' Google tailors results based on user tracking, which is what helps with result accuracy. So the results we as DuckDuckGo see on a query would be different than what you see. Plus, even if we could see your results, we wouldn't want to. We don't think it is good to be filter bubbled, we want to serve you the best results we can without censoring you. It should be your decision to a view a certain website, not ours!
Don't believe for a second that governments are not already banging at the door for access. The only ways to combat is either hosting in neutral offshore datacenter and incorporating there and living there is to not collect "useful" information in the first place on users. I think they went for the latter.
As a DDG user I'd be really happy if you did this and think it would be the best way of identifying searches I'm unhappy with. I go to google now only if I'm not happy with the results from DDG.
DDG is committed to returning literal search results for the exact query that the user typed within double quotes, right? Sometimes I wonder if DDG is going the route of other search engines and trying to return "relevant" results instead of only results containing the exact search term the user asked for within double quotes.
Could you make clear exactly what DDG's policy is on exact search terms within double quotes? Does DDG honor such requests?
That's a great idea - I'm not personally involved in the feedback process, but I'll certainly bring this up to the relevant team!
Still love you though!
Switching over everywhere would probably help me adapt more quickly.
Further, I find that I use shortcuts like !w, !yt, or plain "!" much more often than I use !g due to poor (or simply less targeted) search results.
I'd say I'm happy with the search results I get and never find myself going to google unless I need a map. I also really like that they adopted the quick results if you hit something common and get a excerpt which usually has my answer in it right at the top of the search results.
That puts DDG at less than 1%, but their percentage growth is huge. It can't continue forever, but they could be a significant search engine if it doesn't peter out too soon.
I tried DDG several times, but it finally stuck earlier this year. I'm really happy it's available as an alternative.
It's difficult to define what constitutes a "search" these days. Since google release autocomplete search in 2004 [1], each letter you type into the search bar results in a search query.
So simply searching for "superbowl 2017" would result in 14 searches because there are 14 characters. So, do you count that as 1 search or 14?
[1] http://www.theatlantic.com/technology/archive/2013/08/how-go...
Even now, you can inspect JS traffic and see that usually there aren't 14 searches when you enter [superbowl 2017]. Internal metrics reflect a lot of that nuance and more.
Source: I own a Google Instant shirt because I was in one of the teams involved in months of testing and in the launch.
In spite of its good growth rate, DDG will have to change tack if it wants to make the leap to the next stage.
*FYI, DDG is my default search engine.
Results are decent, though the results page seems harder to skim read.
Also, half of my co-workers. !python, !mdn, etc.
I think the biggest hurdle going forward is the fact that DDG is much better for English results than in other languages (we work half in French), but I hope the project grows and grows. Google needs to go down eventually!
It's interesting to see that duck.com is owned by google. This is what they call competition :)
The duckduckgo team should be smart enough to keep secret an upcoming name change/rebranding. See for example how comodo tried to register "let's encrypt" trademark: https://tech.slashdot.org/story/16/06/23/1946249/comodo-atte... .
It's not easy to beat google at it's own game. Good luck!
As a researcher and doing some coding, it's invaluable for me to filter results that way.
http://apps.axibase.com/chartlab/e8635882/13/
Any good reasons behind it?
What's interesting to me is that there's such a consistent drop in searches around the holidays. I'd expect some drop, but not that big, relatively speaking.
And I moved from Google on my phone due to the AMP crap.
I'm annoyed that I can't set DDG has the default search engine in Chrome (Android). I hold a bookmark though so 'duc' autocompletes to duckduckgo. Not too inconvenient.
Never seen anything comparable in competitive products.
I have, and will continue to, pass good privacy practices on to others, particularly non-tech users, even though people don't always have the time for privacy discussions, they become aware of the issues.
What ? ... did people live in a cave all this time ? It existed for more then a decade all around. Some examples include early Opera, chrome, vimium plugin, Surfraw (linux CLI search tool) etc...
Anyway, DDG is refreshing. You can even turn off adds in the options. Google may be a great search engine but as far as I am concerned, it can delete 'don't be evil' phrase from its mission statement.
>In addition, the search engine is celebrating a
combined total of 10 billion searches performed,
with 4 billion searches conducted in December 2016
alone.
This can't be right, can it? DDG's been around for years, and nearly half of its total use volume has been in the past month?I can throw anything at it carelessly.
Much like when I use bing, I more often than not I must re search for whatever I want on google to find it.
Our community has helped us pull in data from over 1100 sources to make amazing experiences.
The value of this is that you can build out anything you are passionate about. It creates some great niche experiences.
Maybe you want to look up who is in space right now: https://duckduckgo.com/?q=people+in+space&ia=answer
Or you need to look up some NFL games: https://duckduckgo.com/?q=patriots+games&ia=nfl&iai=641
Or you can't remember the details of that card from your Magic The Gathering deck: https://duckduckgo.com/?q=mtg+boiling+seas&ia=games&iax=1
We encourage people to build out the answers they want to see!
DDG has a lot of open source code on GitHub. A list mapping bang search terms to site URLs could be maintained as a simple text file on GitHub so contributors could test them or suggest improvements.
https://bugs.chromium.org/p/chromium/issues/detail?id=348360
I have a blog on DDG's best bangs (and how to use them): http://duckgobang.com