Your user agent | Hacker News Reader

Your user agent

duckduckgo.com

107 pointseksith71 comments

And this is why I've grown to love DDG. My last "wow" moment was when I needed an ascii table two days ago. Searched for it and bam! it's right here in the front page.

To anyone skeptic about DuckDuckGo: I've been there. But I've decided to give it a week worth of a try some time ago (Google is just a !g away, anyway), and it's been my default search engine for months now. You might want to give it a try as well. Maybe you'll get hooked on the !bang syntax, maybe you'll get hooked to the 0-click information box, or maybe you'll just switch back to Google.

Some queries aren't always on par yet with Google, but I feel that it's getting here. It's mostly lacking on queries about specific errors, and local stuff. For everything else, I feel that I barely switch to Google anymore.

And, as a bonus, DDG works perfectly with Firefox Mobile. Google instead has some problems (probably caused by some webkit-specific code).

> Some queries aren't always on par yet with Google

I don't think google search is that great anymore. It's way too fuzzy of a match. If you're trying to remember an actor's name or find some reference with a vague query, it works ok. But google takes way too many liberties with search results and is super focused on showing maps, reviews, and youtube videos, and then results to alternate searches. Bing and DDG are better IMHO.

Google is useful to more people, which means it's less useful to people who are happy with things like the O Reilly Google Hacks.

There's some switch somewhere to force it into "Literal" or "verbatim" mode. https://news.ycombinator.com/item?id=3239452 http://insidesearch.blogspot.co.uk/2011/11/search-using-your... (These are the posts that give me the '1 in 600 searches used the + operator correctly' factoid.)

I think my current problems with Google are

1) Lack of predictability. I have no idea what terms might be substituted, or what they'll get subbed for. I don't know what terms get stemmed or not.

2) Recentism. Google is great at finding things from this year. Finding stuff from before 2000 is harder, and before 1990 is really tricky. (I don't have any examples! Normally I keep them.)[1]

3) Desire to return many hits. I'm happy if I enter a search term and don't get many hits. I'd rather have a few hits, rather than many hits that are less relevant. I'd rather have a few really bad hits, and thus know that I need to tweak my search terms, than get many sort of okay hits.

But for most people Google is great. Having said that, I switched to mostly DDG and I really like it, except I'd like to be able to force a more English (and not US) result cloud; and I'd like to be able to enter a term, have it translated into other languages, and search for those. This would be a way to break out of the English speaking bubble.

[1] I find it weirdly hard to find minor news events from before 2000, even though they're online and covered by newspapers, even if I use Google's time limit settings. I'll start to keep these difficult searches.

> I find it weirdly hard to find minor news events from before 2000, even though they're online and covered by newspapers, even if I use Google's time limit settings. I'll start to keep these difficult searches.

Might be because the pages are updated after the publication date. I don't know that google is able to find that type of structured info in each news page. Actually a specialized search engine which is focused on date functionality would be invaluable. Maybe someone like archive.org would tackle that. It would be great to search even pre-WWW results, such as historical archives or radio transcripts.

I've wired Safari directly up to a verbatim search, and it's made my life much less stressful. Don't know how long that'll be available for though…

Story time: Three years ago, DDG had an image-only ASCII table which meant you couldn't copy any of the values. I emailed Gabriel with a link to a pure HTML version that I had compiled[0], and within a few days, he had added to the site. Glad to hear you like the table, and a huge thanks to Gabriel for being so awesome.

[0]: https://github.com/kyleconroy/ascii-table

The selectable table is great. But confusingly, ddg still shows asciitable.com as 'source' and a "more from asciitable.com" link below it.

https://duckduckgo.com/?q=ascii+table

This blew my mind:

https://www.google.com.au/search?q=avail+etymology&client=ub...

That really is very cool. As an English speaker living in Germany this is really quite useful: https://www.google.com.au/search?q=avail+etymology#q=etymolo...

With regards to the ASCII table: `man ascii`

I searched for it for ages before realizing that it has been right there at my fingertips for ages.

Also useful: "man iso-8859-1" (aka latin-1, the most common Western charset if not using UTF-8). For the Windows version, you'll have to visit http://en.wikipedia.org/wiki/Windows-1252 -- non-UTF encoded files are more likely to be that instead of latin-1 (fortunately either is just a s.decode('cp1252') away from Unicode in e.g. Python)

There's a surprising amount of data files still being moved around in either of those. If your shell is in UTF-8 mode, you'll learn to recognize either encoding (a single funny character) or the UTF-1252 "encoding" (2 or more chars, where Ã is often one) where both utf-8 and cp-1252 were used.

The nice thing about DDG is it is built around competition, because of its emphasis on redirection. In theory, DDG can become the hub around which we access a whole variety of niche search engines, each time choosing which one is best suited to the task at hand. I think that encourages a fundamentally healthy ecosystem.

It would be even better if they could play up that element a bit more, if they could make the different search options in a particular area more discoverable. You want a better way to search for search engines.

Ah, like the good old days. There was a metasearch engine that was fairly popular in pre-Google times. Can't remember the name of it though.

There was probably more than one, but http://www.dogpile.com/ was the one I remember (which I'm quite surprised to see still exists/works).

Metacrawler? If so, its still there: http://www.metacrawler.com/

There were many as the commenters have said, but I always like hotbot.com

I also use DuckDuckGo for most searches, not because of the privacy aspect in the slightest, but because it produces more immediately useful results in most cases. The bangs are great for saving time. I even requested and after a while got given !ngram :-)

As you say, Google is just a !g away, and I do find myself still occasionally using that, but most of the time, DuckDuckGo's own search engine matches my requirements better.

Yes, this is exactly why I started using it. The privacy aspect is only a nice bonus.

I was really impressed with it just this morning. I've recently moved to Emacs and just couldn't work out how to create a new file. Everything just said it should be C-x f. But this just didn't work.

Get into work today where my default search engine in DDG. Thought I'd just have another search, type in "emacs create new file" and there right away in the quick info box is :

Try:

C-x C-f C-f

It should kick you out of ido mode into "normal" find file mode

Ah ha.. There's my answer. I have Ido mode enabled.

A similar search in google has a general (and useless) How to use emacs page as it's first link. The actual answer was a stack overflow question that came up second which I'd just missed before..

> Some queries aren't always on par yet with Google

I think a lot of queries aren't on par with google.

I'm bullish on DDG but it's got a long way to go if they want to deal with the spam issue that plagues tons & tons of queries.

I tried DDG for a week, but I found it much slower than Google which is what ultimately led me to switch back.

Try turning off the "Redirect" option in the "Privacy" settings tab. When it's on, clicking on a result will send you to a DDG interstitial page which hides your search query, and I found it to be slow at times (~1 second).

The slowness was in returning the results, not visiting them.

I wish that Duck Duck Go had more of these useful pop-ups.

I replaced Google with Duck Duck Go as my default search engine in Chrome a few weeks ago, and the switch as a whole has been positive. The only things I miss about Google are the instant suggestions for movies, timers, etc.

I still love Duck Duck Go, though (esp. the privacy benefits), and I'm sure that they'll broaden their offering of instant pop-up results as time goes on.

We'd love thousands and thousands of these.

--Please post ideas at https://dukgo.com/ideas (we're about to migrate all the old ones from http://ideas.duckduckhack.com/).

--If you want to hack on them check out http://dukgo.com/duckduckhack/ (our whole instant answer platform is open source).

--A bit out of date, but there a bunch listed at https://duckduckgo.com/goodies/

I made the switch about two months ago. IMO, Google still beats Duck Duck Go in terms of relevant results and I am curious whether Duck Duck Go can do anything about it as they are not gathering as much data as Google. I try to find my answer on Duck Duck Go first and if I cannot see anything useful within 30 seconds I end up adding !g at the end of the query which redirects to Google with the original search terms.

There's also https://duckduckgo.com/?q=ascii+table which I use so frequently that I should really just print it out and tape it to the wall.

$ man ascii

$ man ascii | cat | lp -

We have a nominee for the useless use of cat award.

Realized after I posted. Too late to fix it. The only reason I made the mistake in the first place is that I wasn't sure if the pager would mess up text output.

(What? You expect me to print three or so pages to test an assumption about how *nix utilities work?)

    man -t ascii | lp -

seriously, just do it. Having it on your wall is faster than any search can get it to you.

The thing I love about ddg is, it links to https versions of pages searched. For eg. Google will link to non https version of Wikipedia results, whereas DDG will link to the https version.

The thing I love about ddg is they don't seem to do that horrible redirect on some results that Google does.

I'll never understand why they do that; if you want to track links, send an async post when I click, don't add an occasional multi-second delay.

It's because browsers are unreliable about delivering async posts when the page navigates elsewhere. Some people proposed extending the web APIs to provide a reliable mechanism but it was shot down by complaints of "tracking". So now we're left with both tracking and horrible redirects.

The EFF'S HTTPS Everywhere plugin (available for Chrome and Firefox) works similarly and for a much greater number of sites:

https://www.eff.org/https-everywhere

We actually implement the same ruleset from HTTPS everywhere :)

Ok, sold.

How many people here just realized their corporate proxy is adding unwanted headers in every request?

Somewhat related.

https://www.google.com/search?q=ip

Similar, but plain text and, thus, easier to parse, if necessary, e.g.:

    $ IP=`curl https://icanhazip.com/`
    $ echo $IP

There's also ifconfig.me, who returns either an HTML version or a plaintext version depending on the user-agent, so `curl ifconfig.me` returns just the plain IP. There's also a list of other URLs on the homepage (for the host, user agent and a bunch of other stuff).

I've found ifconfig.me to be excruciatingly when I actually need it. Like takes 10-30 seconds for a response just to get my IP. Sucks that someone got the vanity domain and put a shitty server behind it.

Is there a reason you can't retrieve it via uPNP or similar?

If you want JSON output, then there is httpbin.

  curl https://httpbin.org/ip

returns

  {
      "origin" : "192.168.0.1"
  }

It also supports getting the user agent [0] and headers [1], among other things.

[0] https://httpbin.org/user-agent [1] https://httpbin.org/headers

FWIW:

    $ curl -4 https://icanhazip.com/
    88.…
    $ curl -6 https://icanhazip.com/
    2001:470:…

(for me it's worth a yay)

curl: (7) Failed to connect to 2606:f200:0:7:bad:f00d:d00d:2: Network is unreachable

Depending on what you use the IP for, asking a third party for your IP address could be an interesting attack vector for a MitM attack.

e.g. you need your IP to tell a remote server to send you something. You connect to the IP address service to ask for your IP, it should return x.x.x.x but it returns y.y.y.y instead. Any connections to any port on IP y.y.y.y get forwarded to a similar port on IP x.x.x.x meanwhile the MitM takes a copy of all of the data going back and forth.

It's not foolproof, and there are a myriad of ways to defend against it, but it's definitely a reason why I wouldn't trust any third party service (even DDG/Google) to get my IP address.

This is magnificent! I wish I knew about that before!

somewhat easier to remember: curl ifconfig.me

And, https://duckduckgo.com/?q=ip

I've used http://ip.jh.gg && http://ua.jh.gg for ip/user agent needs.

On this note, is it possible to completely remove my user agent entirely from all my browsers?

I'd like to get to the point where the only identifying information my computer gives out about me is information I give out about myself.

At the end of the day I shouldn't even suffer as a user because of this because if sites are "doing things right" then they shouldn't be using the user agent for anything at all anymore. They should be using feature testing only.

Since it's been such a useless feature for so long, I'm surprised no browser makers have gone as far as removing it entirely (I'm looking at you chrome and firefox)

You probably don't want to send blank information for everything, this could make you more easily traceable. Just like disabling cookies/javascript might actually decrease your anonymity.

If you use Firefox, you can install something like Blender (https://addons.mozilla.org/it/firefox/addon/blender-1/) which fakes your data to some common data (to make you "blend" in the crowd).

Now I digress, but I suggest disabling (or use click2play) plugins, as they leak a lot of informations about you.

For cookies (which are used a lot for tracking), you can install something like Self-Distructing Cookies (https://addons.mozilla.org/it/firefox/addon/self-destructing...), which accept cookies (like most browsers do), but delete them when you close the tab.

Don't forget something like RefControl (https://addons.mozilla.org/it/firefox/addon/refcontrol/)/ Smart Referer (https://addons.mozilla.org/it/firefox/addon/smart-referer/), as the referer can be used to track you.

And the classic Disconnect.me (https://www.disconnect.me/) or AdBlock + EasyPrivacy (https://easylist.adblockplus.org/en/).

See, privacy/anonymity on the web requires a lot more than just disabling the useragent. There are several other ways to track you, and it might have some legit uses (for example, serve you the right software for your operative system, I don't think there's a javascript api to recognize your os). It might be a good start, but it wouldn't help a lot, and it would break a lot of legacy code.

If your goal is to eliminate information that can be used to identify and track you, being the only person on the internet with an empty useragent string may be somewhat less than ideal...

https://panopticlick.eff.org/self-defense.php has some strategies.

You can fake it with a number of extensions, but there's a load of fingerprinting options that are sneakier than the user agent. The Tor Browser project has a document outlining changes they had to make to make things less fingerprintable.

Disabling the useragent may break some stuff that many sites still do server-side.

Disabling referer may break downloads and images on many sites. You'd think they'd all be able to support blank or non-existent referrers, but many large CDNs only support whitelisting specific REFERER strings (not blank ones). We'll be turning referrer blocking back on for one of my big sites shortly because we have other sites linking directly to file downloads (of 60MB and up) and hotlinking images.

Yes. For example, it disables graphs in Google Analytics. They do not work, if your useragent is empty or if it is Googlebot 2.1 ;-)

There's still a lot of differing behavior in modern browsers that a feature check doesn't expose. Bugs, quirks, performance characteristics (e.g. slow SVG animation in Firefox), and just plain divergences from the spec.

HTML5 media is my favorite example of this: mobile devices will actually straight up lie to you when you call certain methods.

If you're making even moderate use of any HTML5 features, you're going to end up doing user agent sniffing.

It hasn't been removed because of exactly what you say; it would break a lot of sites. If it is a change that breaks a good portion of the web, they rightfully shouldn't make that change.

The user agent is useful for debugging. If users of a certain browser never submit a form, the site owner needs to know.

Anyway, feature testing and benchmarking can be used to create a user agent fingerprint. Hiding the user agent header just keeps it "secret" for a few weeks until the big trackers all switch to fingerprinting.

Just give up on hiding identifying information. The promiscuous model used by web technologies is not securable. For instance, timing analysis can be used to determine which sites you have recently visited.

User agent headers always uneasily remind me of this: https://panopticlick.eff.org/

Is there no way of preventing your browser supplying enough information to uniquely identify your machine?

NoScript can prevent most information from being gathered, but unfortunately "no information" is actually pretty unique.

Using Chrome on Windows with Flash and Java installed is probably the best way to blend in the crowd and make yourself less unique.

Don't try to rely on Chrome's Incognito mode or Firefox's Private mode, either. IIRC, I tested Panopticlick on the former and it was actually more unique on that particular PC than Chrome in "normal" mode.

Panopticlick has been out for a while, so I'm hoping more and more people are going to hear about it and eventually there will be tools developed to combat that sort of device IDing.

I did a quick Google search and I found Secret Agent, a Firefox add-on, that claims to "randomize your Firefox HTTP user agent, to surpress device fingerprinting, and resist web tracking" : https://www.dephormation.org.uk/?page=81

This made me to switch to DDG (for the second time) as default search engine. I'll give it more time now, I didn't know about all these "goodies". For instance, I just searched for a youtube link and it offered me a box with a link to the wanted video and also the possibility to play it right from there. This is so cool :)

Love DDG but discovering your UA on SERP like that isn't very useful. If you're someone who needs to know your own UA, you`ll probably need it several times a day or more. Searching for "user agent" is the long way to do it, there are browser extension for that. Still, nice Easter Egg from the best Duck in town.

I love the bangs on DDG.

arachnids !W (wikipedia)

a train !lyrics

jambi !grooveshark.com

northernlion !YT (youtube)

just shove it in the address bar. no waiting for tabbing or hoping google gets it right.

Just want to share that VIM keys for up and down (j/k) work when selecting search results.

What are we supposed to see here?

Reply on news.ycombinator.com