Hacker News Reader

Comment by benhoyt | Hacker News Reader

As someone who's just written about the "small web" [1], this warms my heart. The lite version is probably a little too bare-bones for most people's tastes, but it sure is tiny -- great for people on very slow or flakey connections. Some numbers:

HTML homepage transfers 34KB (94KB uncompressed) over 6 requests. HTML search results page transfers 133KB (248KB uncompressed) over 31 requests.

Lite homepage transfers 13KB (11KB uncompressed, ha) over 4 requests. Lite search results page transfers 21KB (43KB uncompressed) over 5 requests.

In all cases, all requests are to *.duckduckgo.com, which was very good to see from a privacy perspective. Nice work, DDG!

[1] https://benhoyt.com/writings/the-small-web-is-beautiful/

View on HN

lite is amazeballs as a command-line utility (leveraging any of several console-mode browsers, including w3m, lynx, links, elinks2, etc.).

As a bash (or zsh) function:

  ddg ()
  {
      /usr/bin/w3m "https://duckduckgo.com/lite?q=$*&kd=-1"
  }

Note that this enables any bang searches as well, though you'll need to single quote these to avoid attempted history expansion.

lite is also my default w3m search bookmark entry.

So:

   ddg '!w foo' # Wikipedia article on foo
   ddg '!dict foo' # Dictionary search on foo
   ddg '!etym foo' # Etymology Dictionary search on foo

Note that if the endpoint itself relies on JS for local search, the bang won't be successful (though DDG will do its bit). Reddit, and HN/Algolia, I'm looking at you.

I happened to be running some quick lookups a few days ago while a friend was watching, and they 1) wondered how I was doing that and 2) if they could have a similar feature. Power of the shell.

Wow, that's amazing!

Similarly, in Firefox (and maybe other browsers) you can add keywords to bookmarks, meaning that the browser will open that bookmark if you first type the keyword. It will also then insert anything else you type into the bookmarked URL if it has a %s. So as an alternative, more reliable way to search MDN, I bookmarked this:

    https://duckduckgo.com/lite/?q=site%3Adeveloper.mozilla.org+%s

... and gave it the keyword >mdn (I add ">" as a prefix to all my keyworded bookmarks - avoid accidentally triggering or autocompleting to them).

If I'm browsing and need to look something up, I just hit CTRL+L followed by

    >mdn [whatever I'm looking for]

... and off we go

The benefit to bookmarks / keywords is flexibility and not having to wait for DDG to add (or fix) a given bang search.

The downside is that you're speaking your own language, and unless someone adopts your specific keywords, you can't tell them to, say, "bang dict" (if you can get away with saying that in the context).

This is a fundamental distinction between any private vs. shared language.

(People are very uncomfortable when dropped into my computing environment --- GUI, shells, editors, browsers, etc. They've all acquired several decades of personalisation. Works for me.)

Have you heard of Julian Assange’s surfraw?

  $ alias d='sr duckduckgo -text -ducky'
  $ d '!man surfraw'

  [www-browser opens http://manpages.org/surfraw]
  […]
  DESCRIPTION

   Surfraw provides a fast unix command line interface to a variety of popular WWW search engines and other artifacts of power. It reclaims google, altavista, dejanews, freshmeat, research index, slashdot and many others from the false-prophet, pox-infested heathen lands of html-forms, placing these wonders where they belong, deep in unix heartland, as god loving extensions to the shell.
  […]

Edit: even better, dict(1) (or GNU dico):

  $ dict foo # same text as !dict’s http://dict.org/bin/Dict?Form=Dict2&Database=*&Query=foo

Yep, both the tool and its author.

I also use dict, have Debian's gazeteer TIGER files & miscfiles installed, dwww, RFCs, and more.

Info-on-tap is wonderful.

What is the &kd=-1 for?

edit: To answer my own question https://duckduckgo.com/params

These are great, and I’m glad to be able to use them without direct contact with “Big Tech” and the tracking bloat that entails, but as I understand it DuckDuckGo is more or less a glorified — albeit relatively glorious and pleasant — proxy to Big Tech’s Bing (well, more, not less, because it does add some great conveniences like the IIUC formerly open source Instant Answers¹ and !bangs, and some fraction of results from its own crawler, and likely some from Yandex. But still, the largest fraction of its core service seems dependent on Bing. Not that that’s its fault or there are attractive alternatives).

1: (Edit) https://duckduckhack.com/

Yeah, I wonder what their long-term plan is there. Surely to get substantially better, they'll have to move off Bing at some point (or buy Bing off Microsoft!). In any case, as long as DDG is a "privacy-stripping proxy" (which I believe it is), it's just using Bing technology as a search API, which doesn't seem too problematic.

I agree. Fortunately, to move off Bing doesn’t have to be a sudden flip of a binary switch.

> DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google). — https://help.duckduckgo.com/duckduckgo-help-pages/results/so...

I find their instant answers fairly useful, enough to often not use the other results at all — unfortunately for them enough to use some of the sites some answers are sourced from directly, in particular often using my browsers’ built in Wikipedia searches.

DuckDuckBot I can’t judge, because I don’t know which results come from it (previous comments on HN seem to show the amount of difference between DDG and Bings results vary between search terms¹), but it seems that non-Bing links are a thing and DDG can adjust what fraction of the page they take as whatever gap there is between them and the bigger crawlers’ narrows or widens.

1: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

DuckDucGo's classic link results are verbatim from Bing (or sometimes Yandex instead). DuckDuckBot is only used to grab favicons and a subset of its rich results (instant answers, zero-click info).

Don't take my word for it; you can compare DDG and Bing results for esoteric queries side-by-side. The order of the results may vary since search results aren't deterministic, but you'll find them to be otherwise identical. You can also ask staff in help channels about the details of where link results are sourced from.

I've been using Bing, directly, more and more in the past few years, when Google either gives no useful results at all, or blocks me for trying to use it with more precise queries. Overall its breadth is still lacking, but occasionally it finds useful things which Google doesn't (or won't?) find. It also doesn't try to mangle (only truncates when too long) URLs.

Among other characteristics, I'm finding DDG works when searched via a Tor proxy, and even offers an Onion URL (https://3g2upl4pq6kufc4m.onion/).

Google 1) fails to work without JS for me much of the time and 2) throws up endless ReCAPTCHAs, so I generally don't bother (not just for GWS, but Scholar, Ngram viewer, and other actually-useful tools). My preferred response (https://toot.cat/@dredmorbius/104371588129861216) produces very faded joy over Tor.

DDG lite works great in Lynx! I hope it never goes away.

It always annoys me to lazily type the short “ddg.gg” in and wait for it redirect first to duckduckgo.com, then to https://lite.duckduckgo.com/lite/, but once it’s bookmarked it’s easier to use than SearX.

Is there a lite version of Google search? How does Lynx work for Google searches?

1. My site is also smol, but brand new and probably doesn't warrant a mention here, but respect for smol web!

2. > Because most people only view one or two articles on my site, I include my CSS inline. With HTTP/2, this doesn’t make much difference, but Lighthouse showed around 200ms with inline CSS, 300ms with external CSS.

With respect, this seems to be the frontend consensus and it... just doesn't make sense to me. Unless you expect the majority of your traffic to never hit a cache header on that external CSS. That 100ms perf hit (which is probably a warning sign that something's wrong with your assets or server config anyway) should be a one time affair, and not repeating that payload over and over surely makes up for it in the sub-200ms responses after.

These are fast!

Noticeably faster than the default JavaScript DDG.

The lite version has a broken image at the bottom. Other than that it is very nostalgic of Alta Vista home page back in the days. Kudos to DDG for having such simple non-javascript version. In current days, non-javascript web page is on the rise. All web pages should have a non-javascript lite weight version.

> All web pages should have a non-javascript lite weight version.

Couldn't it be the exact same page, that still works when javascript is disabled?

I'll play devil's advocate as an engineer.

"HTML" version shows no images, just text and favicons. The total size of visible text content shown on the page is about 4KB (just did a count of characters on the page for "steve jobs" query). DDG shows those 4KB using 248KB of data.

1. That is not _that_ small. It's only small compared to the rest of the web, but not in absolute numbers, or in comparison to useful content shown.

2. The content to data ratio comes at 1:62. Or for one character shown on screen you are transferring 62 bytes of stuff. For context, Wikipedia page for Steve Jobs, which also includes multiple images, is 1.08MB and content is 128KB for a content to data ratio of about 1:9. So DuckDuckGo could in theory do ~7x better than this.

Sorry, as an engineer, not impressed at all!

Can you give a more coherent comparison? For example, between a term search in DDG HTML vs the JS version or DDG HTML vs Google.

Comparing the search results page with a wikipedia page is hardly any sort of evidence.

On a side note, saying "as an engineer" doesn't mean anything on HN, especially if you don't even say what type of engineer you are.

Reply on news.ycombinator.com