Show HN: Browser extension to read HN comments for any url, in ClojureScript

Because DNT was the quickest solution. I'm actually rewriting that part over the next couple of days to be opt in instead (via a popup the first time the sidebar is opened, I think).

One problem is the risk of self-selection bias.

I did read your comment (and I didn’t downvote you b/c I agreed with you). The reason I wrote this response was because your comment read as being very aggressive in response to an honest attempt at explaining an issue the dev was having. What your comment lacked was any additional information about better approaches.

I get the concern and share it, but your comment only said that what they did was wrong. It didn’t offer any ideas of how to fix it. Telling someone they’re on the wrong road but not telling them how to get to the correct one leaves them just as lost.

The extension icon should have a number on it that corresponds to the number of comments. My thinking was that you could use that to see if a page had comments or not.

You can at least start by stripping all query parameters and anchors off of the URL. And instead of automatically piping everything to Algolia, just give the user a damn button.

Algolia doesn't even have to (intentionally) abuse their service. Request logs can be leaked through software bugs, malicious actors can gain access to their servers, etc.

Trying to find matches for URL hashes would be a much nicer approach, but you would need a database of hashed URLs for that, so it's not a trivial fix.

> Unfortunately, I don't think there's a way to get that particular UI feature without sending all URLs to some external server.

Look up the protocol that Google's SafeBrowsing[0] project uses. The short version is that your extension should check visited URLs against a locally-downloaded database of HN-posted URLs. For SafeBrowsing there's a bit more optimisation (e.g. hashing URLs & storing only hash prefixes to reduce the size of the database), but that's the short version.

This secures the end user's privacy.

[0] https://safebrowsing.google.com/

How will you tie the relevant minion to the hash?

You'd have to have a lot more than just that hash, the log would at least have to include a static IP or something that you can isolate by window-of-opportunity, for instance all the other minions were at a ballgame and the timestamp indicates that that one minion that wasn't at the ballgame visited tips.fbi.gov right then from an IP not associated with a stadium hotspot.

Regardless, if you're going to leak stuff on 'evil overlord's organization you'd better make sure you don't do it from anything that can be associated with you, so not your laptop, not your IP, not your browser and certainly not with all kinds of weird plugins installed. Burner device and a location and time chosen so it could be anybody in 'the organization' leaking.

How do you get to 'less than a megabyte'?

A reasonable hash is going to be 64 bytes or more to avoid collisions, the total DB would be a least a couple of 10's of megs.

I don't think we need either a strong hash or to submit hashed URLs to a server. We can use an ordinary hash and the only URLs we need to submit to a server are story requests to HN, if we do this thing like this:

Include in the extension a hash table construction as follows:

  foreach ID of an HN story submission
    URL = the URL of the submitted story
    URL = normalize(URL)
    insert_into_hash_table(URL, ID)

insert_into_hash_table(key, val) is a function that inserts val into a hash table with key key. The hashing function does not need to be cryptographically secure.

normalize(URL) is a function that takes a URL and normalizes it. What normalize means in this context is a little fuzzy, but the basic idea is that if URL_1 and URL_2 are different URLs to the same article, normalize(URL_1) == normalize(URL_2).

NOTE: what you include with the extension is the hash table itself. Conceptually it is probably just a sparse array containing HN IDs, with maybe a little more depending on how collisions are handled.

In the extension, do this:

  URL = normalize(URL_of_current_page)
  ID_list = lookup_URL_in_hash(URL)
  foreach ID in ID_list
    story = get_HN_story(ID)
    if (normalize(URL_of_story(story)) == URL
      show_story_comments(story)

The hash is only used for data retrieval from a local hash table, so does not need to be cryptographically secure.

After the hash lookup we have a list of candidate stories on HN that might match the browser story. It's a list because due to collisions there might be more than one HN story with matching hash.

Note that all that is ever fetched from the server during operation of this are HN stories, so there is minimal information leakage.

But wasn't the 16 million estimate based on the increasing ID numbers, which includes comments and self-posts, neither of which don't need to be hashed or checked? The number of submitted URL's that need to be checked against is presumably much smaller, and the number of those that have at least one comment is much smaller than that.

Edit: Estimating quickly from the new submissions for the last couple hours, it looks like there are on the order of 100 submissions per hour. Of these, it seems like less than 10% have any comments. So if we assume that HN has been up for 12 years (100,000 hours) we get about 1 million commented URL's with comments, and that's probably an overestimate (dups, increasing usage, etc).