Show HN: I built a FOSS service to protect against link shorteners tracking us | Hacker News Reader

Show HN: I built a FOSS service to protect against link shorteners tracking us

unshort.link

57 pointsl1am039 comments

Related, the Archive Team realized a while ago, that when a link shortening service goes dark, all the links that went through it instantly break. This has bad implications for future historians trying to figure out how the present-day web was interlinked.

So there's a project called URLteam to index and save them. I just started running a Warrior instance on the URLteam project, and it's really straightforward. I figure folks reading this may be interested. More info:

https://www.archiveteam.org/index.php?title=URLTeam

https://www.archiveteam.org/index.php?title=ArchiveTeam_Warr...

Where and why are you guys still seeing/using shortened URLs? I don't think I've personally made a shortened URL since Twitter built one into their service, and I stopped seeing shortened URLs around the same time.

There are two exceptions:

• I still see service-specific short URLs, e.g. goo.gl, nyti.ms, etc. Because these can only be created by the service in question, they don't introduce security/privacy concerns.

• When a URL shortener is used to create a sort of vanity link to e.g. a Google doc. (OP's service could be useful for these!)

Outside of those times... why would you use a URL shortener? When are you limited by number of characters?

If you use an tracker blocker, you’ll find a slew of app to app and site to site links and callbacks fail because they went through a tracker that sometimes is a short URL service, but often just a long URL redirector.

These are definitely not curated by the service in question.

On iOS, examples include:

    [business].app.link
    [business].onelink.me
    [hash].ulink.adjust.com
    [business].bttn.io
    branch.io
    etc...

Link shorteners are a subclass of this, that do something useful (shorten link) while doing something unhelpful (obscuring the link). But it’s all the same problem.

I use one inside of a big corp so I can put memorable urls like myco.co/news on posters or email signatures. Especially when the real url is some terrible long thing I cannot control, like some internal wiki.

Link shorteners are also used to ubfuscate tracking urls. E.g you can hide https://fb.com/?utm_source=hackernews&other_tracking_paramet... behind a nice looking bit.ly/fb url.

In my research I also found that it is a quite nice thing for phishing to fool you into opening a shady website.

• I still see service-specific short URLs, e.g. goo.gl, nyti.ms, etc. Because these can only be created by the service in question, they don't introduce security/privacy concerns.

This isn't necessarily true; I've seen companies that offer to do the 'custom shortlink' thing for various companies. It's safe to assume that there are privacy concerns with at least some of them.

Bitly runs nyti.ms, so yes, you aren't trusting just the NYT.

Yes, and nyti.ms only ever pints to New York Times, a well known entity (well, that's what's intended, right?)

Sure. Noting that you have to trust that bitly isn't fingerprinting your browser and tracking you across their domains.

I use them all the time to transfer URLs between devices that enxode a lot of state in a long url. For example, Firefox Send pits the password in the URL. Of course, I'd never do this with something I didn't watch associated with me.

On youtube every so often complementary materials are linked via shortened urls, but youtube also tracks those in turn via their redirect urls.

Some subreddits use them to track which links people click on in their sidebar. /r/listentothis in particular.

SMS spam.

Reminds me when security researchers used data Bit.ly exposes to go from one url shortened by Russian military intelligence to the hundreds of other short URLs they made for phishing emails, many of which had state embedded in query parameters that gave away who they were targeting. All because the GRU forgot to set their accounts to private.

https://www.vice.com/en_us/article/mg7xjb/how-hackers-broke-...

Oh wow thanks for all the feedback. Should not post a link on HN and than sleep directly after.

I came to work on this as I wantend to not get bothered with the short link services (and the links behind them) tracking me. So unshort.link also tries to "learn" (by trying different url parameters) which paramters on the long url are required and which are not and are only tracking you (with a HTML diff).

The service learns new shortlink services the moment you enter them on unshort.link and they than are also automatically used in the extension.

Yes yes yes! I hate URL shorteners. It's not just privacy - I can't decide if I want to click it or jot before I do.

I suppose that issue could also be solved with a browser extension.

It'd be nice to do this with DNS rather than an extension though, above point aside, e.g. if already running Pi-hole or similar, send shortener URLs to unshort.link, and redirect immediately.

Pi-hole would only see the hostname, and not the "slug", so that wouldn't work.

What I mean is, you'd send `t.co` and similar to your unshort.link server, which then has the `Host` header `t.co` and the path; so it can do the redirect.

I know it's not how it works today, but I wasn't describing how it works, I was describing how I'd like it to.

Pi-Hole sees the bit.ly DNS lookup,that responds with HTTP/30x to full.domain.tls, which is a second DNS lookup the Pi could catch/filter.

But bitly.com/xyz can be mapped to full.domain/abcdef

The pi-hole doesn't see the /xyz, or any of the http redirects either.

unshort.link has a browser extension for chrome & firefox, you find it on the site.

If you directly want to get redirected without showing the url you can activate that feature in the plugins config.

Or you can send the shortlinks to http://unshort.link/[http://short.link/sd] and it will directly redirect you

Yes, that's the browser extension that I was referring to when I said I'd rather do it (for all devices) via DNS.

    1. Resolve shorterners to unshort.link server
    2. Use `Host` header and path to redirect to unshortened page

Would be awesome if you build something like that! You can get the currently to unshort.link known short link providers in a nice json format from https://unshort.link/providers

Maybe you can run a cronjob setting that urls in some firewall file or so.

You can get the redirect info from unshort link also via its api without redirecting, just replace /d/ with /api/ in the GET request: e.g. https://unshort.link/api/https://tinyurl.com/unshortchromeex...

If you build an DNS solution, please do not forget to add the documentation on how to do it via a pull request to the unshort.link repo, so everyone could profit from it

Pretty cool. You might consider adding some of the more popular vanity names that bitly runs for big sites. Like nyti.ms (New York Times), wapo.st (Washington Post), etc.

You can add them yourself :D Just unshort them on unshort.link (not the browser extensions) and the service learns that it found a new shortlink service and it will be deployed right away.

After the next browser restart also the extension knows about that new service. (No updated required)

You can see if your updated worked if e.g. wapo.st is listed in https://unshort.link/providers

How do we know this site is not tracking us?

TBO you would never know, that's why the project is open source and if you want to be really sure what happens here, run it yourself.

The browser extension can be easily (in the settings) tweaked to use your own server instead of unshort.link

You can run it yourself, and audit the source if you like: https://github.com/simonfrey/unshort.link/

If you run it yourself, do you not lose some of the privacy benefits?

I believe you gain privacy by running it yourself, since you're no longer submitting a portion of your browsing history to a third party. The README for the server claims "You can build & run it yourself for even better privacy".

Shorteners are frequently used to hide/prettify query parameters used for tracking; so if you run your own you're losing the aggregate anonymity-ish, and gaining that OP can't see what URLs you're unshortening.

I suppose it depends on your use, but IMO if you run it yourself as the only user, you lose more privacy than you gain.

If the website you're ultimately visiting can see your IP address anyways, I fail to see the lost privacy. Their traffic will be sliced and diced into cohorts along many axes, and this is only one axis on which you are slightly anomalous. If anything, you'll be targeted less as a result since you won't show up in their funnels.

Is it possible to make such a thing working if using command-line programs such as curl or wget to download a file rather than opening it in the web browser?

It should totaly be possible. Same as with the DNS solution you can get the data via the api of unshort.link and than build a small bash script for it. If I have time I may try to do such a script, but not sure if and when I will be able to make it.

Would be awesome if you build something like that! You can get the currently to unshort.link known short link providers in a nice json format from https://unshort.link/providers

If you build a solution, please do not forget to add the documentation on how to do it via a pull request to the unshort.link repo, so everyone could profit from it

This is what I use which also removes utm_* tracking parameters.

  getloc () {
     curl -sI "$1" | grep -i "Location" | awk '/http.*/{print $2}' | sed 's/[%+]*//g' | sed 's/&*utm_[a-zA-Z0-9]*=[a-zA-Z0-9]*//g' | sed 's/?\s*$//g'
  }

This is really cool. And it's all written in Go? Interesting.

Do you have any plans on releasing the static JS/HTML? Or is there a way this could be run on standard Apache/Nginx/IIS web servers?

No I do not play to make it runnable on Apache/Nginx/ISS as there is more logic in it, than just the redirect.

The js/html assets are in the subfolder server/static/ for your use

Add: Replacement of affiliate or referral links with your user' affiliate link and you can charge for this service.

I wanted to make something like this with my C.GG domain but couldn't refuse an offer to sell the name for $10k

Reply on news.ycombinator.com