Hacker News Reader

The idea is still interesting. I imagine using some dictionary based compression and crawled data to build a dictionary could get us somewhere.

View on HN

Unfortunately the longest urls, the ones you want to shorten the most, are the worst candidates for compression (with long query strings full of things like uuids).

I have an idea to identify the uuid and other identifier type parts and encode them up as numbers. The file radix_coder.js is working toward this. So we cover a few formats like guid base64 digits base36, to get a little bit more gain.

But my initial experiments suggest it's just a little gain. Digits go to 68%, base36 to 90%, and then we also have to add in the prefix to indicate we are switching encodings.

That's basically what a Huffman code is. Once you have a big list of examples, it's just a way to encode it to take advantage of frequency of occurrence.

If it were to use a dictionary that defines a large subset of domain names and common strings found in URLs it could reduce the size somewhat more, but still seems too long to be viable. What might be more interesting is an Ethereum approach to URL shortening that is distributed amongst the Internet.

Reply on news.ycombinator.com