Hacker News Reader

Comment by eps | Hacker News Reader

Compression speed is around 2 KB/sec.

The figure is given in nncp_v2.1.pdf. This is 1000x slower than xz, which in turn is 1000x slower than gzip.

I have always wondered if a compression service could be a viable commercial offering and with NNCP rates and its compression speed, it looks like it could! That's for cases when you need to really compress the heck out of something, e.g. prior to distributing a large static piece of data to a very large number of recipients. I think there was a service like this, used by AAA game developers for distributing patches, but it didn't take because it had a very slow decompression... and, now reading further through the paper, it looks like an issue with NNCP too -

    The NNCP v2 decompression speed is similar to its compression speed.

So it's a no-go for my brilliant and lazy startup idea, but fascinating stuff nonetheless.

View on HN

I think maybe matching the compression speed with the network speed is what you want to do.

In a place with ubiquitous fast broadband, a compromise with regard to decompression speed might be ok.

If you're sending something to mars... 2 kb/sec might make sense.

>If you're sending something to mars... 2 kb/sec might make sense.

Keep in mind that 2KB/s is on something like 3090... Massively power hungry even on the decompression.

> I think maybe matching the compression speed with the network speed is what you want to do.

You might be interested: Zstandard has an adaptive mode that attempts to do this.

Hm, there are likely multiple large files that exist that are very common (DLLs, versions of shared libs, etc etc etc) that might be frequently compressed (SCP, stored in SSDs, etc) since disk space isn't free yet last I checked.

Each file is really its own special theoretical multivariate space to find the best theoretical compression ratio. Usually finding the optimal or near-optimal compression is the hardest computationally.

And it's basically a perfectly cacheable computation if you have some sort of database mapping algorith + parameters to apply to a given file.

Decompression usually isn't as big a deal for performance.

Anyway an online database for a service that has actively tried to take a known common file (match on filesize, hash, etc) down to its best compressed representation so 1) we don't have to do it every time.

Cloud providers would presumably love these, since S3 probably has millions copies of many files and would 1) get to compress the file and 2) get to share its representation across all instances.

Compression speed is around 2 KB/sec.

Apologies if this is a noob question, but could you improve it with training?

> Apologies if this is a noob question, but could you improve it with training?

Considering all other things like GPU, code quality etc. equal, the speed here mainly depends on the size of the network. Larger networks result in better performance (logarithmic), but also slower speeds (linear). There are some tricks like pruning to create significantly smaller networks with similar-ish performance, but good models are often large.

Unrelated to your question, what nobody mentions is that the decompression side also needs to complete network. With 57M and 187M parameters, those files are going to be quite large, > 100MB for sure. That completely annihilates the performance wins for 1 time transfers.

No, as I understand it, the network does not need to be transmitted. The decoder learns on the same stream as the encoder, so you do not need to transmit network parameters.

If I compress a large file at location A using this algorithm, and want to send it to location B, how does location B know how to decompress it?

Basically the same way your favorite codec knows how to decompress. It builds the predictor (in this case, a NN) from the known input. The predictor then assigns a symbol (or a series of symbols) to the next few bits - higher probabilities need fewer bits (I haven't checked for this specific technique, but there are techniques that don't even require an integer number of bits, e.g. range encoding).

See https://news.ycombinator.com/item?id=27244810

As I understand it, when starting up (beginning of stream) - the network does not compress, then guesses the next byte at every turn determininstically from input so far.

Decoding can "read" the fist byte, then proceed to learn and guess the second byte and so on.

see the (current) top comment

https://news.ycombinator.com/item?id=27244810

it's a very clever use of symmetry.

It's clever, but pretty much "standard" in compression. Earliest I'm aware of that used symmetry in the encoder and decoder to prevent explicitly transferring the parameters this way was LZ78 (Lempel, Ziv; 1978), but there could well be predecessors I'm not aware of.

LZ78 used it "just" to build a dictionary, but the general idea of using symmetry is there.

(A fascinating variant on this is Semantic Dictionary Encoding, by Dr Michael Franz in his doctoral dissertation (1994), that used this symmetry to compress partial ASTs to cache partially generated code for runtime code generation)

Yup, it isn't a dictionary based compression. Nonetheless that's a fascinating take

With preprocessing it is a dictionary based compression. It uses a fixed dictionary to improve on compression.

This is what the games industry uses in practice. It is optimized for fast decompression. http://www.radgametools.com/oodle.htm

Reads like a snake oil blurb to be honest. No comparision against any modern compressors, eg lz4, no specs, no pricing. Just "trust us, it's awesome." Looks really strange.

Of course there is comparison with LZ4: http://www.radgametools.com/oodlemermaidselkie.htm

LZ4HC is LZ4 compressed with the slow compressor to achieve better compression. It is decompressed using regular LZ4 decompressor.

They also compare with zstd and lzma, which are the open source competition.

I don't know what "specs" you think are missing. They do say which architectures are supported, what the memory use is, whether multi-threading is supported, etc.

No public price, true.

You can read the blogs of the people who work on this code to get a feel for the caliber:

https://cbloomrants.blogspot.com/

https://fgiesen.wordpress.com/

These are known people with known reputation. The product itself is well known in the industry.

What about decompression speed ?

For real

I know shipping speeds are important, but it seems game developers could try harder with delta compressing their updates, because it's hard to believe a tens-of-GBs download is the best they could do when you already have a BluRay disk or something (unless it's new content, of course)

I wonder how much their bandwidth costs are and how quickly a reduction would pay itself.

Does DRM contribute to this perhaps?

Given the layers of encryption and obfuscation, and leaving aside the merits or otherwise of DRM in the first place, I can imagine it could be hard to do delta patching. At least, without making it easier to crack the DRM - e.g. you’d presumably have to decrypt the asset that needed patching, apply the delta, and re-encrypt/re-sign it all client side.

Would it even pay for itself? It seems to me like bandwidth is pretty cheap compared to computation costs so why bother trying to squeeze everything out. Just do some basic compression and you're good to go. Buy a bigger SSD if necessary, with technologies like reBAR compression just slows down a game anyway.

As far as cloud providers go, bandwidth is where the lock in is. It's priced to make it too expensive to switch to a competitor.

What would pay for itself is p2p, e.g. bittorrent... alas.

Blizzard used to use Bittorrent in some of their installers/updaters a while ago.

I'm not sure how viable this is today, or ever was: Many residential connections are heavily asymmetric. Players presumably aren't too excited either by the idea of continuing to seed while they already play online, with shitty ISP-provided routers suffering hundreds of milliseconds of bufferbloat whenever the uplink is saturated.

>Blizzard used to use Bittorrent in some of their installers/updaters a while ago.

Yes, WoW used to have bit-torrent based update.

>Many residential connections are heavily asymmetric.

this is a US centric-view, though. For instance I have 300Mb up/down. At least for most of the Europe (and I suppose Asia), it's pretty common not to be restricted in any way.

I could leave a bit-torrent client non-stop and won't feel it. Like I've said: alas. There is some stigma associated with p2p/torrents.

I have very rarely seen symmetric broadband plans available in Finland. In the past I've had 10/1, 24/1, 100/10 and now I have 1000/100. With ADSL connections being asymmetric was basically necessary because the bandwidth is so limited - I think most consumers would prefer 24/1 over 12/12. With fiber I think it's more about market segmentation: ISPs want businesses to buy significantly more expensive symmetric plans.

Not sure where you are in Europe, but in Germany you have to search a bit to find a residential ISP with symmetric bandwidth. Mine is 50/10.

Asymmetry is the rule in my country for both copper and fibre lines, although it's less pronounced with fibre. Symmetric lines are a specific business option.

> At least for most of the Europe (and I suppose Asia), it's pretty common not to be restricted in any way.

In the Philippines, 15 Mbps down / 3 Mbps up ADSL with a static public IP is not something that a foreigner could easily buy. Globe Telecom in Cebu asked for evidence that I have sufficient income to pay for this (2600 PHP per month), and their rules only allow documents from the Philippines, so my bank statement was rejected. I had to talk to their higher-level manager to negotiate an exception. And their default router has 2 seconds of bufferbloat, so I bought a TP-Link Archer C7 and installed OpenWRT to compensate.

> this is a US centric-view, though

I live in Europe and I have a 1000/0.5 Mbit/s connection.

My gut feeling says you have a 200/0.5Mbit/s connection at maximum, because just from watching some download monitor out of the corner of my eye while downloading something with 100Mbit/s requires about 250kbit/s ACK packets in the upstream.

No more ACKs? No more DL!

No, I can still hit (almost) 1 Gigabit on the downlink. DOCSIS networks are so heavily asymmetric, they take special measures (ACK compression) to get around that!

Interesting... never been on Cable/DOCSIS.

That seems very atypical. As the link capacity ratio goes past 10:1, even for typical dowloads, speed starts to be limited by the ability to send requests, or even TCP ACKs.

I've seen 20:1 offers on cable providers, but it's mostly marketing BS, as you're unlikely to get full download speed in most circumstances due to upload limitations.

True - my nominal bandwidth (i.e. what I pay for) is 1000/50.

I do reach the full downlink speed though due to (presumably) ACK compression in the modem and ACK synthesis/reexpansion in the CMTS. You are right that otherwise such a high ratio would not be feasible with TCP.

Plain single-stream download-only TCP should fit without compression, probably even without Delayed ACKs.

IME, the line starts to choke when you actually want to send requests and they compete with ACKs and your internet stutters and crawls in both direction. Though that was with a slower connection - it's harder to fill a bigger pipe.

BTW, is the ACK compression counted for consumers benefit?

This is not the standard at all in Europe, maybe that's the standard in your specific country?

I have lived in quite few countries, travelled as well. Someone mentioned Germany (and Bavaria) - indeed, it had one of the worst inet couple of years back. OTOH, Austria was totally fine.

"It's fine" is a different statement from "up- and downlink bandwidths are heavily asymmetric", no?

While asymmetric connections make sense for many residential use cases, P2P is not one of them.

Your claim was that the lines were symmetric and "unrestricted". Neither are the standard unfortunately.

p2p wouldn't solve the main issues, which are excessive download times and poor user experiences for upgrades.

Reply on news.ycombinator.com