Bulat_Ziganshin

i think it's American trauma. outside of the Western hemisphere, sexist and racist jokes are just jokes

xz is a data compression tool, so it's natural to have compressed files for (de)compression tests.

these files are also useful to check that the library we just built works correctly. but they aren't necessary for installation.

we may have more sophisticated procedures that will allow us to use some parts of distribution only for tests. This may significantly reduce an attack vector - many projects have huge, sophisticated testing infrastructure where you can hide the entire Wikipedia.

except that it doesn't support full AVX-512, making the whole idea of backward compatibility between these levels meaningless. "It's Intel!!!"

yeah, exactly. instruction fusing can turn mov+op2 into 3-reg operation, or push+push into push2. but adding new instructions allows to increase the frontend throughput too.

128/256-bit subset of AVX-512 already exists and implemented by P-cores as well as Zen4. They will just enable it via micro-code.

It's actually incorrect, see https://news.ycombinator.com/item?id=32054460

The field I've used in benchmarks is GF(0xFFF00001) which has root of unity of 0xFFF00000 = 0x100000 * 0xFFF order. Since my code implemented only FFT for 2^N sizes, it was limited to 2^20 order for this particular field and thus 2^20 blocks total.

It can process larger amount of blocks in other fields (f.e. GF(2^64-2^32+1)). Also, I have implemented other FFT algorithms, which will be published soon.

I ahve chosen this field for the speed (and million blocks is more than other RS implementations provide anyway), but RS in GF(2^64-2^32+1) will be even faster and allows 2^32 blocks even with the current limited FFT implementation.

There is Leopard codec implementing very similar approach in GF(2^n). AFAIR, it's a bit slower, but overall it looks more useful than FastECC

(OTOH, since we anyway store hashes and other info, we may store there extra overflow bits required for recoding into GF(p). That said, FastECC on practice works like a slightly non-MDS code, but at least there are guarantees how much extra data we need to store)

1. As of 2017, we had quadcores as top desktop CPUs for years, and very little IPC and frequency improvements since 2011 2. But the main reason is that I can say "70 MB/s per Haswell core*GHz" and noone will feel whether it's good or not. Performance stated for top modern CPU is much easier to grok

There is long NNCP thread on the forum dedicated to compression algos:

https://encode.su/threads/3094-NNCP-Lossless-Data-Compressio...