Now using Zstandard instead of xz for package compression
archlinux.org
archlinux.org
How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours ago?
If it's not easy to read, then the problem is between the css and your screen. Not the title rules.
As long as it's on the same line as the title and easily legible, I really don't see a problem.
And it's not a spot of secondary importance. If it was still in the title, making it longer, the spot where you see the url would have title in it.
You would rate the exact same formatting good or bad based entirely on whether the text next to it is a couple pixels larger. That does not sound like "accessibility".
Whether it's hard to read is an accessibility issue. But that's not your complaint.
I'm not disagreeing with your experience, I'm just disagreeing with part of the way you want to fix it.
Yes, that happens... but I'm not sure how it applies to this specific case? The URL is the same two words that were removed from the title.
I'm not defending HN's title edits in general.
Desaturating and down-sizing suggests the information is not important, but in this case is critical to understanding and of equal weight and so should probably have the same visual hierarchy, that's easily achieved by including "Arch Linux" in the title.
should be allowed. But I'm not sure that it is.
Unless one uses a link shortener. Are shorteners permitted on HN?
If people can't read the source site's domain after the headline then I agree there wouldn't be much context, but equally, if they can't read that, surely their best solution is to adjust the zoom level in the browser.
It's clear you won't get complete context from the headline list plus domain, but a hint of it is provided and if you want more you click the link. Maybe I'm being a little uncharitable but I don't see a big problem here.
I'm sympathetic to disliking the change, but that's taking it to an extreme.
When you take out information, don't expect people to have the correct guess.
The question is what's better than a strict "no editorialization" rule.
For a contrived example of how dated the guideline seems, what if somebody submitted a tweet thread criticizing Twitter the company with a headline/sitebit like "Twitter now banning third-party clients. (twitter.com)". Would it have to be renamed to "Now banning third-party clients. (twitter.com)"? That would make it appear to be a more official statement instead of an unsponsored opinion.
I'm picking on Twitter out of recent memory of this submission of mine a couple weeks ago, where the submission title "Tracking down the true origin of a font used in many games and shareware titles" was 100% my own editorializing for lack of title-worthy material in the linked tweet itself: https://news.ycombinator.com/item?id=21667238
I suspect the original intent of the rule was to get rid of pointless redundancy in the title. "The 10 craziest things you don't know about X - clickbait.com" is the sort of thing you see very often in the <title> element, but it adds no new information. Actually, you'll notice even Hacker News posts have " | Hacker News" appended to them.
In an article about Arch Linux, the text "Arch Linux" is much less likely to be redundant than an article about something else that just happens to be on the archlinux.org domain.
Earlier last year I was doing some research that involved repeatedly grepping through over a terabyte of data, most of which were tiny text files that I had to un-zip/7zip/rar/tar and it was painful (maybe I needed a better laptop).
With Zstd I was able to re-compress the whole thing down to a few hundred gigs and use ripgrep which solved the problem beautifully.
Out of curiosity I tested compression with (single-threaded) lz4 and found that multi-threaded zstd was pretty close. It was an unscientific and maybe unfair test but I found it amazing that I could get lz4-ish compression speeds at the cost of more CPU but with much better compression ratios.
EDIT: Btw, I use arch :) - yes, on servers too.
http://pages.di.unipi.it/farruggia/dcb/
Looks like Snappy beats both LZ4 and Zstd in compression speed and compression ratio, by a huge margin.
LZ4 is a ahead of Snappy in the decompression speed.
I have not researched this opinion much
The numbers I know about are wrong: zstd always beats gzip for compression ratio.
I will need to do my own testing.
It compresses about as well as lz4, but more slowly. It also decompresses more slowly.
It is faster than zstd -1, but compresses less well.
It is possible that it does better with certain kinds of data, but 12x remains implausible.
Apparently the current file format has suffix ".sz".
tar -I zstd -xvf archive.tar.zst
Hopefully there's another option added to tar that simplifies this if this compression becomes mainstream.
tar -axf archive.tar.whatever
-I, on the other hand (which, in gnutar, specifies a compression program to run the output through), appears to actually be GNU-specific. BSD tar makes -I synonymous with -T (specifying a file containing the list of filenames to be extracted or archived).
(Please don't use Zstandard if you care about cross-platform compatibility at all, though-- it's fine in controlled environments like an OS package manager, but I don't have it on my Mac, nor do I have it by default on my Ubuntu server (which is still sitting back on 16.04; I should fix that sooner or later).)
Looks like that option got merged in 2013: https://github.com/freebsd/freebsd/blame/24ac2fb6ec0610e60ac...
The textual libarchive changelog isn't super clear on it, and I didn't feel like digging further, but appears to have been introduced some time between libarchive 3.0.4 in 2012 and libarchive 3.1.2 in 2013: https://github.com/freebsd/freebsd/commit/366f42737cba40ceb2...
That'll save me some time next time I need to extract one at the command line; I always end up having to look at unzip's man page to make sure it's actually going to do what I want, and the 7zip command line utility's kind of funky (and not installed by default most places). But 'tar -xvf filename' is permanently burned into my mind and pretty much always does exactly what I want.
[1] https://www.freebsd.org/cgi/man.cgi?query=libarchive-formats...
tar cf example.tar.zst example
tar caf example.tar.zst example
For compression, you can use "-c -I zstd"
tar -acf blah.tar.zst blah/
zstd -cd archive.tar.xst | tar xvf -
Package installations are quite a bit faster, and while I don't have any numbers I expect that the ISO image compose times are faster, since it performs an installation from RPM to create each of the images.
Hopefully in the near future the squashfs image on those ISOs will use zstd, not only for the client side speed boost for boot and install, but it cuts the CPU hit for lzma decompression by a lot (more than 50%). https://pagure.io/releng/issue/8581
Also one more benefit of zstd compression, that is not widely noted - a zstd file conpressed with multiple threads is binary the same as file compressed with single thread. So you can use multi threaded compression and you will end up with the same file cheksum, which is very important for package signing.
On the other hand xz, which has been used before, produces a binary different file if compressed by single or multiple threads. This basucally precludes multi threaded compression at package build time, as the compressed file checksums would not match if the package was rebuild with a different number of compression threads. (the unpacked payload will be always the same, but the compressed xz file will be binary different)
This looks like a very good move. Debian should follow suit.
EDIT: nevermind, this doesn't seem to have made this the default for building packages locally, just for ones you download from the official repos. Guess I'll go change that by hand and then still be sad that I can't have it easily disabled entirely for AUR helpers but build my packages with compression.
In makepkg.conf, ommit compression by specifying:
PKGEXT='.pkg.tar'
PKGEXT=.pkg.tar makepkg
As long as there will be support within the tools for xz individual builders of packages for their own use can use either or more.
I believe you can supply it via an environment variable if your AUR helper has the ability to set those for `makepkg`.
Impressive. As a AUR package maintainer I am also wondering how the compression speed is though.
Was a time when 700 MB mattered; it was as much as you could get onto a CD.
So, there is a place for each. I would set up the process to use Lz4 when testing, and Zstd for actual delivery to download archives.
In some circumstances, particularly when using a shared file server, Lz4 can be quite a lot faster than writing and reading data uncompressed.
FWIW I really applaud Arch here. Even if it's just a small step. Commercial operating systems should take notice. OS updates should really not take as long as they (mostly) do.
"... decompression time dropped to 14% of what it was..." (s/14/actual_value)
Luckily updating libarchive manually with an intermediate version resolved my issue and everything proceeded fine.
This is a good change, but it's a reminder to pay attention to the Arch Linux news feed, because every now and then something important will change. The maintainers provided ample warning about this change there (and indeed I had updated by other systems in response) so we procrastinators really had no excuse :)
But if latency matters you might better use lz4.
Also lz4, of course.
Few (or none?) of Chrome's fairly dramatic improvements to zlib have been upstreamed. https://github.com/madler/zlib/issues/346
Edit: Also, if browsers do adopt zstd, it's likely you'll end up with the same situation where they fork their own implementation of zstd. Upstreaming requires signing Facebook's CLA, which has patent clauses that don't work for most.
https://github.com/facebook/zstd#benchmarks
AFAIK (I haven't looked much into it since 2018) it's not widely supported by CDNs, but at least Cloudflare seems to serve it by default (EDIT: must be enabled per-site https://support.cloudflare.com/hc/en-us/articles/200168396-W...)
That’s a philosophical objection. For a long while Mozilla also was of the opinion that WebP is not “better enough” than JPEG/PNG to warrant the addition of another image format which the entire web must support forever using only one available implementation.
Plus I think there are still some unresolved patent claims on the VP8/9 video codec (which are the basis for WebP).
https://github.com/google/brotli/issues/642 is the best 3rd party documentation of this behavior.
zstd does decompress fast, but this is not free. The cost is the compression density -- and lesser streaming properties than brotli.
For typical linux package use, one could save 5 % more in density by moving from zstd to large window brotli. The decompression speed for a typical package would be slowed down by 1 ms, but the decompression could happen during the transfer or file I/O if that is an issue.
Name | Comp. size | Comp. ratio | C. MB/s | D. MB/s
brotli 11d29 | 21117099 | 8.1 | 0.52 | 515
zstd 22 | 22249772 | 8.6 | 2.32 | 1270
brotli 11 | 22898691 | 8.8 | 0.57 | 662
I think it's fair to say that zstd struggles the most at the extremes. On the fast extreme it loses (marginally) to lz4; on the slow extreme it (maybe) loses (marginally) to brotli. But it's relatively quick across the spectrum and provides a lot of flexibility.
It may make sense to continue to use Brotli or xz for static assets that are compressed infrequently and read often. But for something like HTTP Content-Encoding, where dynamic pages are compressed on the fly? Zstd would shine here, over both Brotli and (vanilla) zlib. (I know Chrome has some hacked up zlib on the client side, but I do not know too much about it.)
Also, parallel zstd must have some way to split up the work, that you could maybe use too.
PKGEXT='.pkg.tar.zst'
After warming up all my caches with a few pre-builds to try and keep it fair by reducing disk I/O, here's a sampling of the results:
xz defaults - Size: 33649964
real 2m23.016s
user 1m49.340s
sys 0m35.132s
----
zst defaults - Size: 47521947
real 1m5.904s
user 0m30.971s
sys 0m34.021s
----
zst mpthread - Size: 47521114
real 1m3.943s
user 0m30.905s
sys 0m33.355s
I did not seem to notice any appreciable difference in adding the `--threads=0` to `COMPRESSZST=` (from the Arch wiki), they both consistently gave me right around what you see above. This was compression only testing which is where my wait time is when upgrading these packages, huge improvement with zst seen here...
pacman:
COMPRESSZST=(zstd -c -z -q -)
devtools:
COMPRESSZST=(zstd -c -T0 --ultra -20 -)
That's still less than XZ, which reaches 64 MB.
Brotli defines memory use as log2 on command line, i.e., 32 MB = 1 << 25
zstd uses a lookup table where the number given by the user is mapped to a decoding-time memory use. The user just needs to look it up if they want to control decoder memory use.
If one benchmarks zstd with `20` and brotli with `20`, zstd may be using 32 MB of decoding memory, where one is specifying 1 MB for brotli. By default zstd tends to use 8 MB for decoding (however it is variable with encoding effort setting) and brotli 4 MB for decoding (not changing with the encoding effort setting).
If you know your use case well and are sure the situation won't change (or don't mind swapping compression algorithms when they do), then lz4 still has a solid niche, especially where compression speed matters more than decompression speed. But in many if not most cases I'd say it's probably a kind of premature optimization at this point, even if you think you're close to lz4's sweet spot.
I have a server that spools off the entire New York stock and options market every day, plus Chicago futures, using Lz4. But when we copy to archive, we recompress it with Zstd, in parallel using all the cores that were tied up all day.
There is not much size benefit to more than compression level 3: I would never use more than 6. And, there's not much CPU benefit for less than 1, even though it will go into negative numbers; switch to Lz4 instead.
I am a little shocked that they bothered; Arch is rolling release and explicitly does not support partial upgrades (https://wiki.archlinux.org/index.php/System_maintenance#Part...). So to hit this means that you didn't update a rather important library for over a year, which officially implies that you didn't update at all for over a year, which... is unlikely to be sensible.
https://git.archlinux.org/infrastructure.git/
Now, official infra doesn't reach hundreds of days. But personal systems might :)
(I wanted the challenge of running arch in production just to learn, good times)
That sort of attention to detail is what continues to impress me about the Arch methodology.
Do you know if Debian is using parallelized XZ or not with apt / dpkg?
The xz tool is not deterministic when compressing. The packaging team might change upstream for a few things, but diving into the innards of a compression tool is expecting a bit much.
For xz, you need to compress with chunking (and may be indexing for more benefit), in order to allow parallel decompression to begin with. Otherwise xz produces a blob which you can't split into independent parts during decompression, which makes using many decompression threads pointless.
But yes, if parallel compression is creating non determinism, you can do all the compression work with chunking without parallelism, still allowing parallel decompression. But I'm not sure why it even has to create non determinism in the first place.
In comparison, the 0.8% of zstd looks like a bargain.
zstd is used at level 20, but it can compress more. Levels can go up to 22, and (complex) advanced commands are available to compress data even more.
edit: Sorry, my fault that was decompression RAM I was thinking about, not speed, although I was influenced by my test that without measuring both xz and zstd seemed instant.
It's not set in stone currently.
If this was netbsd m68k, you'd probably easily understand.
May I ask, why xz over, say, PAQ8PF?
And you're talking to a guy who had his C128D spend all nights crunching intros + pic + game at 1 MHz in Cruel Cruncher 2.5+ and Time Cruncher 5.0 before that. xz(1) on a MC68030 or the Vampire is super fast in comparison.
Solaris 10 as in the last one from Sun (Damn Oracle...) is an interesting choice. Does openindiana not run well on the old hardware?
Is Core2 Quad with 8GB of RAM or a Sun X4100 M2 "older hardware"?
For those figures, this will be better total time for you if your computer network connection is faster than about 1.25mbit/sec. For a slow arm computer with an XZ decompress speed of 3MB/s the bandwidth threshold for a speedup drops to _dialup_ speeds.
And no matter how slow your network connection is and how fast your computer is you'll never take more than 0.8% longer with this change.
For many realistic setups it will be faster, in some cases quite a bit. Your 54MB XZ host should be about 3% faster if you're on a 6mbit/sec link-- assuming your disk can keep up. A slow host that decompresses xz at 3MB/s w/ a 6mbit link would a wopping 40% faster.
This is for html web compression, but the results are similar for other datasets. For internet transfer more compression is better than more decompression speed.
You can make your own experiments incl. the plots with turbobench [2]
[1] https://sites.google.com/site/powturbo/home/web-compression [2] https://github.com/powturbo/TurboBench