Hacker News Reader

There are downsides which most young fellas around here don't really appreciate. Hands up, how many of you remember bzip? Not bz2, but bzip, the original one, mostly seen as "version 0.21"?

Well, at the time it was released, people were making much the same arguments (with kittens). It compressed so much better than gzip, no reason to use the obsolete gzip format and tools, etc. And some of us jumped on the hype bandwagon and started recompressing our data, only to find out afterwards that bzip2 is now the new thing and the format is not only obsolete, but also patent-encumbered and in general needs to be phased out.

From a long-term perspective I'm fine with gzip. At least I know that I'll be able to open my data in 10 years time, which is not the case with bzip-0.21. The jury is still out on "xz", in my opinion.

Agreed, and I think people do not fully understand it because the way it happens is not safety based the way you describe it. The question is not is X safe and Y unsafe, the question is each on its own. Is X safe, and will it still work on old systems or in ten years? Yes, 100%.is y safe and still work in ten years? Well some old system might have issues, and the patents might still exists, and....

If you're a developer, gzip is simply the best option. It's not the best, but it's good enough and it's safe.

I had to explain the same thing to an engineering team the other day. There was a push to switch from a "fast but ok" compression algorithm to a "faster and better" compression algorithm. This seemed like win-win, but I explained that:

* The faster compression made a difference of about 100 milliseconds to a user experience lasting minutes.

* The better compression made a different of about 1 second to most users.

* The change of compression algorithm would take time away from engineering teams, and ultimately introduce bugs.

So in the end it was sidelined until other fundamental changes (file format etc) made it able to be coat-tailed into production.

Same story as yours: don't fix what ain't broke. Tallest working radio antenna in the world -> longest lying broken antenna in the world.

We started packaging things as .tar.xz for a while, but a large number of users were having terrible trouble opening the file.

Mostly Mac, but iirc some Linux users were confused as well. We switched back to gzip because everyone knows how to use it.

The same applies to OGG audio. Better than MP3 but the average user is unable to play it. So everybody just sticks to MP3.

MP3s with a decent bit rate is as good is it gets. Of course something like ogg back in the napster days would have been fantastic, but MP3 at 320 Kbps is fine for anyone who doesn't pay $1000 a meter for speaker wire.

But MP3 was patent encumbered and so a bunch of music creation software had weird work-arounds.

I am not sure what your point is? Yes MP3 files were not the right choice years ago, but today who cares.

If I recall correctly, there are some alive patents in the U.S. until 2017 when dealing with MP3 encoders, requiring purchasing a license per copy distributed.

The workaround that the parent is talking about is usually "get LAME from a different distributor", which is still done by Audacity and others.

That's not true. MP3 simply never gets transparent and you can notice with 5 dollar in-ears. And people in general notice. This leads to absurdities such as bitrates of 320 kbps, even thou these do not sound significantly better than 128 kbps and are still not transparent.

On the other hand 128 kbps AAC is transparent for almost any input. AAC is supported abou everywhere where mp3 is. The quality alone should be convincing. The smaller size make the usage of mp3 IMHO insane.

OTOH "the scene" still does MPEG-2 releases I think.

You are making some bold statements about the general transparency of different audio formats that contradict pretty much everything I've read about this topic so far. Hence, I'd like to learn more, do you have any links that you would recommend?

Well, try it yourself :). Make sure to make it blind-test with the help of somebody else. Ideally such things would be subject to scientific studies. But these are kind of expensive and nobody cares for mp3 anyway. I'm not aware of any recent.

Hydrogenaudio listening tests [1] are studies by volunteers, but they focus on non-transparent compression. Anyway, it also illustrate aswell how bad mp3 is.

[1] http://wiki.hydrogenaud.io/index.php?title=Hydrogenaudio_Lis...

You realize that that test doesn't test anywhere near 320kbps mp3, right?

Yes, that's what the last two sentences are about.

This page says at 128 Kbps all the encoders were the same (a 5 way tie). Their wiki says at 192 Kbps MP3 is transparent [1].

1. http://wiki.hydrogenaud.io/index.php?title=Transparency

I have listened to a lot of MP3 at different bit rates and with modern encoders and variable bit rates I can't tell the difference between anything above 160 Kbps - most of the time it is hard to tell the difference between 128kbp and anything higher. Really at 320kps you are entering the realm of fantasy if you think you can hear any difference.

This is not true. It is trivial for almost anyone to distinguish 320kbps mp3 from uncompressed audio, with built-in DACs and $5 headphones, with as little as 5 minutes of training.

[citation needed]

How can it not be true as I am describing my experience. Are you really telling me that I actually can tell when I say I can't?

You're also describing everyone else's experience:

> Really at 320kps you are entering the realm of fantasy if you think you can hear any difference.

It depends on the encoder, the track, your equipment, and how good you are at picking out artifacts. Some people do surprisingly well in double-blind tests, though I doubt anyone can do it all the time on every sample.

There is no scientific evidence that anyone can do it at all above 192 kbps.

This is why ABX testing is so big in lossy audio circles. People can and do demonstrate their ability to distinguish between lossy and lossless encodings with certain samples in double-blind tests, at all sorts of bitrates. I've done it myself occasionally.

That people have been doing this for many years is one of the big reasons modern encoders are so good - they've needed tonnes of careful tuning to get to this point.

Like the parent comment, more bold statements about perceivable sound quality, with no evidence.

I absolutely heard a difference between 320 and everything below. You can tell me I didn't, but I did. There is a world of difference between 160Kbps and 256, and 128 is a lot worse. If you can't hear it, I understand, but the blame isn't the algorithm -- it is your equipment, your song selection, or your ears.

For those of us that sample from songs that we buy, WAVs are a bit easier to work with because the DAW doesn't have to spend time converting it. That said, since most of my tracks these days are using either the 48k or 96k sample rate, it still needs to be converted from 44.1 :)

But can you tell the difference between a variable encoded 320 Kbps MP3 using a modern encoder and a wav file? I have some reasonable equipment and I most definantly can't.

When I DJed I used a mix of FLAC files and 320 Kbps CBR and high level VBR. On performance equipment I could tell VBR was not holding up. There is also some quality loss that you encounter when slowing down MP3s that is not present for FLAC or WAV, especially when kept in key, but for the most part that is only audible beyond the 8-15% range, and it was not common to alter the tempo that much for me. I ended up settling mostly on FLAC when I can get it and 320 CBR otherwise. I don't think I ever heard the difference.

Can you hear the difference between 320 Kbps CBR and VBR? I have to say I have never tried slowing down the music to try and hear the difference so it is possible under these conditions that it might make a difference.

I would say yes, but again, playing music at very amplified volumes and tweaking its tempo is not a common use-case.

The degree of difference depends on the kind of music you listen to. Live recordings of acoustic ensembles in airy cathedrals -- in that case you can tell the difference. On tracks that have a highly produced studio sound, where everything is an electronic instrument -- not going to be much of a difference.

I tried doing tests like this and I could not find any recordings where I could tell the difference at 160 kbps VBR. I not saying that it is impossible, but the conditions must be pretty rare and the difference very minor - compared to the massive degradation that come from room effects it amounts to nothing.

compared to the massive degradation that come from room effects it amounts to nothing.

Truth.

Chamber music in an echo-y cathedral. With bad encoding, you can hear a noticeable difference in the length of time the reverberations are audible and and the timbre of those reverberations cab be quite different. With lots of acoustic music, the "accidental beauty" produced by such effects can be quite important.

Finding this convinced me to re-encode my music collection in 320kbps MP3 for anything high quality, and algorithmically chosen variable bitrates for lower quality recordings -- usually around 160 kbps. That was quite a number of years ago, though. I'd probably use another format today.

most people can't. They say they can, but under scrutiny it generally falls apart. At some point, you're listening to sound and not music anyway.

That's the thing. With better formats, you don't need 320 kbps for transparency.

We have OGG opus these days which is even better than OGG vorbis.

I actually tried this last year, and found out the hard way after reencoding my mp3 collection to vbr opus at around half the bitrate (I did some light quality testing to make sure it was of similar fidelity, of course you lose some quality going lossy -> lossy) that either opus-enc or gstreamer at the time would produce choppy broken audio.

And it was reproducible on all my computers. I couldn't use my opus collection at all because either the encoder was broken or the playback was broken.

I need to do it again at some point, when I have 8 hours to transcode everything and try again. See if they've fixed it.

I'd expect similarly awful results for any lossy -> lossy transcoding.

Don't ever do that. If you didn't rip CDs to lossless or buy lossless, you are stuck with the format you've got.

Interestingly, it's opened by the same command on both OSes -

    tar -xf some_file.tar.xz 
    tar -xf some_file.tar.gz

If you don't explicitly specify (de)compression method (z for gzip, J for xz), tar will try to guess it.

That's new. Tar didn't use to behave that way, and plenty of people got used to specifying flags.

And many implementations don't behave that way.

I guess a lot of people are or were used to:

  tar -xzf file.tar.gz

To be honest, I only started doing -xf a year ago. I was used to -xjf and -xzf.

tar flags must be a prime example of cargo culting. Hands up everyone who's done or seen someone do

  tar -xzvf file.tgz > /dev/null

never seen that in my life until now - what does it do? Just unzip the file to dev/null? What's the purpose? Does the verbose flag show you what's inside but the dev/null means it's not written to disk while unzipping?

Actually, the files are decompressed to the current directory, it's the just output of the verbose flag that goes to /dev/null. Which makes is even more senseless.

Exactly. I've seen people who always do `tar xzvf` and have no idea removing the `v` is the correct way to make it not print the name of every file in the archive.

I use the -v switch since I want to see what I decompress, however I didn't knew that I could supress the z switch.

Isn't it better though to omit the -v switch and do `ls *` and/or `tree` afterwards? That gives you the same information but structured so it's much easier to understand.

The advantage of -v is that you can see what is being extracted as it happens. This is useful if you have a tarball with thousands of small files, as otherwise it's hard to tell whether tar has got stuck or there are just a lot of files.

You didn't use to be able to suppress the 'z' switch. You had to specify 'z' or 'j' depending on whether you wanted gzip or bzip2 decompression. It's a somewhat recent (sometime in the last 15 years, I think) change to "tar" to make it just detect the compression algorithm.

I don't understand why you'd do that. tar does not compress to stdout.

Redirecting stdout is to cancel the -v flag (verbose, lists every file extracted).

That's gnu tar. I don't think that works on BSD.

Correct, on OpenBSD:

    $ tar -xf foo.tar.xz                                      
    tar: End of archive volume 1 reached
    tar: input compressed with xz

    $ tar -xf foo.tar.gz                                      
    tar: End of archive volume 1 reached
    tar: input compressed with gzip; use the -z option to decompress it

Does that _tell_ you what it's compressed with but then not decompress it? That's the absolute worst way to do it!

It tells you how it's compressed and how to decompress it if it knows how. OpenBSD's tar doesn't support xz so it can't help there, but does support gzip so it suggests using -z.

Not letting untrusted input automatically increase the attack surface it's exposed to is a feature.

>Not letting untrusted input automatically increase the attack surface it's exposed to is a feature.

How is that a feature? The user's explicitly asking for this.

This feature reminds me of vim, that suggests closing with ":quit" when you press C-x C-c (i.e. the keychord to close emacs). It knows full well what you want to do and even has special code to handle it, but then insists to hand you more work.

`tar -xf` is not "explicitly asking" for gzip. `tar -zxf` is "explicitly asking" for gzip.

I don't really care what vim does, that's a different argument. There have been many vulnerabilities in gzip, and in tar implementations that let untrusted input choose how it gets parsed, those vulnerabilities might as well be in tar itself.

Vim suggests closing with ":quit" when you hit C-c; the C-x is irrelevant.

Upon receiving a C-c, it does not know full well what the user wants to do.

When vim receives a C-c from you (or someone who just stumbled into vim and doesn't know how to exit) the user wants to exit.

When vim receives a C-c from me, it's because I meant to kill the process I spawned from vim, and it ended before the key was pressed. I very much do not want it to quit on me at that point.

Showing a message seems the best compromise.

I'm fairly sure bsdtar had automatic compression detection before GNU tar did. Been the default on FreeBSD since version 5.3 (2004).

Also works with non-tar formats like Zip, RAR and 7z: https://www.freebsd.org/cgi/man.cgi?query=libarchive-formats...

Using xz for Linux builds of your software might make sense though, or would do so in the future. Recent releases of Fedora and RHEL already uses xz to compress their RPM packages.

Debian/Ubuntu dpkg supports compressing with xz too -- and it's a hard dependency of dpkg at least as far back as the precise (12.04) LTS release. So I'd say the majority of Linux users already have access to xz.

I just started typing in 'xz compression' in Google to learn more about it, it offered 'xz compression not available'. And some more queries that indicate it's not quite ubiquitous.

It's good to be informed about the capabilities of xz. I will keep using gzip, but consider xz in situations where the size or time matters. I might not care about 100 megs versus 50 very much, but I will about two gigabytes versus one.

Is it likely that a user has gzip on a system but not tar itself? From the article:

What about tooling? OSX: tar -xf some.tar.xz (WORKS!) Linux: tar -xf some.tar.xz (WORKS!) Windows: ? (No idea, I haven't touched the platform in a while... should WORK!)

Tar does not implement decompression. If you don't have xz installed it won't work.

That doesn't seem correct, on osx 10.11 `tar xf` can extract `.tar.xz` yet doesn't fork an xz. AFAIK 10.11 doesn't even come with xz.

tar can link the xz lib without forking.

So tar does "implement decompression" (and compression, by delegating the work to libarchive) and it can work even "if you don't have xz installed".

It would require liblzma, but you are correct that the library is a separate thing from the executable xz.

> It would require liblzma

Yep, in the same way it requires libz and libbz2.

IIRC bsdtar (e.g. on OS X) includes xz.

Pretty sure tar -xf does not actually work on osx unless you download a recent tar.

I have tar that came with the system (latest OSX) and tar -xf works just fine. And it did work fine for as long as I can remember.

It does, and should have since at least 10.6 (I can find references to 10.6's tar being built on libarchive and that's one of libarchive's headline features; 10.5 predates libarchive so it may not have supported that)

On the vast majority of Linux distributions, you can pretty much guarantee that both tar and zlib will be installed.

Both tend to be part of an essential core of packages required to install a system.