Comment by __turbobrew__ | Hacker News Reader

Comment by __turbobrew__ | Hacker News Reader

A few years ago I was working on optimizing scientific data delivery by switching our compression algorithms to use bzip2 instead of deflate64 for our zip files. I checked the zip standard and saw that bzip2 compression was added in 2001 so clients have had 20 years to implement it.

After changing our data delivery pipeline to bzip2 I patted myself on the back for finding an easy win which significantly reduced data size and increased compression speed. Fast forward a month or two later I get word that one of our clients is having problems uncompressing the data we sent them. It turns out that there were still windows 10 versions out in the wild that don’t support bzip2 zip files. I begrudgingly cursed windows and reverted my change, robbed of my small victory.

It truly is amazing how terrible compression support has been in windows. I could compress a +100GB data archive in under a minute using pigz, but since we had to support windows it would take over 30 minutes to compress the file using deflate. I calculated that with the space savings of gzip we could ship a gzip binary embedded with the data in a smaller size than the deflate data on its own. Unfortunately it was a no-go to ship decompression binaries with embedded data, but I thought it would be a cool idea to use cosmopolitan libc to create a platform agnostic binary which when run uses gzip/bzip/lz4 to quickly decompress the data in a platform agnostic way. I imagine such binaries would be picked up by anti-virus but it would be cool to be able to compress data with the latest algorithms without needing the client to have the matching decompression libraries.

Wouldn't have been easier to ask the partner to install 7zip?

Ha! No, probably not.

I don't mean to be flippant. But I recently had to convert Phillips proprietary DS2 audio format file to MP3, and purchased Phillips software to do that. And then financials came down hard because it wasn't on the approved software list.

I know what you mean, but still, I find weird that any company worth 2 cents doesn't have an official compression app. In my experience lots of them use 7zip (free/open source), WinRAR (paid) or even WinZIP (also paid).

But I do know what a pain some IT shops can be.

Why on earth would you think that? Have you never worked with "partners" before?:)

The idea of talking someone through using 7zip gives me nightmares. The fact it's multiple programs, has it's own file managers etc. Nah not worth trying to explain how to use it to a non-tech person.

I know it's a favorite program of a lot of people but personally I just think that's testament to how bad the situation is on windows.

Multiple programs and has its own file managers?

I've not seen this in practice. Sure, it has a clunky gui, but outside of that 7-zip has excellent windows 10 or less integration and works fine in windows 11 even though the jerkwad potentates at M$ decided to hide the usual explorer options behind a second panel.

Sounds like you didn't have to explain to someone after 2020 where X button to close the window is located. I work with about 100 women on my job, only a few of them younger than 40.

Do you typically say ageist and sexist things online?

If you work IT support in any capacity, you should accept that your job is to solve problems for people who are doing things you either can't do or don't want to do.

Their experience is outside of your wheelhouse, and you are there to make the technology transparent to their job function. If you can't do your job with a glad heart then find a different one.

Have some humility.

It is easy to ask, it is nearly impossible for every partner to actually install it. Many were working in locked down environments where they would need to get some very unhelpful central IT team to approve new software and install it on workstations. Even if they could convince the central IT team to install 7zip they probably weren’t using configuration management to install it so there was a chance someone would forget to install 7zip while provisioning new workstations. Unfortunately, when you do not know who is going to consume the data you need to use the most common denominator which is still zip with deflate64.

7z does have two advantages. The main one is that lack of support is obvious, so nobody wastes time thinking the file is broken. The other one is that you don't use bzip2, which compresses worse than lzma and decompresses slower.

Reply on news.ycombinator.com