Hacker News Reader

Comment by distortedsignal | Hacker News Reader

Completely agree.

I performed the steps in the article and reduced the output binary size from ~35 MB for raw go build to around 29 MB for go build with the ldflags all the way down to 8.3 MB for compressed with upx.

Really impressive results.

View on HN

Does UPX still trigger virus scanners, though? As much as I like the idea of tinier binaries, the chance that your user's endpoint security is going to freak out makes it not worth the hassle.

Never mind that, startup times are atrocious with such a huge upx packed binary ...

That's an interesting point.

I hadn't tested the startup times of the upx binary before, so I just did by running `time $binary_name --help`. For my binary, this should print a simple help message and quit.

Big: 0.04s user 0.01s system 101% cpu 0.054 total

Med: 0.04s user 0.01s system 99% cpu 0.048 total

Sm : 0.14s user 0.02s system 99% cpu 0.163 total

So you're right - it makes execution time take about 3x as long. However, in my case, 3x slower is a fine trade off to make.

UPX binaries throw off most endpoint security models and so we have a chicken and egg problem where its more common use is for malware authors than commercial software vendors. But once UPX binary signatures are in significantly wider use new models can be developed to better score legit binaries.

I don't think UPX triggers (my) virus scanner. The McAfee daemon on my work laptop barfs at the EICAR Test File (https://kc.mcafee.com/corporate/index?page=content&id=KB5974...) but it ignores the UPX binary right next to it.

Virus scanners just treat UPX as a compressed file format and scan the binary inside.

Some scanners also treat UPX as generally malicious.

35MB seems relatively small already. What do you get from an 8MB binary that you don't from one 4x that size?

27MB. Where that is important is context specific

> 35MB seems relatively small already

35 MB is huge! Are there any assets inside the binary?

> 35 MB is huge!

For context, 35MB is two and a half boxes of high-density floppy disks. There were whole operating systems, complete with applications, smaller than that.

Were? My initrd + vmlinuz is still smaller than that today on a stock Ubuntu kernel.

I remember when a OO framework like MacApp would produce a minimal "hello world" program that took up hundreds of KB and that was considered almost too bloated to be usable. I can't remember exactly how much, but the better part of a floppy to do nothing was ridiculous.

Seems like a culture clash between “why would you not use your distro’s shared libraries?” and “containerize all the things!”

Aren't Go binaries completely static? That sort of is a large part of an operating system (the libraries minus hardware-related libraries) -- Just add a kernel (tens of MB) and whatever userland stuff you need to get your hardware to work (which could be "nothing" in some cases) and you're ready to go.

QNX used to be shipped on a single high density floppy.

There are no assets (other than a helpfile) inside the binary. This is (compiled) 35 MB of Golang.

What OS/arch? I have a small program that compies to 16MB, 14 with -w on OS X. Compiling for Linuxx with -w results in a 14MB binary too.

This is a binary with ~1800 LoC including comments and generous whitespace. Probably 16-1700 Executable SLoC. Mostly some API handling, file reading, and user input requests. I'm running the following go version:

go version go1.14.4 darwin/amd64

Looks like the Linux and Darwin versions of the binaries are only a few hundred kb different.

For context, the Starbucks app on Android is 92MB. My bank's app is 180MB. They might have assets, but modern app size has become ridiculous.

They are mostly assets or frameworks like React Native, Flutter, Xamarin what have you.

Native Android doesn't need such big sizes.

I'm not familiar with UPX compression so it might invalidate what I'm about to say. However it can be beneficial for performance if you fit more of your executable into CPU cache (actual results may vary based on access patterns and many other factors)

UPX is just a (nice) compresser for binaries, where running the executable really runs a small decompressor on the binary, gets a binary in ram and runs it. So your binary is not smaller in cache when executed.

35mb seems small to you? That's an absolutely massive binary.

But to answer your question, you're going to see some performance improvements if your binary can fit into lower levels of the CPU cache.

Kdb+/q for example is less than 700kb, which mean that the entire thing can fit into the L1 instruction cache on high end server CPUs. And that size is even more impressive considering its only dynamically linked to libc and libpthread.

This small size contributes to the out of this world performance the system has. And remember that's the language and the time series db.

Some of the gains here are from compression, does the compressed size matter for fitting into cache? Surely it's stored as the uncompressed size after expansion?

Yeah, if you're simply compressing the binary with upx or something, the uncompressed size is what actually matters. Though I guess it will decompress faster, the smaller it is. But I doubt that upx decompression speed is particularly important.

Hello! You must be a fellow Googler.

(The program we at Google use just to build our bloated programs is itself 180MB.)

Cough python cough cough

... nothing except a vague feeling of self superiority, judging from the answers so far.

I mean, you're not entirely wrong.

But also, since this is client side code, a smaller download means smaller utilities on client machines. Which is nice.

Reply on news.ycombinator.com