Very cool. I'm not an expert, but does JPEG generally have a ton of flexibility in compression? Why so much difference in sizes?
1) YUV420 vs YUV444. Guetzli practically always goes for YUV444.
2) Choosing quantization matrices.
3) After normal quantization, choose even more zeros. JPEG encodes zeros very efficiently.
When doing the above, increase the errors where it matters least (certain RGB values hide errors in certain components, and certain types of visual noise hides other kind of noise).
On top of that, you could tweak the quantized values themselves to make them more compressible.
edit: second read-through I found the paper [1] which explains it. The answer is basically "yes", where the large scale distortion function is basically activity masking. Normally this would be implemented with delta-QPs, but because JPEG doesn't have that, Guetzli uses runs of zeroes instead.
Butteraugli uses a 8x8 FFT, but computes this every 3x3 pixel creating coverage at block boundaries. In later stages of butteraugli calculation values are aggregated from an even larger area. Block boundary artefacts are taken into account by this and impact quantization decisions.
Butteraugli operates neither in RGB nor YUV. It has a new color space that is a hybrid of tri-chromatic colors and opponent colors. Black-to-yellow and red-to-green are opponent, but blue is modeled closer to tri-chromatic. In more simple explanation it is possible to think of it as follows: first apply inverse gamma correction, second apply a 3x4 transform for rgb, third apply gamma correction, fourth calculate r - g, r + g and keep blue separate.
It seems like that's where most of the magic lies. Also peculiarities of human vision are one of my oddball interests, after compression of course. :)
The more bits you lose during quantization, the more ringing and artifacts you can expect after the IDCT process.
So the tradeoff is quite literally artifacts for smaller size.
this compressor seems to be cleverer about where to lose data than libjpeg.