What you're describing (moving complexity into the algorithm) _is_ an increase in Komogorov Complexity, though... so I think we're saying the same thing?
i.e. Kolmogorov complexity K(x|y) (x, given y) is the length of the shortest program that on input y, outputs x.
So, when you say "move complexity into the algorithm", this is an increase the length of the shortest program that given input <data to compress> outputs <compressed data>.
Or have I missed or misunderstood your point?
Fewer bits does not always mean better compression, particularly if the data has other patterns which is destroyed by the packing.
With Apache ORC, I found out that if you bit-pack data to say 7bits vs leaving it as 8 bits, the 8 bits version compressed much more with Zlib than the 7 bit version.
This had to do with the data getting a bit offset into the previous byte sequence, until what was a sequence of repeating bytes turned into a pattern which repeats far less often.
Leaving the extra bit in place, helped Zlib dictionary encoding and huffman work much better than trying to save a bit.
The final kicker was that the 24 bit sequence was faster to read than a 23 or 21 bit sequence, but purely due to the fact that the word aligned stuff can be decoded in SIMD.
I'm no better at guessing what would work - "whatever works ... works, so try them.".
https://news.ycombinator.com/item?id=13049894
>It's crucial to evaluate encoding space usage in the context of compression. For instance gzip(base16(data)) is often smaller than gzip(base64(data)) for practical data. Even though base64 is more efficient than base16, it breaks up data across byte boundaries which then makes gzip significantly less efficient.