When I worked on a semi-similar problem (a realtime-ish feed with lots of numbers), we found that it was better to just eat the extra RAM costs.

Once you have variable width values, it gets much more expensive to answer questions like "Here's a file with every stock price, we know that Berkshire is the 1,000th one in the list, what is their current price?" Instead of being able to jump straight to the relevant bytes, you end up needing to parse (potentially) the entire file. Even with better CPU support for variable integer sizes, you're still essentially turning an O(1) operation into one that is O(N)