Oh hey I did something like this a while ago, but much less polished. I basically made an rANS encoder and hooked it up to GPT-2 (via the excellent transformers library from HuggingFace) to see what compression ratio I could get. Bellard's approach seems to use a custom model, which is probably advisable as big language models like I used are very slow. I didn't try to optimize much and only ever tested on my little personal laptop with just CPU, but I only got encoding speeds of ~0.1 Kb/s (limited mostly by available memory). However I was able to get very good compression factors of 6.5x using standard GPT-2 and 7.3x using the medium version.

Here's the code if anyone's interested: https://github.com/trsigg/lm-compression . I haven't looked at it in a while except for updating it today to work with more recent versions of the transformers library, but I think it should still function and is pretty easy to try out with different models. I thought about using something like a hidden Markov model instead which would be much faster (even more so because it would enable using tabled rather than range-based ANS), but I haven't gotten around to it.