I suspect that's the best solution: install Linux onto a USB drive. I liked that Knoppix was a curated setup, and that it was on read-only media.
That's the bit that used to annoy me a lot. Invariably I wanted to add something to the LiveCD, or to transfer a largish amount of data. The read-only aspect of things got in the way.
With a complete 'system on a stick', you can install extra packages if you want, you can even do development at a pinch, mainly bash scripts.