> isn't this a temp container that disappears when you shut it down

For this part, the container of Code-Server in OOTB Code-Server suite is a VM-like persistent and mutable container powered by LXD, so it doesn't disappear even if you shut it down.

As for the use case for Code-Server, it allows to access single server-powered environment from any computer as you guess. I personally use Code-Server from my iPad and laptop.