Privesc is also trivial on desktop linux :P so if your barrier is "but I'm an unprivileged user" it's likely not enough. You need a proper sandbox if you want to make escalation difficult.
ex: Desktop Linux's running X have a trivial escalation path; any program can read all keystrokes across all users. You type your sudo password in, you're screwed.
Or if the attacker is running as your user they can just modify your bashrc, aliases, etc, to do a whole bunch of things - like having `sudo` go to an attacker controlled binary - that one will work on the server, too!
So yeah sandboxing builds is super important because "unprivileged users" are almost always one trivial step away from full root.