ex: Desktop Linux's running X have a trivial escalation path; any program can read all keystrokes across all users. You type your sudo password in, you're screwed.
Or if the attacker is running as your user they can just modify your bashrc, aliases, etc, to do a whole bunch of things - like having `sudo` go to an attacker controlled binary - that one will work on the server, too!
So yeah sandboxing builds is super important because "unprivileged users" are almost always one trivial step away from full root.
Beyond just minding privileges of the user, building in a container/chroot/etc is nice from a cleanliness/repeatability perspective.
For those interested in the Fedora packaging ecosystem -- look into fedpkg and mock
https://docs.fedoraproject.org/en-US/package-maintainers/Pac...