noexec Do not allow direct execution of any binaries on the mounted filesystem. (Until recently it was possible to run binaries anyway using a command like /lib/ld*.so /mnt/binary. This trick fails since Linux 2.4.25 / 2.6.0.)
nosuid Do not allow set-user-identifier or set-group-identifier bits to take effect. (This seems safe, but is in fact rather unsafe ifyou have suidperl(1) installed.)
nodev Do not interpret character or block special devices on the file system.
Of course this is just one less (perhaps improbable) attack vector. Never the less; many of the mentioned partitions should never have these kind of files unless it's for a malicious purpose.Edit: fixed pre and clarification
take /var/log as example. if someone floods your server with traffic and you log every tiny bit (or even worse you have the debug level set to on) the logs will quickly take lots of space. if the space is full no further write access is possible
if it's a seperate partition you can't write more logs... if it's on the same partition you won't be able to log in by ssh as example because your drive is full and you can't spawn the sshd agent ;)
With a system like ZFS, you create logical filesystems essentially at a folder level. So you can say that /tmp is noexec and has a quota of 20GB but a reserved space of 2 GB. You can also make sure that your root drive has at least 10GB reserved, so even if /var fills up, you still have at least 10GB for your root space.
And this is without heavyweight partitioning, things are easily modifiable on the fly, and you still get constant-time snapshots.
Fortunately, logical volume managers exist and do help.