But anything running as your user account can also steal your ssh keys and gain persistence via cron or .bash_profiles or any of a multitude of other methods. How does a writable ADB binary let an attacker do anything more than they could do before?
I have no answers on Linux sadly (but you can disable user-side cron so that you can ensure that cron entries are not editable), but on Windows you can lock down your system by not allowing to run executables from non-whitelisted directories (AppLock), so you can only execute programs from specific directories. Does not really prevent stupidity ("I accidentally run NotAVirus.exe from our build directory") but you frustrate attackers from running their own executable (especially that most drive-by attacks rely on the Download folder or Temporary folder being executable). Of course, you need to monitor your build directories for harmful executables but you significantly reduce the attackers' footprint into the system. Additionally SSH keys can be stored in a format where you need to have an active password in order to decrypt them.