If the binary could be distributed, then there's no reason you'd need to install Rust to use it.
One catch is that (especially if you're using C libraries other than libc, but sometimes with libc) you'll need to worry a little bit about forwards- and backwards-compatibility of those libraries. It's somewhat safer to find a semi-old distro like Ubuntu 12.04, build there, and see if it works everywhere newer.
This is kind of a pain, and the long-term answer is to get Rust into the distros so that they can build software written in Rust. (This doesn't require rustc being installed on any machine other than the builders.) In the short term, a cross-distro package generator for binary crates sounds like a maybe-fun project....