sscache has pre-built binaries on their releases page. Or you can do:
cargo install --no-default-features
To avoid building support for all storage backends. You can then add back any backend you're actually using.