I'm definitely a Rust fanboy, but the single-quote syntax for lifetime annotations can be irritating. Several editors I've used automatically insert a second quote to match, and I am frequently unable to disable that behavior without losing all paired delimiter insertion (like for parentheses or braces).
It's a minor quibble to be sure, but it's the only language symbol that bothers me when writing Rust. Not sure what I'd suggest replacing it with...backtick, maybe? Pipe? @? ~?
There aren't many other special characters on a QWERTY board that aren't already used in Rust. Which I think gets at one of the stumbling blocks that I see in the various Rust syntax bikesheds among those who haven't worked in the language. It's just alien until you've used it a bit, especially if you're writing a lot in pseudocode-y dynamic languages.