Ooh that's a fun question. I think it comes from reusing the C/C++ notation for references (https://stackoverflow.com/a/2094715/2242866). But I have no idea where the original idea for using those characters came from.
There's one other thing in Rust that would be kind of difficult if passing a reference were the default, rather than the owned value. Rust has implicit returns so the last thing at the end of a block or function is returned. If you needed to explicitly ask for ownership, you'd need to do that at the end of every function. Also, Rust will yell at you if you try to have a function return a reference to a value that's going to be dropped.