AFAIK, the sigils were used to express differences in data types (wrt to memory allocation scheme), which is quite different from ASCII operators. This doesn't bother me nearly as much, but it may be my Perl background.
You are right, they were mainly around pointers. As long as they are a handful and their meaning doesn't change (as opposed to using in a DSL), its easy to get used to them. Still, I find the explicit keywords a familiar territory and better to read.