1. The name, saddling it with the legacy of Perl. (I say this as someone who really liked working with Perl back in the waning years of the old millennium, and believes that Perl was visionary in pointing the way to the state of modern programming... but also that it had some limitations that have only become more glaring over the years.)
2. The coyness around whether it's "done" or not. Right now today, as someone who has paid attention to Perl 6 and even written some toy code with it, I do not understand its release status at all. That's absurd.
3. The insistence on branding sub-components of it and exposing that branding. Yes, it has a VM. Yes, it has a compiler. But why do people talk about ParrotVM and Rakudo as if they're different things? Java has a compiler and a VM, they're called "javac" and "jvm." This wifty "no, Perl 6 is the standard, the software you're running is Rakudo and Parrot" distinction is totally irrelevant to anyone actually wanting to write code, and weirdly confusing.
4. In a distant, distant last place: any actual functional stuff about the language.