I say don't worry about the criticism and popular opinion: The specific language you use is pretty far down the list of problems that a given software might have. You'll build up some useful libraries in your niche and eventually other people will see that the most time-effective way to solve a problem in that niche is to use Perl 6 to get access your libraries.
Every other language I use derives from its specific purpose: Ruby because vendors like PayPal or AWS publish Ruby gems(and because of Rails); Haskell for writing parsers, interpreters, and compilers(ADTs and parsing monads); C if I'm trying to understand some code and need it to be portable or operationally simple; Perl 5 if it's a quick prototype script. Just the other week I needed a Bayesian variant of Prolog and would've used Perl 6 if it had a library for it.