Perl was my first language that I really engaged with, it started my career. Why did it fall out of favour? (I left web development to write applications/C, then returned some years later to write APIs but everyone had stopped using it).
As soon as it became clear that the move from 5 to 6 was going to be discontinuous, if you wanted to be a Perl developer you had to decide if that meant being a Perl 5 developer or a Perl 6 developer. Being a Perl 6 developer was the clearly marked path to The Future, but Perl 6 was years away from being ready to use in production environments, and it never seemed to get any closer. Perl 5 was practical for real-world use, but choosing to be a Perl 5 developer meant worrying that you were investing time and energy learning skills that were already marked as obsolete and writing code that would end up having to be re-written.
That kind of uncertainty is fatal to a language. Why put up with it when there's so many other languages out there with clear upgrade paths? Especially when the upgrade path for the one you're using now appears pretty similar to learning a new language anyway?
So developers drifted away from Perl to other languages, and that, as they say, was that.
Perl(5) just needed classes and clean up a few bits of historic cruft (like the need to end modules in "1;", indirect object syntax, etc) and it could have been an awesome language. Perl6 decided to throw the baby out with the bathwater.
The ambitiousness of perl 6 really warranted a different name for the language, and more explicit breaking ties with Perl's established and productioned legacy. This would have helped both projects I think. Even working on (vs in) perl 5 is somewhat considered a dead end, because it has perl 6 looming over it. It ends up doing a disservice to the livelihood of both projects.
It is very actively developed. Some (small group of) people use it every day. Some (very, very small group of) people even use it in small portions of their business.
My main issue though is that static languages are improving to the point where I start to wonder about the relevance of any dynamic language.
IMO static typing can be cumbersome for exploratory programming. That said, I really like the added benefits it can impart. I think an optionally typed language may be the best of both worlds, but my experience with them is minimal.
After 3 or 4 years of crying wolf, I and the rest of the industry simply moved on.
When 6 finally started coming out, the complex release mechanism and the difficulty in firing up a fresh install of *-nix and typing "perl" really left a bad taste in my mouth.
Lots of that has finally cleaned up, but Perl6 is not a default package in any OS yet, and there's still years of performance improvements yet to do before it's really production ready.
I think the very fast progress of Go has also made me rethink spending time with Perl 6 and instead thinking of taking some time to get familiar with Go.
(I say all this as a very huge Perl fan who's solved some very hard problems in the past with Perl and had it help me make a successful career out of being able to solve those kinds of problems without lots of fuss).
TIMTOWTDI makes it hard to share code, even between masters. I believe it's the source of the "Perl is unreadable" meme. Because everyone is capable of defining their own personal dialect of Perl, no solid, common subset emerged.
Compare with the Zen of Python: "There should be one – and preferably only one – obvious way to do it." In Python, it's generally very easy for two similarly-skilled Python hackers to share code.
It looks like a Wikipedia editor shares this view: http://en.wikipedia.org/wiki/There%27s_more_than_one_way_to_...
That's wrong. This common subset converged through consensus and finds expression in e.g. the Modern Perl movement, the module collection Task::Kensho and the updated motto "There is more than one way to do it, but sometimes consistency is not a bad thing either".
Wild speculation follows:
There's always a need for a solid "just get shit done" language, but maybe any good language in this space is doomed to die by its own success. Crufty, hard to maintain code accumulates over time, because quickly banging out code you don't expect to still be maintaining in 10 years is the whole point. As that cruft accumulates, people start noticing it and inevitably blame the language. That leads to casting about for the next "just get shit done" language, and the cycle repeats.
Note: You and your in this case is meant generally, as it most definitely includes me.
My collection of Programming Books Which I Will Discard Before I Ever Move Them Again includes a book about CGI programming with the Korn shell. I am not making this up.
As you well know, back in the day when CGI was magic and server-side includes were amazing, there weren't a lot of options for server side programming. You had C, if you were a Unixy systems programmer and didn't mind doing string processing in C. You had Tcl, if you wanted to go the AOLserver route. You had whatever shell script you wanted, if you wanted to pipe and stream together Unixy commands. After a time, you even had server-side JavaScript, if you spent a lot of money on Netscape server products.
You also had Perl, which ran on just about every Unix under the sun and behaved the same way pretty much everywhere. Even if you couldn't rely on one Unix utility behaving the same on every Unix variant (unless you somehow managed to get the GNU tools installed), you could rely on Perl being just about everywhere. Better yet, it had a fair library of code being developed and published because people were sharing it.
In the early days, you might have to use a Perl fork to connect to an Oracle or a Sybase installation, but you even had that option. (Yeah, you could do that in C too, but that would be borrowing pain far beyond string handling in C.)
It's pretty easy to see why Perl ended up with a lot of that market. It fit that niche which hadn't previously existed, and there wasn't much competition. (Korn shell. Korn. Shell.)
The market expanded rapidly. Then the competition expanded rapidly because the market expanded rapidly. It's innumerate to suggest that the relative position of market share will remain the same or even grow when the market grows so rapidly and so much competition appears.
I used very, very poor wording, and on re-reading what I wrote, it looks like I'm trying to say something I'm not.
When the market expands rapidly, but you share does not should have been when the market expands rapidly, but your users do not at a similar rate
My main point just being that Perl has less relevance to this market than it did before. This is obvious and expected with much more competition. On the other hand, it wasn't a foregone conclusion that Perl would cede a high position of relevance (if not the top position) once the competition heated up, which it has. I think that's important and it's useful to recognize that (as obvious as it may be to some) so we don't become complacent by falling back on platitudes of "we are growing, so everything is okay". We've lost a lot, and I still want that to be a motivator for doing great things and telling people about them.
I consider Ruby, for example, a more evolved Perl in a lot of ways.
Perl used to be the go to language for scripting and automation (practically what it was designed for) but now that niche is also filled with Python, Ruby, node, power shell, and improved automation tools like puppet, chef, or ansible.
Perl also used to be the go to language for dynamic web applications but it fell behind as other languages gained ground and especially as sophisticated frameworks like rails put Perl development to more and more of a comparative disadvantage.
Hopefully not in strength of module system, automated testing, extension mechanisms that aren't monkeypatching, quality of core development practices, metaprogramming (if you count Moose or p5-MOP), documentation, lexical scoping, or Unicode support.
Ruby got jRuby and Rubinius, PHP got the excellent work Facebook did, Python got PyPy, and many cool languages started to appear on the JVM.
Perl got stuck with /usr/bin/perl.
Having said that, the internals of Perl continue to improve in the current interpreter, and there's some long standing amazing work such as the incredible performance of the DBI drivers (look at the techempower benchmarks to see how well Perl shines even versus Node.js when it comes to database access).
I still use Perl for small scripts, but I'm just not building anything big in it these days. Perl6 was a huge lost opportunity, and a massive failure in vision and leadership.