In pig, any reasonably large workflow has had me hooked for hours working with stub data trying to figure out how data flows and comes out.
In fact I see this nothing new to Perl. If you are working with any serious application. A good touch over the language plus experience with handling things with patience is a must. Python is no exception to this rule, neither is any language I know of.
>>I may be completely missing the mark and actually readability is much better. I really hope so.
It depends on how you define readability. I would say verbose code is equally unreadable especially if you work with large applications. If you simply don't like regular expression you have a different problem. In some cases use of regular expression is simply unavoidable. IMO Python works very well for applications that interact with standard data sources(Databases, API's, XML's). Once you get the data to some very easily ingestable form, and there is little work to do from there on, Python works just fine. But Perl works fine for other use cases, heavy lifting and other kinds of data work. Perl may not be the right language for all use cases, but neither is Python or Java.
>>Vapourware is vapourware. Perl 6 was meant to be production ready a decade ago.
No, I don't think they ever gave out a date when they wanted a finished version out. Besides Rakudo is feature ready already. They are just working out last bit of performance issues and good to have things. And plan to have it out by 2015: https://fosdem.org/2015/schedule/event/get_ready_to_party/
http://perl6.org/compilers/features
>>On the other hand, I'm interested in knowing what you think Perl 6 brings that will give it the edge over Python?
Grammars, Macros, Concurrency, Traits, Types, Improved regexes, Better OO, Functional programming, User defined operators, Mutable grammar, Junctions, Currying, Continuations, Multiple VM backends to name a few.
The full list is at : http://perlcabal.org/syn/
>>Do you see it displacing Python as rapidly as Python replaced Perl 5 as the science "glue" language of choice?
Perl and Python both co exist and will continue to, albeit the web frameworks market is dominated by Python and Ruby at this point in time, Perl is still pretty big in a lot of places. This isn't Perl or Python, people use tools which work for them within their project limitations.
But I guess people will use what they find useful. And that only individual projects can decide as to what works best for them.
>>With the apparent rise of Julia as well, could it just be too late?
Reading the Perl 6 spec will tell how different Perl 6 is with regards to everything else. Even with the delay Perl 6 has suffered, I don't see any other tool that has the feature set Perl 6 has to offer.