Ask HN:Why is Perl so dwarfed in data science by Python?
Quite baffling. Naturally far more efficient, how come Perl lags far behind or gets dwarfed in data science by Python ?
- Perl actually was really popular for a while back then, especially in the Bioinformatics/Genomics space. It was all over that field, I think partially because it's really easy to think of a genome as just a text string of ATCGs and Perl was really convenient for manipulating text.
- I originally switched to Python for some projects because I had to do simple GUIs and visualization. Pygame and TKinter were much nicer to deal with than Perl's options. If you were just reading data in from one file and writing it out to another, Perl was fine, but the GUI toolkits were miserable.
- Numpy ("Numeric" at the time) was what really sealed the deal and probably pushed a lot of others to switch. Perl had PDL and similar but they weren't as fast or as easy to understand and use. Even if Perl is "Naturally far more efficient" than Python, it's still slow enough that for non-trivial calculations you would not want to use it directly. You have to use one of these other libraries where the actual number crunching is happening in highly tuned FORTRAN or C libraries. Python's options back at that time just leapfrogged Perl's and Numpy had really nice integration with Pygame, which made visualization in your app really smooth. Perl might've caught back up, but that was around when Perl6 was announced and sucked all the air out of the room.
My first internship was re-writing a Perl program that parsed the output of `samtools mpileup`[0] into a C program that used htslib[1] to directly read the BAM file and extract the relevant data. This preprocessing (the Perl version) was the slowest part of the pipeline, even slower than the actual data analysis.
[0]: http://www.htslib.org/doc/samtools-mpileup.html [1]: http://www.htslib.org/
So readability matters. And Python is one of the most readable languages out there. It remains relatively readable even as the code and data structure get very complex. Its syntax resembles math and pseudo-code more closely than other languages. It feels more like a tool of thought, not just a bunch of alien hieroglyphics you have to write to crudely and inefficiently express how you really think about the problem.
Perl, on the other hand, is one of the least readable languages for a general audience. It is not a tool of thought for non-experts. Its syntax is ugly, obscure, overly symbol-laden, and beloved only by gurus. The supposed "feature" of TIMTOWTDI just makes it more obscure.
The thing I can never get over with Perl is that it didn't even have a sane, idiomatic notation for functions until recently (and even today I suspect it's not widely adopted). There was no such thing as just defining a function f(a,b) of multiple named formal arguments. You had to use some special magic variable and people start talking about shift operators or some such nonsense.
This is the point at which I start saying "yes, all the language choices are Turing complete, but that doesn't mean they're all equally effective choices".
Just so it's clear, you do call a function in that way... some_function($a,$b).
It's how the function gets a hold of those arguments that you're talking about..the args are in an array called @_, like:
sub some_function { ($first_arg,$second_arg)=@_; }
And, like most things in Perl, there's lots of different ways to deal with it. Using shift(), or passing args as a hash, etc.
The concept behind sigils is very simple and once you've learned it, it's not that threatening. And when talking about data structures, it is a benefit to have it more structured and know that you are dealing with an array or hash or scalar.
In my opinion, Perl is not used for data science because it lacks(or people are not aware of) libraries like numpy, pandas, etc.
I vehemently disagree. When choosing among languages today, performance is mostly good enough across all languages and readability becomes the most important consideration. It's "subjective", but human beings are subjects, and subjective things are important to them.
> I've seen a lot of unreadable code written by data scientist in Python. And a lot of readable Perl code. It's not about the language, it's about the best practices.
This is a bad and cliche argument. Yes, there is well-written code in languages that deprioritize readability and badly-written code in languages that highly prioritize readability. That doesn't mean that any differences in the inherent, built-in readability of X and Y are moot. And "best practices", which are not widely agreed upon or followed, absolutely do not suffice to paper over this.
> The concept behind sigils is very simple and once you've learned it, it's not that threatening.
"Once you've learned it" or "get used to it" is the biggest programming cliche in the world. People say this about all their favorite languages and language features. I'm sure I could complain about doing data science in Brainfuck and get someone in the comments who would say it's not threatening "once you get used to it".
Well sorry, but I don't want to get used to your favorite language's line noise. That's why I choose another language. That's why everyone else chooses another language.
> And when talking about data structures, it is a benefit to have it more structured and know that you are dealing with an array or hash or scalar.
You can keep track of types in Python with gradual typing. Most people don't bother, so evidently it's not that important in most cases, but you can do it.
Other than that, no, I don't need to fill my code with more line noise to constantly remind myself what type of container I'm dealing with. That's obvious enough from the methods I'm using to manipulate the container.
> In my opinion, Perl is not used for data science because it lacks(or people are not aware of) libraries like numpy, pandas, etc.
No, this is after the horse has already left the barn. Those libraries were developed in Python because the authors of those libraries preferred to work in Python. The authors preferred to work in Python because of the legibility considerations I've already mentioned.
Python's design and adoption were both heavily driven by a vision of readability from the very earliest stages in the 1980s/90s. People got into Python because they liked how the code looked:
> A friend of mine who knows nearly all the widely used languages uses Python for most of his projects. He says the main reason is that he likes the way source code looks. That may seem a frivolous reason to choose one language over another. But it is not so frivolous as it sounds: when you program, you spend more time reading code than writing it. You push blobs of source code around the way a sculptor does blobs of clay. So a language that makes source code ugly is maddening to an exacting programmer, as clay full of lumps would be to a sculptor.
- Paul Graham (http://www.paulgraham.com/pypar.html, August 2004)
I absolutely agree that readability is most important consideration. I just disagree that it depends on the language. As said before, there is a lot of write-only code written by data scientists in Python.
> Well sorry, but I don't want to get used to your favorite language's line noise. That's why I choose another language. That's why everyone else chooses another language.
You don't have to. It's called a programming language paradigm, not noise. You either learn it or not when studying a language. If you want to understand something without studying it, it's your choice. I am proficient in both of these languages and can see the strengths and weaknesses in both of them.
> Those libraries were developed in Python because the authors of those libraries preferred to work in Python. The authors preferred to work in Python because of the legibility considerations I've already mentioned.
Perl was extensively used by scientific community, but Python was the lucky one that was backed by Google in 2005. And that's ok. It is better than Perl for data science now, but not because of its syntax :)
> Perl was extensively used by scientific community, but Python was the lucky one that was backed by Google in 2005.
I'm curious about this. Can you point to any specific evidence that being "backed by Google in 2005" was a momentous event for Python? AFAIK Google is not a major player in the Python scientific computing / data analysis / ML community outside of TensorFlow, which was introduced after Python was already huge.
Here are some stats with the popularity of various languages over time: https://statisticsanddata.org/data/the-most-popular-programm...
In 2005 Google hired Guido van Rossum, where he spent half of his time developing the Python language. As you can see, before 2006, Python was little less popular than Perl(9th vs 8th). By the time Guido left Google(December 2012), Python became 4th most popular programming language.
To quote from the magazine article on Python for Scientists and Engineers:
"During these early years, there was considerable interaction between the standard and scientific Python communities. In fact, Guido van Rossum, Python's Benevolent Dictator For Life (BDFL), was an active member of the matrix-sig. This close interaction resulted in Python gaining new features and syntax specifically needed by the scientific Python community. While there were miscellaneous changes, such as the addition of complex numbers, many changes focused on providing a more succinct and easier to read syntax for array manipulation. For instance, the parenthesis around tuples were made optional so that array elements could be accessed through, for example, a[0,1] instead of a[(0,1)]. The slice syntax gained a step argument— a[::2] instead of just a[:], for example—and an ellipsis operator, which is useful when dealing with multidimensional data structures."
Personally I'd much prefer D language to replace python for data science due to its Python like programming vibes but with more powerful real-time programming constructs, and its suitability for beginners or type A data scientists [3]. Python is already showing its limitations especially for the B type of data scientists [4]. There's recent discussions on Dlang forum for the potential improvements of D for data science and there's also recommendation for Walter to join such SIG as was Guido [5].
[1]Numpy History:
https://en.wikipedia.org/wiki/NumPy#History
There's even recomendation from Dlang forum for Walter to join such SIG [3].
[2]Python for Scientists and Engineers:
https://www.computer.org/csdl/magazine/cs/2011/02/mcs2011020...
[3]Teaching D from Scratch: Is it a viable first language?
https://news.ycombinator.com/item?id=29711800
[4]Type A Data Scientist vs. Type B Data Scientist:
https://www.projectpro.io/article/type-a-data-scientist-vs-t...
[5]Dlang forum: Pandas like features:
https://forum.dlang.org/thread/zuebxhkycgxsoipsgnbk@forum.dl...
The other person claimed that Google magic somehow led to Python becoming dominant in the sci/numeric/data computing community, and that just strikes me as wildly implausible. Google has done negligible foundational work on scientific Python. No one in that community has ever cared what Google thinks.
Anyways, I'm curious why you think D has potential as a Data Science language? Why not a more popular language like Julia or Rust?
Performance of applications written in Python is good enough only were there is a very low requirement for fast code (e.g. glue code / shell replacement of which there is a lot and, as computers get faster, more and more) or there are native code libraries (e.g. numpy). Of the popular languages, only Ruby is slower. I have high hopes for Julia (at least in the data science / numerical computing domain) which performs much better, while preserving readability and ease of use of Python.
I strongly disagree.
As someone who had to interview for VLSI design positions (both sides of the table) back when Perl was popular, I used to take a printout of a small tool I developed in Perl with me to every interview (parse a VCD file and check the results).
This forced the interviewers into MY subset of Perl. That made a huge difference into how competent they thought I was with Perl.
Most of the interviewers gave up after we went through about 10-20 lines of code because they had never seen half of the stuff in my code--and I wasn't doing anything particularly obscure (decorate-sort-undecorate/Schwartzian transform would lose most of them).
Both sorts have the same O(n log n) number of operations performance. However, in Perl, using an internal comparison for integers was a significant constant faster than using a user specified comparison function--at least an order of magnitude and sometimes two. Consequently, the 2*O(n) to decorate and undecorate the list with integers to sort by got swamped by the 1/10*O(n log n) sort improvement.
I believe that this overhead eventually got reduced. However, I remember using this a LOT in Perl 4 and early Perl 5 days.
As for best practices, Python does insist on writing readable code a lot. Python people always nudge you to write readable code. Guido himself said he designed Python to be a readable language.In general, Python has the philosophy of one way of doing something which contrasts with Perl way where you can accomplish a task in numerous ways. You can't be too clever with Python (little exaggerated position). However, the same is not true with Perl.
However, for data science, the biggest drawback vs Python is how you work with complex data structures. Because everything in Python is an object, (de)referencing things is generally straightforward, like:
somelist[12]['whatever']="abc"
mylen=len(somelist)
$somelist[12]{'whatever'}="abc";
$mylen=length(@somelist);
Edit: A real example from the perl data structures (perldsc) man page:
print "it turns out that $TV{$family}{lead} has ";
print scalar ( @{ $TV{$family}{kids} } ), " kids named ";
print join (", ", map { $_->{name} } @{ $TV{$family}{kids} } ); print(f"it turns out that {TV[family]['lead']} has ", end=‘’)
print(len(TV[family]['kids']), " kids named ", end='')
print(*[kid.name for kid in TV[family]['kids']], sep=', ')
Somehow Python took the niche of “executable pseudocode”, “glue language” for me.
Minus the sigils and crazy wrapping, like: @{ $TV{$family}{kids} } to refer to the array, but a different sigil/form if you're talking about one element, etc.
Which may not look intimidating on its own, but it's leaving out % and the backslash \$reference syntax, $object->method and other things that get confusing.
You can't, for example really make a "list of hashes/dicts". Try it, and it just turns the keys/values into array elements. You have to specifically make a "list of references to hashes/dicts". So you do something like:
push(@list,\%hash);
foreach my $hashref ( @list ) {
print keys %$hashref;
}
print("""It turns out that {} has {} kids named {}""".format(
TV[family]['lead'],
len(TV[family]['kids'],
', '.join(*[kid.name for kid in TV[family]['kids']]*)
)
As far as the perldsc page example, I think that comes down to having a better personal style that makes your code readable. I would make the top level variable a hash reference rather than a plain hash, and purposefully not insist on dereferencing multiple levels all on one line.
As someone pointed out in another comment, you can make the Python version look just as lousy by virtue of a lousy style.
My version of code doing the same task would be something like this:
my $familyData = $TV->{$family};
my $lead = $familyData->{lead};
my $kidsList = $familyData->{kids};
my $kidCount = scalar( @{ $kidsList } );
my $kidListStr = join (
", ",
map { $_->{name} } @{ $kidsList }
);
print "it turns out that $lead has ";
print "$kidCount kids named $kidListStr\n"; dataframe["A"] + dataframe["B"]
Both Python and Perl made bold transitions, Python from 2 to 3 and Perl from 5 to 6.
The Python transition was difficult but ultimately successful. On the other hand, Perl rolled the dice and lost. In some alternate universe it might have been the other way around.
Perl 6 (Raku) is effectively a different language. Porting from 5 to 6 would change most lines of code, in other words, a complete rewrite. To ease transition, Perl 6 made it so that both could coexist, but if you want to go v6, then it is all or nothing.
Python 2 and 3 are not that different. You can easily write code that is compatible in both, and porting is often not much more than using 2to3.py most of it is replacing "print" with "print()" and converting between bytes and str.
Perl strategy was that to create a modern language inspired by Perl, in indeed, Perl 6 has quite a few features ahead of its time. And Raku (the new name for Perl 6) is still one of the most advanced language today.
Python strategy was to keep the same language but address a few pain points that require breaking compatibility, Unicode in particular.
Perl 6 took a much bigger gamble.
Here's a bunch of things I was dealing with on a daily basis at the time:
* No revision control - sometimes files got deleted and research was lost
* Code that worked by accident, e.g. accessing the first element in an array using @array[$0] instead of @array[0], which worked only because $0 evaluates to "myscript.pl", and @array["myscript.pl"] returns the first element in the array.
* Some people just liked their code left-indented (read: no indentation at all anywhere)
When Python came along, with batteries included and with science-friendly libraries like numpy and scikit (so not just text processing, but proper numerical processing) you could get away with coding all your stuff in Python (though a lot of stuff is still Matlab). And on top of that it was easier to learn and to read than Python and it was fast enough. It was a no-brainer, so newcomers stopped learning Perl.
That said, Python's biggest contribution to this world, in my opinion, was forcing people to indent their goddamn code, because, as super-smart as these science guys generally were, they were also super-stubborn.
As for me, I moved from academia into the industry and it's so much nicer to work with other software developers. Sometimes I wonder if I should stop making some billionaire richer and go back to contributing to the scientific field, but, unfortunately, I have bills to pay. Maybe when I retire.
I do come across some perl every now and then but it looks like Bash on steriods to me (perhaps because of the $variables), and I never see any DataFrames or similar structure that looks familiar to me.
Idk, Perl never occurred to me and nobody ever recommended it. Is Perl good for data science? Do you have any examples? I never ran into "efficiency problems" with Python btw, I'm not using it at that scale, at what scale would I notice this? I have run out of ram at times, but that's usually when a dataset on disk is already larger than my ram. But then I usually find a tool to deal with the data anyway (for example pyosmium for super large osm/o5m fles).
Edit: I feel that Python also gives me other nice things to get started with, things like Django and Snakemake. This also leads me to recommend Python to other people, it's a broad basis for a lot of stuff. That's why even though some people recommended R when I got started, I choose Python anyway. I have no regrets, unless you are going to blow my mind with some examples...
Python also got heavyweight libraries such as Numpy and Pandas which put it in the front. Perl do not have have such well known libraries as far as I know.
Amen to this point. I am currently the single engineer in a small software biz and after the last engineer left I thought to myself "well, at least I won't have to decipher his code anymore"...needless to say, I've been shocked by how much time I can spend on a system only to revisit it within a year and have little to no memory of how or why I did something a certain way!
This. For the kind of tasks data science involves, you either write C++ or you need the equivalent of Numpy. No Numpy means no Pandas, which means no sane way to wrangle datasets.
For all the hate it gets, Numpy is a world-class library.
Looking now (it's open source) there's two keywords I have forgotten. It's a Perl binding to a C library using an old version of Swig. There's lots of Perl or C helper code to smooth out the bumps. It also looks like it hasn't been updated (bummer).
> Naturally far more efficient
What does this mean exactly? One big reason for Python’s success in data science is numpy, which is far more efficient (especially on large data) than vanilla Python. I’m unaware of the state of the Perl ecosystem, does it have something similar?
> how come Perl lags far behind or gets dwarfed in data science by Python?
Perl seems to be lagging behind in general, no? Is there anything specific about data science where Perl should be shining?
Perl has PDL (http://pdl.perl.org), which I think predates numpy and does the same kind of array manipulation.
As for the original question, Python's dominance for "data science" projects is just a matter of momentum at this point: it's where Google and others have put their money, and that is now affecting undergraduate and graduate courses. Originally, I think it was just that some people didn't like Perl's syntax.
Personally, I prefer to use a mix of Perl and Julia, each for its respective strengths, and find Python all-around mediocre.
In the cgi-script mode Perl had to start a new process and compile all your code for each request. There was "mod_perl" which was more efficient but frequently you struggled with memory leaks and other reliability problems.
PHP came out and then Apache Tomcat, web hosting systems for Ruby, etc. all of which had efficiency similar to mod_perl but easy and reliable environments to work in. Generally there were many modules in CPAN that were essential to web development (HTML escaping) for which bugs were not getting fixed and that added to the feeling that Perl was slipping behind.
As for more general scripting I think people found Python was better. Even though it is cross-platform, Perl has a strong UNIX feel to it. Python doesn't feel like it belongs to Windows, UNIX or any other environment, rather it feels comfortable anywhere.
I mean, even the author of Learning Perl, said "sometimes Perl looks like line noise to the uninitiated, but to the seasoned Perl programmer, it looks like checksummed line noise with a mission in life."
$ python3
>>> dir(dir(dir))[(not False)*41].upper()
'INDEX'
>>> type(dir(dir(dir))[(not False)*41].upper())
<class 'str'>I've seen (rarely) old shell scripts and even assembler made readable by copious use of comments. I think the claim should rather be that you can't write easy-to-read self-documenting code in all languages.
I think we could make better languages than Python that fix these issues. Like an implicit boolean -> int conversion does more harm than good, IMO (and it appears in your example). But I'm still not surprised that of the languages we have had when we have had them, that Python absolutely dominates this space.
However, it might be a bit too much if you are from other fields and just want to get your sums in order.
Then a whole ecosystem was built on top of numpy, with scipy, pandas, PIL, et cetera. Everything that uses n-dimensional arrays used numpy as their base and as a result all those things can be combined trivially. That's very powerful.
Then later came ipython, a much improved interactive shell, and Web based notebooks that are very useful for data science work.
That the language involved is Python isn't even important, imo. Numpy + ecosystem replaced Matlab. All Python has to be is be a better language than Matlab, and it is.
One thing to consider is that scientists are not necessarily programmers, there is an overlap, especially in data science, but from my experience, they tend to write terrible code. It is not to be dismissive, it is just a different skill set, there would be no need for professional programmers if scientists could do better and vice versa.
And Python is very interesting in that it is actually difficult to write terrible code with it, forced indentation and clean syntax certainly helps, it also heavily promotes the one "pythonic" way. Contrast with Perl "there is more than one way to do it" philosophy.
It is not that you can't make a huge mess with Python, but it only tends to happen at the intermediate level, like when you are starting to write libraries but have not yet reached mastery.
perl doesn’t have the numerical chops to keep up, and if it started to fix that now, it has 20 years of headwind to fight through for probably marginal gains. Good luck.
I think R is worse but it is bundled with a very good stats library. R users aren't so interested in "programming in the large" or even the small so it is OK.
Other than the meaningful indentation Python strikes me as a generic and normal programming language. I'm quite amused at how Python and Java got pattern matching at about the same time with a broadly similar approach to "fit this new feature in this language so that it feels like it belongs and plays well with decades of existing practice".
However the community is untalented, they historically produce very bad code, it all started with Matt's script archive. They do create horrible sites, PerlMonks is a good example, It looks horrible and its usability is horrible. The code posted by PerlMonks users is mostly very bad. CPAN that is often pointed as positive is just a mostly poorly documented repository of very bad code. There are very few usable modules on CPAN.
Perl community has also written some of the worst technical books ever.
Talent attracts talent, that is the reason Perl is dead for any kind of usage and Python is popular.
For that matter, I'm not sure what you mean when you say Perl is "naturally more efficient" than Python. There's nothing about Perl that makes it easy to run faster, and the ability to write incomprehensible one-liners is not a very satisfying measure of "efficiency".
As for why people choose Python over Perl, Perl is a pain in the ass in a lot of ways, and I say this having written thousands of lines of Perl back in the day. Dollar signs on variable names? Obvious bad code turning out to be syntactically correct but do weird stuff, because of strange irregular legacy syntax rules? Library code being unreadable because of the aforementioned incomprehensible one-liners?
Python is bad enough about not finding errors until the code blows up in weird ways, but Perl is worse.
Don't believe me? Look at how numbers are stored: https://rushter.com/blog/python-integer-implementation/
But it doesn't really matter. Computers increase in speed over time. Libraries allow speed sensitive sections to use more sane data types. As you pointed out, the Perl language's design decisions cause much worse problems -- hard to identify bugs.
In this way, the statistical packages and plotting software in Python is better than Perl. I would say R is even better than Python and Perl for certain statistical analyses and quickly plotting complex data in different ways.
Perl might beat other languages in wrangling certain types of data like comma separated values or other 2D arrays in terms of writing expressive one-liners. Perhaps that is what you mean by "naturally efficient"? How much one can do in one line of code?
There is the "time to first plot" issue, and I found that the "efficient as C, easy as Python" motto really means "efficient as C XOR easy as Python", but all in all, it's very easy to write stuff very cleanly, and the path to effiency is quite natural if you know where to look, and the metaprogramming makes it that more powerful.
System administration never fully replaced awk/sed/bash with Perl and the new wave was all configuration management, like chef and puppet.
Python was considered to be a "clean", algol-style language, so universities started teaching it twenty years ago. Only logical after teaching Pascal for decades. Students kept using Python, so now there are lots of data science projects around.
It's been a while since I was in that field, but I suspect those kind of low level operations are now heavily optimized in faster languages as sequences to operate on became longer and operations more complex.
One of the main issues I have with perl is the `there is more than one way to do it` slogan, which, for me, means that each person what writes perl, writes practically a different language. This make the bar of starting a new project much higher, even if you have use perl before.
As noted above, Python generally has a less cryptic syntax and tends to be much easier for non-experts to read.
The main barriers to good communication of data science is the translation between the real world goals, data, algorithms, assumptions and priors, scientific and statistical methods being applied, testing, the deployed model's performance and ongoijg monitoring and management, etc .. and the actual code and artifacts produced to achieve those ends.
Python wins because there is a wide ecosystem around those translation efforts.
* Visualization and statistical profiling are first class citizens, probably the killer app in terms of communicating difficult mathematical concepts
* Easy to extend, very "framework-friendly", so you can "speak the same language" between data engineering, DS, analysts, MLOps folks
* Needless to say, network effects of a community used to Pythonic idioms
If I were writing something where I actually cared about low-level performance (a 'let's see what we can get the compiler to inline and unroll' sort of code), I guess I'd
1) start by writing pseudocode
2) code it in C or Fortran.
The fact that there's a runnable version of pseudocode called Python means that often people will stop at the first step, realize computers are incredibly fast, and be happy enough with not writing the Fortran (just sprinkle in some NUMPY for the crunchy bits).
Lots of cases can be handled with large calls to heavily tuned libraries anyway, where most programmers won't beat the library in C or Fortran, let alone Perl.
Every time I look at an example I am turned off and my mind is in a way "boggled" ...
I just opened the tutorial on the Raku site I see this
say looks_like_number "foo";
right from the beginning, the say turns me off, computers do not speak, at least I really hope the computer is not actually speaking when I type that. Then there are the weird rules of quoting to not quoting ... I can't quite tell the rules are
the example is unpalatable, why does the looks_like_number word have underscores? is that a odd variable name that needs underscores? but it looks like a message to the user, what a terrible choice either way!
what does this print? I can't tell ...
I mean things like https://youtu.be/7Ivoh-fzDw8 , https://stackoverflow.com/questions/60510667/how-can-i-defin... and https://en.m.wikibooks.org/wiki/Raku_Programming/Grammars
The example for "looks_like_number" has probably be chosen, because many experienced Perl programmers are familiar with that.
In Raku, you can use kebab-case, and if there would be such a routine in Raku, it would probably be called "looks-like-number", and return a Bool, as in either True or False.
"say" is also builtin in Perl, at least for the past 12 years or so. And if you are on a Mac, running "say this is your computer talking to you', it will talk to you.
In any case, the thing you appear to refer to, is in no way a tutorial, but rather documentation intended for a specific group of people: Perl programmers.
> right from the beginning, the say turns me off, computers do not speak
That said, `say` is just a normal function, and you could always bind a new function and use that instead.
Quick addendum: data science != computer science, most data scientists learn coding on top of another skillset, not as their primary area of expertise, so things like under-the-hood efficiency are often second order concerns to learn-ability, ease of use and maintenance.
Perl is.... perl. If you read it it's not at all obvious what's happening. They don't believe in the one and only one obvious way principle.
What? No it isn't
.
> how come Perl lags far behind
Because nobody wants to use it, same reason it lags far behind in literally every other field in programming, too
If you actually want to understand this, learn why CGI doesn't point at Perl anymore. That was its last real stronghold.
(Only half joking, everything is dwarfed by Python in the data science community.)
> Naturally far more efficient
Most data science languages are interpreted with a few higher-level routines. Same with J, for instance - it has adequate performance as well.
Also because Python is readable by most software engineers and data scientists, unlike R.
Exploring Bioperl for genetics leads to several big rewards, but you need to want to take the extra-mile and doing it. And is not easy. In the same way as exploring R in the past where none used it, was definitely benefical to me. R is awesome.
All is readable if you find the correct people able to read it. People at the university just keep using the same as their boss so you really can't always choose. Maybe Perl wizards were highly sough and didn't remained in the university. Maybe is just a question of different generations of programmers and Perl is a little older. Dunno.
[1] Correct (do what it's supposed to do). [2] Maintainable (understandable, debuggable). [3] Fast.
In that order.
That's why I won't touch it.
At the time, I did actually use perl a fair bit for data munging. Perl is a lot nicer for anything that required lots of system calls and involved a lot of pure text processing, but that only goes so far. The standard pattern was pre-process ascii data in perl, write ascii or simple binary formats out to disk, invoke some system executable (often something written in fortran) on the file you've written out, read back in the output. Perl is definitely nicer than python for that workflow. However, that workflow has severe limitations. The roundtrip to disk / stdout / etc is pretty crappy for some things.
There really weren't good numeric data containers in perl, at least that I was aware of at the time. Even before numpy, there was numeric. Numpy/numeric focus on c-like in-memory arrays that can be semi-directly passed into / referenced from low-level libraries. That's huge -- suddenly it's easy to manipulate large numeric datasets in memory and _maintain memory efficiency_. No linked lists, very clear rules about what creates intermediate copies, etc. You then can pass these directly into C / Fortran routines without a copy in many cases. (Okay, that last part is non-trivial, especially at the time, but very possible.)
Then there's plotting. Folks forget just how interactive matplotlib is, and was from the very early days. From the perl side, I was using gnuplot/etc (and even more of a domain specific tool called GMT). That meant static figures. Matplotlib meant I got an interactive figure and something that I could easily embed in Tk to make quick GUIs.
I also used Matlab heavily at the time, but it was pretty difficult for the things that needed to interact with everything else (read: old F77 routines and proprietary domain-specific data processing tools). Licensing was also an issue, as there were a limited number of matlab licenses, and you couldn't reliably count on being able to check one out, especially for cron-esque jobs.
Python bridged the two. You had a matlab like environment, decent data munging ability, a good language, and also a good environment for building other tools. This was all possible in Perl, in principle, but the key tools weren't there in Perl, even almost 20 years ago. Basically Perl couldn't replace Matlab easily and Python could.
So why didn't they get built in Perl instead of Python initially? I suspect the short answer is operator overloading. Python is _really_ nice for that, and it's a very nice way of having flexible array manipulation syntax. Second to that is that Python is more readable, and readability matters in the long term.
Also, don't discount how big of a deal having Tk support by default in python is, though. Yeah, sure, these days folks completely ignore desktop GUIs, but at the time web apps were pretty irrelevant. Desktop GUIs were everything. Being able to whip up a quick reusable gui data processing application that a random lab assistant or new grad student could easily use was/is a very big deal, and that was way easier in Python than most other things, especially at the time.