Happy to answer questions as far as possible. I have used Pandas extensively but I don't have deep experience with all of these libraries so I learnt a lot while summarising them.
If you know more than I do and I made any mistakes, let me know and I'll get them corrected.
Thanks for the article, before it I only knew of Dask as a real alternative.
P.D. I just remember that I wanted to try Pandarallel as well, so you have any insight on this library ? Thanks!
To be fair, that's one of the reasons that Spark ML stuff works quite well. Be warned though, estimating how long a Spark job will take/how much resources it will need is a dark, dark art.
I've worked professionally with data scientists, and we've used both Dask and Ray with some success. Scaling pandas will be an issue for a long time to come with a lot of data science code being written in Python with Pandas.
For example, as of the beginning of this year, the latest version of Pandas was 0.25. ((t jumped to 1.0 in late January.) This despite it having been a core part of the standard professional toolset for years and years now.
That said, it's hard to keep SQL readable when doing more complicated data analysis, and you'll probably want the flexibility of Python the moment you start to do anything more custom.
Spark is ace as it has an SQL API available cross-language, which makes ETL much more effective, and ML models (though I've always been sort-of suspicious about their maturity).
tl;dr - demonstrate the speed of running regressions in Spark, and many (most) data scientists will invest the time in learning the tool.