I've always found pandas really hard to use or reason about. I eventually get there but I don't like the code. Obviously subjective. I've never used another "data science" language though so I've no experience beyond it.
It has a rather different api, and is significantly faster. Highly recommend it.
So you are one `df.to_numpy()/df.to_pandas()` away to `X` libary you want to use.
Or you can convert it directly into a pandas Df
Background: I first gained some experience with J, where I first learned to appreciate the advantages of array languages. The main advantage is actually having your own way of thinking about processing multidimensional data. Now, recently, I've gotten into the notation of APL, and it's really even cooler than the ASCII "noise" of J. The symbols make it easier for me to both write and read programs. Admittedly, for more complex operations with data it takes a lot of learning that you usually don't have. But for simple transformation APL is already quite fast usable and convinces beyond the "mainstream".
From https://docs.python.org/3/howto/functional.html it's got map/filter/currying and plenty more, what's misting in your view?
nothing intrinsically 'functional' about this, though it's nice
> syntax for partial application
<https://docs.python.org/3/library/functools.html#functools.p...>
> and composition,
Hmm, to my surprise I can't find this but TBH it's trivial to write, here's an eg <https://stackoverflow.com/questions/16739290/composing-funct...>
> a typing system which can express structural types,
typing is orthogonal to FP (though very nice to have)
> a generalised list comprehension
IDK what this means, what's ungeneral about comprehensions currently?
Pythons failure to provide this, indeed, outright hostility to doing so is half the reason pandas is a mess of incomprehensible syntax
By prioritising assignment expressions and implementing pattern "matching" as a statement, they're clearly showing a lot of hostility to one of the major use cases of their language
(incidentally, recall that C# introduced LINQ in 2008, a generalised comprehension! That's C#.)
There isnt the syntax to reimplement to support natural phrasings of data transformation, in lieu of this, pandas exploits weird operators such as `.loc[a,b]`.
At some point the dam is going to have to break, and python is going to have to introduce something to resolve this mess. However, i'd bet it'll be a decade of tooth-and-nail fighting about it. It isnt a software engineering language for education any more, and i'm not seeing this reality being acknowledged
Available in 3.10: https://peps.python.org/pep-0636/
> syntax for partial application...
from functools import partial
somefunc_arg1_arg2 = partial(somefunc, arg1, arg2)
A native compositional syntax would be nice.
> a typing system which can express structural types
# mypy will typecheck this code and see `MyString()` has a `.read()` method, so it's a `Readable` even though it doesn't sub-class `Readable`
from typing import Any, Protocol
class Readable(Protocol):
def read(self) -> Any: ...
def read_something(something: Readable) -> Any:
return something.read()
class MyString:
a_string: str = "something"
def read(self) -> str:
return self.a_string
read_something(MyString())
I agree this would be nice.
that isnt syntax for partial appication, and the protocol system for structrural typing is syntactically and practically absurd
the question is "why do data processing libs in python look syntactically illegible" and the answer is largely, as youve posted above, that python doesnt support useful syntax
Commenters here are keen to say "but dont you know!" -- and, yes, I do.
Python does FP fine and FP via an ugly library or via elegant inbuilt syntax is still FP. I get you want nicer syntax but it wasn't clear.
I dunno, write a PEP and see if it gets support. I hope you succeed!
https://github.com/otsaloma/dataiter
Here's a comparison of dplyr vs. Dataiter vs. Pandas, which should give quick overview of the similarieties and differences.
https://dataiter.readthedocs.io/en/latest/_static/comparison...