According to these benchmarks: https://h2oai.github.io/db-benchmark/, DF.jl is the fastest library for some things, data.table for others, polars for others. Which is fastest depends on the query and whether it takes advantage of the features/properties of each.
For what it's worth, data.table is my favourite to use and I believe it has the nicest ergonomics of the three I spoke about.
In the Julia world the one which optimizes to be fully non-dynamic is TypedTables (https://github.com/JuliaData/TypedTables.jl) where all column types are known at compile time, removing the dynamic dispatch overhead. But in Julia the minor performance gain of using TypedTables vs the major flexibility loss is the reason why you pretty much never hear about it. Probably not even worth mentioning but it's a fun tidbit.
> For what it's worth, data.table is my favourite to use and I believe it has the nicest ergonomics of the three I spoke about.
I would be interested to hear what about the ergonomics of data.table you find useful. if there are some ideas that would be helpful for DataFrames.jl to learn from data.table directly I'd be happy to share it with the devs. Generally when I hear about R people talk about tidyverse. Tidier (https://github.com/TidierOrg/Tidier.jl) is making some big strides in bringing a tidy syntax to Julia and I hear that it has had some rapid adoption and happy users, so there are some ongoing efforts to use the learnings of R API's but I'm not sure if someone is looking directly at the data.table parts.
Agreed, and the DF.jl developers are aware and very open about this fact - the core design trades off flexibility and user friendliness over speed (while of course trying to be as performant as possible within those constraints).
One thing that hasn't been mentioned so far is InMemoryDatasets.jl, which as far as I know is the closest to polars in Julia-land in that it chooses a different point on the flexibility-performance curve more towards the performance end. It's not very widely used as far as I can tell but could be interesting for users who need more performance than DF.jl can deliver - some benchmarks from early versions suggested performance is on par with polars: https://discourse.julialang.org/t/ann-a-new-lightning-fast-p...
I have not tried it. I like that the project makes broadcasting invisible, I dislike that it tries to completely replicate R's semantics and Tidyverse's syntax. Two examples: firstly, the tuples vs scalars thing doesn't seem very Julia to me. Secondly, I love that DF.jl has :column_name and variable_name as separate syntax. Tidier.jl drops this convention (from what I see in the readme).
> I'm not sure if someone is looking directly at the data.table parts
I believe there was some effort to make an i-j-by syntax in Julia but it fell through or stopped getting worked on. By this syntax I mean something like:
# An example of using i, j, and by
@dt flights [
carrier == "AA",
(mean(:arr_delay), mean(:dep_delay)),
by = (:origin, :dest, :month)]
# An example of expressions in by
@dt flights [_, nrows, by = (:dep_delay > 0, :arr_delay > 0)]
data.table also has some nifty syntax like
data[, x := x + 1] # update in place
data[, x := x/nrows(.SD), by = y] # .SD = references data subset currently being worked on
The conciseness and structure that comes from data.table and its tendency to be much less code than comparable tidyverse transformations through some well-informed choices and reservations of syntax make it nicer for me to use.
Personally, my main usability gripe is that it's difficult to do row-wise transformations that try to combine multiple columns by name. I know one can do ``` transform(df, AsTable() => foo ∘ Tables.NamedTupleIterator) ```
But this is 1) kind of wordy and 2) can come with enormous compile times (making it unusable) for wide tables
Similarly make sure you research the ecosystem because everything in Julia is very fragmented, IE pandas.loadcsv will require two or more packages in it's Julia equivalent.
Furthermore, Bogumil Kaminski, one of the main developers behind DataFrames, makes sure that the DataFrames tutorials he has created here (https://github.com/bkamins/Julia-DataFrames-Tutorial) are updated on every new release.
No, it isn't. I'm using Pandas, DF.jl, and even polars at work. DF.jl is by far the best/easiest/quickest to use, as its syntax is consistent. Polars is a bit more annoying as its syntax is further along the learning curve that I have gotten yet.
Pandas ... what to say about a library that will happily return a pd.Series in one moment, and a pd.DataFrame in another, for the same function call. This means you need extra code like
if type(ret_thing) = pd.Series: # then do something to coax it back to a df.
lest your actual code break.
This is of course the same language that has API differences that make no sense in, say, re.match vs re.findall vs re.search. I've been burned by all of those.
So, look, we get you hate Julia. That's fine. Go live your python life to its best. But really, stop with the misinformation/FUD. This speaks volumes about you, and tends to make the case precisely the opposite of what you think.
And yes, I use Python, Julia, C++, and many other languages in the $day_job.
Yes the ecosystem is very fragmented. It's done so by design. One of the major contributors to the language wrote a paper about it.
At its most basic the obviouses becomes different:
Let's try to get a very simple object: a 3,3 matrix of random booleans in both languages: Julia:
A = rand(Bool,3,3)
Python: No standard support for Matrices. I could really do it a disservsice and comapre the "core language", but that''s obivously stupid so we'll bring in some external libraries to make it easier. Of many ways to skin the cat here's one.. Python
import numpy.numpy as np gen = np.random.default.default_rng() B = gen.choice([True,False],(3,3))
BTW julia has this choice function built into the command as well, so rand(["Which", "Word", "Will", "I", "get?"]) produces exactly what you'd expect.
----
Acutally I can't think of any cases at all off the top of my head. Sorry, I mean my np.somenamespace.another.namespace.sparce head :) I mean just going down the list of things that make code easier in Julia..
* Python requires a third party library for any kind of linear algebra. and matrix multiplication:*
A =[1 4;6 7"; B = [2 ;3]; A*B doesn't work, i.e. you literally even multiply a matrix! This is madness.you'll need yet again a third partly library
import numpy as np
import math #sigh
x = np.array([1, 2, 3, 4, 5])
f = lambda x: sin(x)
Even basic string interpolation in Julia is a much nicer "trivial task" than in python alone
No special brackets, just clean "$myvar"
# Reading files is easier
Julia
readlines("my_test.txt")
open("my_test.txt").readlines()
What a stupid option in 2? if I call readlines on a filename, 99% of people, 99% of the time, want to read the lines on the file at that path. Why require two function calls?
I really don't understand what you are trying to complain about. Namespaces are nice. Dumping everything into the global namespace sucks.
For one thing, I've found it very useful when folks use syntax like `import numpy as np`, because then when I see `np.foo` I can trace back where `foo` comes from and look up the relevant documentation.
The complaint about `open("foo.txt").readlines()` vs `readlines("foo.txt")` is a red herring IMO, because nothing stops anyone from implementing a generic `readlines()` function in Python that can take a string file name or a file handler object. It's just that nobody really cares enough to because it's a complete non-issue.
>Julia
A = rand(Bool,3,3)
>Python
import numpy.numpy as np
gen = np.random.default.default_rng()
B = gen.choice([True,False],(3,3))