Pandas vs. Julia – cheat sheet and comparison
datascientyst.com
datascientyst.com
https://pandas.pydata.org/docs/reference/api/pandas.DataFram...
In R you can pipe a data frame into any function from any package or one you just wrote, so you use %>% for any piping that happens. In pandas, you have special pandas methods that don't need the pipe, but to pipe with any other function, you have to write .pipe.
The comparison is not really between %>% and ., it's between "you just use %>% for everything" and "you use . for a bloated, somewhat arbitrary collection of special pandas methods, and .pipe for everything else".
The ability to pipe shouldn't be tied to whether a function is a method of a class.
a(b(c(d))) vs d |> c |> b |> a
but I'm not convinced pipes are better than more verbose code that explains each step:
step1 = c(d)
step2 = b(step1)
result = a(step2)
I've written a lot of tidy R and do understand the specific use cases where it really doesn't make sense to use the more verbose format, but generally find when I'm building complex mathematical models the verbose method is much easier to understand.
In programming the code will be read multiple times and good names will help the future readers. But in data science the calculation will be most likely will not be reused. So efforts to name things will be waste of time.
I suggest that trying to strictly only bind output to a symbol if it will be used in multiple places.
So when I read code and I see some "intermediary" value bound - it tells me immediately "this thing will be used in several spots". Thereby bindings actually start to convey extra information
Anyway, it's just something that's worked for me. In all other scenarios I will use threading/pipeline (maybe Clojure specific). If steps are confusing/complex then you make a local named lambda or add in the extreme case.. comments
(Yes I know what you mean, but yes you know what I mean!)
In the end, chains are about readability and logical flow; even if you don't like pre-wrapping in more meaningfully named functionals like the example, and accept the slight readability cost of using the occasional in-spot lambda or partial, I feel that this still becomes a lot more readable than "treat this symbol unconventionally in this context as a positional placeholder" hacky syntax stuff.
> Provides link to R.
Is there an example of this in Julia? I use R now, and every time I give Julia a shot I go back to R because of the insane TTFP. I don't use anything remotely close to big data, and the 90-120s compile times just to replot my small data (using AlgebraOfGraphics.jl in a Pluto notebook) just kill me.
>Julia
A = rand(Bool,3,3)
>Python
import numpy.numpy as np
gen = np.random.default.default_rng()
B = gen.choice([True,False],(3,3))
Similarly make sure you research the ecosystem because everything in Julia is very fragmented, IE pandas.loadcsv will require two or more packages in it's Julia equivalent.
Furthermore, Bogumil Kaminski, one of the main developers behind DataFrames, makes sure that the DataFrames tutorials he has created here (https://github.com/bkamins/Julia-DataFrames-Tutorial) are updated on every new release.
No, it isn't. I'm using Pandas, DF.jl, and even polars at work. DF.jl is by far the best/easiest/quickest to use, as its syntax is consistent. Polars is a bit more annoying as its syntax is further along the learning curve that I have gotten yet.
Pandas ... what to say about a library that will happily return a pd.Series in one moment, and a pd.DataFrame in another, for the same function call. This means you need extra code like
if type(ret_thing) = pd.Series: # then do something to coax it back to a df.
lest your actual code break.
This is of course the same language that has API differences that make no sense in, say, re.match vs re.findall vs re.search. I've been burned by all of those.
So, look, we get you hate Julia. That's fine. Go live your python life to its best. But really, stop with the misinformation/FUD. This speaks volumes about you, and tends to make the case precisely the opposite of what you think.
And yes, I use Python, Julia, C++, and many other languages in the $day_job.
Yes the ecosystem is very fragmented. It's done so by design. One of the major contributors to the language wrote a paper about it.
At its most basic the obviouses becomes different:
Let's try to get a very simple object: a 3,3 matrix of random booleans in both languages: Julia:
A = rand(Bool,3,3)
Python: No standard support for Matrices. I could really do it a disservsice and comapre the "core language", but that''s obivously stupid so we'll bring in some external libraries to make it easier. Of many ways to skin the cat here's one.. Python
import numpy.numpy as np gen = np.random.default.default_rng() B = gen.choice([True,False],(3,3))
BTW julia has this choice function built into the command as well, so rand(["Which", "Word", "Will", "I", "get?"]) produces exactly what you'd expect.
----
Acutally I can't think of any cases at all off the top of my head. Sorry, I mean my np.somenamespace.another.namespace.sparce head :) I mean just going down the list of things that make code easier in Julia..
* Python requires a third party library for any kind of linear algebra. and matrix multiplication:*
A =[1 4;6 7"; B = [2 ;3]; A*B doesn't work, i.e. you literally even multiply a matrix! This is madness.you'll need yet again a third partly library
import numpy as np
import math #sigh
x = np.array([1, 2, 3, 4, 5])
f = lambda x: sin(x)
Even basic string interpolation in Julia is a much nicer "trivial task" than in python alone
No special brackets, just clean "$myvar"
# Reading files is easier
Julia
readlines("my_test.txt")
open("my_test.txt").readlines()
What a stupid option in 2? if I call readlines on a filename, 99% of people, 99% of the time, want to read the lines on the file at that path. Why require two function calls?
I really don't understand what you are trying to complain about. Namespaces are nice. Dumping everything into the global namespace sucks.
For one thing, I've found it very useful when folks use syntax like `import numpy as np`, because then when I see `np.foo` I can trace back where `foo` comes from and look up the relevant documentation.
The complaint about `open("foo.txt").readlines()` vs `readlines("foo.txt")` is a red herring IMO, because nothing stops anyone from implementing a generic `readlines()` function in Python that can take a string file name or a file handler object. It's just that nobody really cares enough to because it's a complete non-issue.
According to these benchmarks: https://h2oai.github.io/db-benchmark/, DF.jl is the fastest library for some things, data.table for others, polars for others. Which is fastest depends on the query and whether it takes advantage of the features/properties of each.
For what it's worth, data.table is my favourite to use and I believe it has the nicest ergonomics of the three I spoke about.
In the Julia world the one which optimizes to be fully non-dynamic is TypedTables (https://github.com/JuliaData/TypedTables.jl) where all column types are known at compile time, removing the dynamic dispatch overhead. But in Julia the minor performance gain of using TypedTables vs the major flexibility loss is the reason why you pretty much never hear about it. Probably not even worth mentioning but it's a fun tidbit.
> For what it's worth, data.table is my favourite to use and I believe it has the nicest ergonomics of the three I spoke about.
I would be interested to hear what about the ergonomics of data.table you find useful. if there are some ideas that would be helpful for DataFrames.jl to learn from data.table directly I'd be happy to share it with the devs. Generally when I hear about R people talk about tidyverse. Tidier (https://github.com/TidierOrg/Tidier.jl) is making some big strides in bringing a tidy syntax to Julia and I hear that it has had some rapid adoption and happy users, so there are some ongoing efforts to use the learnings of R API's but I'm not sure if someone is looking directly at the data.table parts.
Agreed, and the DF.jl developers are aware and very open about this fact - the core design trades off flexibility and user friendliness over speed (while of course trying to be as performant as possible within those constraints).
One thing that hasn't been mentioned so far is InMemoryDatasets.jl, which as far as I know is the closest to polars in Julia-land in that it chooses a different point on the flexibility-performance curve more towards the performance end. It's not very widely used as far as I can tell but could be interesting for users who need more performance than DF.jl can deliver - some benchmarks from early versions suggested performance is on par with polars: https://discourse.julialang.org/t/ann-a-new-lightning-fast-p...
I have not tried it. I like that the project makes broadcasting invisible, I dislike that it tries to completely replicate R's semantics and Tidyverse's syntax. Two examples: firstly, the tuples vs scalars thing doesn't seem very Julia to me. Secondly, I love that DF.jl has :column_name and variable_name as separate syntax. Tidier.jl drops this convention (from what I see in the readme).
> I'm not sure if someone is looking directly at the data.table parts
I believe there was some effort to make an i-j-by syntax in Julia but it fell through or stopped getting worked on. By this syntax I mean something like:
# An example of using i, j, and by
@dt flights [
carrier == "AA",
(mean(:arr_delay), mean(:dep_delay)),
by = (:origin, :dest, :month)]
# An example of expressions in by
@dt flights [_, nrows, by = (:dep_delay > 0, :arr_delay > 0)]
data.table also has some nifty syntax like
data[, x := x + 1] # update in place
data[, x := x/nrows(.SD), by = y] # .SD = references data subset currently being worked on
The conciseness and structure that comes from data.table and its tendency to be much less code than comparable tidyverse transformations through some well-informed choices and reservations of syntax make it nicer for me to use.
Personally, my main usability gripe is that it's difficult to do row-wise transformations that try to combine multiple columns by name. I know one can do ``` transform(df, AsTable() => foo ∘ Tables.NamedTupleIterator) ```
But this is 1) kind of wordy and 2) can come with enormous compile times (making it unusable) for wide tables
But I agree this should have been reflected in the title of the article.
df = pd.DataFrame( {'col_1': [11, 12, 13], 'col_2': [21, 22, 23]}, index=[0, 1, 3])
julia> setprecision(1024)
julia> a=BigFloat(1.0E-300)
1.000000000000000025059091835208759685696146807703705249925342319900466043184051484676302812181950100894962306270278254148910311464998804130812246091606190182719426627934584275510414782787015070222639260603793613924359775094030143866141479125513590882591017341692222921220404918621822029155619541859418525883262e-300
julia> a=BigFloat("1.0E-300")
9.999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999988e-301
Now we can add it to 1.0 to validate the precision of a calculation and use the @printf macro for C-style formatting to round the output to 308 decimal digits:
julia> b=BigFloat("1.0")
julia> using Printf
julia> @printf("%.308f\n", (a+b))
1.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000
Um. You said the answer earlier:
> because the parser treats the literal as a double
BigFloats aren't built in to the syntax of the language (and probably should't be, so you need to escape the parser somehow -- either pass a string to the constructor, or use the @big_str macro to get a non-standard string parsed into a BigFloat.
It makes no sense to an end user that expects an argument you pass to BigFloat to be treated as a BigFloat. As an end-user I would rather have a warning or even error than to have my argument silently treated as a double.
BigFloat(x::AbstractString) is identical to parse. This is provided for convenience since decimal literals are converted to
Float64 when parsed, so BigFloat(2.1) may not yield what you expect.
...
Examples
≡≡≡≡≡≡≡≡≡≡
julia> BigFloat(2.1) # 2.1 here is a Float64
2.100000000000000088817841970012523233890533447265625
The solution here is a good linter though, not adding more work to the already overstressed compiler. It comes back to the issue of Julia needing more mature, easy-to-work-with tooling, that could say "hey, this is technically allowed, but you probably didn't mean this".
And this is not an advantage to the designer exclusively, it is very much an advantage for the end user that the treatment is explicit, consistent and predictable, instead of 'magically' reinterpreting the meaning of literals based on guessing the intent of the user.
Basically, you seem to be saying that when passing x to BigFloat, x should not be treated as the value x, but as some nearby value that might be the one the caller intended (based on some rounding logic perhaps?) Or are you perhaps saying that
x = 1e-300
y = BigFloat(x)
y = BigFloat(1e-300)
I think it was a mistake. Invariably, a new user discovers the discrepancy and is thoroughly confused. Let's not repeat the same mistake with big nums.
But this is a language flamewar thing, probably not a constructive comment, sorry.
Pure Julia is faster than pure python, but there are non-pure python tools available in the python ecosystem for a ton of things.
[0] -- https://www.pola.rs/
Yes, I agree, on average polars is a bit faster for many of the simple workflows, but I certainly don't think that's unconditionally true. It's especially less true when you might want to do something out of the ordinary with your series --- in Julia it's trivial to just extract that as a vector and loop over it (fast!). In polars, one would have to make sure their function can be appropriately vectorized.
``` df = DataFrame(CSV(File("name.csv")))
data = JSON(File("name.json")) ```
instead of the usual hodge podge of methods ``` df = CSV.read("file.csv", DataFrame)
data = JSON.parsefile("file.json") ```
I opened it on mobile because I was interested in seeing how they differ, even though I’m not using either right now.
Not everything needs to cater to mobile.
Nothing has to cater to anything but I'm still allowed to be annoyed that I can't read it on mobile.
It happens what when I cannot stop thinking of a problem and really want to find a solution.
For instance, consider someone who has limited access to desktop computers and have to go by with a mobile device. These individuals do exist, and their access is as legitimate as any other.
It's virtually impossible to make a website that is well designed on both desktop and mobile. As long as the affordances of mouse+keyboard and touchscreen are as different as they are, one of the user groups needs will suffer a detrimental compromise.
After that, what you can do is ruin it, targetting a webpage for a specific screen size.
A phone screen just isn't big enough to display tables with more than two or three columns. There's no reason desktop users should need to be crippled in this fashion.
Tables are a very powerful tool for conveying lots of structured data in a way that's useful and intuitive. Avoiding them means losing out on this.
This is the exact problem the website we're discussing is having. You just can't show the relatively small amount of information side-by-side on mobile the way they are trying to. The only solution to it is to make it less intuitive and show them on top of each other in a complete jumble.
Julia's DataFrames library is more consistent than Pandas by a mile, but it's still a bit weird.
JuliaDB's IndexedTable and NDTable was a really awesome API design, it's quite a pity that JuliaDB is now unmaintained. :(
mydf = pd.DataFrame({'a' : [1, 2, 3]})
print(duckdb.query("SELECT SUM(a) FROM mydf").to_df())
> The SQL table name mydf is interpreted as the local Python variable mydf [...] Not only is this process painless, it is highly efficient.
It might be painless and convenient at first, but I feel like this could get you in trouble down the line. Is there a way to avoid this?