FireDucks: Pandas but Faster
hwisnu.bearblog.dev
hwisnu.bearblog.dev
> By providing the beta version of FireDucks free of charge and enabling data scientists to actually use it, NEC will work to improve its functionality while verifying its effectiveness, with the aim of commercializing it within FY2024.
In other words, it's free only to trap you.
Basically it's a debate about how many dark patterns can you squeeze next to that "upfront language" before "marketing" slides into "bait-n-switch."
I nearly made the mistake of merging Akka into a codebase recently; fortunately I double-checked the license and noticed it was the bullshit BUSL and it would have potentially cost my employer tens of thousands of dollars a year [1]. I ended up switching everything to Vert.x, but I really hate how normalized these ostensibly open source projects are sneaking scary expensive licenses into things now.
[1] Yes I'm aware of Pekko now, and my stuff probably would have worked with Pekko, but I didn't really want to deal with something that by design is 3 years out of date.
Vert.x and other frameworks are far better and easier for most devs to grok.
Plus the license isn't stupid.
I would imaging the non-Scala use case to be less than ideal.
In Scala land, Pekko - the open source fork of Akka is the way to go if you need compatibility. Personally, I'd avoid new versions of Akka like the plague, and just use more modern alternatives to Pekko/Akka anyway.
I'm not sure what Lightbend's target market is? Maybe they think they have enough critical mass to merit the price tag for companies like Sony/Netflix/Lyft, etc. But they've burnt their bridge right into the water with everyone else, so I see them fading into irrelevance over the next few years.
I'm sure that Lightbend feels that their support contract is the bee's knees and worth whatever they charge for it, but it's a complete non-starter for me, and so I look elsewhere.
Vert.x actor-ish model is a bit different, but it's not the that different, and considering that Vert.x tends to perform extremely well in benchmarks, it doesn't really feel like I'm losing a lot by using it instead of Akka, particularly since I'm not using Akka Streams.
[1] Normal disclaimer: I don't hide my employment history, and it's not hard to find, but I politely ask that you do not post it here.
I didn't know the licence and had a look, but I don't see what is bullshit with it. It's not a classical open source licence, but pretty close and much better than closed source
> and it would have potentially cost my employer tens of thousands of dollars a year
If your employer is not providing its software open source, there is nothing shocking to have to pay for the software used
I just think it's a proprietary license that is trying to LARP as an OSS license. It sneaks in language that makes it so it's unclear how much it will actually cost you to use it. It makes me terrified to import anything touching it because I don't want to risk accidentally costing my employer millions of dollars.
I don't really see how it's "pretty close" to an OSS license. Part of an OSS license is that I can use the code for whatever I want, which is decidedly not the case with BUSL. I do appreciate that stuff eventually becomes Apache, so I guess that's better than nothing, but I'd rather just avoid the stuff entirely, or only use the Apache licensed stuff.
I also don't really like the idea that I could contribute to Akka, have my contributions being monetized by Lightbend, but I'm not even allowed to use my own contributions without paying them a fee. I know that CLAs aren't exactly new in the OSS world, but at least if I were to make a contribution to Ubuntu, I'm still allowed to run Ubuntu server for free, with my contributions included.
I guess the license just kind of feels "Bait and Switch" to me. It tries to get you to think that it's OSS and then smacks you with a "JK IT'S PROPRIETARY".
> If your employer is not providing its software open source, there is nothing shocking to have to pay for the software used
Sure, except in the case of Akka there's enough competition in the Java library world that I don't think that it's worth it. Vert.x is comparable, and the license is less likely to accidentally cost me lots of money.
I mostly think that Akka's licensing is way too expensive too, again especially when you consider that there's a good chunk of concurrency libraries in Java-land that have more business-friendly licenses.
We did a long podcast and a couple blogs that offered transparency to the rationale on why we moved from Apache to BSL, which still downgrades to Apache after 36 months. See Emily Omier for the specifics.
It came down to survival. The company faced a bankruptcy event as customers were using the software without contributions and after exhausting alternatives needed to change the license model to create a more sustainable approach.
The consequence of this choice was that there was less adoption from OSS and ISVs who need a flexible licensing model for embedding and redistribution. It also encouraged the Pekko fork which is a branch that is 2.5 years old. And that branch helped older projects and OSS distributions to maintain their position without financial consequences.
It is not cheap to maintain Akka, and after 15 years we have turned a profit, albeit barely. We are growing, finally, and have a prosperous future and most of our spend goes into development. It did allow us to create Akka 3, which is a simpler model for devs within enterprises mixed with a consumption based model that should be significantly cheaper than the traditional libraries, and cheaper than the cost to adopt most any other framework. We can debate the merits of different business models but we couldn't have maintained the 50 CVE fixes and create a modern version of Akka if we hadn't taken this step.
We need a better strategy on how to appeal to the OSS community once more. To appeal to startups and academics, we have free commercial licenses and subscriptions, which nearly 200 accounts have signed up in the last 18 months.
Surely it is also not cheap to maintain Spring Framework either, no?
We are a pure play app dev platform and that gets to the heart of why the business model is different. I'd argue that we are very motivated to make sure that customers are successful with app dev as that is our bottom line where our rivals are financially incentives by infrastructure sales, not app dev outcomes.
That said, and I realize that this is crass but it's also honest: Akka's profitability isn't my problem. When I am looking to import a library for my job, I try my best to weigh pros and cons of each (as we all do), and when I see a BUSL that's an immediate red flag; if Akka were the only cool concurrency library in the JVM world then I'd just put up with it, but when there are viable alternatives like Vert.x it's extremely hard to go to my employer and ask them to spend $5000/month + $0.15/Akka-hour [1], especially since we run thousands of individual JVMs, and running a comparable thing in Vert.x cost us nothing (albeit with having to do tech support ourselves). Whether or not it's "fair" that Vert.x is a pet project from Red Hat or VMWare and therefore doesn't have to worry about financing is sort of orthogonal to whether or not I choose it or Akka.
This isn't meant to shit on Akka, it's very cool software, I'm just frustrated by the BUSL because it gives the illusion of an OSS license, the initial marketing around it looked like an OSS license, and I wasted about 15 hours writing some Akka code only to realize that I had to throw it away because there was no chance I was going to get my employer to approve a PR with BUSL-licensed libraries that would have cost us hundreds of thousands of dollars a year.
Again, apologies that this is rude, and if Akka/Lightbend/Typesafe is making a profit then of course all the best to you, but this is just my rationale.
ETA:
Re-reading this, I apologize for how hostile I come off. You're not trying to sell me, you're just giving justification, which is fine even if I'm not a huge fan of the license.
In fact, what's stopping the pandas library from incorporating fireducks code into the mainline branch? pandas itself is BSD.
(.venv) [fireducks] ritchie46 /home/ritchie46/Downloads/deleteme/polars-tpch[SIGINT] $ SCALE_FACTOR=10.0 make run-polars
.venv/bin/python -m queries.polars
{"scale_factor":10.0,"paths":{"answers":"data/answers","tables":"data/tables","timings":"output/run","timings_filename":"timings.csv","plots":"output/plot"},"plot":{"show":false,"n_queries":7,"y_limit":null},"run":{"io_type":"parquet","log_timings":false,"show_results":false,"check_results":false,"polars_show_plan":false,"polars_eager":false,"polars_streaming":false,"modin_memory":8000000000,"spark_driver_memory":"2g","spark_executor_memory":"1g","spark_log_level":"ERROR","include_io":true},"dataset_base_dir":"data/tables/scale-10.0"}
Code block 'Run polars query 1' took: 1.47103 s
Code block 'Run polars query 2' took: 0.09870 s
Code block 'Run polars query 3' took: 0.53556 s
Code block 'Run polars query 4' took: 0.38394 s
Code block 'Run polars query 5' took: 0.69058 s
Code block 'Run polars query 6' took: 0.25951 s
Code block 'Run polars query 7' took: 0.79158 s
Code block 'Run polars query 8' took: 0.82241 s
Code block 'Run polars query 9' took: 1.67873 s
Code block 'Run polars query 10' took: 0.74836 s
Code block 'Run polars query 11' took: 0.18197 s
Code block 'Run polars query 12' took: 0.63084 s
Code block 'Run polars query 13' took: 1.26718 s
Code block 'Run polars query 14' took: 0.94258 s
Code block 'Run polars query 15' took: 0.97508 s
Code block 'Run polars query 16' took: 0.25226 s
Code block 'Run polars query 17' took: 2.21445 s
Code block 'Run polars query 18' took: 3.67558 s
Code block 'Run polars query 19' took: 1.77616 s
Code block 'Run polars query 20' took: 1.96116 s
Code block 'Run polars query 21' took: 6.76098 s
Code block 'Run polars query 22' took: 0.32596 s
Code block 'Overall execution of ALL polars queries' took: 34.74840 s
(.venv) [fireducks] ritchie46 /home/ritchie46/Downloads/deleteme/polars-tpch$ SCALE_FACTOR=10.0 make run-fireducks
.venv/bin/python -m queries.fireducks
{"scale_factor":10.0,"paths":{"answers":"data/answers","tables":"data/tables","timings":"output/run","timings_filename":"timings.csv","plots":"output/plot"},"plot":{"show":false,"n_queries":7,"y_limit":null},"run":{"io_type":"parquet","log_timings":false,"show_results":false,"check_results":false,"polars_show_plan":false,"polars_eager":false,"polars_streaming":false,"modin_memory":8000000000,"spark_driver_memory":"2g","spark_executor_memory":"1g","spark_log_level":"ERROR","include_io":true},"dataset_base_dir":"data/tables/scale-10.0"}
Code block 'Run fireducks query 1' took: 5.35801 s
Code block 'Run fireducks query 2' took: 8.51291 s
Code block 'Run fireducks query 3' took: 7.04319 s
Code block 'Run fireducks query 4' took: 19.60374 s
Code block 'Run fireducks query 5' took: 28.53868 s
Code block 'Run fireducks query 6' took: 4.86551 s
Code block 'Run fireducks query 7' took: 28.03717 s
Code block 'Run fireducks query 8' took: 52.17197 s
Code block 'Run fireducks query 9' took: 58.59863 s
terminate called after throwing an instance of 'std::length_error'
what(): vector::_M_default_append
Code block 'Overall execution of ALL fireducks queries' took: 249.06256 s
Traceback (most recent call last):
File "/home/ritchie46/miniconda3/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ritchie46/miniconda3/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/ritchie46/Downloads/deleteme/polars-tpch/queries/fireducks/__main__.py", line 39, in <module>
execute_all("fireducks")
File "/home/ritchie46/Downloads/deleteme/polars-tpch/queries/fireducks/__main__.py", line 22, in execute_all
run(
File "/home/ritchie46/miniconda3/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/ritchie46/Downloads/deleteme/polars-tpch/.venv/bin/python', '-m', 'fireducks.imhook', 'queries/fireducks/q10.py']' died with <Signals.SIGABRT: 6>.
make: \*\* [Makefile:52: run-fireducks] Error 1
(.venv) [fireducks] ritchie46 /home/ritchie46/Downloads/deleteme/polars-tpch[2] $So many foot guns, poorly thought through functions, 10s of keyword arguments instead of good abstractions, 1d and 2d structures being totally different objects (and no higher-order structures). I'd take 50% of the speed for a better API.
I looked at Polars, which looks neat, but seems made for a different purpose (data pipelines rather than building models semi-interactively).
To be clear, this library might be great, it's just a shame for me that there seems no effort to make a Pandas-like thing with better API. Maybe time to roll up my sleeves...
You can do the same with Polars, but you have to start messing about with datetimes and convert the simple problem "I want to calculate a monthly sum anchored on the last business day of the month" to SQL-like operations.
Pandas grew a large and obtuse API because it provides specialized functions for 99% of the tasks one needs to do on timeseries. If I want to calculate an exponential weighted covariance between two time series, I can trivially do this with pandas: series1.ewm(...).cov(series2). I welcome people to try and do this with Polars. It'll be a horrible and barely readable contraption.
YC is mostly populated by technologists, and technologists are often completely ignorant about what makes pandas useful and popular. It was built by quants/scientists, for doing (interactive) research. In this respect it is similar to R, which is not a language well liked by technologists, but it is (surprise) deeply loved by many scientists.
I've had trouble determining whether one timestamp falls between two others across tens of thousands of rows (with the polars team suggesting I use a massive cross product and filter -- which worked but excludes the memory requirement), whereas in pandas I was able to sort the timestamps and thereby only need to compare against the preceding / following few based on the index of the last match.
The other issue I've had with resampling is with polars automatically dropping time periods with zero events, giving me a null instead of zero for the count of events in certain time periods (which then gets dropped from aggregations). This has caught me a few times.
But other than that I've had good luck.
> my_df.resample("BME").apply(...)
Done. I don't think it gets any easier than this. Every time I tried something similar with polars, I got bogged down in calendar treatment hell and large and obscure SQL like contraptions.
Edit: original tone was unintentionally combative - apologies.
Reviewing my work, only needed an hourly aggregation, which was similarly easy in polars and pandas (I misspoke about being easier) -- what I found easier was grouping by time data that wasn't amenable to `resample`.
In polars I had no problems using a regular group_by with a pl.col.dt object, whereas in pandas I remember struggling to do so, even though it seemed straightforward.
Sorry, I wish I could remember more details; this was probably 5 years ago that I was writing the pandas code and just converted it to polars about a year ago, so it's possible that I just got better at python in the meantime (though I was writing much more python back then). And of course a rewrite is likely to feel easier the second time.
The other confounding issue is that the eager pandas code crashed with OOM regularly and took several minutes to run, whereas polars handles it very well (which I'm sure to some degree is it optimizing things that I could have done manually), but this made iterating on this codebase feel much less onerous.
`.join_where()`[1] was also added recently.
[1]: https://docs.pola.rs/api/python/stable/reference/dataframe/a...
But I'm guessing it's something like this:
import pandas as pd
def calculate_monthly_business_sum(df, date_column, value_column):
"""
Calculate monthly sums anchored to the last business day of each month
Parameters:
df: DataFrame with dates and values
date_column: name of date column
value_column: name of value column to sum
Returns:
DataFrame with sums anchored to last business day
"""
# Ensure date column is datetime
df[date_column] = pd.to_datetime(df[date_column])
# Group by end of business month and sum
monthly_sum = df.groupby(pd.Grouper(
key=date_column,
freq='BME' # Business Month End frequency
))[value_column].sum().reset_index()
return monthly_sum
df = pd.DataFrame({ 'date': ['2024-01-01', '2024-01-31', '2024-02-29'], 'amount': [100, 200, 300] })
result = calculate_monthly_business_sum(df, 'date', 'amount')
print(result)
Which you can run here => https://python-fiddle.com/examples/pandas?checkpoint=1732114...
df.resample("BME").sum()
Done. One line of code and it is quite obvious what it is doing - with perhaps the small exception of BME, but if you want max readability you could do:
df.resample(pd.offsets.BusinessMonthEnd()).sum()
This is why people use pandas.
> df.resample("BME").sum()
Assuming `df` is a dataframe (ie table) indexed by a timestamp index, which is usual for timeseries analysis.
"BME" stands for BusinessMonthEnd, which you can type out if you want the code to be easier to read by someone not familiar with pandas.
It so easy for my analyst team because of daily uses but my developers probavly will never thought/know BME and decided to implement the code again.
Considering switching from pandas and want to understand what is my best bet. I am just processing feature vectors that are too large for memory, and need an initial simple JOIN to aggregate them.
I previously had a pandas+sklearn transformation stack that would take up to 8 hours. Converted it to ibis and it executes in about 4 minutes now and doesn't fill up RAM.
It's not a perfect apples to apples pandas replacement but really a nice layer on top of sql. after learning it, I'm almost as fast as I was on pandas with expressions.
My conclusion was that pandas is not for developers. But for one-offs by managers, data-scientists, scientists, and so on. And maybe for "hackers" who cludge together stuff 'till it works and then hopefully never touch it.
Which made me realize such thoughts can come over as smug, patronizing or belittling. But they do show how software can be optimized for different use-cases.
The danger then lies into not recognizing these use-cases when you pull in smth like pandas. "Maybe using panda's to map and reduce the CSVs that our users upload to insert batches isn't a good idea at all".
This is often worsened by the tools/platforms/lib devs or communities not advertising these sweet spots and limitations. Not in the case of Pandas though: that's really clear about this not being a lib or framework for devs, but a tool(kit) to do data analysis with. Kudo's for that.
It doesn't work for me so it can't work for anyone?
Writing pandas code is a bit redundant. So what?
Who is to say that fireducks won't make their own API?
I agree that pandas does not have the best designed api in comparison to say dplyr but it also has a lot of functionality like pivot, melt, unstack that are often not implemented by other libraries. It’s also existed for more than a decade at this point so there’s a plethora of resources and stackoverflow questions.
On top of that, these days I just use ChatGPT to generate some of my pandas tasks. ChatGPT and other coding assistants know pandas really well so it’s super easy.
But I think if you get to know Pandas after a while you just learn all the weird quirks but gain huge benefits from all the things it can do and all the other libraries you can use with it.
I 100% agree that pandas addresses all the pain points of data analysis in the wild, and this is precisely why it is so popular. My point is, it doesn't address them well. It seems like a conglomerate of special cases, written for a specific problem it's author was facing, with little concern for consistency, generality or other use cases that might arise.
In my usage, any time saved by its (very useful) methods tends to be lost on fixing subtle bugs introduced by strange pandas behaviours.
In my use cases, I reindex the data using pandas and get it to numpy arrays as soon as I can, and work with those, with a small library of utilities I wrote over the years. I'd gladly use a "sane pandas" instead.
I get it doesn't follow best practices, but it does do what it needs to. Speed has been an issue, and it's exciting seeing that problem being solved.
Interesting to see so many people recently saying "polars looks great, but no way I'll rewrite". This library seems to give a lot of people, myself included, exactly what we want. I look forward to trying it.
I find I basically never write myself into a corner with initially expedient but ultimately awkward data structures like I often did with pandas, the expression API makes the semantics a lot clearer, and I don't have to "guess" the API nearly as much.
So even for this usecase, I would recommend trying out polars for anyone reading this and seeing how it feels after the initial learning phase is over.
https://hexdocs.pm/explorer/exploring_explorer.html
It runs on top of Polars so you get those speed gains, but uses the Elixir programming language. This gives the benefit of a simple finctional syntax w/ pipelines & whatnot.
It also benefits from the excellent Livebook (a Jupyter alternative specific to Elixir) ecosystem, which provides all kinds of benefits.
However, I still find myself using pandas for the timestamps, timedeltas, and date offsets, and even still, I need a whole extra column just to hold time zones, since polars maps everything to UTC storage zone, you lose the origin / local TZ which screws up heterogeneous time zone datasets. (And I learned you really need to enforce careful manual thoughtful consideration of time zone replacement vs offsetting at the API level)
Had to write a ton of code to deal with this, I wish polars had explicit separation of local vs storage zones on the Datetime data type
IMO Polars sets a different goal of what's the most pandas like thing that we can build that is fast (and leaves open the possibility for more optimization), and clean.
Polars feels like you are obviously manipulating an advanced query engine. Pandas feels like manipulating this squishy datastructure that should be super useful and friendly, but sometimes it does something dumb and slow
For a comparison, dplyr offers a lot of elegant functionality, and the functional approach in Pandas often feels like an afterthought. If R is cleaner than Python, it tells a lot (as a side note: the same story for ggplot2 and matplotlib).
Another surprise for friends coming from non-Python backgrounds is the lack of column-level type enforcement. You write df.loc[:, "col1"] and hope it works, with all checks happening at runtime. It would be amazing if Pandas integrated something like Pydantic out of the box.
I still remember when Pandas first came out—it was fantastic to have a tool that replaced hand-rolled data structures using NumPy arrays and column metadata. But that was quite a while ago, and the ecosystem has evolved rapidly since then, including Python’s gradual shift toward type checking.
That's because it's a bad way to use Pandas, even though it is the most popular and often times recommended way. But the thing is, you can just write "safe" immutable Pandas code with method chaining and lambda expressions, resulting in very Polars-like code. For example:
df = (
pd
.read_csv("./file.csv")
.rename(columns={"value":"x"})
.assign(y=lambda d: d["x"] * 2)
.loc[lambda d: d["y"] > 0.5]
)
Column-level name checking would be awesome, but unfortunately no python library supports that, and it will likely never be possible unless some big changes are made in the Python type hint system.
You are probably thinking about `df.apply(lambda row: ..., axis=1)` which operates on each row at a time and is indeed very slow since it's not vectorized. Here this is different and vectorized.
.loc[lambda d: d["y"] > 0.5]
[df.y > 0.5]
For the rest of your comment: it's the best you can do in python. Sure you could write SQL, but then you're mixing text queries with python data manipulation and I would dread that. And SQL-only scripting is really out of question.
Big problem with pandas is that you still have to load the dataframe into memory to work with it. My data's too big for that and postgres makes that problem go away almost entirely.
(I'm the first to complain about the many warts in Pandas. Have written multiple books about it. This is annoying, but it is much better than [df.y > 0.5].)
But what are you saying, that typing wrong things you might get wrong results? Yes, coding is like that.
What's your point, make a point.
Here's my alternative: https://github.com/otsaloma/dataiter https://dataiter.readthedocs.io/en/latest/_static/comparison...
Planning to switch to NumPy 2.0 strings soon. Other than that I feel all the basic operations are fine and solid.
Note for anyone else rolling up their sleeves: You can get quite far with pure Python when building on top of NumPy (or maybe Arrow). The only thing I found needing more performance was group-by-aggregate, where Numba seems to work OK, although a bit difficult as a dependency.
Yeah, Pandas has that early PHP feel to it, probably out of being a successful first mover.
I have not yet used siuba, but would be interested in others' opinions. The activation energy to learn a new set of tools is so large that I rarely have the time to fully examine this space...
A pity their compares don’t have tidyverse or R’s data.table. I think R would look simpler but now it remains unclear.
Also, having (already a while ago) looked at the implementation of the magic `_` object, it seemed like an awful hack that will serve only a part of use cases. Maybe someone can correct me if I'm wrong, but I get the impression you can do e.g. `summarize(x=_.x.mean())` but not `summarize(x=median(_.x))`. I'm guessing you don't get autocompletion in your editor or useful error messages and it can then get painful using this kind of a magic.
Polars rocked my world by having a sane API, not by being fast. I can see the value in this approach if, like the author, you have a large amount of pandas code you don't want to rewrite, but personally I'm extremely glad to be leaving the pandas API behind.
Is there anything specific you prefer moving from the pandas API to polars?
Say you want to take an aggergation like "the mean of all values over the 75th percentile" algonside a few other aggregations. In pandas, this means you're gonna be in for a bunch of hoops and messing around with stuff because you can't express it via the api. Polars' api lets you express this directly without having to implement any kind of workaround.
Nice article on it here: https://labs.quansight.org/blog/dataframe-group-by
Is this shitware? It seems to be very high quality code
How would you define "quality" in this context?
One could also say that quality is related to the functional output.
Right, I said nothing that contradicts that ("High quality code isn't just code that performs well when executed, but also ..."). High quality functional output is a necessary requirement, but it isn't sufficient to determine if code is high quality.
My point was that it's all subjective in the end.
Imagine writing a very good program, running it through an obfuscator, and throwing away the original code. Is the obfuscated code "high quality code" now, because the output of the compilation still works as before?
Do you mean how well it was written, or do you mean how well it performs? Or do both matter? Equally, or one more/less than the other?
It probably depends on whether you're the developer taking over the codebase, or the customer running the code in production..
Take video games.. A lot of it is messy spaghetti C++ code, not modular or well structured, full of hacks and manual optimizations, to give the best possible performance on available hardware.
It might be impossible to parse or maintain, but it does the job about as well as possible, which is really all that matters to the end user. I would call that high quality code.
So again, subjective...
GitHub always been a platform for "We love to host FOSS but we won't be 100% FOSS ourselves", so makes sense they allow that kind of usage for others too.
I think what you want, is something like Codeberg instead, which is explicitly for FOSS and 100% FOSS themselves.
I don't understand what it means. It looks like a contradiction. Does it have a BSD-3 licence or not?
> While the wheel packages are available at https://pypi.org/project/fireducks/#files, and while they do contain Python files, most of the magic happens inside a (BSD-3-licensed) shared object library, for which source code is not provided.
Edit: To use, redistribute, and modify, and distribute modified versions.
Imagine being like "the project is GPL - just the compiled machine code".
https://fireducks-dev.github.io/files/20241003_PyConZA.pdf
The main reasons are
* multithreading
* rewriting base pandas functions like dropna in c++
* in-built compiler to remove unused code
Pretty impressive especially given you import fireducks.pandas as pd instead of import pandas as pd, and you are good to go
However I think if you are using a pandas function that wasn't rewritten, you might not see the speedups
They are showing a 20-30% improvement over Polars, Clickhouse and Duckdb. But those 3 tools are SOTA in this area and generally rank near eachother in every benchmark.
So 20-30% improvement over that cluster makes me interested to know what techniques they are using to achieve that over their peers.
> Future Plans By providing the beta version of FireDucks free of charge and enabling data scientists to actually use it, NEC will work to improve its functionality while verifying its effectiveness, with the aim of commercializing it within FY2024.
Its freeware under an open source license. Really misleading.
It looks like something you should stay away from unless you need it REALLY badly. Its a proprietary product with unknown pricing and no indication of what their plans are.
Does the fact that the binary is BSD licensed allow reverse-engineering?
Reversing and re-compiling should count as modification?
I've had the chance to play with it on some of my code it queries than ran in 8+ minutes come down to 20 seconds.
Re-writing in Polars involves more code changes.
However, with Pandas 2.2+ and arrow, you can use .pipe to move data to Polars, run the slow computation there, and then zero copy back to Pandas. Like so...
(df
# slow part
.groupby(...)
.agg(...)
)
def polars_agg(df):
return (pl.from_pandas(df)
.group_by(...)
.agg(...)
.to_pandas()
)
(df
.pipe(polars_agg)
)Where can I find the code? I don't see it on GitHub.
> contact@fireducks.jp.nec.com
So it's from NEC (a major Japanese computer company), presumably a research artifact?
> https://fireducks-dev.github.io/docs/about-us/ Looks like so.
Is it actually? Do people see that level of compatibility in practice?
It should be pretty close, though.
I wonder how much of this is fundamental to the common approach of writing libraries in Python with the processing-heavy parts delegated to C/C++ -- that the expressive parts cannot be fast and the fast parts cannot be expressive. Also, whether Rust (for polars, and other newer generation of libraries) changes this tradeoff substantially enough.
This really does seem like a rare thing that everything speeds up without breaking compatability. If you want a fast revised API for your new project (or to rework your existing one) then you have a solution for that with Polars. If you just want your existing code/workloads to work faster, you have a solution for that now.
It's OK to have a slow, compatible, static codebase to build things on then optimize as-needed.
Trying to "fix" the api would break a ton of existing code, including existing plugins. Orphaning those projects and codebases would be the wrong move, those things take a decade to flesh out.
This really doesn't seem like the worst outcome, and doesn't seem to be creating a huge fragmented mess.
Don't come to old web-devs with those complains, every single one of them had to write at least one open source javascript library just to create their linkedin account!
Q: Why do ducks have big flat feet?
A: So they can stomp out forest fires.
Q: Why do elephants have big flat feet?
A: So they can stomp out flaming ducks.
My easy guess is that compared to pandas, it's multi-threaded by default, which makes for an easy perf win. But even then, 130-200x feels extreme for a simple sum/mean benchmark. I see they are also doing lazy evaluation and some MLIR/LLVM based JIT work, which is probably enough to get an edge over polars; though its wins over DuckDB _and_ Clickhouse are also surprising out of nowhere.
Also, I thought one of the reasons for Polars's API was that Pandas API is way harder to retrofit lazy evaluation to, so I'm curious how they did that.
We found `numpy` and `jax` to be a good trade-off between "too high level to optimize" and "too low level to understand". Therefore in our hedge fund we just build data structures and helper functions on top of them. The downside of the above combination is on sparse data, for which we call wrapped c++/rust code in python.
I wrote a nice article about chaining for Ponder. (Sadly, it looks like the Snowflake acquisition has removed that. My book, Effective Pandas 2, goes deep into my best practices.)
(Disclaimer: I'm a corporate trainer and feed my family teaching folks how to work with their data using Pandas.)
When I teach about "readable" code, I caveat that it should be "readable for a specific audience". I hold that if you are a professional, that audience is other professionals. You should write code for professionals and not for newbies. Newbies should be trained up to write professional code. YMMV, but that is my bias based on experience seeing this work at some of the biggest companies in the world.
EDIT: I've found some benchmarks https://fireducks-dev.github.io/docs/benchmarks/
Would be nice to know what are internals of FireDucks
```
>>> df['year'].dtype == np.dtype('int32')
True
```
While for most of my jobs I ended up being able to evade the use of HPC by simply being smarter and discovering better algorithms to process information, I recall like pyspark decently, but preferring the simplicity of ballista over pyspark due to the simpler installation of Rust over managing Java and JVM junk. The constant problems caused by anything using JVM backend and the environment config with it was terrible to add to a new system every time I ran a new program.
In this regard, ballista is a enormous improvement. Anything that is a one-line install via pip on any new system, runs local-first without any cloud or telemetry, and requires no change in code to run on a laptop vs HPC is the only option worth even beginning to look into and use.
Unless I had thousands of files to work with, I would be loathe to use cluster computing. There's so much overhead, cost, waiting for nodes to spin up, and cloud architecture nonsense.
My "single node" computer is a refurbished tower server with 256GB RAM and 50 threads.
Most of these distributed computing solutions arose before data processing tools started taking multi-threading seriously.
an interesting angle on 50 threads and 256G: your data is probably pretty cool (cache-friendly). if your threads are merely HT, that's only 25 real cores, and might be only a single socket. implying probably <100 GB/s memory bandwidth. so a best-case touch-all-memory operation would take several seconds. for non-sequential patterns, effective rates would be much lower, and keep cores even less busy.
so cache-friendliness is really the determining feature in this context. I wonder how much these packages are oriented towards cache tuning. it affects basic strategy, such as how filtering is implemented in an expression graph...
The promise of a 100x speedup with 0 changes to your codebase is pretty huge, but even a few correctness / incompatibility issues would probably make it a no-go for a bunch of potential users.
I haven’t seen that in other system like Polars, but maybe I’m wrong.
edit: I know pandas uses numpy under the hood, but "raw" numpy is typically faster (and more flexible), so curious as to why it's not mentioned
At some point I think it's more honest to say "the python ecosystem keeps getting more awesome".