Rust Dataframe: Update 1
github.com
github.com
The right way to design a data analysis DSL is to do it the way Python's dask does it: build an operation graph and execute it as the last step. The trouble is that Dask doesn't get it right either because, as part of its graph formation, it computes sizes of operands, and computing operand sizes can involve huge amounts of computation, and so dask, in effect, is also an "immediate mode" system.
What I really want to see is something lazy that can do sane query planning and that can work within limited system resources. Maybe one day I'll open source the work I've done in this space. Query languages are infinitely nicer for analytics than data processing libraries.
Fun story, about 18 months ago I was consulting for a telco. One of my tasks was to run a segmentation model on their subscriber base. They wanted me to use SAS in their cluster, but because I had 'brought my infra/laptop', I told them that I could do it with Python instead. I got the gist of it working on Pandas, but it was 'slow'. So they asked me if I was going to run it on a sample, but I said I'll port it to Spark in the evening.
The following morning I presented them with results, and the porting to Spark took about 2 hours (I hadn't used SparkML before), and the thing took an hour to run on the whole population.
Pandas does not also defer computations, which means computing Boolean functions that include the same data multiple times must make multiple passes over said data. Absurd.
Disclaimer: have never used numpy, have used python fairly extensively.
for x in np.nditer(a, op_flags = ['readwrite']):
x[...] = cos(sqrt(sin(pow(x, 2))))
Edit: just read about `Ellipsis`. I'm a fan, even if it's sort of nonstandard across libraries. Those readwrite flags are a travesty though, but at least you can paper over it with a helper function.
Something like:
def np_apply(a, f):
for x in np.nditer(a, op_flags = ['readwrite']):
x[...] = f(x)
np_apply(x, lambda x: return cos(sqrt(sin(pow(x, 2))))))
def np_apply(a, *argv):
for x in np.nditer(a, op_flags = ['readwrite']):
for f in argv:
x = f(x)
x[...] = x
np_apply(lambda x: return pow(x, 2), sin, sqrt, cos)
Either way see https://news.ycombinator.com/item?id=22786958 where I figured out the syntax to do it in place, which shouldn't result in any allocations at all and should solve both your concerns.
1. Memory allocation isn't free 2. Doing multiple loops over the buffer (once per operation) is going to be slower than doing all the operations at once, both because of caching and the opportunity to do the operations on a register and write at the end (though who knows if the python interpreter will do that).
Now I have some more reading to do. Thanks.
It is an ongoing effort to built an intermediary representation that numerical library could share to produce an optimized instructions tree.
1. Without CTEs SQL isn't that expressive and with CTEs queries can be hard to understand. Logical query plans on the other hand are rather easy to understand and can express more queries than SQL can.
2. SQL is hard to modify programmatically. To most software, SQL is simply a string without any inherent meaning. A logical query plan on the hand is a tree structure that can be introspected and modified.
That's basically the same thing you want, except that you want it implemented on dataframes while I want it implemented for database systems.
Eigen does this via expression templates; it essentially optimises the computation graph at compile time.
That's wrong too. The right query execution plan can depend heavily on the sizes and statistical distributions of the operands. Eigen doesn't have enough information at compile time to make good decisions in all cases. This kind of computation really calls for a JIT.
- Don't put methods on it like pandas.DataFrame. You won't get the API right the first many times and you'll end up with a million methods.
- Make chaining easy.
- Use pure functions. No mutation.
- Get inspired by R's dplyr and data.table.
I was actually thinking about this for a bit. Since the data layout seems unlikely to ever backwards-incompatibly change, can you not define the API entirely in traits on the struct, and package them into one super trait (e.g. DataFrameAPIVersion1)?
Then you can simply import the DataFrame struct as well as the version of the API you'll use, and be 100% compatible with any other code regardless of API version they use to manipulate the dataframes.
One of the common pure alternatives is to make them lazy - i.e., all those pure methods don't do anything but rather just collect the information on what to do with your data. But then you need to write an execution engine for arbitrary computation graph, which ain't easy.
This isn't necessarily true due to Rust's strong ownership. Methods can take `self` (without reference) as an argument which means the method takes ownership of itself. There is no copy for 'double' in the below example, yet it is pure:
#[derive(Clone)]
struct A {
v: Vec<i32>,
}
impl A {
fn double(mut self) -> Self {
for x in &mut self.v {
*x *= 2;
}
self
}
}
fn main() {
let a = A { v: vec![2, 3, 4] };
let b = a.double();
dbg!(b.v[1]);
}
I'll write an experiment on this in the coming weeks/months, to see if my assumption would work.
frame.foo_columns().bar_rows().baz()
1. Import data using hand-written SQL from SQL server (read_sql)
2. Perform various filtering, aggregation, math (loc, iterrows/itertuples, groupby, agg)
3. Push results either out to delivery file or back to source server (to_csv, to_excel, to_sql).
Really bread and butter stuff, but the relative ease and stability of using pandas to do it is the attraction.
- two signals X and Y should always be within 5% of each other while condition Z is true.
- signal A should transition from value M to N within time window t1-t2
Would love something in rust to match the rest of the infrastructure. Python also has plenty of warts with regards to packaging, cross compilation and deployment
1) Speed and memory efficiency, I've often worked in python/pandas and been frustrated by its performance. If I have a job that takes 12 hours to churn through all of the data, flipping over to rust may accelerate my iteration time to ~7.2-72 minutes.
2) Straightforward FFI bindings into other languages. Like C you can trivially bind rust code into python. Language choice can then be a bit of a two-way door back to python.
In most of my projects I'm unable to avoid using a udf/python data manipulation at some point.
http://www.oilshell.org/blog/2018/11/30.html
(this is my article, but it will also show up if you Google the question :) )
Although, one might run into a "true level" situation. This is when Morty feels what true level is like, then says "everything is crooked, reality is poison" when he has to live in the "fake level" world.
As a side note, a small usage example in the readme would be good.
We can and frequently do. Rust even seems to have one of the better packaging stories around building C++ libs, albeit with several heavyweight dependencies (C++ compiler + toolchain, libc, etc.). Granted, calling into it from Rust means we need a C ABI, but that's rarely a terrible problem.
> What novelty would a Rust version of the same thing bring really?
I find Rust easier to cross compile for starters. I still haven't tamed emscripten for compiling C++ as WASM - the Rust toolchains by comparison make compiling Rust to WASM trivial. Rust's larger stdlib, standard package manager, and healthy library ecosystem means whatever I'm cross compiling is less likely to need to resort to platform specific nonsense in need of manual porting.
But the main draw of Rust to me is the ability to write memory safe code and race-safe native code, reducing how many heisenbugs I encounter and the portion of the codebase where they might lurk. If your codebase is large, and mostly memory/race safe, this is a huge advantage. If your codebase is large, and only 50% memory/race safe, you're still frequently in debugging hell inside a large haystack.
If you take a super solid C++ dependency, maybe you kinda retain that advantage, but most C++ dependencies I take on don't meet that high bar (including those me and my coworkers write!) Alternatively, I could take on a relatively mediocre Rust dependency, and I'll probably still retain that memory/race-safety advantage. There are exceptions to this of course thanks to Rust's many opt-in escape hatches - but I also find it much easier to audit Rust code for their use and the possible problems they may cause, versus C++ where every array access is suspect.
That said there are plenty of reasons why they shouldn't just import c++ libraries, one of them being that c++ is not the same language as rust.
They do do a pretty damn good job of giving people this impression.
This mindset may be appeased by demonstrating why you don't believe your actions are actually a waste of effort.
If you wait long enough, any stable working solutions will be reinvented in its own Language.
It's completely unproductive. But there is reason to that: compatibility with the language toolchain and understandability.
If someone hadn't implemented the libpq protocol in Rust, I likely wouldn't have written in binary copy support on the dataframe in one afternoon.
This reduces the barrier for people to productively use their favourite languages. It might be unproductive at first, but we see the benefits as time goes on.
It is vastly unproductive to reproduce anything useful in 10 different language just because our cross-language tooling awfuly suxxe.
The lack of cross-language dev-oriented lead us in the ridiculous situation where every language has its own package manager and his completely unable to use a package made for an other language.
There is reasons to that... but much more political than technical and that's the sad state we are in.
Let me give you an example but this time with Groovy. Groovy runs on the JVM and that means it benefits from the existing JVM ecosystem, right? Well, of course you get the benefit of existing libraries but there was a crucial difference between using them through Groovy/Grails than natively through Java. There were lots of grails plugins that were very thin layers above the Java libraries yet somehow the usability increased a hundredfold. A small plugin with maybe 1000 lines of code ended up providing more value to our business than the part of the library that was written in Java.