Pandas on Ray – Make Pandas faster
rise.cs.berkeley.edu
rise.cs.berkeley.edu
import ray.dataframe as pd
https://github.com/ray-project/ray
Unlike Dask, Ray can communicate between processes without serializing and copying data. It uses a shared-memory object store within Apache Arrow:
http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-obj...
Worker processes (scheduled by Ray's computation graph) simply map the required memory region into their address space.
What the %$#@ is Ray?
I make a habit of doing this myself whenever I do a post like this. Sure, I was able to look up Ray from the Riselab and figure this out myself, but I wish I didn't have to.
From the Ray homepage:
Ray is a high-performance distributed execution framework targeted at large-scale machine learning and reinforcement learning applications. It achieves scalability and fault tolerance by abstracting the control state of the system in a global control store and keeping all other components stateless. It uses a shared-memory distributed object store to efficiently handle large data through shared memory, and it uses a bottom-up hierarchical scheduling architecture to achieve low-latency and high-throughput scheduling. It uses a lightweight API based on dynamic task graphs and actors to express a wide range of applications in a flexible manner.
Check out the following links!
Codebase: https://github.com/ray-project/ray Documentation: http://ray.readthedocs.io/en/latest/index.html Tutorial: https://github.com/ray-project/tutorial Blog: https://ray-project.github.io Mailing list: ray-dev@googlegroups.com
Pandas on Ray is an early stage DataFrame library that wraps Pandas and transparently distributes the data and computation. The user does not need to know how many cores their system or cluster has, nor do they need to specify how to distribute the data. In fact, users can continue using their previous Pandas notebooks while experiencing a considerable speedup from Pandas on Ray, even on a single machine. Only a modification of the import statement is needed, as we demonstrate below. Once you’ve changed your import statement, you’re ready to use Pandas on Ray just like you would Pandas.
In my imagination, this article is about some guy named Ray teaching Panda's how to run faster.
"biological data" is a bit vague, but for the data I know to be that big, sequence and array data, it does not naturally have the structure of a dataframe nor is pandas the tool of choice.
pandas.read_hdf has beaten out ray.dataframe.read_csv in terms of speed on the few files I've just initially tested now. But I imagine the programmable flexibility csvs have over hdfs (I've never used a Unix command to edit a hdf for example) is why this new approach could get some traction.
Doesn't Pandas target only structured (tabular) in-core datasets? (Unless you use something like Blaze / Dask / Ray on top.)
Does anyone really work with Pandas on "100's of terabytes of biological data"?
The fact that Dask also has high level collections which it knows how to parallelize is also interesting. For workloads which are more related to nd-arrays, matrices and scientific computing, my understanding is that is is more efficient than Spark.
The integration with your ecosystem is also important. If you have to ingest from the (Java) big data ecosystem for instance, Spark has had a lot of work put in its integration with it, it just works for the most part.
I don't think anything is inherent, more about priorities and momentum. For example, Spark devs have been working on cutting latency, and Conda Inc is/was contributing to the Arrow world. I had assumed the pygdf project would get to accelerating arrow dataframe compute before others, so this announce was a pleasant surprise!
pip install ray
¯\_(ツ)_/¯For node/data hackers: Our team is trying to bring the full & accelerated pydata world to JavaScript. We started with Arrow columnar data bindings (https://github.com/apache/arrow/tree/master/js). Next stop is Plasma bindings in node for zero-copy node<>pydata interop. That enables nearly-free calls from node web etc. apps to accelerated pandas-on-ray. If others are interested in contributing, let me/us know!
For example:
concurrency = 4 # Num of cores
pool = multiprocessing.Pool(processes=concurrency)
results = pool.map(fn, df.group(...)) # fn would be a callable for computing on a chunk.
pool.close()
return pd.concat(results)