Comment by hardwaresofton | Hacker News Reader

Comment by hardwaresofton | Hacker News Reader

Parent Full thread

Am I the only one that just can't grok what Arrow is doing and how minio in particular made it faster? I read/skimmed this submission and I've worked out this much:

- Arrow is an efficient object format that helps skip serialization

- Spark works in memory

- Minio makes storage of data scalable and cheaper than hard disks

- (from the article) Spark jobs don't (didn't??) saturate the network

- (from the article) Spark jobs needed to write the same data to many buffers to get anywhere (exporting?)

- memory mapped IO is here to help (this seems like they're mixing memory mapped device IO

Is the gist that because arrow does not require much serialization/deserialization, memory mapped network devices can shoot it out to another server (w/ S3 as an intermediary) at network speed, skip the disk, and get used at the destination faster? I guess the other method was program-1 -> disk -> network -> disk -> program-2 and now it's program-1 -> s3 -> program-2 ?

You're not the only one. I've worked quite a bit with Minio, Arrow, and Spark, and I don't really understand the point the article is trying to make or how it's related to Minio. It's either badly explained or just a fluff piece throwing together a bunch of technologies.

To me, the article says "columnar formats that don't require deserialization are good, oh, and you can use Minio to store data!"

I was involved in Arrow project for a while. The goal of the project is make transfers easier and faster (zero-copy) between dev eco-systems (java->python, python->go) and across network (to database, s3, spark server etc).

And yes it has nothing to do with minio any more than it has with s3 or any cloud FS. There are easy to use writers and readers that also handle streaming.

Last time I checked,

> The most common usage is parquet handling.

> python to JAVA IPC was in place.

> The communication across systems uses protobuf and that is optimised for web-level transfers and not large scale data transfers yet. So transfers still have work to be done

There was also an in-memory object sharing module. I think its no longer part of the repo now.

Yeah so I didn't know about Arrow and I'm glad I looked it up (while trying to figure out what exactly was going on), it's a really cool technology. A lot of server time is spent encoding/decoding JSON -- but wonder if it could help that use-case? I know how to compare thrift/protobuf/cbor/etc but I don't really know if Apache can work for the general use case -- it seems like it deals in tables and sets.

Most of the time I don't necessarily want disparate languages in one system but it seems inevitable and knowing that Arrow exists seems like it will help me one day.

Reply on news.ycombinator.com