Hacker News Reader

This does use Apache arrow, that's how the data is transferred from spark to pandas and back.

Great, glad this was confirmed as I am about to solve a similar problem and my expectation is that Apache Arrow will be needed. So pyarrow = Apache Arrow.

Reply on news.ycombinator.com