Hacker News Reader

Could this same problem be solved by using Apache Arrow, to convert to/from pandas, and cut down the complexity in the process?

View on HN

This does use Apache arrow, that's how the data is transferred from spark to pandas and back.

Great, glad this was confirmed as I am about to solve a similar problem and my expectation is that Apache Arrow will be needed. So pyarrow = Apache Arrow.

Reply on news.ycombinator.com