Hacker News Reader

arrow/feather is really the best format these days for tabular data transmission.

anyone who disagrees I’d be very interested to hear your thoughts on alternatives.

What about compression - is this part of arrow itself?

It's not part of Arrow, but Arrow is columnar so just a basic LZ4/ZSTD will work pretty well.

Arrow looks super complicated.

Are data types useful for data to/fro web/mobile clients? Encode type into the column header?

data types are absolutely helpful. when you know a column stores Float64 data, you don't have to write out float to base 10 and parse it back. You just dump the bytes.

Or parquet, for compression?

Arrow is meant as the “in-memory” dual to Parquet, which is meant as the “on disk serialisation format”.

Many parquet supporting libs will load Parquet files into an Arrow structure in memory for example.