Hacker News Reader

It looks quite nice, and I will have to explore the performance comparisons with Dask more.

I have recently started using Xarray for some projects, and really appreciate the usability of multidimensional labelled data. Are the memory mapping techniques used for speedup here only applicable to tabular data?

The support for Apache arrow is quite nice. Have you considered any other formats, such as Zarr?

View on HN

Thank you. Memory mapping could be used for other data as well, and I have looked into zarr (even opened an issue for that https://github.com/zarr-developers/zarr/issues ). Memory mapping of contiguous data makes life much easier (for the application as well as OS), chunked data could be supported, but is more bookkeeping.

I'll need to have a closer look later, but would vaex fit in with somewhat indexed mapped files?

E.g. parquet supports column indexes now: https://issues.apache.org/jira/browse/PARQUET-1201

Reply on news.ycombinator.com