Our Journey with Apache Arrow (Part 2): Adaptive Schemas and Sorting
arrow.apache.org
arrow.apache.org
- cost (more maintainable from the tech overhead required).
- storage (lower size)
- compute (faster reads and indexing)
Such that it really depends who you are talking to about which format you should use. Ideally, it seems, support both. Allow rapid ingress and egress of data to/from Excel in whatever way that you can. CSV is the common stopgap between the two, with loads of sharp edges on loading it into Excel.
- These processes and SOPs are actually quite optimal for low scale use-cases but as organizations grow they find themselves in a tough space where even after you have a hiring spree going on everything seems to be falling apart and not maintainable.
- You almost always do lose out on an audit trail. Now, you don't always need an audit trail but if you find your org sharing tons of excel docs being shared on a regular basis you do need a tech integration to solve it. The lack of it means a gap in identifying things that can be made simple and no way to identify / fix things that may have gone wrong over a period of time.
- I fully agree with the statement . Ideally, it seems, support both. Allow rapid ingress and egress of data to/from Excel . The data in systems in software systems is useless it's almost always meant to be consumed by humans (mostly). And that just means while I do most of the authoritative processing in file format that computers do well with I allow it to be converted into a more human readable format which happens to be excel in some cases and visualizations in others.
Same thing, largely, seems to apply to documents. Yes, you could farm out a book to a hundred authors and editors. We could build a tool that allows active collaboration that shows what everyone is doing. And we even have some exciting data structures that seem to make this a viable path forward. My experience has been that that isn't the case, though. Specifically, the data structures are fine and nice, but getting even a dozen folks working on the same doc at the same time usually reduces the document to a glorified database with each person doing an optimistic lock on their part so that they can do what they want to do.
I recently used pymongoarrow to speed up data extraction from mongo in python as the deserialization from bson to python is super slow. Using pymongoarrow helped achieve nice speed improvements (average 3x to 5x)