wesm

See also https://wesmckinney.com/blog/mythical-agent-month/

See also The Mythical Agent-Month https://wesmckinney.com/blog/mythical-agent-month/

Super well! I don't work without this tool running in the background supervising all the agents' work

I've been building https://roborev.io/ (continuous background code review for agents) essentially as a cope to supervise the poor quality of the agents' work, since my agents write much more code than I can possible review directly or QA thoroughly. I think we'll see a bunch of interesting new tools to help alleviate the cognitive burden of supervising their work output.

I was especially excited to learn that RZ is built on Apache Arrow internally, which makes it easy to integrate with other Arrow-based applications and the emerging "Composable Data Stack". Really exciting stuff, they're just getting started!

If you read my slide decks over the last 7 years or so (while I've been working actively on Arrow and sibling projects like Ibis) I've been saying exactly this.

See e.g. https://ibis-project.org/

Almost no database systems support multidimensional arrays. So they are not appropriate for many use cases?

* BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhouse: no * Dremio: no * Impala: no * Presto: no ... list continues

We've invited developers to add the extension types for tensor data, but no one has contributed them yet. I'm not seeing a lot of tabular data with embedded tensors out in the wild.

I challenge you to have a closer look at the project.

Deserialization by definition requires bytes or bits to be relocated from their position in the wire protocol to other data structures which are used for processing. Arrow does not require any bytes or bits to be relocated. So if a "C array of doubles" is not native to the CPU, then I don't know what is.

> Arrow's serialization is Protobuf

Incorrect. Only Arrow Flight embeds the Arrow wire format in a Protocol Buffer, but the Arrow protocol itself does not use Protobuf.

There's no serde by design (aside from inspecting a tiny piece of metadata indicating the location of each constituent block of memory). So data processing algorithms execute directly against the Arrow wire format without any deserialization.

Microsoft is also on top of this with their Magpie project

http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf

"A common, efficient serialized and wire format across data engines is a transformational development. Many previous systems and approaches (e.g., [26, 36, 38, 51]) have observed the prohibitive cost of data conversion and transfer, precluding optimizers from exploiting inter-DBMS performance advantages. By contrast, inmemory data transfer cost between a pair of Arrow-supporting systems is effectively zero. Many major, modern DBMSs (e.g., Spark, Kudu, AWS Data Wrangler, SciDB, TileDB) and data-processing frameworks (e.g., Pandas, NumPy, Dask) have or are in the process of incorporating support for Arrow and ArrowFlight. Exploiting this is key for Magpie, which is thereby free to combine data from different sources and cache intermediate data and results, without needing to consider data conversion overhead."

(Wes here) I appreciate the Arrow shout-out but note that Apache Arrow has been a major open source community collaboration and not something I can take sole credit for.

There is no “JIRA politics” blocking the LZ4 work, only a lack of volunteers to do the development and testing.

Keep in mind that Arrow Java<->C++/Python interop has been in production use in Apache Spark and elsewhere for multiple years now. We have avoided some of the mistakes of past projects by really emphasizing protocol integration tests across the implementations.

Again, I have to object to your use of “arrow/parquet”. These are not the same open source projects and while people use them together it isn’t fair to the developers of each project for you to discuss them like a single project.

See

http://arrow.apache.org/faq/index.html#what-about-arrow-file...

You can store them long-term if you want (and you'll still be able to read them 5 years from now) but we aren't optimizing the Arrow IPC format for the _needs_ of long-term storage.

What you've written sounds like a criticism of the JVM data analytics ecosystem (the Java Parquet library in particular) and not Apache Arrow itself. Parquet for Java is an independent open source project and developer community. For example, you said

> It's barely useable and the dependencies are horrific - the whole thing is mingled with hadoop dependencies - even the API itself.

These are comments about http://github.com/apache/parquet-mr which is a different open source project.

For C++ / Python / R many of the developers for both Apache Arrow and Apache Parquet are the same and we currently develop the Parquet codebase out of the Arrow source tree.

So, I'm not sure what to tell you, we Arrow developers cannot take it upon ourselves to fix up the whole JVM data ecosystem.

hi, Wes (Apache Arrow co-creator and Python pandas creator) here! If you're wondering what this project is all about, my JupyterCon keynote (18 min long) from 3 years ago is a good summary and the vision / scope for what we've been doing since 2016 has been pretty consistent

https://www.youtube.com/watch?v=wdmf1msbtVs

This isn't accurate -- there are multiple query engine subprojects within Apache Arrow.

Arrow:

* Standardizes binary interop and "serialization" of large structured data, removing all conversions / serialization at ingest and export boundaries. This alone can mean > 2-100x performance improvement in an application that processes a lot of data

* The Arrow in-memory format is an ideal data structure to code analytical algorithms against.

Check out my 18min talk from a few years ago about the vision for the project https://www.youtube.com/watch?v=wdmf1msbtVs

I wasn't aware of it but will take a look. Thanks!

To be clear for anyone reading, we're parsing and generating the data-related protobufs ourselves, and retaining ownership of the memory returned by gRPC to obtain zero copy.

The C++ details are found in

https://github.com/apache/arrow/blob/master/cpp/src/arrow/fl...

No. If you want to talk about it come on the Apache Arrow mailing list

Too big of a discussion for Hacker News! Come on dev@arrow.apache.org if you want to talk about it

Dask does not have "experimental Arrow integration". It supports using Arrow to read Parquet files but no Arrow-based computational functionality.

I'm looking at the code I linked, and you are serializing in the general case, it is not zero copy. Unpacking a bitmap is not free.

Note: Vaex has its own memory model. If you input Arrow, it converts to the Vaex data representation. Details here:

https://github.com/vaexio/vaex/blob/master/packages/vaex-arr...

One of the primary objectives of Apache Arrow is to have a common data representation for computational systems, and avoid serialization / conversions altogether.

RAPIDS is partly powered by Apache Arrow. So we are all collaborating on a common next-generation computation ecosystem.

In general we are talking about O(n) algorithms, and the gains are due to better CPU cache utilization and fewer instructions per value, which LLVM helps do

Arrow Flight is still being developed and may be while before it drops to real user code. See ARROW-249. Indeed having a open standard runtime memory format for tabular / columnar data sets is key to improved performance