Hacker News Reader

Comment by efaref | Hacker News Reader

I don't really see how ION tables are an improvement over arrays of arrays, e.g.:

    {
      "headers": [ "a", "b", "c" ],
      "rows": [
        [ 1, 2, 3 ],
        [ 4, 5, 6 ],
      ]
    }

Furthermore, ION appears to require you to know the length of your data up front, whereas you could use CBOR unspecified length arrays to stream data from your database without precalculating the table length. It seems like quite a niche format, though. Most data is not truly tabular.

Also, in the table it claims "Yes" under support for "Cyclic references", and yet further down the page:

> ION has support for expressing cyclic references between objects. At this point this support is not 100% finalized.

So surely this should be "Yes(*)"?

View on HN

First of all, lots of results sent back from backend services (or databases) are arrays of objects. So no, tables are not a "niche" format - tables are heavily used.

Second, an array of arrays could mean anything. You have not semantics telling whether the arrays are independent or if the first array is an array of columns for the following arrays. ION Tables add that semantic information.

Third, yes, we could encode everything as text or as raw bytes and leave it up to the user to make sense of it. But that is exactly what we are trying to avoid with ION. We want to give devs a decent standard data format to use, that doesn't require a lot of data encoding choices up front. The encoding options have been thought through already, and sensible choices already made which you can just follow.

Fourth, you can nest ION tables inside ION tables, and thus create a more compact representation of an object graph. Using JSON / CBOR you would need a lot of nested arrays inside arrays to emulate that. Possible, but not exactly pretty.

Yes also, the TLV nature of ION means that you have to buffer the ION data while writing it, in case you don't know the full size of the embedded ahead of time. However, this is really only a problem for very big messages. HTTP has worked like that for a long time already, and HTTP servers have proven capable of scaling pretty well, won't you agree?

Additionally, if a message does not include its length up front, you are just trading faster write time for slower read time. A node receiving dynamically sized data then has the problem of knowing how much data to allocate for the full message, plus the receiver has to inspect the data as it comes in to see where it ends. With an ION message the receiver knows within the first 4-5 bytes (typically) how big that message will be, and can thus copy the following bytes directly into the perfectly allocated memory area, without having to examine them any further. Since data is most often read more times than written (e.g. data written to a file), we felt that making a tradeoff that favors read speed over write speed made sense.

If you're going to send structured records and care about knowing the structure: use protobuf. If you're going to be binary JSON: use RFC 7049 CBOR. http://cbor.io/

Reply on news.ycombinator.com