Amazon Ion

In 2013 I was generating code from SDL definitions which could also be used for data validation during serialization/deserialization. I had plenty of “model” packages which were just SDL definitions, build config, and maybe some unit tests to validate the schema constraints. (Edit: SDL is “schema definition language” which was a schema definition tool written for Ion with definitions in Ion.)

These were used in services and reactors which never touched raw Ion (at least not in any way different from Coral or BSF).

Full disclosure, I spent a lot of my free time working on Ion, both the supported implementations as well as my own. The additional data types are worth it alone, imho. Having to use JSON for most things now I’m frustrated at what is “missing”.

I built the first version of IntelliJ plugin[1] to make working with the reactor stuff easier. Doesn't look like there have been much improvements to it.

1: https://github.com/amzn/ion-intellij-plugin

But do you see any reason why such tooling could not be built?

I suppose it's mostly an under-investment of time, not a shortcoming of the format itself.

ion precedes PartiQL by many years, maybe even a decade. The sole reason ion exists is to make parsing json faster and less ambiguous so that a few specific edge cases are handled efficiently. So far so good, right?

The problem is that it spreads, like an infection, to surrounding services. Inside Amazon there are literally hundreds of libraries that duplicate standard json libraries in various languages but support ion instead of json. All of this is just to deal with interoperability.

Ion is slower than protobufs and less universally understood than json. Honestly it's just an annoyance.

Yeah It’s more like a spec, they expect you to write the query optimizer, analyzer basically a new query engine. They offered a document based “reference” implementation on Kotlin though so I think expert could follow it.

Point is AWS might not make up its mind yet on whether this does more harm to their DB business or not

Given Postgres already supports document types, do you know what is the main advantage of using AWS' instead?

That sounds great, but is Kotlin really the only current implementation?

So it didn't really catch on?

Just from reading their page and being familiar with the formats you mentioned:

vs. protobuf: ion is self describing, vs needing a schema

vs. thrift: similar, thrift needs a schema to interpret a binary file

both thrift and protobuf are really binary formats, though they have a canonical textual representation, it's not actually used to serialize. Sounds like ion supports serializing as text as a first class concept.

vs. msgpack: ion has a corresponding text format, whereas msgpack is only binary. Additionally, ion has a symbol type, msgpack doesn't.

I think the biggest benefit here is that it's a new chance for a format that fixes some of json's rough edges to gain critical mass. There's probably nothing ultra special about it that hasn't been solved in other formats, but maybe the timing will be right and everyone will just adopt it as a json replacement (sort of how people just gave up on xml and switch to json seemingly overnight). It's impossible to predict stuff like that.

Edit: upon noticing that it was released in 2016, it seems less likely everyone will jump on the ion bandwagon ...

There are some painpoints that are being addressed:

1) timestamp : I have had issues with a round-tripping timestamp representation quite a bit 2) decimal : currency is denoted in decimal rather than float and shows the Amazon retail heritage. This is very useful. 3) symbols : I've had cases where symbol table/dictionary would have made big difference in serialized size

Protobuf/Thrift are schema-based serialization systems.

Ion is directly comparable to JSON/MessagePack/BSON/CBOR.

I would expect Ion will have slightly different time/space tradeoffs than the other binary schemaless formats.

I wish these comparisons would occur to the documenters without the need for a prompt. Also Apache Avro .. which is common with Kafka?

It is probably closer to something like avro, but with more features and less java focus.

I left Amazon a bit over a year ago, after being there seven years. It always struck me as a combination of "not invented here" syndrome and a solution in search of a problem. It has no real world benefits over JSON, the tooling is limited, but you inevitably have to deal with some other team that regrets choosing it and now it's their API. I'm so happy I never have to look at it ever again, and seeing this post today is a real throwback to wasted engineering effort. Just let out go, Amazon.

Depends on the part of Amazon but it is pretty prevalent in Retail. The fact that it is both binary and self describing makes it pretty good for data at rest. You can still parse and understand that archival transaction data from 8 years ago.

The support for S-Expressions is both a blessing and a curse. The ability to write logic with native data structures in it is fundamentally interesting, but it leads to lots of reinvention of somewhat crappy Lisp implementations.

The tooling ecosystem has been slowly improving outside of JVM, particularly the latest JS implementation.

In a vacuum, the support for type annotations, timestamps, decimals and binary serialization make it superior to JSON for use cases where self describing data is appropriate.

The easiest way to get an interview at Amazon is to get a referral. If you can demonstrate competent programming abilities and have a half decent attitude, it shouldn't be too hard to get a referral from someone at Amazon, regardless of what projects you have under your belt.

I’ve published two libraries for Amazon products. They didn’t care.

I was thinking about spinning up a support library for Haskell.. but it’d be a pretty serious investment of time when everyone’s employment is cut back or up in the air already. It would be nice to get a crate of sanitiser or something.

Who's going to maintain it? Are you just doing that for an itw or are you offering real support to the library? That's the reason why people are paid to work on software vs someone on its free time.

Why would you want that? There are so many better companies to work for that don’t abuse their most vulnerable workers.

Have you tried applying?

Haven’t you heard it sucks working there? If you are replaceable by a grad student, they will let you go before your options are vested.

PHP is a banned language internally Amazon and Amazon subsidiaries, so they will not care.

... why am I getting downvoted for offering direct experience as an AMZN engineer? Amazon InfoSec forbids PHP. See also: https://news.ycombinator.com/item?id=23030330

iOS shopping app is a web app and it uses plain JSON.

> So, we just use JSON, and our solution to this problem has been to pass 64 bit unix timestamps around.

Passing an integer does not have the same semantics as passing a timestamp. Relying on out-of-band info to parse a document is a problem in the making.

> but for most use cases it is more than enough practical range & precision to get the job done.

Parsing s-expressions would also get the job done, even if it's a primitive s-expression that only supports cons cells and a string data type. However, people find value in enabling the parser to validate booleans, arrays, and objects.

ION is just a logical next step. Timestamps are quite naturally a fundamental data type in comm between web services, particularly in binary form.

Pass 64-bit Unix timestamps around as JSON numbers? That's a bad idea, seeing as they're 64-bit floats. You're better off formatting your 64-bit integers as strings.

I'm rather attached to things like time zones.

Nope, backend if anything. For example, their new QLDB product uses it to get consistent hashing of documents on account of Ion being a canonical format.

For the public API, customers want JSON, so they get JSON. Internally there's Coral, and something like Coral/Protobuf outright superior for the use case of an API where a schema can be distributed in advance. The only real use case for Ion is when you have data that's already JSON-formatted for whatever reason and you want to compress it for storage or transit.

Last time I checked, QLDB is in AWS

I think "superset" is a clear relationship. It means "legal JSON is legal Ion", just like "legal JSON is legal YAML". I don't think it's inappropriate to point that out. In fact, it's an excellent feature.

It kind of does:

    class SQL(str):
        ...

    query = SQL("""
        select * from table
    """)

Any of the elements of an ISO 8601 time element can have decimal fractions added to any number of digits. But only the lowest element (according to Wikipedia because I don't have the actual standard in front of me). But you can definitely have a timestamp of 2020-07-23T12:37:55.758145Z

The standard acommodates timezones as offsets from UTC, because it's a representation of a timestamp, not a local time at a particular geographical location. So things like daylight savings time periods are not relevant.

This is awfully negative. JSON explicitly does not declare the represented range of floats or integers, and doesn’t have a distinct arbitrary-precision decimal type. I haven’t read the Ion spec, only the description, but since it’s advertising arbitrary precision, presumably any implementation that does not support that is not a correct implementation at all.

If I recall correctly, Ion preceed even Google's protobuf, and is 20+ year old technology. This isn't result of "yet another standard" but parallel evolution

This is a funny joke.

Also, this is the way any progress is made. Between 15 competing standards, some win over.

Were it not so, we'd still use whatever Cobol used for data serialization.

Ion is actually two formats, with Ion data having a canonical representation both in binary and in human-readable text. The text format's file extension is ".ion" and the binary format's file extension is ".10n", and I think that's the entire motivation.

Google has already protobuf for a long time.

Ion predates JSON FYI. I heard Amazon invented it internally many years ago