Getting Started with Apache Avro | Hacker News Reader

Getting Started with Apache Avro

idarlington.github.io

28 pointsluu12 comments

Yeah... you're only going to save money if you're an absolutely enormous scale. I'd also really question that point at all when you can compress JSON as it's being transmitted (GZIP, baby).

Next, I have yet to see somewhere... anywhere, not at the scale of something like Uber, or a high-performance game server, really need to worry about serialization and deserialization of JSON. Almost all serialization libraries are written in C for MAXIMUM PERFORMANCE regardless of the language you're using, or some sort of hand-tuned-magical bytecode fuckery which has the same effect.

I honestly can't imagine this being useful except for obscenely high throughput service to service communication or as an internal serialization protocol (for something like FoundationDB where you're storing the Avro encoded instance to disk). Truly, JSON and REST are good enough for 99.999% of services and the overhead of using something non-standard will likely never pay off due to the increased developer overhead and maintenance.

This just smells like tech debt and an architecture that can't be debugged.

The compression soon mounts up in savings. It’s quicker to send over the network, disks cost money and on the cloud even low terrabyte sized data starts to mount up in cost.

JSON is unstructured. Every one who parses the data needs to write a parser. As it’s unstructured how do you manage schema changes?

Using a binary format like avro you get strong typing (basic validation), schema change modes, no need to write clients. You can publish your avro schemas to a registry and you then get a catalog of data available anyone can browse to see what data is available and how it looks.

> This just smells like tech debt I’d say the opposite. Using JSON is tech debt. Everyone writing parsers/serialisers, not knowing what fields are available, what values the fields can accept, how to handle new fields / deprecating old fields.

Debugging is simple, AVRO can be inspected unlike over formats with its Generic record format that can also be represented and dumped as JSON.

people should use binary strongly typed formats more. It makes things so much easier for the developer,data evolution and data processing. You also get benefits that your application is no longer running in debug mode.

> you can compress JSON as it's being transmitted (GZIP, baby).

Gzip doesn't buy much advantage on small records, because there's not much repetition in the source document. And there's no law against compressing binaries, last I checked.

> Next, I have yet to see somewhere... anywhere, not at the scale of something like Uber, or a high-performance game server, really need to worry about serialization and deserialization of JSON.

I built a discrete-event simulator[0] which presents its findings with a simple web interface from the localhost. It's about the simplest possible setup. The simulator can process millions of simulated events in a few seconds, but it takes much longer to stream the JSON to the browser for rendering. Even with "GZIP, baby".

As an experiment I converted it to stream records in Avro format. There are several javascript libraries and Golang libraries. The conversion took me an afternoon. The amount of data streamed went down by approximately half. I didn't gain on rendering time because it currently completes the entire stream before conversion. If that wasn't the case I expect the Avro version would noticeably faster.

So if "it took one random person an afternoon to hook up some libraries" is tech debt, then I guess I better call my bank, because I do that pretty much every day.

[0] https://github.com/pivotal/skenario

I'm toying with Avro right now for a project and our common JSON payload went from 1400 to 70 bytes, which makes it much more suitable for use over UDP

Also using JSON in a statically-typed language is tedious and error-prone.

How much of your data is repeated as part of a cyclical graph? You may want to try jsog + compression.

[1] https://github.com/jsog/jsog

If the author is around:

> Translates to less money saved on provisioning hardware.

I read that as it will cost more. Saving less money is bad.

I assume that is a typo?

Okay, so the article claims (without much evidence) those pieces where Avro shines.

I'm more interested where it fails. It's the places where a library falls down that's likely to burn me when I'm 24 hours from shipping a product.

We’ve used Avro in production for a number of years now. We use it to define the schema for Kafka messages passing between parts of the system and it has worked well - but you want some downsides, so..

Compatibility: Avro deals with this well, but there are gotchas especially when you need/want to have “forward compatibility” as well as “backward compatibility” in your messages. Adding a new type to a union? Breaking change. Adding a new element to an enum? Breaking change. Want to add a new member to a type? Best give it a default, or.. breaking change. We use the Confluent schema registry to manage our schemas on kafka and we find ourselves having to do carefully managed rollouts (stop all producers, put schema-reg into compat mode, upgrade consumers, upgrade and start producers) every now and again

Performance: we frequently see Avro as the “hot” code in our apps - mainly I suspect because we’ve got a fairly large schema with lots of union types etc, and our messages are up to 2kb or so in size. This results in our applications topping out at low tens of thousands of messages per second, though it scales linearly with number of cores. The most recent java version has a “fast read” flag that promises to speed up our use-case but we haven’t tried it yet

> Adding a new type to a union? Breaking change. Adding a new element to an enum? Breaking change.

This is incorrect. I've personally gone from type X to union {X,Y} in C++, you just need to use the resolving decoder that takes both schemas. Enums can also be resolved based on names. Check out the schema evolution rules here:

See https://avro.apache.org/docs/current/spec.html#Schema+Resolu...

Of all the binary serialization formats of this ilk (proto, thrift, flatbuffers etc), Avro definitely has the most schema flexibility.

It’s good for server-to-server type applications.

It’s good for storage at rest.

It’s good for using common big data tools on your data.

It’s not as trivial as using JSON client side in a web browser although you can, also I’ve never seen it done.

The JSON schema format tedious to write and difficult to read. I recommend anyone using Avro to use the IDL language[0] and use the Java tool to convert to the JSON schema format.

java -jar avro-tools-1.9.2.jar idl yourschema.avdl

[0] https://avro.apache.org/docs/current/idl.html

When managing millions of transactions through Kafka that have to be serialized to s3, read by hive natively, translated to parquet by spark, support nullable complex types (map, struct, array), allow schema evolution forward and back, then Avro is a good choice for your base data format.

Reply on news.ycombinator.com