Show HN: Avsc – a pure JavaScript Avro implementation

Thanks!

I like to think of Avro in two (related) parts:

+ A very compact binary serialization, which lets you efficiently store or transfer data over the wire. (Avro actually also defines a way to encode your data as JSON in cases when you need a human-readable representation.)

+ A way to define "data types". For example this schema [1] defines what we expect a "Human" entry to look like, and we can now do things like check whether we are missing any information, make sure all fields are valid, sort, efficiently copy objects, etc.

This is glossing over many things (files, RPC, ...) but I hope this helps a bit! You can find a lot more information in the official Avro documentation [2] if you're curious.

[1] https://github.com/mtth/avsc/blob/master/etc/benchmarks/sche...

[2] https://avro.apache.org/docs/current/index.html

I'll start with the second question since it's easier to answer: some of the other libraries don't support all Avro features (for example I don't think `node-avro-io` supports encoding records defined inside arrays), so they're unable to run some of the benchmarks. Both the reference Java implementation and `avsc` are able to parse all these schemas so don't have any -1s in their columns.

Performance-wise, the main benefits came from doing as much preprocessing as possible. Here are a couple examples of what is done when a schema is parsed:

+ All methods for encoding and decoding nested types get fully resolved, removing the need for lookups (for example an array of integers would have a reference to the method for encoding integers).

+ A constructor, and encoding/decoding methods gets programmatically generated for each record type. This avoids having to iterate over the fields each time. Same for unions. (Having a "Class" for each record type also lets us attach useful methods to its prototype, available for all record instances.)

Then there are a few other general ideas like making sure v8 is able to optimize all the methods on the critical path, or reusing buffers as much as possible to avoid allocating too many objects.

I haven't tested this but my guess would be that performance would be similar as they both run on v8. The only difference is the buffer implementation which is roughly as fast it seems (benchmarks here [1]).

[1] https://github.com/feross/buffer#performance

Cool, we use avro a lot at my work, but mostly handle it on our python server. This could be useful for some client-side code gen for schema validation before sending to the server.

We define all of our records in protocols in avdls, I'm assuming this doesn't handle avdls as of yet?

Why did you name it avsc? Are you concerned that it might cause confusion with the avsc file extension that avro uses for schema files?

https://avro.apache.org/docs/1.7.7/gettingstartedjava.html

I don't know enough about Thrift to compare them unfortunately.

Not that it's better, but Avro is a widely supported format in open source 'big data' tools.