How I learned to stop worrying and love the schema

The guys at Snowplow Analytics are really into applying schemas to event streams, and will write or bang on about it at any opportunity:

http://snowplowanalytics.com/blog/2014/07/11/on-the-benefits...

http://snowplowanalytics.com/blog/2014/05/15/introducing-sel...

https://github.com/snowplow/iglu

I think there's a lot to be said for knowing for sure what shape your data is.

My experience might be insignificant.

I've been working with several APIs, both XML and JSON, and I'm yet to see an example where a schema have helped me writing an automated parser. Validating the response/request? Sure.

Maybe I did not have the right tools, maybe I lacked the knowledge, but the fact is, I've been writing such parsers by hand. I'm vaguely aware that there are various C++ parser generators from XML schemas, but I could not use those because of license/compatibility issues.

I suspect it would be possible to write a schema->parser converter using meta-programming, but how many languages are capable of that without huge amounts of pain? I could probably do that in Python or LISP, but not in some other language.

And I have not seen a JSON schema being used... anywhere?

A good way to enforce schema backwards compatibility is to have something like a pre-commit hook or an auto-revert that triggers when the compatibility check fails during the build of your schema class library (assuming Java).

In addition, consistent use of schemas is also essential to operating something like Kafka audit:

http://www.slideshare.net/JonBringhurst/kafka-audit-kafka-me...

I'm, not sure I agree that the solution is always to use a schema... In general, yes, it's a good idea. However, it seems to me that, in this particular example, the correct format could be determined by the device_id in the JSON formatted text. That is, the date format when device_id=123 is X, the date format when device_id=watch_345is Y, etc... If there are a small number of device_ids then it's a workable solution, maybe even preferable if the amount of data to be transferred needs to be minimized.

However, once JSON was used, then I would almost certainly have put in a schema/format/version field in a header.

Two things:

1) The people on the "temp-o-meter watch team" are all retarded

2) He fails to show a schema that will solve this, and I seriously do not know what he's talking about. What would fix this:

"Had this company been using schemas, the watch and phone teams could have simply reused the same schema, avoiding the need for the watch team to reinvent the wheel, "

Given that the different languages are used [1], what kind of magic fixes this? Without requiring all the parts to support something common technology I might add..

[1] if not, there wouldn't be a problem

There's nothing prettier than a pretty ERD!