How would this work for my 18th century agrarian business?
52 karma · joined March 17, 2008
However, since Singer is built around piping data between applications, your suggestion - to code something that sits between taps and targets - makes perfect sense. The whole "flow" would look like:
$ tap-mydatasource | do-aggregations | target-mytarget
We'd be eager to hear from anyone who tries this approach!
- JSON doesn't have a robust set of data types, and specifically lacks a datetime/timestamp type. With a schema, Taps can, for example, denote fields in the JSON that contain datetimes represented as strings, and then targets can convert those to proper datetimes and handle them accordingly.
- Dealing with un-structured or flexibly-structured data is hard. Requiring a schema forces a Tap author to think about the structure of the data up front. By validating each data point against a schema, the Tap author should be able to more quickly identify nuances in the data set - like missing fields, nullable fields, mixed-type fields, etc - and either decide to clean them out of the data (if appropriate), or provide the right schema to inform downstream applications about them. Identifying and handling these problems requires an understanding of the source data set, so it is best done as close to the data source as possible.
Additionally, we'll be releasing a Java client library any day now, with other languages and platforms to follow.
We have found that this approach works well for our users, who prefer to get the rawest possible data, and the systems we target like Redshift that are themselves powerful transformation engines. This gives the user unlimited flexibility for defining transformations, and a full audit trail for understanding how their data has changed.
We are always evolving, though, so if there's a use case that you think requires this approach, I would be eager to hear more about it.
Is this the behavior that HN's vote ring detector is trying to discourage? I understand that these things are a slippery slope - but if so, it's too bad, because quality content like robertjmoore's last three posts is getting lost, and I would imagine that other authors at small companies like ours are unwittingly falling into the same trap.
If there have been other posts explaining the DOs and DONTs of the HN vote ring detector, I apologize in advance for not having read them.