Transit: JSON Data Interchange Format
github.com
github.com
http://blog.cognitect.com/blog/2014/7/22/transit
JSON has become the go-to schemaless format between languages, despite being verbose and having problems with typing. Transit aims to be a general-purpose successor to JSON here.
Stronger typing allows the optional use of a more compact binary format (using MessagePack). Otherwise it too uses JSON on the wire.
Anyone who knows more, please correct me.
XML has become the go-to schemaless format between languages, despite being verbose and having problems with typing. JSON aims to be a general-purpose successor to XML here.
XML also doesn't suffer the problems of a limited set of pre-defined types that JSON has. Its type system is extensible through schemas.
Yes, it is, though there are a variety of schema schemes that can be used with XML.
> It was designed to be extensible through schemas.
It was designed to be extensible through custom tags (in fact, all tags are custom tags); it can be optionally be restricted through the use of schemas, not extended.
> XML also doesn't suffer the problems of a limited set of pre-defined types that JSON has.
Yes, it does, with an even smaller number of types. (Strings, with a couple different representations.)
> Its type system is extensible through schemas.
No, it isn't. Its structure can be constrained through schemas (XML Schema, Relax NG, etc.), which can provide a means of modeling a different type system on top of the data structure supported by XML, but that isn't really "extending" XMLs type system any more than using a Haskell compiler to generate x86 machine code is "extending" the type system of x86 machine code; its just providing a completely separate type system on a different layer.
Also, no matter how you view this, its not a distinction between XML and JSON, since you can do the same thing with JSON, and just as there are a variety of different schema standards that do this for XML, there are a variety that do the same thing for JSON.
Floating point keys just seem like a terrible idea.
I don't see why floating point (with a defined precision) keys are any worse fundamentally than any finite-domain scalar keys. The applications for which they are useful may be limited, and they may be the wrong choice for some uses, but that's true of anything.
If it's fixed-precision it's not floating point, it's basically an integer we interpret with a decimal point somewhere when we display it. If you didn't mean fixed-precision, then I'm not sure how actual floating points could sanely act as a hash key.
Which makes it hard to write a well-defined hash.
I'm not referring to fixed point, but fixed-size representation.
> If you didn't mean fixed-precision, then I'm not sure how actual floating points could sanely act as a hash key.
I don't see what the problem is. Certainly, there's problems if you do certain floating point calculations and blindly look for the result as a key in a mapping, but that's not a problem with floats as keys, its a problem with that particular usage pattern.
I'm still not sure what you mean as any IEEE floating point (which in reality all anyone uses) is fixed-size. If you decide to truncate, round, do w/e to this value, you may still not get exactly the same answer for similar computations or even the same computation on different machines. There isn't a reason you can't (well, shouldn't) do float_val == 0.0 -- it won't work in all circumstances.
Please explain your method of making sure float_val == 0.0 while retaining the essence of being floating point (i.e. not using fixed-point).
Doing client-side programming with things like CLJS, Figwheel, Reagent and core.async feels miles ahead of what we have in moden-js-land (es6/7, babel, webpack, React, promises).
If you were to start a startup today, would you be comfortable going with something like Clojure/script?
Everything you've mentioned has been awesome for us. The arrows we take as pioneers are things like having to roll libraries for email-as-a-service, stripe, etc. Granted it's usually easy but can be time consuming.
Biggest wins for us are:
- ClojureScript / figwheel as an awesome front end development combination.
- The combination of functional programming with immutable data structures
- Lisp "magical powers" (macros, interactive REPL etc.)
- Ability to exploit the Java library ecosystem whenever you need it
Biggest downside = Lack of types.
In terms of non-component organization, I believe re-frame is a significant improvement over redux. Reactions are good when you don't control the endpoints, splitting out the reducers into pure functions is good, but adding middleware on them is the real win. On the other hand, it wouldn't be that hard to adapt the model to redux.
The next phase of organization is integrating Relay/Falcor concepts. David Nolan gave a talk about this in Om Next at NYC Clojure last month and there is a video. Om Next as presented is very compelling if you're on Datomic and less so otherwise.
As to your startup question:
I've been a full time cljs dev on a b2b app at Reuters for the past 9 months. I took up the job specifically because I wanted to write cljs. I had been involved with the Clojure community (I care about state) but only working on toy projects for the previous ~4 years.
My experience with Clojurescript is that it was less of an improvement on modern js than I was expecting. The biggest advantages are protocols and the standard library being both rich and standard. Nice to haves but non-critical are native syntax for immutable maps and multimethods. I guesstimate I write ~10% less code in cljs versus js but you're ultimately writing the same stuff.
Problems I've run into:
Full build time for this app is long. Our app is in the 15k LoC range across ~150 files and cold compile is 140s on a 2012 MBA. It's annoying but incremental compilation times are sub-second after some build config tweaking.
We have one component in particular that tends to get lost when switching branches and the missing namespace forces a fresh compile. Our cljs version is from June so this may be fixed. I've also spent a number of hours debugging problems that turned out to be stale build issues.
I tried a couple times unsuccessfully to get Emacs (cider) to connect to a figwheel repl. After a few evals things simply become unresponsive. Just using figwheel is good enough but I miss the in-editor repl. Haven't tracked down the reason, could be my lack of emacs knowledge.
If you're using core.async, the main loop has a try/catch/rethrow. This causes Chrome Dev Tools to break in the outer loop instead of actually at the problem. You have to explicitly err.stack in the console (which is not source mapped) and don't have access to the locals unless you manually set a breakpoint at the error and reload. You also get to learn to read the JS representation of Clojure literals. None of this is impossible and if you're working in a tight loop you tend to have a pretty good idea of what the error is without jumping through debugger hoops but if you're doing something like switching branches or refactoring it's annoying.
I like Reagent but I've had a number of times where its behavior doesn't match my expectations. In particular, figuring out what part of the vdom is invalidated on a ratom change caused me problems. There's a gotcha that sequences must be forced with doall or you'll get weird behavior. At the moment I have a very expensive reaction (list processing ~6k items) that's getting run 8 times in response to a single key change in the source ratom so I'll be tracking that down tomorrow.
I don't consider this list a reason to not adopt Clojurescript. I can make a similar list for the Babel stack.
As for the question of would I be comfortable, I like writing clojurescript but I'd only really recommend it if you're committed to full stack Clojure. It takes a number of weeks for a new frontend hire to ramp up on the language. I've discussed this with the other frontend specialist on the team and our consensus is that cljs is a better language but we're not that much more productive in the language compared to ES2015 so I'm not really convinced the weeks of ramp up time are worth it. Our experience with hiring has been that we've had very few candidates but they've all been skilled.
OMG! Is that what is causing that behaviour?! It's frustrating (though not a show stopper) and I was unsure of the cause. Thanks for pointing that out. I will have a deeper look.
Immutable, js-csp, etc... were all inspired by Clojurescript, I guess that's the point the grandparent was trying to make.
{ number: 1.0000000000000000000000000000001 }
Type this in your address bar for an illustration:
javascript:alert(1.0000000000000000000000000000001);
{ number: "1.0000000000000000000000000000001" }
XML:
<number>1.0000000000000000000000000000001</number>
<xs:element name="number" type="xs:decimal"/>
This is a real world problem we encounter in the financial sector every day.
javascript:alert(1.000000000000001);
{ "number": "1.0000000000000000000000000000001" }
{ "number": { "type": "ModelReal", precision: 64 } }
<number>1.0000000000000000000000000000001</number>
<xs:element name="number" type="xs:decimal"/>
All minimally conforming processors must support decimal numbers with a minimum of 18 decimal digits (i.e., with a totalDigits of 18).
You have 32 decimal digits in 1.0000000000000000000000000000001, so "minimally conforming processors" are, according to spec, actually free to drop everying after 17th place after comma in this case.
You say "constraints of the type are undefined". I don't see how "double-precision 64-bit format IEEE 754 value" from ECMAScript spec is lesser defined than "decimal numbers with a minimum of 18 decimal digits".
You're right about decimal precision however so I conceded there but the precision and capability is defined.
{ number: "1.0000000000000000000000000000001" }
How is it worse than XML?
It's conceptually easier to think about JSON with simple data sets but it's terribly inflexible and you have to think about how things are represented inside strings. A couple of thought exercises on JSON:
1. How do you represent an image inside JSON?
2. How do you represent a reference to another part of the data in JSON (consider a DAG for example)?
3. How do you represent an ordered set or an unordered set in JSON?
4. How do you represent an unsigned value in JSON?
Use the right tool for the job.
Sure, if you don't consider "human-readable" a good justification. Some of us do.
No, they don't.
> Imagine HTML in JSON.
Yes, sure, JSON is a crappy text markup language, and would be much less readable than HTML for that purpose.
OTOH, readability when used as a markup language for content consisting largely of prose text and readability when used as a structured serialization format for data that doesn't mostly consist of large blocks of annotated prose isn't necessarily the same thing.
That's a highly-subjective and controversial point.
(To me, XML is the Java of data languages -- its a lot worse than the best alternative considered on its own for almost any purpose -- though the best alternative will vary by purpose -- but it has a fairly wide range of uses for which its not intolerably bad, and its often a better choice than its inherent features would suggest because of the strength and maturity of the ecosystem around it.)
For 100% of the XML feature-set I don't actually know of a viable alternative. If you are using XML for the right reasons and the right way (rare) there is currently little or nothing that can replace. That being said, there are a vanishingly small amount of problems that actually require XML - namespaces and extensibility are two of them.
Real problems rarely need 100% of the XML feature set to solve. The breadth of the feature set is why there are lots of problems for which XML is a tolerable solution based on its inherent features (which in turn is a big factor in why it has such a large ecosystem), but they often don't make it the best solution (especially before considering the ecosystem, which is important in choosing a tool, but not a reason to avoid developing a new alternative, since any new alternative is going to start with an ecosystem disadvantage, but with adequate inherent value should be able over time to gather an ecosystem of tools.)
#/usr/bin/python n = 1.0000000000000000000000000000001 print n # 1.0
//Javascript n = 1.0000000000000000000000000000001; console.log(n); // 1
from decimal import Decimal
n = Decimal("1.0000000000000000000000000000001")
Run it through JSON lint ( http://jsonlint.com/ ), then fix the error (unquoted number text), then run it again and watch the data loss occur due to my original point...
And what exactly do you mean by "semantics around type handling are implied", especially in `{ number: 1.0000000000000000000000000000001 }` case?
Edit: So why the downvotes? How about a conversation instead?
And now with Schemas and editor support for them, I think it is an acceptable replacement personally.
I think you made a good argument, although I personally prefer the more mature XML tooling and metadata support for versioning.
On the contrary, I don't remember one single case when the object was described easily using "node/attribute" separation, but couldn't be described as easily using JSON. In fact, I don't even think it's possible, as you can always make two children for each object: "attributes" and "nodes".
So I guess it actually is unnecessary complication and not the benefit of using XML over JSON.
Its a nice soundbite, but it ends up being less than useful in practice, because all metadata is data, and almost any data can be viewed as metadata, a distinction which is both subjective and strongly influenced by the use to which a consumer is putting the data rather than being determined on the basis solely of the inherent nature of the data.
(A soundbite is something altogether different.)
JSON is a lot more "honest" in this respect, in that it's core data model is already useful for many applications, even without additional standards bolted on top of it. (Though those exist, this article being one of them)
So JSON succeeded because XML is not verbose enough? Really did not see that one coming.
And it's a good data exchange format if you're okay with its loose typing.
But if I need a strongly-typed, extensible markup language, I'd think really hard about inventing my own…
Well, I can dream..
No, it's a decimal number with attribute-dependant accuracy settings, with at least three different accuracy levels used in the same object used for different purposes.
(Just something I came across today at work. Thankfully, my SOAP library handles that automatically…)
That's what types are: enforced documentation.
That's an interesting way to think about it, but I'm not sure it applies here. From the example given about money being a string or a float. A type would just enforce either or, but the consumer of the data would still need to know what it is, and convert it to whatever is necessary for them.
I'm not convinced that "strongly-typed" is an attribute that can meaningfully be possessed by a data exchange format. "Strongly-typed" is about allowed actions in processing data, not about the inherently-static format of data exchange or serialization.
http://blog.codinghorror.com/xml-the-angle-bracket-tax/
http://nothing-more.blogspot.co.uk/2004/10/where-xml-goes-as...
SOAP's serialisation of RPC predated the popularity of JSON, and has largely been replaced by JSON for web-RPC through REST APIs. Why? Because it's needlessly verbose and complicated.
(I do worry that new serialisation formats are being developed in a vacuum, and we'll reinvent ASN1 or something)
An interesting and useful criticism would first engage with the strongest arguments of Rich Hickey (creator of Transit and edn). If you find something in his Language of the System talk (https://www.youtube.com/watch?v=ROor6_NGIWU) that you either (a) disagree with (b) think XML solves already, I and many others would certainly be interested in having that conversation.
Note that this doesn't mean the exposition of "Why Transit" can't be better, but that calls for constructive criticism on how explain the ideas better, or a question made in qood faith. What it doesn't call for is a hostile reply saying in effect "pff, already been done already, stop reinventing the wheel".
Personally I'm a fan of YAML for both being fast and human readable.
It's worked awesome for updates, and using Transit to keep the transmissions minimal has let us focus on the API for a realtime system.
[1] - https://github.com/ptaoussanis/sente#sente-channel-sockets-f...
It was a point to point system. Sente seemed to have better support for point to multipoint (like a chat app). It was overkill for what we were doing and chord fit the bill nicely.
There are other reasons as well, but deserialization latency is a big one.
I'm more partial to the way Avro does it, where the encoded JSON remains type-tag and cruft free, and a separate schema (also JSON) is used (and required) to interpret the types, or encode to the correct binary encoding.
Another feature Transit has is that it caches identical keys (can also cache values with some additional code), giving you a smaller footprint.
At the same time, I don't see how you can do anything sensible with Transit except throw it in to a hashmap/dict alongside runtime type information. This is natural for dynamic languages, but, without a schema, you're left to write you're still left to write your own error-prone structural validation, and can't fully leverage the efficiency of static languages. MsgPack has this problem as well.
Optimising the JSON representation just doesn't seem worth it to me. Avro for example encodes enums by exposing the type names as JSON keys. It sucks, and it's something I'd change, but at the end of the day you're building on a poor transport. Caching reminds me a lot of DNS label compression, or a poor mans preset zlib dictionary.
Another great thing (again, for someone using functional languages) is that Transit can ensure identity for equal values.
When it comes to structural validation, that is something I use prismatic.schema for. Of course, it helps that I can share the validation code between backend and server (using Clojure and ClojureScript).
When using plain JSON, I had to convert types myself after decoding/encoding, which was error-prone and gave a surprising amount of bugs (especially regarding dates).
Transit solves a big problem for me, while still retaining the easy to read syntax which JSON has, and also has great tooling in browsers.
Also, caching isn't only about smaller size. It allows for faster parsing of the initial data, allowing Transit to be just as fast as plain JSON for certain payloads, even considering that it has to expand from cache and decode values.
http://swannodette.github.io/2014/07/23/a-closer-look-at-tra... http://swannodette.github.io/2014/07/26/transit--clojurescri... http://swannodette.github.io/2014/07/30/hijacking-json/
There's also a) it's fully schemaless and b) you can add new primitives. Which you may or may not need. But if you're looking for something that does a) and b) while being fast for browser communication, transit's currently your best option.
I generally prefer JSON as a serialization format for a variety of reasons beyond what you cited:
1. Simplicity/plain text. Even a total newb could read a JSON file and probably puzzle out how the data works. This is important if I'm providing an export function or otherwise expect my users might want to shop somewhere else to deal with this data.
2. Forward compatibility/future proofing. If somebody finds my JSON file in twenty years with no documentation, (1) ensures that it's not going to be lost to them and they'll be able to extract the data with minimal difficulty.
3. Cross platform. It might not be the best format, but it's going to be supported in pretty much any language I would ever want to add support for the data to in a new system.
Either way, the advantages of binary often outweigh the difficulty of having a human read the data. As the saying goes, code is executed far more often than it's read.
(Most binary formats have some fairly obvious possible text representations, at least for key fields (they're just encoding ints, floats, strings, bitfields, etc.). When you're having difficulty, or you're simply curious, you can print them out. If the format is any good, malformed messages are easy to check for in code - this code is not harder to debug than any other. If anything it's actually easier than with some kind of text format.)
I regularly MITM a connection between a mobile app and a HTTP server, when something isn't going quite right. A look at the JSON they exchange exposes the problem in under a minute more often than not. If I want to test something out quickly, I simply modify the incoming / outgoing JSON by hand. It's rare to get the syntax wrong and involves no context switches. I can then go back to the code, find the relevant section and make the changes I need to make. I find this a very convenient way of debugging and I don't think it would be as nice to do this with a binary format.
What does it even mean that something is string based?
Here: https://github.com/eishay/jvm-serializers/wiki a JVM bench where JSON is pretty competitive with binary formats [and results would be event better if project owners actually merged PRs ;( ]
Transit is self describing using tags, and encourages transmitting information using maps, which have named keys/fields which will repeat often in data. These overheads, typical of self-describing formats, are mitigated in Transit by an integrated cache code system, which replaces repetitive data with small codes. This yields not only a reduction in size, but also an increase in performance and memory utilization. Contrast this with gzipping, which, while it may reduce size on the wire, takes time and doesn't reduce the amount of text to be parsed or the number of objects generated in memory after parsing.
The extension mechanism is writing handlers in all languages communicated with, since its stated purpose is cross language value conveyance.
In contrast, a schema language allows extensions to be described once, in one language.
I was expecting this to be a sort of macros for data notation (an inline schema language), but it seems more like an extendible serialization library.
No, it doesn't. A schema language allows that to be done for the syntax, but requires the semantics to be implemented for each language. Schema languages often include, as part of their specifications, core types which must be supported; when an application restricts itself to these core types and types whose only important semantics are derived from them (e.g., restricted subsets of core types in most cases), then the fact that every full implementation that supports the core types will already have this work done for every language means no additional work is necessary. But that's not a product of a schema language preventing types to be implemented for all host languages, that's a result of the fact that a predefined set of core types accompanying the schema language means that all implementations are required to have already done the work of implementing the core types for all languages.
Are folks using it for internal services?
http://www.lispworks.com/documentation/HyperSpec/Body/02_dh....
Or use bookmarks :)