JSON Schema

I'm trying to find time, among many blockers that I have to take a look at Vega Lite - as another alternative/addition to kibana or grafana, also last night found DejaVu - https://github.com/appbaseio/dejavu - and yes, JSON schema comes off often enough, since we moved a lot of our configuration there, and hated or not XML had the XSD sanity...

This light bulb is exactly why many people prefer statically-typed programming languages, especially above a certain codebase size/complexity and team size.

Make my tools do the grunt-work for me, thanks, so I can focus on the actual problem domain.

Working link: https://vega.github.io/schema/vega-lite/v2.json

https://jsonschema.net may help you get started without the pain of writing boilerplate code.

Unfortunately it doesn't perform validation, nor implement all validation annotations - but it definitely helped me get started.

It is considered essential that JSON schema is expressible as JSON. There are advantages to that, but it could be only a wire representation instead, with some more concise syntax alternative.

Because json-schema is language agnostic. You can have a schema that can be shared between servers/clients with different languages. If you want to do it the TypeScript way, then you can generate the types from json-schema (probably via an automated tool)

There is nothing wrong with Typescript definitions if that works for your use case. Json schema is more flexible:

- it can do checks at runtime (to validate untrusted input, for example)

- there are libraries for many other programming languages

- it can express constraints on values as well as on types (size of arrays, string regex matching, integer value ranges, ...)

I built a tool to generate JSON Schema from typescript interfaces: https://www.npmjs.com/package/ts-json-schema-generator

The reason for having the schema are that they are language agnostic and there is wide tool support. For example, vscode supports autocomplete and tooltips when you use json schema.

So, in typescript a number is unsigned by default? If not, your typescript representation has lost some information compared to the JSON schema example, and would theoretically allow for negative ages.

Additionally, what you've chosen is a very simple example, and even in that case you're having trouble correctly mapping the types to what is expected. What about a slightly more complex one[1]?

    {
        "$schema": "http://json-schema.org/draft-06/schema#",
        "type": "object",
        "properties": {
            "/": {}
        },
        "patternProperties": {
            "^(/[^/]+)+$": {}
        },
        "additionalProperties": false,
        "required": [ "/" ]
    }

 1: http://json-schema.org/example2.html

No, there's nothing wrong with that and it's usually significantly more readable than JSON Schema. Good idea!

Of course there are use cases that JSON Schema works better for (eg most other replies you got) but I fully agree that TypeScript type definitions are a great way to specify the structure of JSON data.

To make it feel more language-agnostic, you could consider using type aliases instead:

    type Person = {
        firstName: string,
        lastName: string,
        age?: number // Age in years
    }

This is equivalent to the interface you described, but potentially less confusing to readers who don't know what an "interface" is (eg because they mostly used Ruby or C++). You don't even need to tell people that it's TypeScript - this could be a perfectly sensible schema language for JSON. A bit like what RELAX NG[0] is for XML.

At my company we actually use typescript-json-schema[1] so we can specify the structure of our API payloads in TS instead of JSON Schema and still benefit from JSON Schema's better tool support.

[0] https://en.wikipedia.org/wiki/RELAX_NG#Compact_syntax

[1] https://github.com/YousefED/typescript-json-schema

AFAIK typescript interfaces don't exist at runtime, so you can't validate incoming JSON with type any.

Also you can't really hand a third party a typescript interface and tell them to conform with that.

And finally, I would imagine a good schema could enforce more things like the length of identifiers, whether a number can be negative, etc.

TypeScript provides, primarily, a type verification system.

JSON-schema provides a type assertion system.

Both serve as type documentation systems.

TS says "if you create objects in my world, I will use my type system to verify that you're creating the right kind of objects, and you can look up their definitions to see what's in them". JSON-schema says "if you give me external, un-annotated objects, I can tell you whether they comply with a type definition, and you can look at that definition to see what's in them".

TS has a limited assertion facility, but it's not nearly as capable as JSON-schema. Projects like io-ts (https://github.com/gcanti/io-ts) add more functionality to those assertions, but they're still much less easy to use, in my opinion, than JSON-schema for this purpose. If you're using TS types to assert, though, you can definitely describe more complex constraints, so it may make sense as an alternative to JSON-schema for some people.

As others have pointed out, JSON-schema is also much easier to use on non-TypeScript platforms than TS assertions, which, even if you use an assertion library like io-ts, will still be limited in use to JavaScript (and JS-to-$other_language runtimes).

We have a similar setup for extending our app with custom fields / forms using ruby-json-schema and react-jsonschema-form [1].

ruby-json-schema has worked out quite well, we added a custom schema reader so that we could generate enums dynamically based on records in the database via custom refs eg. app://schemas/organizations?type_eq=education&tags=one, any of these internal refs are inlined at the API layer.

One issue we did run into with ruby-json-schema, is the schema caching is not thread safe. We opted to clear the cache after each validation, this ended up causing race-condition issues. In the end we had to use a mutex on write and cache the validation result (not ideal, but our app isn't that write heavy).

[1] https://github.com/mozilla-services/react-jsonschema-form

My first thought on seeing the headline was "continuing to reinvent the XML ecosystem...in JSON"

XSD gets thumbs up from me.

Generate useful docs. Easy to read to get a handle on data structure. Also can be used as runtime contracts. somewhat similar but v. limited compared to Racket Contracts and Clojure spec.

Not to mention super tooling for interacting with schema (like XMLSpy)

In conclusion, having schemas for data structures(especially at boundaries) is absolutely necessary to maintain control & visibility over data.

Yep, I use it heavily in FormAPI (https://formapi.io). One of the core features is schema validation, to make sure that PDFs can only be generated with valid data.

Yep, I used it to design and document a schema for a recent client of mine. What was really nice was the backend developer directly used the schemas to validate requests to our flask http server just by adding decorators to the routing functions. It was a great tool for us to coordinate API changes across front and back end teams.

We use it for validation for data that goes into our Postgres JSONB fields. It's a nice middle ground from forcing some unstructured data to be serialized/deserialized from protobuf to enforce its schema and some half-baked server side validation that lives in the application code.

We used to, the idea being we could reproduce validation in JavaScript and Python. It kind of worked but it wasn't that clean. Ended up switching the server to Marshmallow schemas and just having more custom logic on the client side. It's a classic DRY ... this is where "duplicate" code was better for more control and less shoehorning.

I have used it a few years back when designing a customer facing integration API, it was definitely good to be able to tell customers "here's the JSON schema, you can use these libraries to work with it, and use it to validate your requests for correctness before sending them to us". This said I felt expressing non-trivial constraints was quite verbose (things like if field X is set to 123 then fields Y and Z are mandatory, otherwise Y is not allowed, if X is set to 456 then no other field is allowed, ...) and hard to extend once written, I definitely felt the need for a "JSON schema generator" at times where I could've created a table of some sort and have it output the schema.

In another much more generic customer facing API I instead used a custom schema, as I needed to do other things that would not have been possible in it like embedding the API documentation in the schema so the front-end could display help on-the-fly and use the schema to dynamically generate forms. The advantage was also that being fully in control of the syntax I could create ways to express constraints like field X is valid only in searches, field Y is valid only in PUT/PATCH but not POST, etc. etc.

This said especially for straightforward APIs I feel JSON schemas are quite useful, being able to have a single source of truth on what your messages look like that you can share with front-end, back-end and customers is quite powerful. One thing is though that validation errors are not very user-friendly unfortunately. This was all around draft-04 time IIRC.

Yes, but many of the implementations are a nightmare to use if you have a complex schema and you want to return a meaningful, useful error to API users. To be fair, there can be ambiguity making it difficult for the library to return the correct error such as using "oneOf". Two things make a difference.

1. if/then/else which was added in Draft-07.

2. Custom errors. For Node, I use ajv and ajv-errors.

With the combination of if/then/else and custom errors, I have found that I can always return a single, useful error to the API user regardless of the schema complexity.

We did at one company I was at to do some code generation (specifically in Java[0]). Since we never ended up using it to generate client libraries the gains weren't as significant as originally planned.

[0]: https://github.com/joelittlejohn/jsonschema2pojo

I have done it a bit lately. To document to other companies what they can send or expect to receive. Trying to avoid having too much business rules in it, though, mostly just fields and their type. Compared to xsd for xml I find the tooling lacking.

Yes, Snowplow uses JSON Schema extensively for event definition and validation (https://github.com/snowplow/snowplow/).

jsonmerge is heavily based on JSON Schema.

https://github.com/avian2/jsonmerge

Heroku does, and has built some nice open-source tools around it: https://blog.heroku.com/json_schema_for_heroku_platform_api

Yes, we use it in Nakadi (https://zalando.github.io/nakadi/). It's an event broker platform based on Kafka, and we use json-schema for event validation.

Yes, found it highly useful for a JSON-based configuration with a GUI for editing (towards not so technical users). The GUI was fully based upon the schema (i.e., dynamic by using things like the descriptions, types, and constraints for displaying / input validation).

The only thing that is counter-productive is the different schema iterations; at the time v3 and v4 have been different in some key areas. Haven't looked into v7 yet, but I hope its not "breaking as much" as v3 vs v4 did.

Yes because it documents what should go in the corresponding JSON, and some editors will automatically add completion.

We export massive amount of JSON downloads for our customers, with often different features enabled. So for each customer we generate a different schema for validation.

Yes. A library I write (Keras-RCNN) uses it to validate input before starting the expensive augmentation process.

Shameless self plug here, few months back I created a project https://www.contenttagger.com that gives you hints for Schema tags based on a parent Schema attribute. You can also validate it with Google devtools + Facebook graph.

It's used to great effect by Vega and Vega Lite - see e.g. here: https://vega.github.io/vega-lite/examples/bar.html

See also my comment here: https://news.ycombinator.com/item?id=16407707

Not since I discovered GraphQL. It's hard to imagine any other abstraction on top of REST APIs that packs as much robust functionality in something so easy to use.

Absolutely. When you have more than one team in different locations and cultures working on stuff in parallel you end up specifying your contracts. Many ways to do so, but nothing as binding as a JSON/XML Schema including samples.

Yes, I use it to trigger actions on certain messages in one of our APIs instead of hardcoding them.

Incoming data -> is it a valid messages at all? -> if it is a message of type x, hand it over to class X -> if it is a message of type y, hand it over to class Y -> etc

I found that to be a very easy way to setup a general-purpose processing pipeline.

Could you provide a link to your startup's page? I'd like to check this out.

I built this same thing in the past month for automated questionnaire rendering in react, saving and merging the responses into word documents.

I use that in production code for checking input for correctness.

I wouldn't say JSON without schemas is necessarily too problematic, I've used that a few years ago. Now I prefer JSON Schema for extra confidence and more convenience in pointing out problems.

We "use" this in the form of Swagger, which is a web service that can download a specification of how an HTTP API works and generate a minimal UI to use that HTTP API from that specification. While the specification of the API is in "Swagger", the actual documentation of types used by the API is JSON-Schema.

The UI is okay; however, we've had two big downsides:

* The UI needs to be able to fetch the JSON schema document from the API; that means that it needs to correctly respond with CORS headers if you host the web interface on a separate domain. (We do, s.t. each API service doesn't need to repeat the mundane task of hosting it and keeping it up to date.)

* The schema needs to be useful. Far too often, people will document an API endpoint as taking "JSON" and returning "JSON"; this is true and correct, but not helpful. What keys exist? What are the valid values for those keys? etc. The author of the schema needs to take the time and thoroughness to document the API, and all too often either they do not (the workmanship is sloppy) or they can not (management provided deadlines that were too tight to get the job done right).

Neither of those are really problems with JSON-schema per se. You also don't need JSON schema to document a JSON API: you can always do so with prose, by using the type system of a language your devs are familiar with and providing a set of rules to map those structures/types to JSON object, etc.

My biggest nit with JSON schema: it's painful for humans to interpret. (And we actually do it in Swagger, which is in YAML, so it's slightly easier to parse and we get comments. But it's considerably verbose compared to most type systems; the upside is that it can encode more information such as descriptions, or valid ranges/values if the raw type like "number" or "string" doesn't suffice.)

Does oneOf not cover it?

I agree wholeheartedly (I have so many use cases for a sum type!), but I think that opportunity was lost when JSON itself was created; JSON-schema, built on top of JSON, would have to make some assumptions about how to map any sum type onto JSON, since JSON doesn't natively support them. JSON-schema — reasonably, I think — doesn't.

Seems like a lot of work that a Google search would have obviated...

Your site's CSS makes it unreadable on wide screens: https://imgur.com/a/WlYOA

It's Groundhog Day all over, and the cycles are getting shorter and shorter.

You'll love this talk by Robert C. Martin ("The Future Of Programming"): https://www.youtube.com/watch?v=ecIWPzGEbFc

What, JSON shouldn't have schemas because they exist for other formats? Or JSON shouldn't be used because XML is good enough?

While I agree that tech should reevaluate its sisyphean upgrade treadmill, I'm not sure how that relates here.

Please comment civilly and substantively or not at all.

https://news.ycombinator.com/newsguidelines.html

I was looking for a comment that explained how json schemas are similar or not to xsd but your rant was the closest I could find amongst all the hype that this is great.

Did you have other technologies in mind or were you also thinking of xsd?