Testing in production: using JSON Schema for 3rd party API response validation | Hacker News Reader

Testing in production: using JSON Schema for 3rd party API response validation

56 pointscachehit22 comments

I tried to introduce `zod` (https://github.com/colinhacks/zod) to achieve the same thing within my current team.

The idea was that API responses that weren't reflected in the types would cause hard errors that we had to fix, and that we'd be forced to make sure that the mocks in our tests were up-to-date valid responses. Similarly to the article, I also made it so that in production they would log warnings, because I didn't want to expose us to the possibility that validation could take down the app.

However, TypeScript is new to my current team, and so nobody wanted the introduction of a validation API on top. It ended up being ~20 lines of code I couldn't get merged. Since TypeScript was new to the team syntax like `z.infer<typeof User>`, generic code and a whole new way to define types at runtime made them feel like it was too complicated.

Did others have similar problems introducing something like this?

I struggled introducing fp-ts into my last team but io-ts was a much easier sell, as was fast-check.

It's a lot easier to get buy-in when you can demonstrate that it will reduce bugs thereby improving the customer experience and reducing development burden.

Something like io-ts or zod gives you confidence that the types in your application are correct, so you don't need to code or unit test defensively. This ties into the idea of pushing unsafety to the edges of your application. It can also be a quick way to discover that types don't align because of a backend API change; we had our io-ts decoding hooked up to Sentry.

Likewise with fast-check I argued its utility in its introduction PR and made clear that it's an opt-in method of testing, and it caught a bug we hadn't spotted within a few weeks. You can't really argue with results like that.

At a certain point though, if your team is unwilling or unable to recognise the limitations of TypeScript and therefore the necessity of constructs like `z.infer` or `t.TypeOf`, you might just have to accept it or move on. In my experience you can't move a team that far from where they'd already be heading without your presence. If you're surrounded by folks without experience in static typing, it's already a major win to have introduced TypeScript at all.

Yes definitely. I have trouble just introducing Python type hints, despite the clear insanity of working with a large convoluted untyped codebase.

> and a whole new way to define types at runtime made them feel like it was too complicated

How you made this happen? My impression was that TypeScript doesn't have runtime types and stopped looking at TypeScript after that as it basically makes TypeScript useless for myself and my team.

Libraries like io-ts and zod take care of this for you. For io-ts for example you just need to define a type guard function that asserts "x is Y" and away you go, you've got a runtime codec; it knows how to "decode" data, and it knows what that represents to the type system at compile-time. You'll generally only need to use the built-in combinators though e.g. `t.type({ prop: t.string })` produces a codec for `{ prop: string }`, and you can get that type at compile-time without effectively rewriting it using `t.TypeOf<typeof myCodec>`.

It's a nice library, although I personally prefer explicitly writing the type out beside the codec. Otherwise you have to translate the codec to the type in your head when reading the code without an ide that can show you the type.

This also gives you a "test" if you will that the codec is parsing the structure you think it is

There's a variant of this in $enterprise-dayjob: production system Foo is a service exposing a FooAPI with a spec documented in some machine readable format (SOAP ; Swagger/OpenAPI ; gRPC ). You don't need to infer the spec, you just get a copy of the spec in a machine readable format. You code generate a FooAPI client from the API spec, maybe you rig your API client to fail if it cannot decode a message with respect to the spec (it might do this by default if your code generator is any good). Maybe your new thing-with-a-FooAPI-client goes into production and you find out, lo and behold, the production FooAPI implementation doesn't honour its own spec, and encodes its data in some wacky way 5% of the time.

One stupid solution to this, that I would love to deploy, would be a circuit-breaker to detect API implementation defects. Similarly to how a circuit-breaker attempts to infer if a service is down/overloaded and temporarily disable the integration, an API-spec circuit breaker could be deployed with a copy of the API spec the external service is meant to be serving, and it could infer if that service doesn't actually implement the API spec. If it detected a counterexample from a message sent by the external service, it could disable the integration for an appropriate backoff retry interval to give the team maintaining the service enough time to align their spec or implementation (sleep 3 months to give the portfolio an opportunity to prioritise a fix; retry; sleep another three months... ).

Less stupidly, a lame but pragmatic solution can be to figure out the absolute minimal subset of the external API your use case needs in order to function. If an external service publishes an API spec with a huge service area [+], but you only need to use 1% of it, you might be advised to prune a copy of that API spec down that only focuses on the 1% you need. Validate that part of the message. Then if the external service does something weird in parts of the response you don't care about, you can ignore it instead of rejecting the entire payload.

[+] "service area" was a mistake, but it seems appropriate. maybe i need a circuit breaker between my brain and the keyboard.

Json schema validation is cpu expensive in nodejs, so you do not want it in production for every request. We made a proxy [1] that only allows requests that are conform to the swagger json which is easy to hook between frontend and backend in dev-mode

[1] https://github.com/EXXETA/openapi-cop

Back in the early years of Snowplow, we adopted a similar approach for our support of 3rd party webhooks[1] as an event stream source supported natively by Snowplow (https://github.com/snowplow/snowplow). (The important background here is that all events and entities flowing through Snowplow are described using versioned JSON Schema).

We built a kind of Maven Central for 3rd party webhooks' schemas[2], and got to work adding schemas for various popular webhooks, e.g. Sendgrid and Pingdom. But, we never met a single SaaS vendor who was interested in 'adopting' the JSON Schemas for their webhook, let alone publishing versioned JSON Schemas for their whole API. This meant that at Snowplow we have stayed on the hook for keeping these vendors' webhook schema definitions up to date in our registry.

It's a real tragedy of the commons - oceans of developer time wasted on tedious low-leverage work (updating API client code) so that each SaaS vendor can 'move fast and break things'. The ultimate irony is that if these vendors were to adopt, publish and respect versioned schema definitions internally, using something like OpenAPI, they would see huge productivity gains (think enhanced CI/CD testing, auto-gen of client code etc).

1. https://docs.snowplowanalytics.com/docs/getting-started-on-s... 2. https://github.com/snowplow/iglu-central/tree/master/schemas

Seems like this is a feature you could demand if you were a big enough client of their enterprise products. Better yet, get a critical mass of big clients together and push for it. You could probably do this by showing up at a product-specific conference and networking with the other devs who care enough about the tool to show up.

Also, if this is a genuinely useful thing to have, you could probably monetize it directly. Decouple the schema repository from your primary application and sell access to well-validated, performant client libraries for popular SaaS APIs.

As someone who's about to assist a frontend team to implement testing against a backend I've developed (my first time doing something like this), I can see that mocking backend APIs is inherently tedious and error-prone. And I'm not confident that we have the resources to keep the mocked endpoints consistent with the real backend given the pressure to develop the "real product".

What are the downsides of of using a real API (e.g. dev environment) for automated testing?

  > What are the downsides of of using a real API (e.g. dev 
  > environment) for automated testing?

It can be difficult to recreate the situations required to produce particular responses, and if you are using this development environment to deploy backend fixes, downtimes or bugs will sometimes block the engineering work of your UI team.

> What are the downsides of of using a real API (e.g. dev environment) for automated testing?

Slower, harder to set up state the tests need.

Front end teams should also be testing their code against a broken back end; eg http 500 responses.

Consider adding "additionalProperties": false as you tweak the JSON Schema. That way, you will also get alerts as new fields are added. https://json-schema.org/understanding-json-schema/reference/...

I like this approach a lot. I wonder if it's a good idea to keep the assertions in prod as well. I guess this is a trade-off between safety and convenience..

As a side note, what do people think about JSON schema? I find it quite verbose and cumbersome (compared to, say, typescript type definitions)

> As a side note, what do people think about JSON schema? I find it quite verbose and cumbersome

Agreed. But if you want to have validation and autocompletion in most editors, it is still the way to go. We use it to validate JSON content files for a CMS [1]. It is a little repetitive in our case (e.g. the `:name`/`:description` keys are repeated), but not that bad. Newer versions of JSON Schema [2] have improved in that respect, but most editors (or plugins) and tooling in general is still based on older versions. So we have to stick with those for a while.

[1] https://github.com/frontaid/schema/blob/master/frontaid-sche...

[2] https://json-schema.org/specification.html

If you have to deal with JSON a lot, you might want to checkout https://www.schemastore.org/json/ which has schemas for a large number of file types.

I used https://github.com/ForbesLindesay/typescript-json-validator for a service I built recently. It parses TS types and turns them into a validator function - seems to work very well so far.

There’s a conversion of the type to JSON schema under the hood but I don’t have to worry about doing it by hand.

I've used JSON schema a fair bit for config files for provisioning infrastructure and its been great. For example I have a kubernetes clusters JSON file cleanly schematized which specs all the clusters I need (node pools, metadata etc), which I generate TS types from using 'json-schema-to-typescript', and then just iterate over these in pulumi. All I need to do is update the JSON file to update my infra, and the coding experience is brilliant with pretty much complete type safety.

I wouldn't write JSON schema definitions directly. Check out typebox. It lets you define a schema in memory and immediately have access to the types https://github.com/sinclairzx81/typebox

Combined with ajv you can even have more advanced custom validation with custom error messages.

It seems fine for validation, but when I want an IDL it seems to be vague in describing the indended data structure. This is in terms of cross platform dev and far from JS. It’s fine for describing the string like things, but when getting into sum types it doesn’t give more than `dependencies` to describe a tag descriminator. Going the other way, that doesn’t say more than both need to be there when the sum type is present.

It's a bit of a shallow analysis but I lack experience about this so excuse me in advence. I see more and more people importing a lot of libraries to write Typescript code. Part of it is libaries to do things (like in the article), part of it is libraries to change the language (like fp-ts). In my (limited) experience I saw the same thing with Scala, with people using cats or equivalents. I wonder, does this happen in other languages to? Does it lead to ecosystem splitting? Is this the sign of a language becoming "enterprise" or "mature"?

I think fp-ts exists because TypeScript is entrenched with a strong job market. If that weren't the case most users would just use something like PureScript.

Reply on news.ycombinator.com