APIs as infrastructure: future-proofing Stripe with versioning
stripe.com
stripe.com
We're in the early stages of deploying a new RESTful stack, and versioning is a hot topic (along with getting people out of the RPC mindset and into a resource-based paradigm). While version bumps should be much less common, we'll probably end up doing something similar to our cascading transformations. Essentially, the old version becomes a consumer of the new version, and as long as the new version continues to hold to its API contract, everything should work with minimal fuss. Of course, that's assuming that we don't change the behavior of a service in ways that aren't explicitly defined in the API contract...
As a raw concept, I really like this idea. Let's say you're bumping from 2.5 to 2.6 and there's a breaking change in-between.
You replace the old api code with a new thin layer that consumes 2.6 and puts it in 2.5 format. This would be easy to write a series of tests for to make sure that 2.5 is still providing what it should, and it has the added benefit of giving your engineers the opportunity to consume the new api internally in a structured way, so that would be a good way to catch any bugs you might not otherwise find.
And of course, the 2.4 version already only consumes the 2.5 version, so it's turtles all the way down.
That's a great concept. You would have to still deprecate APIs or provide an incentive for users to update to the newest version, otherwise the oldest versions would create a long chain of requests that add unnecessary overhead. But otherwise I love it.
The problem I guess is breaking changes come easily - it's fine if say we have example.com/homeaddress and now I add a zip code field in 2.6 - but a 2.5 request has no zip code and if 2.6 makes that mandatory it's really 3.0 - that just feels wrong. But it's a breaking change so, ok. But you get a deeper discussion then over why make a lack of zip code a breaking chnage ? Interesting
1. Request for 2.5 comes in
2. Version 2.5 calls v2.6, takes the data, and transforms it into 2.5.
3. Since a 2.5 response shouldn't have a zipcode field, it's dropped from 2.6 before the 2.5 response is returned.
Interestingly looking at stripe changelog https://stripe.com/docs/upgrades#api-changelog they still do breaking code changes every month or two so despite the effort and with a smallish api they break a lot of contracts.
The odd thing is that they keep a different api version for each customer, tagged at customers settings, so I suspect your approach (spin up api based on incoming version) might work better - but boy I think they may be in a lot of pain trying to maintain that backwards compatibility
And SOAP is usually schema-verified, so adding fields will generally be a breaking change.
To the point of the XSLT being so trivial that maybe you didn't need the change, I recall one of our changes being around a field that got split into 2 fields. We certainly could have put special case code in to handle old version vs new version, but the great thing is, we didn't have to put that code in. It really keeps the interface a lot cleaner.
Also, XSLT is capable of some pretty advanced transformations, so I wouldn't dismiss it out of hand as triviality.
Regarding the documentation, we build all our docs off Javadoc and a few annotations, so docs were always current. As a bonus, when your Javadoc is used to generate public docs, you tend to treat it with more respect than Javadoc usually gets.
We did something like this in the on a message bus: the channels had the version in the name and and older version service always asked for the same data on the next version channel (the only other version it knew about), and then transformed it before sending it on. This meant you never had to maintain anything accept your newest version. It also meant you could never completely remove data because versions going obsolete needed a way to find or derive it to create their older version.
For anyone else who's interested, they've written/talked about this a few times over the years, to fill out the picture:
- http://amberonrails.com/move-fast-dont-break-your-api/
- https://www.heavybit.com/library/video/move-fast-dont-break-...
- https://speakerdeck.com/apistrat/api-versioning-at-stripe
- https://brandur.org/api-upgrades
- https://news.ycombinator.com/item?id=13708927
It sounds like their YAML system has changed to be implemented in code instead, which maybe allows the transforms to be a bit more helpful/encapsulated. If anyone from Stripe is here, it would be awesome to know if that's true and why the switch?
Additionally, devs would need to specify what their changes were independent of where they made the change, which meant that our API reference (which can display warning flags next to changed fields) was missing changes. With the new system we can enforce that the change being made is properly documented, since we know that it's encapsulated inside of the change class itself.
In general, the concepts employed by Stripe really encourage better design choices. All changes, responses, request parameters, etc should be documented and then handled automatically by the system. We took this approach in our design, although we don't do it with an explicit "ChangeObject" like Stripe does; it's a great idea though.
Hoping to be able to put out a blog post once we start implementing the system and getting feedback on what works and doesn't work well.
For example, in those cases where the response object was fundamentally different, I've added a whole new Query field (and Mutation field) at the top level, that returned the new object. Under the covers there was still a bunch of complexity needed to support both top-level query fields, but old clients continued to access the old query, and new clients went to the new query.
Also, two other things I really like: With GraphQL it's trivial to log which clients are accessing which fields (at some point you can just decide to kill old ones). Also, the @deprecated support in the GraphQL schema IDL integrates wonderfully with the GraphiQL tool, so anyone browsing your schema for the first time always sees the recommended latest version that they should use.
This is a really smart way to do it.
One question is, over the years, wouldn't you add a lot of overhead to each request in transformation? Or do you have a policy where you expire versions that are more than 2 years old, etc? (skimmed through parts of the article so my apologies if you already answered this)
Stripe probably tracks the relative usage of each API version. If they found a lot of their users were stuck on an old version, that would point to a bigger problem than just some per-request overhead.
I'm not sure how true that is. In my experience, integrations with online payment services are mostly write-only code: you do it, you test it, and then no-one goes near it once it's in production unless there's some sort of known bug or security issue.
Literally the last thing I want to do with working, tested code integrating with the service that collects money for a business is make unnecessary changes that might break it. I know several businesses that use versions of APIs that are several years old with services like Stripe, because they have no need to change.
- They apply transformations to convert responses from the current version to the previous version, step by step.
- This means the overhead for old versions only applies to consumers of the old version.
- They support all versions going back 6 years.
Does anyone know of packages that do this already? I have been contemplating creating one in PHP/Laravel for a long time but haven't had the time yet...
And it's underlying Transformers object API .... ?
https://github.com/bwillis/versioncake
"Version Cake is an unobtrusive way to version APIs in your Rails or Rack apps."
I wish other payment services treated their long-time clients with the same respect (looking straight at you, GoCardless).
Hey, @pc with all the spare time your team has accumulated by using this api model maybe you could put it to good use. Might I suggest it's time to divert most of your tech resources into creating the next Capture the Flag? Because those were just awesome!
I'm joking, in case it's not obvious (but I would absolutely love another Stripe CTF).
From a consumer of Stripe's API's perspective, doesn't this make debugging or modifying legacy code a real pain? Let's say I'm using Stripe.js API's from a few years ago; where do I go to find the docs for that version? Do I need to look at the API change log and work backwards?
"We also tailor our API reference documentation to specific users. It notices who is logged in and annotates fields based on their account API version."
I opened the docs in another browser to compare the differences and I am indeed seeing my older version when signed in. It's all very behind the scenes...I would have gotten tripped up by this had I tried to make changes to my code, and I wouldn't be surprised if many others have run into this issue.
My preferences would be for there to be multiple versions of the docs and have your version be very clearly laid out to you rather than automagically updating docs based on your pinned version.
It was a delight to get a peek behind the curtain. :)
I don't understand why they didn't just use the X-GitHub-Media-Type header as a request header.
Nothing in the spec says the Accept and Content-Type must match.
I get that from a practical standpoint the server needs to send back a JSON Content-Type, otherwise a lot of clients won't understand that it's JSON and won't decode it properly. But given this, shouldn't they have picked a different header to declare the API version?
What's the benefit of using the Accept header in this fashion? AFAICT there is no benefit at all over just using a custom header.
No it isn't. If you're doing REST and your resources are not completely trivial (arguably, even if they are) you should be defining a custom media type for them (think HTML5).
>but then they're not actually using those media types as the Content-Type of the response.
Ok, that is very weird.
Have run into more than one issue with the vary: Accept being ignored, resulting in everyone getting the result from the first man in regardless of header variance.
Especially noticeable when we tried supporting both JSON and XML via the accepts header, with the version in the mime.
Say it is splitting of street into street1 and street2 for an address element. How would I know what version to target to get this feature?
1. Mandatory new field.
2. Field is split. For example, address field is now divided into street1 and street2.
3. Change in datatype.
In the above three cases, we had to force users to upgrade their versions.