Implementing Microsoft REST API Filter | Hacker News Reader

Implementing Microsoft REST API Filter

sergeykibish.com

67 pointsskibish43 comments

don't make me tap the sign:

https://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypert...

> I am getting frustrated by the number of people calling any HTTP-based interface a REST API. Today’s example is the SocialSite REST API. That is RPC. It screams RPC. There is so much coupling on display that it should be given an X rating.

> What needs to be done to make the REST architectural style clear on the notion that hypertext is a constraint?

You can tap the sign as much as you want, that battle was lost a long time ago. REST is just the common term people use for HTTP+JSON RPC.

DONT THINK I WONT

"Ugh... REST is too much work! We'll just change the meaning to suit us."

Why should people do hypermedia if it is too much work?

The Web (the prototypical REST application) had a lot of good ideas, not just hypermedia. It certainly makes sense to attribute those good ideas to REST.

> "Ugh... REST is too much work! We'll just change the meaning to suit us."

See also: Agile. DevOps.

It's pretty common once you look.

You gotta say RESTful if you want to emphasis RESTfulness. But I personally don't care. I don't mind JSON RPC.

Then say that?

Rest pedants don’t care if you don’t use rest, as long as you don’t call it rest. It’s hardly a difficult concept.

Yea, but nobody cares what REST pedants think.

True that. I use it incorrectly with non-technical folk as they may understand what a REST API is, but when I say "it's more like an RPC interface" their eyes glaze over.

… and it's not just the hyperlinking that's problematic. (Yeah, MS's unRESTful "RESTful" APIs do a huge amount of coupling in the form of URL building.)

E.g., in Azure, which is also a "RESTful" API that has no idea what REST is about, MS completely misses Fielding points that most of the effort of definition should be spent defining the content / data's format, not things like URL structure. That way we can speak about MIME types / content-types, and know what structure we're describing. But Azure will happily describe in JSONSchema a single type, and declare that it is used for both PUT/GET, and … it's not. And discovering the additional constraints that exist on the type in the PUT is gleaned only through calling the API, certainly not through Azure's docs. And that's assuming you get a usable error in response.

JSONSchema is also a bit of a disappointment. On the one hand — yay, a spec? But on the other hand, it fails to capture so, so much. Half the fields in the type will be required … and the schema will say they're optional. Sum types of any kind are particularly badly handled, and half the time are just "string" though I think this is more of a failing on MS/Azure than JSONSchema, for simple string-like enums; but more complicated sum types, IDK if JSONSchema can't cut it or if MS just doesn't get it or what. For example, to instantiate a VM, the request body looks something like:

  body: required struct {
    properties: optional struct {
      storageProfile: optional struct {
        imageReference: optional struct {
          communityGalleryImageId: optional string,
          exactVersion: optional string,
          id: optional string,
          offer: optional string,
          publisher: optional string,
          sharedGalleryImageId: optional string,
          sku: optional string,
          version: optional string,
        }
        osDisk: optional struct {
          createOption: optional enum { "Attach", "Empty", "FromImage" }
          image: optional struct {
            uri: optional string,
          }
          managedDisk: optionalStruct {
            id: optional string,
            // omitted fields
          }
          vhd: optional struct {
            uri: optional string,
          }
        }
        // omitted fields
      }
      // omitted fields
    }
    // omitted fields
  }

I've listed only the fields used in determining where to source the VM's OS disk from. And it's nuts! "properties" and "osDisk" are actually required; if you specify "imageReference" or "image" or probably "vhd" (but I've never used that myself), "createOption" must be "FromImage", if you specify "managedDisk" it must be "Attach", and the docs don't describe what meaning "Empty" has. You can specify only one of those, because otherwise, you're saying to source the image from two things which would be nonsense (but is permitted by schema/docs?).

"imageReference" itself is really a sum type; you must specify (offer, publisher, sku, version[, exactVersion]), or communityGalleryImageId, or sharedGalleryImageId. You could image it being,

  enum ImageReference {
    FromMarketplace { offer: String, publisher: String, sku: String, version: String, exactVersion: Option<String> },
    SharedGallery(String),
    CommunityGallery(String),
  }

And we've not even touched VHDs, managed disks, or VM images yet! And you don't need createOption.

I think, again, I'm going off what I've learned the hard way about how Azure works. I shudder to think what the validation logic looks like. (I'm also reading the docs. Reading JSONSchema is … painful to start with, but Azure's schema's directory layout structure makes it triply painful.)

But even that sum type is to miss the point of REST entirely. The RESTful definition would be:

  image: URI-reference

and that's it. The Content-Type of the content at the provided URI provides the type of image that it is.

Oh and while I'm here: don't choose a boneheaded page size if you paginate an API call. Half of Azure's services will trickle-feed you 100 records at a time, and so the response body is like 60 KiB. Since the payload also has the next page's URI, your calls get decimated by latency. Some bad offenders: listing images in a repo in ACR gets ~ a phone modems worth of overall throughput. It takes minutes to download single-digit megabytes of image metadata. The Azure pricing APIs are similar: it's ~58KiB per page. The entire VM pricing data is something like 131 MiB, and that requires 2,235 HTTP calls to fetch.

I'm not sure how you're blaming JSONSchema for "Half the fields in the type will be required … and the schema will say they're optional". But JSONSchema has plenty of ways to define sum types: http://json-schema.org/understanding-json-schema/reference/c...

Hmm. Thank you for that, I seem to be mistaken there. I wonder if that was added after I acquired my knowledge of JSONSchema? Or if it was some weird thing w/ Swagger where it wasn't supported there.

Swagger (OpenAPI 2) and OpenAPI 3 never supported JSON Schema. They support a subset of the draft 4 with custom additions. OpenAPI 3.1 added support for additional schemas where you can use whichever draft you want.

Might just be confusion with swagger / openapi, which is a complete mess of procedural constraints reminiscent of XML Schemas.

Am I right to call out most of this problem is caused by incessantly reusing types (because defining an additional type for i.e. CREATE vs PUT actions is soooo hard.

If you've already gone to the length of defining your DTO with 20+ typed fields including comments/annotations etc., why should you need to duplicate it between CREATE and PUT types? PATCH is probably the trickiest one to do cleanly - while I've used the model of "re-use the CREATE type, but all fields are now optional" in the past, I wouldn't actually recommend it, and probably promote proper JSONPatch format instead, the only real disadvantage being that it's a harder to manually construct the request body, which isn't something you should be too concerned about end-users having to spend time doing.

> gone to the length

Right there. Because typing 20 lines of source code should not be an argument for weakening your model layer. Re-use would be great if types were identical between actions, but in the real-world they're not and they accrue subtle differences over time.

Disclaimer: I work for Microsoft as an API Architect.

I am not working on this specific API, so I am not going to comment on anyway. I hear your complains about Microsoft API guidelines (which is an entire different conversation) but I wanted to add my two cents with regards to JSON Schema.

The problem that I have been having with JSON Schema since forever - is that the data that is being modeled is complected with contextuality of its usage. For instance, if I have

type user = { name: string, surname: string, password: string }

IN JSON Schema it is very hard to give contextuality on it, and most of the times involves having two separate types.

Here is an example:

If I am creating a new user, then name, surname are mandatory, while password is not because the system is autogenerating it. If I a logging in - then I want ALL of the fields.

As of today, it is very hard in JSON Schema to express this.

Basically speaking, I am arguing that the data structure is a thing, another one is its usage in a context, where there can be requirements and complicated validation logic involving even other fields

In my experience, the only thing that has been very very close to what I have been looking for when modelling systems is Clojure. Most of the people laugh to my face when I say that primarily because it is a LISP 2 and yet... In particular, spec (and even better spec2) have the tooling to express data structure as sophisticated as we want without a type system and with the contextuality constraints that are fundamental for a real type reuse.

I'm not sure what you mean by "it is very hard to give contextuality on it"; OAS does supports referring to a type by reference, so that higher level types can reuse the definition of structs they might contain.

But even so, here the problem is that the APIs aren't actual PUT/GETs: they payload types aren't the same going up as they are coming down. It is really two separate types, one for PUT, one for GET.

Some of that is to be expected (there will be some information after the create that is only added by the VM coming into being) but how Kubernetes handles this with a separate "status" for the item I think ends up letting the rest of the type (spec, in k8s's case) be the same type. (… ish. K8s has variants of this problem, too.)

To expand a bit, I'm largely relegated to the API docs themselves. Browsing the actual schema is hard:

  Start at: https://github.com/Azure/azure-rest-api-specs
  Descend into specification.
  Descend into … so many choices … compute.
  Descend into resource-manager.
  Descend into Microsoft.Compute
  Descend into stable
  Descend into — and this is tricky!
    the latest version isn't the latest version.
    The latest version is 2022-04-04, but for VM creation it's 2022-03-01.
    The only way I know to determine this is to seek backwards, or find it in the docs.
  Descend into ComputeRP
  Descend into virtualMachine.json.

And it's 3.3k LoC! Some of this verbosity is JSONSchema, to be sure… but still. And then you might have to wade back up to common.json, though I forget what circumstances cause me to need to look there.

I’ve debated back and forth on whether it’s a good idea to have separate input and output models for each endpoint, because trying to have a generic structure that’s usable everywhere makes it really easy to pipe output back to input for a PUT, but it’s difficult to express constraints like, this field cannot be updated, only created or these fields are required but only for create and update supports a different subset of the fields again.

I think ideally you want seperate structures but you need tooling which helps you map between output/input structure automatically (in strongly typed languages, it’s easy in Python or JavaScript) and that’s just lacking currently.

Been there too. I'm a big fan of explicit data models (DTOs) for each endpoint, as I think any kind of interface description should be as sound, concise and precise as possible. That means never having any property sent that is redundant or unnecessary (=gets ignored by the backend). But I do see the problems and increased engineering effort to achieve this. Especially since a lot of BE frameworks and languages do not support union and intersection types (looking at you, C# and Java) while OAS does. The endresult is usually larger shared DTO models, where any given subset of properties is set to "null" depending on the endpoint it is used for. Not a clean interface design, but decreases a lot engineering effort.

A true REpresentational State Transfer API would give each version of each of the API's L7-protocol request and response messages its own media-(sub)type; and would then rely on HTTP content negotiation to allow clients to specify which messages they're expecting to receive in response to a request; where the difference between throwing/nonthrowing, sync/async, new/old variant, etc., would all come down to which response message types the client is declaring themselves to accept.

Don't think I've literally ever seen this done in practice. Don't think I've ever seen a "REST client" library that even offers the possibility of negotiating with a backend that's attempting to do this — let alone doing it in a streamlined manner.

no, the whole media type thing was a moderately-interesting-idea-turned-ridiculous-navel-gazing turn that, among many other pedantic blind alleys, turned everyone away from the deeply innovative aspect of REST: the uniform interface, and, in particular, HATEOAS

https://htmx.org/essays/hateoas/

any idiot (such as myself) who has ever made a web 1.0 app has created a better REST API than 99.9% of all REST API engineers, so called

I have created Hypermedia APIs using HTTP/JSON for service-to-service interactions, and it is definitely a lot of work for questionable benefit.

The biggest problem with the bad style of RPC-style programming was the assumption that remote calls could be treated like local calls. Modern HTTP-based RPC APIs don't make that assumption and thr problems of distributed computing are handled explicitly.

The semantics of HTTP verbs, semantically meaningful namespace trees for resources in the URL, standard authentication and encoding, etc are all nice benefits of HTTP APIs even without hypermedia.

That said, I haven't given up on hypermedia APIs! I liken them to "fluent" APIs of OOP. It's a powerful design approach, but I just wonder how to capture the benefits of that approach over what most people are doing today.

My current take on the issue is that REST advantages can only be realized if there a sufficiently smart client (aka human) driving the interaction.

https://intercoolerjs.org/2016/05/08/hatoeas-is-for-humans.h...

That basically validates the conclusions I have come to as well.

However, I would add that I think you can build machine APIs using hypermedia, but the API contract would different. You would essentially define an OOP API with MIME Types as class definitions. That is partly I think why Toy Fielding put so much emphasis on defining those types.

But the real pwer of hypermedia - the ability to change those hypermedia relations dynamically - is really only useful for human-driven interactions.

*Roy Fielding. Darn auto-correct.

HATEOAS is its own thing. A good thing, IMO, but not the same thing as REST. Saying that "the innovative aspect of REST is HATEOAS" is like saying "the unique thing about rectangles is that all four sides are the same length." That's squares, not rectangles. Rectangles have a uniqueness all their own, whether they're square or not. REST has innovative aspects all its own, whether you're doing HATEOAS or not.

REpresentational State Transfer is, like it sounds, a very specific concept: it's about how the semantics of HTTP verbs + headers (e.g. Cache-Control headers, Allow headers, Vary headers, etc.) interact with clients and gateways (esp. caching gateways, clients in offline mode, etc); and how these interactions work better and "in harmony with" those HTTP semantics — improving cachability, decreasing number-of-origin-fetches, improving PWA offline graceful degradation, avoiding inconsistency and any need for cache-busting, etc. — when you model your exposed state-representations a certain way.

A REST model — a projection of your data as "REpresentations of State" — is a transformation of your data along lines such that each representation becomes an ADT with semantics that recapitulate those of an object store / WebDAV folder / HTML page as edited in the original WorldWideWeb.app. Each representation can be PUT; POSTed to; PATCHed, DELETEd; GET-ted and then cached; and HEAD-ed without triggering heavy computation. REST's REpresentations are objects that work in harmony with being passed around through the HTTP protocol.

However! Every "RESTful" API you've ever seen (at least if you do CRUD web-dev), is very likely a trivial RESTful API, because it's taking abstractions — resources — that are already inherently ADTs with REST-alike semantics, and just 1-to-1 exposing them via HTTP. (A lot of work may have been put in by some dev at some point in the past, into coming up with a best-practice design for those internal data structures such that they can work for the business-domain while also being inherently RESTful; but it's not REST itself carrying that weight, and it's not REST itself that determines whether it's possible to do that.)

It's great if your internal data is such that you can do that ; but that's not what makes REST powerful; it's not what it means to wield REST as a tool.

REST is a thing like GraphQL: a gateway adapter that reformulates an internal data model into a different, outward-facing data model. It's what frontend devs would call a data-binding layer — but, like GraphQL, one done on the backend. This is why REST talks about representations: representations are to REST as graphs are to GraphQL. They're a modelling layer on top of your data, that reshapes it. Except that, unlike GraphQL, which is almost as bad for HTTP semantics as SOAP, REST is (by definition) whatever transformed form allows your data to be handled most optimally in an ecosystem of HTTP clients+servers+gateways.

This concept is most powerful when applied to things that aren't inherently very RESTful at all; where it's challenging to state them in RESTful terms. For example, an API for controlling an ephemeral pushdown-automata, like an SQL connection. Or an API for managing subscriptions and consumer-groups in a message-queue broker. Or an API for scheduling jobs onto a workload manager.

Re: that last one, Kubernetes is a great example of what it means to "do REST" in a nontrivial way: it takes resources that aren't inherently RESTful, and creates abstract REpresentations of these resources (YAML manifests) that work perfectly under HTTP semantics, transferring the state of these representations to clients in a way where the client can then do things like modify the representation and PUT or PATCH it back. You can stick a caching proxy between you and a Kubernetes control-plane, and everything keeps working. You can query a Kubernetes control-plane from your browser address bar.

But note how, when you create a nontrivial abstraction like this, you end up with very custom types for your REpresentations. Because there are no inherent REST-shaped resources in your backend to manipulate; and because you want your REpresentations to be long-lived documents in their own right (in caches; on the client as something for it to hold and manipulate and patch; etc), you have to define an application-layer formal semantics for what those documents are, and do. You have to ensure that a REpresentation a client fetched a year ago, and is pushing back as a PUT today, will either work as-is; be "migrated" in a forward-compatible way when received by the backend; or be able to be cleanly rejected, by your REST server, despite that document having a schema that is now older than the REpresentation you're currently emitting from that endpoint.

How do you do that? People who don't understand REST use ugly URL hacks for this — like putting the version in the URL, the format as a file extension in the URL, etc. But these hacks fight against the semantics of HTTP.

Say you have an internal Foo ADT, that you first exposed as a FooV1 representation, and then later additionally as a FooV2 representation. If you do that by exposing e.g. /foos_v1/1 and then /foos_v2/1; then a PATCH to /foos_v1/1 can't possibly inherently invalidate a reverse-proxy cached representation for /foos_v2/1. But if you've just got /foos/1, that Vary's on Allow? Then it all "just works."

To make that work, FooV1 and FooV2 have to send separate Content-Types.

QED? ;)

Hey, I’m kinda new to this area, can you show with examples what the differences are? I read the article, he does a lot of explaining but doesn’t do much showing. Like why is a specific api more RPC and why a different one is a perfectly constructed REST api?

REST was the term Roy Fielding coined as description of the original web architecture. The big distinguishing feature of REST-ful systems vs. non-REST-ful systems was something called the uniform interface, where clients had zero knowledge about a given server's API surface beyond a URL entry point. Everything after that was encoded in hypermedia, giving us the term "Hypermedia As The Engine of Application State" (HATEOAS).

Here is an article I wrote on it:

https://htmx.org/essays/hateoas/

RPC-style APIs require shared knowledge about a given end point: what arguments it expects, what data it returns, etc. Unfortunately, for historical reasons, we've come to call all HTTP JSON APIs "REST APIs". It's actually pretty funny.

Thanks! I was really confused with this topic, and your post cleared things up. But from what I gather, proper REST API's wouldn't pair nicely with modern day UI development. Maybe this is because I usually work with APIs that wouldn't be considered restful (from my understanding, my company's public API is here [https://developer.veevavault.com/api/22.1/]), I have a hard time seeing how a modern React webapp would parse the HTTP form response given in your article.

That's right, REST concepts are largely wasted on modern SPA applications, which are fundamentally RPC-style clients.

Instead, REST should be used in its original context: hypermedia clients (browsers) exchanging hypermedia (HTML) with the server. I have written an alternative front end library that takes this exact approach:

https://htmx.org

Some more essays on the topic:

https://htmx.org/essays/spa-alternative/

https://htmx.org/essays/hypermedia-apis-vs-data-apis/

Thank you! This has been a point of confusion for me for a while, maybe I should get around to reading Roy's dissertation. Your post did clear things up :)

One suggestion I've seen (eg here: https://cloud.google.com/blog/products/api-management/unders...) is to distinguish between REST and OpenAPI designs.

Because you're right, REST is a specific thing, but we do need some name for what everyone is talking about, and just saying "RPC" is much too vague.

Why isn’t the meaning of REST allowed to drift? In the comment section of that page Roy mentions how the meaning of hypertext has drifted over time. Why shouldn’t REST evolve as well?

It has, regardless of what Roy believes - I've never come across any API that calls itself RESTful that fits with his idea of what REST is, nor is it clear what advantages it would have.

REST isn't REST anymore. Barely anyone does it "correctly." The term has gained an overload: anything that happens over HTTP that OpenAPI is likely able to describe.

REmote Structured Transactions. Problem solved.

If a teacher has one failing student, blame the student.

If a teacher has 99 failing students, blame the teacher.

Or, in this case, maybe:

If the teacher has no students, blame the topic.

People just don't want HATEOAS as much as some would like.

I did the same a while back.

I used ANTLR4 to generate a .NET tree walker that would build up an IQueryProvider expression. IQueryProvider would then compile to Expression<T> and self-optimise (simplifying boolean algebra and removing redundant expressions). We hooked this up to npgsql and viola - a sane, typesafe, query string DSL without a single line of SQL.

Nowadays I hack together Terraform and Python ML- I miss .NET dearly... it was a simpler time.

To me, REST focus on Resource first. RPC is Producedure first.

Reply on news.ycombinator.com