Comment by RyanMcGreal | Hacker News Reader

Comment by RyanMcGreal | Hacker News Reader

Parent Full thread

It's yet another implementation detail that drags the reality of RESTful web service development farther away from the principle that it's easy because it's based on HTTP and we already know HTTP.

I'm not writing this to dump on REST. I still think it's the right approach to build a web service over HTTP, but I'm also conscious of the diminishing returns on each incremental step toward pure RESTfulness. I'm just trying to make a web service, and suddenly I'm defining new content-types?

For a client, is it worth the trouble to learn a new content-type to determine what's a hyperlink in my JSON response, when they can just look at the response and see something like the following?

    GET /articles HTTP/1.1

    {
        "articles": [
            {
                "url": "/articles/1",
                "title": "This is my first article",
                "pubDate": "2012-01-05T08:39:47.625000
            },
            {
                "url": "/articles/2",
                "title": "This is my second article",
                "pubDate": "2012-01-13T11:07:35.219000
            }
        ]
    }

After a while it starts to feel like the HTML v. XHTML debate.

What trouble? Are you not documenting what you return anyway? What's the cost of just changing the mediatype to some different string?

For a client, is it worth the trouble to learn a new content-type to determine what's a hyperlink in my JSON response, when they can just look at the response and see something like the following?

Personally, I think that mindset will hold us back. People still think of RESTful services as closed silos instead of open, truly uniform interfaces, and that will prevent us from growing, in my opinion.

Your view is of a developer painstakingly coding against each API (s)he wants to use, manually prodding and poking it and writing a custom library that doesn't work with any other service.

My view is of generic, easily adaptable clients, using machine-readable specification of formats¹ and providing graphical UIs where even non-programmers can input the single endpoints of various services and easily combine the data and functionality to create applications that cater to their specific needs.

I see a service like IFTTT[1], but more flexible and where the services are not hardcoded but imported dynamically by the user and then strung together to fit their needs.

Using mediatypes, even if custom, is a step towards that integration and standardization that makes this possible.

¹ RDF seems like the best bet to me - possibly encoded as JSON

[1]: http://ifttt.com/

What I'm trying to suggest is that the point at which a schema is powerful and generic enough to work as a general-purpose web service API for an arbitrary use case, it has become more difficult and complicated to learn, configure and use than a much simpler, discoverable ad-hoc schema that closely matches the functionality of your web service.

In general, REST principles give more than they take, which is why I try to follow them; but you're always going to have to find the sweet spot of abstraction that provides the right amount of power and flexibility combined with the right amount of discoverability and predictability.

> What's the cost of just changing the mediatype to some different string?

I'm happy to change a server's content-type from application/json to application/vnd.my-adhoc-service+json, but that won't change the client's need to read my documentation and play with the service before they can figure out how to use it. That's true even if I move the documentation out to an RFC.

One real advantage I can see is to use the media type for versioning, but even that doesn't remove the need for a client to intervene if they want to upgrade their code to work with the newer API version.

So it's not really clear to me that a custom media type is significantly more useful for the client, even if it is more 'proper'. When a constraint feels ceremonial, I become suspicious that it exists to drive purity-for-its-own-sake rather than a real benefit.

What I'm trying to suggest is that the point at which a schema is powerful and generic enough to work as a general-purpose web service API for an arbitrary use case, it has become more difficult and complicated to learn, configure and use than a much simpler, discoverable ad-hoc schema that closely matches the functionality of your web service.

Is RSS difficult and complicated to learn and use? Now think what a pain it would be to develop a feed reader if each blog had its own "simpler" ad-hoc schema.

Standards don't have to be complicated, especially if they use an extensible encoding like JSON or XML so that instead of having the kitchen sink they can remain simple and be extended for specific purposes if really needed, without affecting generic clients.

I'm happy to change a server's content-type from application/json to application/vnd.my-adhoc-service+json, but that won't change the client's need to read my documentation and play with the service before they can figure out how to use it. That's true even if I move the documentation out to an RFC.

The end goal is for different services to agree on standard formats for common resources. Using custom mimetypes is a step in the right direction because it decouples the format recognition and parsing from the service-specific code - it allows clients to use more modular code that works across different services with plenty of reusable code.

Is it terribly important? No. But since the costs are essentially none - changing a couple of strings here and there - not doing it seems unreasonable to me. You asked why, I ask why not?

First of all, I want to thank you for your thoughtful, clearly-articulated comments. Frankly, you've done a better job of explaining and defending dedicated media types than anything else I've seen and I appreciate you taking the time to engage my comments with seriousness rather than condescension.

> Is RSS difficult and complicated to learn and use?

No, but RSS is a schema with a specific, narrowly defined purpose: to syndicate article-based web pages (and even so, there are 5 or 6 different versions, not including Atom - though your point stands and I don't want to get sidetracked over a quibble).

A more apt comparator may be WSDL, which theoretically allows a SOAP web service's remote procedure calls to be discovered programmatically but in practice is such a messy, complicated, leaky abstraction that it's often easier just to build XML templates manually and POST them to the endpoint.

I'd be happy to use and support some kind of lightweight schema for JSON that puts URLs in predictable places in the response object and still allows flexibility to provide the functionality a given service needs. I'm not aware of anything that fits this description, but I'm certainly open to it if you have any recommendations.

> You asked why, I ask why not?

The model of HTTP resources and methods is useful: it unambiguously delivers real, tangible benefits to the developer and the client over ad-hoc protocols riding on top of HTTP like `GET /create_user.php`.

Similarly, it makes sense to use the HTTP status codes we already have instead of reinventing your own arbitrary set of status codes and, e.g. sticking them somewhere in the response body. If someone e.g. does a POST request on /users/1337 to create a user, it makes sense to return a 405 status. (However, I tend to put status codes in the response headers and the response body, just to make it easier for clients to notice what's going on.)

It also makes sense to put URLs inside the response body in an obvious way so that the client can "click through" to subsidiary resources in a manner analogous to how web pages do it. Again, it makes the web service more or less self-documenting to the client, especially if it uses URLs and HTTP Methods in a sane, predictable way.

I'm struggling to understand the benefit of a custom media type beyond merely satisfying one of Roy Fielding's constraints. If the only reason to use a custom media type is to satisfy a constraint, the constraint might be more ceremonial than useful.

If two web services are identical except one serves `Content-Type: application/vnd.some-arbitrary-format+json` and the other serves `Content-Type: application/json`, I get suspicious about the benefits of REST when people say the latter is doing it wrong and doesn't deserve to be called RESTful.

No, but RSS is a schema with a specific, narrowly defined purpose: to syndicate article-based web pages (and even so, there are 5 or 6 different versions, not including Atom - though your point stands and I don't want to get sidetracked over a quibble).

A more apt comparator may be WSDL, which theoretically allows a SOAP web service's remote procedure calls to be discovered programmatically but in practice is such a messy, complicated, leaky abstraction that it's often easier just to build XML templates manually and POST them to the endpoint.

Well, two points:

Firstly, I think WSDL (and WADL) are working at the wrong abstraction level. I think one of the reasons why REST is cleaner is because unlike SOAP it's data oriented, not RPC. And I believe it's possible, as long as your API is data oriented too, to map that to generic formats that don't feel messy.

I had already mentioned RDF (Resource Description Framework), but I'll mention it again; RDF is a layer between the serialization and the concrete schemas, giving very specific but open ended semantics to its documents.

Unlike WSDL, it's very simple: it consists only of four concepts: triples, which are composed of objects (which is always a resource, and is represented as an URL), subjects (which can be other resource or a literal value) and a predicate (which relates that object with the subject in some way, and which are specified by a schema). Then, a document is just a bunch of statements about one or more resources.

This model is hypermedia (since it clearly points to resources), and it's at the same time constrained enough to not end up being messy, and open enough to define mostly anything. I think it really captures the sweet spot.

There's no JSON serialization, unfortunately, but I like Turtle[1].

---------

The second point is that even if this day dream of mine never comes to pass, I think we'd still benefit a lot from using more of those standard "schemas with specific, narrowly defined purposes" and keep the custom formats to when they're really needed. Instead of defining your own user profile schema - which 99% of the APIs out there need - why not just share a single format, which can be easily:

1) Parsed by generic clients, libraries and plugins

2) Shared between services and applications

But for that, mediatypes make perfect sense, since they make it easy for a single HTTP client to dispatch the responses to the right readers, without involving a bunch of hairy rules that say "when response is JSON and the service being called is X, call parser Y".

This post is against WADL, but I think it expresses that power of using proper mediatypes: http://bitworking.org/news/193/Do-we-need-WADL

I'm struggling to understand the benefit of a custom media type beyond merely satisfying one of Roy Fielding's constraints. If the only reason to use a custom media type is to satisfy a constraint, the constraint might be more ceremonial than useful.

I might be wrong, but I'm convinced some great things could come up from it. Unfortunately, I doubt that anyone will seriously explore the possibilities unless there are already many APIs with support for it, and in typical Prisoner's dilemma style, very few API developers will consider supporting it unless they see a short term benefit.

If two web services are identical except one serves `Content-Type: application/vnd.some-arbitrary-format+json` and the other serves `Content-Type: application/json`, I get suspicious about the benefits of REST when people say the latter is doing it wrong and doesn't deserve to be called RESTful.

Well, I think as long as it supports HATEOAS it's RESTful enough, but on the other hand, I can see the their point. There's nothing wrong with not fully implementing REST, but still calling it REST gives false expectations to the clients.

Of course, at this stage nobody expects a RESTful API to be actually RESTful, but then again that's the problem, isn't it?

Well, sorry for the WOT :|

Maybe I'm missing something, but why the resistance to returning a "meaningful" content type in a response? It's just a name for the structure you have outlined there.

Two reasons that I can see...

If the server generates custom mimetypes, clients have to ensure they "Accept:" the correct mimetype for every request. And if you have multiple data types (and you will have multiple data types), then this "Accept:" header will vary for every request. Getting this right is an irritant to client developers with no actual benefit to them, and a great potential source of bugs.

Much easier is if the client can just send "Accept: application/json" or "Accept: application/xml" for every request.

If the server generates custom mimetypes, and you update your API to v2 which produces an article listing in an incompatible format, you either have to introduce a version number into your mimetype (which seems messy and pointless busywork), or stick with the same mimetype and accept that your custom mimetypes aren't actually particularly meaningful as their meaning can change substantially over time (so why did you have them in the first place?)

Much easier is if the server had just used "Content-Type: application/json" or "Content-Type: application/xml" which will always be correct.

If the server generates custom mimetypes, clients have to ensure they "Accept:" the correct mimetype for every request.

No, he doesn't. First, the server can send whatever it wants - nothing breaks if you reply with your custom media type to a generic "application/json" request.

Second, even if you want to be correct, that's what wildcards are for: the client can simply send always the same header:

     Accept: application/*+json

If the server generates custom mimetypes, and you update your API to v2 which produces an article listing in an incompatible format, you either have to introduce a version number into your mimetype (which seems messy and pointless busywork),

I still don't get what's the problem with changing a string. I think you're using a bad framework.

I doubt that sending a mimetype which the client has said it can't accept would be on anyone's list of best practices.

This is a good point, though, not sure why it escaped me:

    Accept: application/*+json

I think you're using a bad framework.

Probably. This isn't why, though.

So if you don't communicate that the version of the structure of your response has changed through the Content-Type, where would you do it?

I'm happy to change a server's content-type from application/json to application/vnd.my-adhoc-service+json, but that won't change the client's need to read my documentation and play with the service before they can figure out how to use it. That's true even if I move the documentation out to an RFC.

One real advantage I can see is to use the media type for versioning, but even that doesn't remove the need for a client to intervene if they want to upgrade their code to work with the newer API version.

So it's not really clear to me that a custom media type is significantly more useful for the client, even if it is more 'proper'. When a constraint feels ceremonial, I become suspicious that it exists to drive purity-for-its-own-sake rather than a real benefit.

Reply on news.ycombinator.com