We have broken HTTP
gadgetopia.com
gadgetopia.com
This came up recently on HN in connection with GoGo WiFi on airliners, using satellite links. GoGo doesn't have the satellite bandwidth to let users view streaming video, so they block YouTube and some other sites. The problem is telling the user what's going on. They used an awful hack involving a self-signed fake SSL certificate for YouTube to try to get a coherent message to the user.
According to the IP standard, they should send back an ICMP Destination Unreachable message with a code 13, "Communication administratively prohibited". Most IP/TCP/HTTP/browser stacks won't get that info all the way up to the end user, especially the "Communication administratively prohibited" part. Even if the error code made it to the user, many customers would not interpret "Communication administratively prohibited" as "No, you can't watch Youtube because you're on an airplane and we don't have the bandwidth for that".
Agreed. However, I've discovered that even folks who really should know better don't know what to do with anything other that port-unreachable:
In the past, I tried setting my ip[6]tables REJECT targets to send back admin-prohibit. For... reasons[0], I was REJECTing IPv6 traffic to Google's networks. I discovered that YouTube on a Nexus device running Android Jellybean would not fail over to IPv4 but would, instead, sit and spin for multiple minutes before reporting a fatal connection failure.
[0] At the time I had a IPv6 tunnel that (for some reason) transferred packets destined for Google very, very slowly.
These are obviously only available mostly during US domestic flights. For international flights, the in-flight internet is provided through satellite uplink but GoGo doesn't provide this service.
I remember trying hard to get URNs to work, and proposed them for one project. They required a URN resolver, which is entire new set of infrastructure.
I tried using path segment parameters (I believe the author refers to syntax like 'http://a/b/c/g;x=1/y' rather than '/key1:value1/key2:value2/'.) At the time (mid-1990s) the tools I used didn't support it, and I never came up with a need for them. Still haven't. The current spec (ftp://ftp.rfc-editor.org/in-notes/rfc3986.txt) says that the text after a ';' is a "non-zero-length segment without any colon" and that the contents are "considered opaque by the generic syntax." "[I]n most cases the syntax of a parameter is specific to the implementation of the URI's dereferencing algorithm."
Regarding "404 (traditionally “Not Found”) means it was never here. 410 (traditionally “Gone”) means it was once here but is now gone"; the spec says:
> [404] The server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent. The 410 (Gone) status code SHOULD be used if the server knows, through some internally configurable mechanism, that an old resource is permanently unavailable and has no forwarding address. This status code is commonly used when the server does not wish to reveal exactly why the request has been refused, or when no other response is applicable.
It doesn't say anything about 404 meaning "was never here", and in fact includes the possibility that perhaps the server doesn't want to give more information.
An origin server that wishes to "hide" the current existence of a forbidden target resource MAY instead respond with a status code of 404 (Not Found).
I am certainly not a proponent of web as a platform for everything but there are certainly some things it is rather good at, and making content discoverable in a consistent manner is one of those. If you hid the content in an app it's rarely exposed.
Twitter etc are corporately farmed content, and they all provide web and app access. The access method is just a front end.
Personal content in apps isn't usually all that interesting. It's hard to think why anyone would care about my Angry Birds scores, or what level I'm on in Candy Crush. (If I played Candy Crush, which I don't.)
If you want to make an app that shares content on the web, it's not difficult. I don't see a problem with the fact that it's extra effort, and if you're going to do it you need to justify that effort in some way.
Similarly, we have standards boards, who are often very divorced from the realities of actually building real-world software and we also have the engineers building real-world software. Which version of HTTP is correct, the one codified by the W3C or the version codified by the engineers building Chrome, Firefox, IE, Apache and Nginx?
As a learning exercise, I implemented an HTTP server a few years ago and remember finding it fascinating that none of the popular user agents implemented the Keep-Alive header per the specification. And yet they all implemented it in a common, non-compliant manner. It made me realize that as much as I was using the spec as my guide, my implementation wouldn't be correct if I followed what it said for that header.
As a human, I can usually communicate with someone who uses a dialect or makes language errors. Computers are much worse at doing that, they basically need to know about every standard and non-standard behavior to correctly handle it. As such there is a very good reason for standards and thus standard bodies to exist.
There is a reason why protocols are not languages. A protocol is a well-defined standard that engineers are supposed to follow. There really is no space for "descriptivists" in technology and especially in standardized protocols. What is that, the protocol is useless? Well, maybe we should come up with a new standardized one instead of bastardizing the already-existing one and creating a lot of confusion and ambiguity (which machines have a hard time to deal with).
I can't think of any other protocol where even the most unskilled developer is given the tools and encouraged to determine how the protocol is used.
It's not like it ever was universally conformed to. It's not like there was this golden era. What would be the golden era? The web before AJAX? Geocities and Hotmail? Frames broke URLS more than almost anything. PHPNuke forums? Web apps sucked back in the early days.
Makes me think of a fun Alan Kay quote:
> The Internet was done so well that most people think of it as a natural resource like the Pacific Ocean, rather than something that was man-made. When was the last time a technology with a scale like that was so error-free? The Web, in comparison, is a joke. The Web was done by amateurs. -- Alan Kay.
It's really refreshing to code with websockets and just write an application naturally. HTTP is good for GETing static web pages and that's about it.
Simple verbs work for simple tasks - only. There is more to life than REST.
If you need to update multiple resources of different types atomically, then you probably need a new resource abstraction that covers both of the resources anyway. Then just POST to that.
There are developers who do use most/all of HTTP. I'm one. I think I've used most of the obscure features he mentions, including URL parameters and all those status codes (excepting 402 Payment Required, which I've never had a need for, but I do know it exists).
Most of the features he mentions aren't very useful for browser interaction, but really shine for APIs. If you're trying to write an HTTP API, and you haven't at least got to grips with all the status codes and methods, you should and take the time to understand them all and when they should be used. It's a real joy to use an API that uses methods and statuses properly. If in doubt, read the Github API docs - their API is (mostly) very good in this regard.
URN's I had not heard of, but the article didn't make it clear to me what problem they would solve.
Of course I know about HTTP verbs PUT and DELETE and HEAD-- is there really any web dev who doesn't? And I am pretty rigorous about using the appropriate one (unlike with status codes).
I don't have an opinion on how apps are isolated from the web and don't have interlinks between them. Don't really use mobile apps myself except utility ones like Spotify and Netflix.
The idea of a globally unique, permanent identifier that would resolve to a URL to indicate the current location of a resource is fairly commonplace, isn't it? We've tried to bolt all sorts of equivalents onto the existing URL infrastructure without them.
I actually prefer the Wikipedia page[1] exactly because it includes more than just the canonical subset. For example, 451 is not yet official, but pretty much as close as you can get to a standard without having an actual standard (well, there's a draft).
[0]: http://www.iana.org/assignments/http-status-codes/http-statu... [1]: http://en.wikipedia.org/wiki/List_of_HTTP_status_codes
If a client isn't already aware of the code, it decides what to do based on the first number:
- 1xx - continue receiving content
- 2xx - finished receiving content
- 3xx - follow redirect
- 4xx - client error occurred, show error page
- 5xx - server error occurred, show error page
So there is no harm in using better (but less commonly known) codes
- 2xx - You Don't Need Refresh
- 3xx - We're dealing with Refresh for you
- 4xx - Don't Bother Refreshing
- 5xx - Refresh, You Might Get Lucky
Actually there aren't. There are like 60. It really shouldn't take too long to read over them: http://en.wikipedia.org/wiki/List_of_HTTP_status_codes
Yes: I've interviewed some. Very few realise there's any more than GET/PUT/POST/DELETE, which astonished me: surely everyone's left an HTTP debugger running while updating an SVN repo or browsing a WebDAV share?
It's a universal namespace, just like UUIDs, but with just enough extra sauce to make it theoretically possible to do interesting things with it. So 'urn:isbn:0451450523' might represent The Last Unicorn, and if you fed it to your Kindle it might enable you to buy it, and if you fed it into Nook ditto, and if you fed it into Project Gutenberg you might get a notice that it won't fall out of copyright until A.D. 3172, and if you fed it into your browser it might do any of those things or just look it up on ibiblio or Wikipedia.
Unfortunately, like many good ideas, it didn't take off because folks couldn't imagine a profitable use for it.
<!-- AWS Elastic Beanstalk application version --> arn:aws:elasticbeanstalk:us-east-1:123456789012:environment/My App/MyEnvironment
<!-- IAM user name --> arn:aws:iam::123456789012:user/David
<!-- Amazon RDS tag --> arn:aws:rds:eu-west-1:001234567890:db:mysql-db
<!-- Amazon S3 bucket (and all objects in it)--> arn:aws:s3:::my_corporate_bucket/*
http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-na...
I don't know why they didn't use URNs. Probably because nobody else does.
URNs are interesting but difficult to work with in practice; 402 Payment Required has been "reserved for future use" forever. Then there's the suggestion to use custom HTTP methods, which is ill-advised [1].
http://webcache.googleusercontent.com/search?q=cache:0PUMGvU...
(Server is not responding at the time of writing, from two locations.)
I think the situation is improving and serious developers are using a broader range of HTTP features. Proper status code and verb usage has become very common with the proliferation of RESTful APIs.
Except HTML(5) forms do not support any methods other than GET and POST.
The author was ranting about unused features, but not all features are good ideas. Sometimes things that look good on paper don't work so well in practice.
Those who forget SOAP are doomed to recreate it, badly. I worry that things like JSON-schema are headed in that direction.
* Until recently, used stateful numbers for all links * Uses the same location for different resources (e.g. https://news.ycombinator.com/item) * Serves the same resources at multiple different locations, rather than redirecting you to a canonical location. * Does posting, editing and deleting via POST, not CREATE/PATCH/DELETE * Does approximately no content negotiation * Does not support non-browser clients at all, which, well I suppose you could make a perverse argument that that is HATEOAS.
In short, it behaves not at all like the system described in the REST paper.
Can you substantiate that? Why shouldn't I be able to POST a new resource to a server and update an existing resource with PUT using Forms? Why shouldn't I be able to delete a resource using a DELETE method using a Form?
Doesn't make sense to me.
I would not use such a framework, because it did not understand HTTP and HTML specs in the first place.
I'm not sure what you're talking about:
1) It's user input and the user can craft a DELETE request just as well as add a _method=DELETE to the form body.
2) Most frameworks use the _method parameter as the http method, meaning when routing or when the application asks for the method so as to take specific actions. No general framework just deletes or creates stuff for you (unless you specifically set it up to, and even then you can normally add an authorization check hook).
The reason HTML uses an enumeration ("get" and "post") and maps those to HTTP methods ("GET" and "POST") is that they have special meaning for user interaction. With "get" a form becomes an URI builder. With "post" the user can send data to the server. You can call it a coincidence that the values of the enumeration match the HTTP methods. The reason a "post" form is mapped to a HTTP POST is that a server should always process user inputs - even if the behavior of your application results in deleting a resource.
So yes, okay, the frameworks behave correctly, because they process user inputs. My point was to question the intent or reason behind a "_method" parameter of those frameworks. If such a framework invented the parameter because it felt limited by the HTML specification, it did not understand the specification. If it is about automating resource management, then I'm fine with that.
BTW. which frameworks do you mean? I know none (never saw that in Spring Web or WS RS implementations like Jersey).
A DELETE has no payload.
OPTIONS and HEAD make no sense for a browser, because there is no information to show to the user.
So that leaves us with POST and GET. As I already stated, POST is the best form to send user data to a server. And a form with GET is just an URI builder.
Edit: Well no, that does not leave us with POST and GET, because there are also CONNECT and TRACE. But who wants to concern a user with those?
You can still use other HTTP methods from a browser (using XmlHttpRequest), just not through a form.
Because not supporting Get would break a lot of current web applications and html5 had a pretty clear goal of being mostly backwards compatible. The previous form element didn't restrict the method and GET and POST are used everywhere. Restricting GET would have caused too much confusion to be worth it.
It's worth noting that sometimes the reason for developing a new solution is to be free of the baggage that comes with all the failed attempts on the old infrastructure...
e.g. JSON-based solutions over the many attempts to use XML (XML is a capable base, but many find it unwieldy to say the least)
You throw a JSON string at a simple parser and get an object back. That's why it won a place in my heart. Nothing prevented xml from being that except that's just not what the lib authors did. There was so much talk about XML being the One True Data Structure I think it ended up being a jack of all trades, master of none.
Take that away - plain unnamespaced tags, with no semantics beyond "this is a character sequence", common-sense business-specific tag names, and no schema - and you get something that's just as nice to work with as JSON is. And there are plenty of libraries (e.g. Jackson) that will serialize back and forth to objects. But too many people drank the semantic kool-aid and tried to autogenerate classes from schemata or other such silliness.
In my experience, though, when working as a front end developer I've had to deal with developers (mostly in the .Net camp) who will gladly return errors with 200 OK and an xml/json object with the typical "StatusCode: xxx", where XXX is some made up error code that you'll need to reference in their application.
I believe this is because many backend developers are in web development by accident, and don't really care much about the web as a platform.
> Don’t even get me started about URL parameters. No, not querystrings – there was originally a specification where you could do something like “/key1:value1/key2:value2/” to pass data into a request.
I couldn't find a specification, but I did find [1] which led to [2], both of which indicate that there's an idea of parameters within the URL hierarchy itself. It seems to me that it could be useful for API versioning at least.
As to the article itself, I completely agree. The problem is, though, that there's just not enough incentive for folks to actually use the Web standards suite the way (I want to believe) it was meant to be used. How much would cost to make a fully-semantic, JavaScript-extended (but not -requiring), machine-usable website which is also adaptive and beautiful and appealing to humans? And how much benefit would one realise?
Rather than do things the Right Way, folks just hack through and do it A Way, and get on with life. I wish that it weren't so, I truly do, but it is. At this point, railing against it feels like railing against the tides.
[1] http://doriantaylor.com/policy/http-url-path-parameter-synta...
It is amazing to look back to my own development history and remember that I too used to care about proper status code, using metadata elements, and the like.
But after adopting so many, and seeing so many either fail to catch on, or infighting tear them apart, or the big names pick them up and then put them aside ... it's not surprising that current developers don't think about them or don't care.
> And, technically, you’re supposed to make sure GET requests are idempotent, meaning they can be repeated with no changes to a resource. So you should be able to hit Refresh all day on a GET request without causing any data change.
I think it's important to note here that idempotence does not mean that the resource will never change. It means that the resource will not change as a result of a get request (barring some meta requests, for example `GET /my-last-http-request`). A resource can definitely change. I think the author would agree with me but found his language there a bit ambiguous. Allow me to quote the good reverened himself, Dr. Roy Fielding:
> "More precisely, a resource R is a temporally varying membership function MR(t), which for time t maps to a set of entities, or values, which are equivalent. The values in the set may be resource representations and/or resource identifiers." (emphasis mine) [1]
Immediately prior this quote he references "the weather for Los Angeles today" as an example of a resource complete with its own resource identifier (perhaps GET /weather/la/today, most-to-least specific). The contents at the end of that URI will most certainly change, but they shouldn't ever change based on the request.
[1] Section 5.2.1.1 https://www.ics.uci.edu/~fielding/pubs/dissertation/fielding...
It means that the act of GETting the weather right now will never change the weather. (If you want to change the weather via HTTP, you'll need to do it with a method like "POST /weather/hill-valley" and a body like '{"rain": "stop"}'.)
Think of it like a getter function that gets syntactically hidden by the language: you wouldn't want "if (user.accesses < limit || user.accesses > threshold )" to have two different values for "user.accesses" based on the number of times you referred to it.
In a less ludicrous example, it also means "GET /tokens/reset/dsf46JtCb385PDs2" shouldn't log you in and invalidate the token, something that most password-reset pages fail to comply with.
"My point is that a lot of web developers today are completely ignorant of the protocol that is the basis for their job."
One thing I'm not sure I agree with, though, is the mobile application "linking" he mentions. When have non-web applications ever linked? What of all of the old "desktop applications" that everyone used to use? Those don't typically link, why is there a different expectation for mobile applications? Oh, and most mobile apps do support linking (in some sense, though it's not really a primary feature). Perhaps I'm misunderstanding what he means with this.
However, most of the apps I've seen have a Share feature which one can use to retrieve the link.
Making the analogy to desktop apps is fair, but I think the better question would be why not? The URI scheme registration mechanism in Android and iOS allow the dev to create a new OS level namespace for his or her app and all 'pages' inside. For example, http:// is the scheme for browser app based addresses just as venmo:// is the scheme to address all payments or user profiles in the Venmo app. Why not leverage this technology to make app content more open and easier to access?
Now, if we could just get rid of the app store enforced install process and have apps automatically retrieved and cached, things would be a lot smoother...
I totally see what you did there ;). http://xkcd.com/1367/
I feel the lack of linking in apps is done partially because of laziness, and partially on purpose. Most apps that are neither games nor utilities seem to be made to control the viewing experience and/or help the authors trick the users into parting with their money. Increasing interoperability seems to be counterproductive if the only reason for your app to exist is to earn a quick buck from the less savvy users.
I disagree with your last comment as you could make the same argument for websites in general - there will always be spectrum of usefulness/scamminess. Utilities or games might not have the concept of 'pages' as a news app or a social photo upload app, but imagine being able to link to specific constellations in the star chart app, the various calculator faces of some popular calculator apps, or a user's city in Clash of Clans. It's going to be awesome.
I'm imagining it, and in most of those cases I'm imagining the horrible fragmentation clusterfk that will have to be solved in Yavascript - linking to "some popular calculator apps" (plural) means either providing a mess of links for the user to decide, or somehow guessing which one to show.
Two alternatives that both land in the "NAH GANNA HAPPEN" category:
* "everybody agree on a standard prefix for basic things like calculators" HAHAHAHAHAHAHAHAHAHAHA
* "allow client JS to detect installed apps" see also the detect-visited-sites-with-CSS info leakage for why this would be terrible...
"everybody agree on a standard prefix for basic things
like calculators" HAHAHAHAHAHAHAHAHAHAHA
urn:calc:rpn
You click the link and your mobile phone then looks to see if any of your calculator apps satisfy that urn.
If they find any they open it up or present the user with a choice between them or open the one the user configured as their choice.
If they don't find any they check the app store or web and offer the user the opportunity to install one that satisfies that urn.
Actually this is exactly what androids intents do as well minus the app search and install. But I don't think they use urns under the hood.
...except the main cache on a web browser is nothing more than a speed boost, and cannot be relied on.
On APIs, I'd say it's a sin to not return proper HTTP headers, but when the end user on a web site sees a File Not Found that's 200 Status OK versus File Not Found that's 404, it doesn't really matter.
And while it matters for things like the Googlebot, over the years developers stopped caring, because delivering an attractive and descriptive status page was more important than delivering an error status and not rendering a page at all.
I'm not saying it's right, but it was largely a ornamental step in the early days of the Web, so I understand why it disappeared. If browsers proactively provided data to users about HTTP status codes, I think they'd be adhered to much more.
To a client, the URI should be opaque. If a client has to start to make meaning of the contents of an URI, the server loses the ability to change its implementation (or just move resources around).
Edit: Ah, its the same author for both blog posts. That explains it.
(And yes, I know Gadgetopia’s URLs are somewhat lame in this regard. There are reasons why I did it, but changing it now - See more at: http://gadgetopia.com/post/6346#.dpuf)
In providing APIs for third parties that are rate limited, our APIs return a '509' error to tell you that you've exceeded your query rate. Which if you just sleep for a bit and then send a request you'll get an answer, but no, as a 'soft' error too many developers just retry it immediately which is useless. Some folks want only 3 errors, 'ok', 'dead', or 'try again' and can't deal with any more subtlety than that.
In general though, it's just hard to know exactly when and where to implement what standardization. It's not just HTTP. The entire world of programming and computer science would be much more elegant with intelligent standardization, but it's literally a hard problem. The source of information is the hard problem. It is subject to change.
How can you standardize something that is consistently evolving, evolving to the point that it is expected that the functionality of the system is going to exceed all known expectations of it's operation and behavior? As developers, we are consistently striving to exceed our own boundaries, to do things better, more optimally, more safely, more legibly, more precisely, more correctly. Our expectations of ourselves is what screws us when we think about 'the big picture', because we apply the same model of reasoning to 'the big picture'.
Yes, we do not know what we do not know. However, we additionally do not know how to determine the difference between 'that which exists that we do not know' and 'that which does not exist that we do not know'. That is the core problem that lays underneath the lack of implementing standards, and it the reason people standardize things in the first place. Standardization prevents us from re-inventing the wheel. Standardization makes things safe. Standardization makes complex systems predictable even when information about adjacent systems are unknown. Standardization is a vehicle for implementing trust.
People get confused because there are too many words for the same thing - too many ways to describe the same things, too many languages to do the same thing, too many explanations meant to teach the same thing. Either we have to learn the hard way (by observing massive failure, or by experiencing it), or we have to hope we are implementing things the most correct way, and trust we will get there.
There's a reason why URNs are not used in the code, there are not enough by themselves, if you need use URIs.
There's a reason people don't use every status code in every webapps. It's a waste of time, better spent elsewhere.
Most webapps uses more then GET.
Etc. Anyway, HTML is not broken, it's used everywhere by billions of users every seconds.
It's pointless ultimately.
The real question is how can we improve a protocol so it's better used by most people. Hence why it's evolving. Nothing should be set in stone forever and then expect everyone to comply to it or die.
How many of us got our start in web development by viewing the source of the page? By looking at how others did things? Hell, I still do that now.
Apps are the Microsoft of the Internet -- closed, proprietary and secretive. Open source and github has helped this some, but meh.
So, so true. In fact, this happens depressingly often when "technology" is defined as narrowly as "the app a developer is working on" and "history" is as short as "a month ago".
This has always happened, but it feels like it's gotten worse since "agile" became widespread. As if the principle of YAGNI ("You Aren't Going to Need It") has been turned into a poisonous assumption that nobody before you could have put any thought into the future. Rather than trying to understand the natural extension mechanisms that might already be in place, people just jump in and write more special-case code.
It's already too late to make HTTP something sane. What we need are developers who understand and appreciate simplicity and minimalism, to make something awesome with a chance of actually catching on. (Please don't xkcd/927 me.) And they'll have to be fierce in protecting it against third-party extensions (which may not even be possible with something this popular.)
That's unlikely to happen, so I guess we'll just keep playing this perverted game of Jenga ad infinitum.
418 I'm a teapot (RFC 2324) This code was defined in 1998 as one of the traditional IETF April Fools' jokes, in RFC 2324, Hyper Text Coffee Pot Control Protocol, and is not expected to be implemented by actual HTTP servers.
http://http2.github.io/http2-spec/index.html#ErrorCodes
A quick Google shows that it's previously just been used by Twitter to demonstrate rate limiting.
I have a server that returns a 401 when the Authorization header is missing, but I also like to return a JSON body with a little more information as to what went wrong (ie. the header is flat out missing or the credentials aren't being sent in the correct way, with a link to the documentation), and superagent allows you to output the `response.body` in the `end()` method callback.
I think of superagent as the requests[1] of js. In fact, I usually require it under the name requests as they feel so similar.
0: https://github.com/visionmedia/superagent
1: http://docs.python-requests.org/en/latest/
EDIT: link formatting
http://bugs.jquery.com/ticket/7868
http://stackoverflow.com/questions/5884286/jquery-ajax-with-...
I'm not sure what this is, but it could be some sort of expectation where all responses are expected to get a 200 along with wrappers like this:
{"status":"error","reason":"not found"}
{"status":"success","response":{...}}
When someone used to doing things like the above talks to someone more accustomed to RESTish ways of doing things, quite a bit could get lost in translation and communication often completely breaks down.
Treating your application and data as resources (REST is the style of the web) and leveraging the power of HTTP so powerful. You are not a crabby old man. If you do count me in, 33 years old. I could make a career consulting and only fixing http and resource abstraction/usage problems.
IMO though educating our fellow developers on HTTP is easy compared to the REST HATEOAS constraint.
Developers ignore most of the status codes because they introduce unnecessary complexity on both ends (client and server) and do not always map well with the application logic. With JSON payloads, HTTP is used as a "light vehicle" to carry the application logic. Since the payload is going to be specially interpreted anyway, I don't see a benefit in cramming legacy HTTP cruft in the headers and in status codes.
The really annoying thing is the web server spitting out HTML on error conditions instead of the expected JSON.