Open Letter to Calendar Developers about HTTP | Hacker News Reader

Open Letter to Calendar Developers about HTTP

lists.w3.org

61 pointsbpierre33 comments

What do we learn from this? If you provide a public-facing API or service for direct use by applications over the net, use a unique host name / DNS entry for it so you can point it elsewhere or delete it at leisure. DNS entries are cheap, millions of HTTP hits might cost a bit more.

For even less fun, there have been numerous incidents of consumer accesspoints/routers hardcoding NTP servers by IP address[1][2].

[1] http://pages.cs.wisc.edu/~plonka/netgear-sntp/

[2] http://web.archive.org/web/20060408150213/http://people.free...

I would guess the problem is that most sites are going to serve up a 404 in this situation, and handling 404s is a bit trickier, since they might be transient (like a proxy gets in the way, or the hosting service has temporary issues). So they just don't handle it, since polling a server and getting a 404 is going to work well enough for most use cases. Maybe put a little exclamation point next to the calendar or something, but still poll it.

And then they handle a 410 the same as a 404, even though a 410 has very little chance of being transient. It's just rare enough that they don't even realize they don't support it properly.

I'd wager 99% of codes have a big if(200) { // Do stuff } else { // Try again later }

Actually, a bunch of them probably don't even check for response code.

Edit: If dataaccessd is really the iOS native CALDAV integration, that's a bit annoying. Of all people, they should be able to handle that correctly.

  if(200) { // Do stuff } else { // Try again later }

That's like an "idiot-filter" interview question. "Is the following code valid?"

This is a very interesting observation! I imagine, there could be a great conversation derived from this seemingly easy question.

Seems like it should be paired with a "Is the following code useful?" about code which specifically handles every error code listed here http://en.wikipedia.org/wiki/List_of_HTTP_status_codes and contains code to handle unlisted ones.

Then "where do you draw the line?". Sometimes if(200){}else{} is plenty.

This is a silly objection. Not every code merits specific handling because there are groups of codes that merit the same handling.

if(200){...}else{retry} is never plenty.

Assume it only queries your server. Assume your server will only ever meaningfully return 200. Assume the method has no reason to ever move (so no 301 redirect (edit: and redirects are generally handled by your request library anyway)). Assume the only time it should fail is if your server falls over on itself, which should be temporary.

What else would you do then? These are not crazy expectations. Exponential backoff might be reasonable, but I'd argue that's part of "else{retry}" since it's inherently retrying.

> These are not crazy expectations.

Sure they are. Choosing to communicate over HTTP means allowing for the presence of proxies between "your" client and "your" server. In HTTP, you don't control the whole path, even if you control both sides.

Proxies are also fuzzy. I assume you're referring to e.g. a 407 or 511 error? Many times they're handled by something between you and the network library, often the operating system. Sometimes they're not handled at all at the system level (hooray, Android (until recent versions)!), so if you didn't put a proxy in there, you don't have one.

VPNs can also exist, but they sometimes send custom error status codes[1]. How do you handle those? Retry if `code % 3 == 1`? Will your program even be informed if the VPN connection is restored, or will you have to just blindly retry until it works?

[1]: http://compnetworking.about.com/od/vpnsetup/tp/common-vpn-er...

I was superficially commenting on the fact that '//' comments everything to the end of the line[1], but as @jimbobimbo pointed out there are multiple layers here, and it would be a good lead-in to a conversation in an interview.

[1]: It just struck me as one of those "low bar" technical interview questions.

haha, that kinda makes me want to make a language that prevents comments from breaking scopes, so "}" would implicitly also be a comment terminator :)

Off the top of my head, Vim has a folding mode where "{{{" and "}}}" are used to mark the start and end of a fold. So you can see this on the code:

  // ==== Section 1: Code ==== {{{

  BLAH

  // ========================= }}}

Granted, since this is Vim-specific, you usually only see it used in Vim plugins, because you're unlikely to have someone editing with another editor.

"What port does ping work over?"

Depends on the position you're hiring for. I imagine a lot of smart people might not know the answer to this because they've never really thought about ICMP or the OSI model.

Am I the only person who after about 3 months would have started returning random and numerous events? Then clients would voluntarily unsubscribe.

Nope. I'd be hard pressed to not do a little iCal trolling. I'm sure the moon would get destroyed by the Death Star at least a few times per year. Not to mention a new phase of the moon: Space station.

"I have a 2PM with 'Goatse.cx', I don't remember this, let me just click the info link. URGHWTF."

> Then clients would voluntarily unsubscribe.

Or write nasty blog posts about you that pollute Google results forever.

I have exactly the same problem with a project I run. I generate iCalendar versions of the timetables for each course at my university and let students download them. The timetables are static and do not change, yet I still get hundreds of hits every day from dataaccessd, even in the middle of summer when nobody has any classes.

EDIT: http://notlate.co.uk/

EDIT2: Not that it's enough load to be noticeable, but it's a little wasteful on the part of dataaccessd IMO.

I have a feeling many of these are from rubbish applications that have mostly been abandoned by their developers but still remain installed and running in the background for a lot of users.

Except dataaccessd is how iOS's CalDAV sync process identifies itself, so I'd say it's not exactly a "rubbish" application that's been abandoned...

Fetching stuff over HTTP can be incredibly stupid. I there is a security list out there that some little firewall company decided to have their devices automatically download. The list doesn't change that often, but every single device that company made likes to check every 10 minutes or so. Nothing can stop them, they follow Wilya's algorithm [1].

[1] https://news.ycombinator.com/item?id=6268919

Side note: it's sad to see that Apple is a huge perpetrator in this one. I hope they fix that bug.

I'd never even noticed there was such a thing as a 410 until this. And it's not surprising - what incentive is there for a team in the real world with a shipping product to honor it?

There's a pretty obvious incentive to ignore it: it takes developer time to implement, and the result of the 'feature' is that your customer may get permanently locked out of a net resource. What happens if a website admin accidentally puts up a 410, or - worse yet - if a previous owner of the domain once put up a 410 for a page with the same URI.

I suppose there's an RFC for the code somewhere, but it seems like a bad idea. I pity the tech support agent who has to explain why his/her customer's browser refuses to visit foo.com/bar when everyone else can view it and there's no longer any way of knowing that once, for an afternoon in 1997 the page threw a 410!

A long time ago I published an XML file on my website for some ill-fated Google attempt at providing extensible search results. I took the file down some time later, but the Google crawler continued to hit the URL.

Amusingly, I asked around within the company to try to find who was responsible for the crawler repeatedly fetching the 404 but ultimately couldn't figure it out. (It seemed like the decommissioned service had maybe registered the URL with some still-running automated refetching service?) It eventually stopped on its own.

Google will continue to request 410 resources for quite a while too. As far as I can tell it treats them the same as 404.

This is another case for rate limiting

This is probably a case where an exponential backoff would be preferable. When the client is faced with a fetch error the first time, it should wait and try again in e.g. 10 minutes, and double the wait for each repeated error. This way, after a week of downtime at the source, each client would send less than four requests for the resource per month.

No, this is a case where obeying the HTTP specification would be preferable. 410 GONE means: Gone. Don't try again.

Agreed. Incremental backoff makes sense for the 5xx-class errors, where the error is known to be server-side. But 404 is the only 4xx-class error that it makes any sense for: in any other case, it's pretty certain that your side is doing something wrong. Even when you're not -i.e. if you're absolutely certain that this is the right URL for what you want- a 404 should still trigger some kind of notification so that a person can look into it.

Using incremental backoff for 410, in particular, is to miss the point entirely. 410 very explicitly means that you should not try again. The fact that it's so rare only lowers the chance that it's being used incorrectly: someone savvy enough to set up a 410 knows enough to mean it.

That said, I'm starting to think that if you're going to implement 410 on your site, you should also start logging activity. When a host keeps looking, either incrementally or otherwise, start refusing connections from that host at the firewall level. A program that sets 4xx-class errors to fail silently probably isn't doing anything about refused connections, so an IP-ban should wake the developers up. Even if it doesn't, you won't have erroneous requests clogging your server logs, so you still benefit.

> The fact that it's so rare only lowers the chance that it's being used incorrectly: someone savvy enough to set up a 410 knows enough to mean it.

I don't see how your conclusion follows from your premise. I fairly often come across cases where obscure options are mainly used by those who don't understand them, and in the absence of data I wouldn't assume that the same couldn't be the case for HTTP responses.

Exponential backoff is a quite reasonable compromise for those kind of situations.

Why should I believe a server knows a URI will never work ever again? If someone hijacks google.com and returns a 410, should we never try again?

That's the problem with writing specs that include all sorts of "useful" codes. Like SIP's 6xx "globally unavailable", as if a client is going to trust that some server knows the exact state of the known universe.

Reply on news.ycombinator.com