Comment by colanderman | Hacker News Reader

Comment by colanderman | Hacker News Reader

Some IPSes (Intrusion Protection Systems) that perform deep-packet inspection won't pass such traffic.

But this isn't really a "bug" per se; the TCP model is a stream is a stream is a stream. There's no notion of time, packets, or correlation between streams. So browsers (and the OS) are acting only as they can; by treating a TCP connection as two independent streams.

(Though, how could it be otherwise? Assume HTTP over SCTP (sequenced packets). We can't require, or even allow, HTTP servers to ignore response packets that arrive "too early", since it's possible that observers of the client (e.g. Wireshark) may not observe the exact same timing, which would lead to divergent interpretations of the conversation.)

Amazon does this too. Upload APIs will return 4xx errors well before the body is uploaded in the event there's an issue with the headers. Not that (a) most HTTP clients pay attention to this, or (b) they can do anything about it without closing and reopening the connection.

It is fine to respond early to a HTTP request, e.g. returning a permissions error when a user tries to upload a file. Beware however that some clients will go badly wrong if they don't get to send all their data.

What can happen is:

* Client starts to send HTTP request (e.g. POSTing a file). The file is large, so it will take a while to upload.

* Server spots that the user isn't allowed to upload the file and immediately returns a 4xx error of some kind.

* Server then thinks all is finished, closes connection.

* Client, still sending the file, gets an error as the write() fails. Complains about the broken connection to the user but never notices the actual HTTP response.

Many HTTP clients are written in a simple 'send my request, then (and only then) read the response'. They don't react well to getting an early error message. Often you have to work around this by not closing the connection to the client, and continuing to slurp up any further data received.

>> Some IPSes (Intrusion Protection Systems) that perform deep-packet inspection won't pass such traffic.

We had a vendor who's product did this. Everything worked except one feature (the main feature) and a quick glance at the firewall logs showed 'malformed tcp packet' flooding the console.

It was a simple thing to disable (from just their appliance, not the whole network), but I still found it odd that they did that.

> Some IPSes that perform deep-packet inspection won't pass such traffic.

So will those ISPs also not pass Amazon's upload APIs responses? That would be pretty sloppy!

I think he meant "IDS" as in "Intrusion Detection System"

No, I meant "IPS", "Intrusion Protection System".

The difference, if anyone's still playing along, is what action the device takes. An IDS (detection system) is a monitoring and alerting device; traffic still gets through. An IPS (prevention system) drops the flagged traffic.

As you alluded to, the distinction between IDS and IPS is largely configuration and mode of operation.

Years ago, IDS and IPS were separate products, where IDS was the earlier, more primitive version of the other. Now-a-days, you are buying an IPS, which is run either in alerting mode (operating like an IDS) or in "shunning" mode, where the device tracks some defensive action (such as dropping traffic, bandwidth throttling, blacking the IP for a fixed period of time, etc).

"Shunning" mode can be dangerous, since you are essentially building in a feature to "Deny service to X for Y amount of time" into your network.

Attackers can spoof attacks to deliberately trigger the shunning of legitimate users. Because of this, it is less common to see an IDS/IPS with shunning enabled in production. It depends on where "Access to service for legitimate users" and "stopping and possibly hurting attackers" fall on the priorities list.

IPS (Intrusion Protection System), not ISP. And yes, they'd drop the connection on such an error response if configured to do so. (Which is likely what you want to do anyway in this case. Amazon's engineers had the foresight not to respond early in the case of a successful upload thankfully.)

Ah! Complete reading fail on my end. (thanks for the edit, it is much clearer now, I apparently substituted ISP for IPS).

I don't think it is possible to respond early in case of a successful upload, after all, that means the upload can still fail for a variety of reasons. Success indicates that you can move to the next state, and an 'early success' might still turn into a late failure.

Reply on news.ycombinator.com