The curious case of slow downloads
blog.cloudflare.com
blog.cloudflare.com
Sounds simple enough, right? Disregarding accept(2)ing connections,etc... it's just going to sit in an endless loop, select(2) on the sockets then read what comes in, and write that to the quakeworld server.
To debug this, everytime someone sent some data over the network, it would write a simple message ("blah") into a file on my home drive, called "~/blah" (very sophisticated debugging, I know).
Now, the semantics of select(2), cousin of poll(2), are interesting.
My understanding: select(2) blocks until you can read data from the socket.
Actual behaviour: select(2) blocks until a subsequent read on the socket does not block.
Fun fact: if there's not data available on the socket, read does not block, but returns with an error immediatly.
Result: Over the weekend, the process was spinning in an endless loop, writing 'blah' into a file on the nfs-home drive as quickly as the filesystem allowed. This being a university scale solaris NFS server, that's pretty fast.
End Result: A test file called 'blah', filled with about 800 GB of 'blah'. Good thing the admins where nice people and didn't shoot me when I went to explain the next monday why the home drive was filled with "blah" XD
EDIT: read(2) doesn't actually error on EOF, just returns 0, effect is the same though.
They should be the ones explaining why they don't have disk quotas set up on a shared system.
But yeah, maybe them being somewhat at fault as well helped me out a bit :)
https://en.wikipedia.org/wiki/QuakeWorld
Now, 800Gigs sounds like a lot, but I distinctly also recall that time as the era when we ran across 8GB clip issues. It wouldn't be unheard of for drives to actually be 20-60GB in size around this era, maybe larger for the expensive SCSI kind.
Getting 800GB online storage with something like RAID5 or RAID6 would probably take two fully loaded controllers; lets say 10 drives of usable storage across two arrays. (20 drives). That back of the envelope math is about 40GB per drive.
I could see this being an easy case at a LARGE university in some tech centrist area at about the timeframe of 16-18 years ago.
For example, Microsoft may be changing their image, but their core software is closed source. <cloud provider> may be great, but can you pull off something like this when you've got an issue? The importance of being able to debug and patch your mission critical systems is hard to overstate.
Please encourage your employer to fiscally support the open source code that they rely upon!
Not OSS != no source access. A cloud provider on top of the MS stack would most likely have Shared Source Initiative licenses.
`git clone` is an incredible power. Companies should reward those who are generous with their code.
Careful with those goalposts, you're tearing up the pavement.
> Companies should reward those who are generous with their code.
So companies "should reward those who are generous with their code" but not compensate them for it?
No, they absolutely should compensate them for it. Buy OSS.
This issue didn't get solved particularly quickly, either. Nor was "source to the whole stack" required -- kernel source made for a illustrative blog post but was not really part of the process.
Do you have source files for the CPU? The motherboard? Your DRAM controller? At some point you draw the line and make shit work. Tools plus docs plus code is great, but you can make do with just one of 'em in a pinch.
Licensees may use the source code as a reference or view within a debugger,
but may not modify the code or use it to create derivative works.Furthermore, no source access != no way to fix problems.
People have been fixing things with no source for ages, EULAs be damned. If the fix is literally flipping a single bit, the decision is easy. Just do it. The vendor doesn't need to know nor care what we did to fix it, but often giving a detailed description to them of the problem --- one that comes out of the process of fixing it --- will help them fix it quickly and distribute to their other customers too. I've had this experience a few times.
Obviously, "the byte at 0x73F441 of somefile.dll should be 31, not 32" is not how you should communicate such things to the vendor, but they do appreciate your effort in debugging the issue instead of immediately blaming it on a fault in their software. ;-)
(And relatedly, no source access != no way to find problems either, as the security community shows quite plainly...)
When this works well, it's great. You've got the experts on the problem. Not all support is good though, obviously, and if you have a mission critical problem that you need to fix overnight, nothing beats the 'self-support' that you can do with open source software.
Reminds me of the heroku shaming posts that (then) rap genius had to do regarding dyanamo scaling and performance - http://genius.com/James-somers-herokus-ugly-secret-annotated They had no other way forward, even though they were paying the "experts"
This is why companies should pay open source contributors more frequently :-)
In early TCP implementations, if you read from a TCP connection one byte at a time, and the sender was writing faster than the receiver, the sender TCP would get an ACK opening one byte of window. The sender TCP would then transmit a packet with only one byte. This often came up if a TCP connection was driving a slow device. The fix was to not send a window update until there was at least one full packet worth of window available.
This is the same problem, but at the OS buffering level. Linux has a similar solution. The problem is 1) servers today have to protect themselves against readers that are "too slow", because that can be used as a denial of service attack, and 2) the approach to doing this was not well designed.
This is a tough one for Cloudflare, because they're in the DDOS-prevention business. They have to decide whether a slow reader is an attacker or just someone on a slow connection. Still, insisting that a reader consume at least 5MB/minute is a bit much. Some people are still on dialup.
I'm looking at you Arbor.
You can't use a reset just because you want to be fast or don't feel like sending any more data. Resets are for something abnormal happening. A timeout isn't abnormal to a TCP connection. It's part of the process.
A TCP connection may terminate in two ways: (1) the normal TCP close sequence using a FIN handshake, and (2) an "abort" in which one or more RST segments are sent and the connection state is immediately discarded.
and
A host MAY implement a "half-duplex" TCP close sequence, so that an application that has called CLOSE cannot continue to read data from the connection. If such a host issues a CLOSE call while received data is still pending in TCP, or if new data is received after CLOSE is called, its TCP SHOULD send a RST to show that data was lost.
So this behavior seems fine unless another RFC specifically prevents it.
It's also a simple and state-free way of influencing other devices on the network. Arbor is an interesting example - when doing DDoS mitigation, an approach which doesn't require a lot of resources or statefulness is extremely useful.
On one hand, I want to agree with you: this is terrible and makes everyone else's life harder.
On the other hand, maybe its TCP's resource requirements that are at fault.
HTTP does not have a way of sending an error in-mid-transfer, so a TCP reset has to suffice. The other option would be to blackhole the connection, but that would be worse for both sides.
That's not correct. Both Content-Length and the chunked encoding scheme would allow the client to know that the connection was terminated before the end of stream was reached.
The problem here is that (TIL) on linux the "writable socket" state behaves completely differently than the "readable socket" one, and a socket can be effectively writeable even though poll(2) reports that it isn't.
Having a timeout on a poll (or select) is not "something extra". it's something entirely normal and necessary for any non-trivial software, especially public-facing ones.
nginx called send(file), then polled the socket for write with the timeout, closing the connection if the timeout was hit. It has everything to do with poll/select timeouts, and especially with Linux letting poll hit timeouts when waiting on effectively writable sockets.
The goal of NGINX here is not to measure whether a socket can consume any data whatsoever. A connection that consumes 1 byte every 15 seconds is still considered stalled, and should be killed. Using the time between sendfile invocations is not the goal, it's a means toward implementing a minimum rate, and they implemented a minimum rate the wrong way. It's an X Y problem and that's not the kernel's fault.
> [SO_RCVTIMEO and SO_SNDTIMEO] only have effect for system calls that perform socket I/O (e.g., read(2), recvmsg(2), send(2), sendmsg(2)); timeouts have no effect for select(2), poll(2), epoll_wait(2), and so on.
And from my understanding would have the same effect as a timeout on select/poll, so it's not the correct way to do anything.
It is a problem entirely of NGINX's devising, not a unix stack problem.
In a nearby comment, LoSboccacc notes that they've hit that issue [removed: on S3].
Slow requests (like slowloris) were much more effective.