"Trickle" attack on Minecraft servers, or Don't Use Thread-Per-Connection | Hacker News Reader

"Trickle" attack on Minecraft servers, or Don't Use Thread-Per-Connection

corbinsimpson.com

44 pointsAndyKelley26 comments

Lots of purpose built servers are vulnerable to slowloris type attacks. Which doesn't mean that they shouldn't be fixed, just that it's hardly remarkable to find one. While certainly everyone is entitled to do disclosure however they see fit, it comes off kinda jerky to drop a working POC on the community just to advertise your unfinished competition.

You're entirely correct. I should explain that this PoC was written not to advertise Bravo, but to explain the nature of the attack to a handful of members of the community. I don't advertise Bravo -- the people that it was written for were using it back when it was called Beta, and the wider community really doesn't care about custom servers very much. Additionally, mineserver is a much better server than Bravo; if I'm plugging custom servers, I will plug them instead.

Agreed. Yes, it is easy to attack video game servers. Indie game servers even more so.

If you want to DoS a server so that players can't die wool or smelt iron or hit pigs for porkchops, you should probably reevaluate what you're spending your time on.

Paul Tyma's counterpoint from 2008, that thread-per-connection is scalable to tens (and maybe hundreds) of thousands of threads, using recent Java & Linux NPTL:

http://paultyma.blogspot.com/2008/03/writing-java-multithrea...

Indeed, I'm reading this wondering how 400 connections could even crash a JVM on an OSX machine. It's surprising.

I suspect it's not a threading issue in the sense he's describing. 400 threads probably shouldn't give you an OOM unless your heap size is set perilously low (in which case just about everything will give you that error).

It's also painful to then see him argue that epoll/select/NIO is the magical path to freedom. Erlang and Akka developers are trying to to cry and laugh at the same time when they see this.

400 threads with a thread stack size of 8M means that 3.2GB of virtual address space needs to be committed, which suspiciously close to the usual limit on a 32 bit system.

If you want to create that many threads, you need to start by severely tuning down the stack size.

Good point. It does require tuning.

And use 64-bit systems.

Or use 64 bit systems. That 8M isn't physical memory, it's just allocated address space. Only the tiny handful of pages at the end of the region actually get faulted in.

The thrust of my argument is merely that one thread for every connection is bad. (Minecraft happens to use two per connection, but that doesn't change the fundamental argument.)

You are correct that the problem is not directly due to thread resource usage, but to the additional resources attached to each thread. I was able to raise the heap size, but not able to get past around 600 connections on my 32-bit x86 machine.

NIO/MINA is the right answer on Java. If you are using Erlang, well, I would suppose you would not need a Java-based solution. :3

"NIO/MINA is the right answer on Java."

I think my point, which admittedly is only informed by the work of some brilliant people, is that your point is not supported by data gathered on modern hardware with modern operating systems running a modern JVM. At least, not in all cases. With care and some tuning, people can get extremely good performance out of the thread-per-connection approach. Please check the document linked in the parent to this thread (http://paultyma.blogspot.com/2008/03/writing-java-multithrea...)

What's more, even the systems that are very good at this do not use something like NIO, epoll, or select (as anything other than a very buried implementation detail). Erlang, for example, doesn't restrict you to the kind of model that raw NIO does. NIO is a fine thing for what it is, and a potential building block for many things, but it's not a very compelling as a server framework. MINA is a better suggestion, but still not a universal catch all for these sorts of problems.

I'm not trying to say, "NIO is never right, tpp is never wrong." I'm just saying your categorical statement is not always correct. Making that statement is wrong, and it's misleading, and I don't think the problem you found was really due to the unavoidable overhead of the threading.

Okay. I concede.

I didn't even know a person COULD enter these words into an internet text box. I thought the internet itself would reject them. ;)

I salute you, sir.

One thread per connection might not scale but in this case other things could be done to mitigate the potential problem before re-writing the whole server. Perhaps use a special encoding for the authentication that doesn't use unbounded strings for username but a fixed size array of chars? Then maybe start a timer and if authentication isn't performed during a given time period, disconnect that client.

Say you kill connections after 1 second (pretty aggressive) and you can hold 1,000 connections open (that's a fair number of threads)

All I have to do is create 1,000 connections per second and I'll hog all your thread resources. That's not too hard to do.

Killing connections after a full second has elapsed probably isn't at all aggressive in the context of a game server. If the connection is legit but takes a full second to send authentication info, then there's no hope for the game actually being playable over that connection.

Depends on the game. Minecraft is hardly twitchy, especially if you're building, not fighting; the client proceeds as if each action has already happened, so you don't have to wait for the roundtrip between actions.

1 second might not be too agressive for initial log-in -- that is where the real problem he is trying to solve is. If the username + password are not unbounded strings, but say, fit in an array of a 100 chars, then you should expect to get that data a lot sooner than 1 second (maybe 200ms?). So then maybe you can handle 5K connections... Yeah you are right, this is still not as good as a good async framework but the point was that some smaller things can be tried before a complete re-write. It might just be good enough.

Nobody has really publicly talked about it, but there is now code in the Notchian and CraftBukkit servers to limit connections from single IPs to once every 5 seconds, and to disconnect slow logins by killing the thread after 30 seconds, effectively limiting the attack to 6 threads used per attacker. To actually bring down a server this way would then take 100 attackers or so.

This really is just a proof of concept, after all, and not a tool which can bring down entire servers single-handedly. (At least, not anymore.)

Exhausting ports or generating a lot of TIME_WAIT states would also cause similar problems. How to protect against those ?

You'll be hard-pressed to exhaust all ports: modern operating systems track connections by source address + source port + target address + target port. I wouldn't be surprised if the TCP sequence number is also part of the mix.

TIME_WAIT times can be tweaked with the net.ipv4.tcp_tw_recycle and net.ipv4.tcp_tw_reuse sysctls, on Linux systems anyway.

Exactly, you can only really exhaust the ports from the server address/port to your single host, which is effectively only DoSing your own access to the server. Ive seen over a million sockets in time_wait and it did not effect the servers.

However if doing something like behind a single software load balancer then the number of ports available will be limited to the 65k connections since the source address/port and target address(the lb) are fixed.

I seem to recall that there are five numbers involved: Source address, source port, target address, target port, and a fifth number. I want to say it's the file descriptor, but that just sounds wrong when I type it out.

Running out of ports is a poor way to put it, but it can be read as "Run the target machine out of fd (file descriptors)." In unixy systems there is a system parameter limiting the maximum number of file descriptors that can be open at one time (actually 2, per-proces and system-wide limits). These limits need to be turned up for big servers.

The why is a pretty boring story: the number exists as a check against runaway processes or big jobs (e.g. busy servers) causing an I/O overload, so the numbers are set to a "reasonable value" by default. Of course the reasonable value is usually much lower than the system can handle, and at tuned for older common hardware.

There are two ways to avoid getting TIME_WAIT states in the first place:

1. Convince the client to close the connection. The initiator of the close will be the one holding the TIME_WAIT socket (for ~2 minutes)

2. Send RSTs to clients that refuse to close. RST avoids creating a TIME_WAIT state. I think this is slightly more dangerous than a normal FIN close, because wandering duplicates could interfere with the next connection. From Python, you can close a TCP socket in a way that causes a RST to be sent:

  # Set SO_LINGER to 1,0 which, by convention, causes a 
  # connection reset to be sent when close is called, 
  # instead of the standard FIN shutdown sequence. 
  self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, 
      struct.pack("ii", 1, 0))

(From the patches and branch in http://twistedmatrix.com/trac/ticket/78)

Except if your language/platform hides the actual thread implementation in an epoll loop.

coughhaskellcough

Reply on news.ycombinator.com