Hacker News Reader

Comment by sgc | Hacker News Reader

I am ignorant as to how most bots work. Could you have a second line of defense for bots that avoid this bomb: Dynamically generate a file from /dev/random and trickle stream it to them, or would they just keep spawning parallel requests? They would never finish streaming it, and presumably give up at some point. The idea would be to make it more difficult for them to detect it was never going to be valid content.

View on HN

You want to consider the ratio of your resource consumption to their resource consumption. If you trickle bytes from /dev/random, you are holding open a TCP connection with some minimal overhead, and that's about what they are doing too. Let's assume they are bright enough to use any of the many modern languages or frameworks that can easily handle 10K/100K connections or more on a modern system. They aren't all that bright but certainly some are. You're basically consuming your resources to their resources 1:1. That's not a winning scenario for you.

The gzip bomb means you serve 10MB but they try to consume vast quantities of RAM on their end and likely crash. Much better ratio.

Also might open up a new DoS vector on entropy consumed by /dev/random so it can be worse than 1:1.

Entropy doesn't really get "consumed" on modern systems. You can read terabytes from /dev/random without running out of anything.

As mentioned, not really an issue on a modern system. But in any case, you could just read, say, 1K from /dev/urandom into a buffer and then keep resending that buffer over and over again?

That's clear. It all comes down to their behavior. Will they sit there waiting to finish this download, or just start sending other requests in parallel until you dos yourself? My hope is they would flag the site as low-value and go looking elsewhere, on another site.

This article on Endlessh also shows how to implement a resource-light http tarpit: https://nullprogram.com/blog/2019/03/22/

For HTTP/1.1 you could send a "chunked" response. Chunked responses are intended to allow the server to start sending dynamically generated content immediately instead of waiting for the generation process to finish before sending. You could just continue to send chunks until the client gives up or crashes.

[0]: https://en.wikipedia.org/wiki/Chunked_transfer_encoding

This will waste your bandwidth and resources too

The idea is to trickle it very slowly, like keeping a cat occupied with a ball of fluff in the corner.

Cats also have timeouts set for balls of fluff. They usually get bored at some point and either go away or attack you :)

Yeah but in the mean time it's tying up a connection on your webserver.

If the bot is connecting over IPv4, you only have a couple thousand connections before your server starts needing to mess with shared sockets and other annoying connectivity tricks.

I don't think it's a terrible problem to solve these days, especially if you use one of the tarpitting implementations that use nftables/iptables/eBPF, but if you have one of those annoying Chinese bot farms with thousands of IP addresses hitting your server in turn (Huawei likes to do this), you may need to think twice before deploying this solution.

Yes but you still need to keep a connection open to them. This is a sort of reverse SlowLoris attack, though.

You've got the option of abandoning the connection at any time should resources be needed elsewhere.

(Or rather, the tarpit should be programmed to do this, whether by having a maximum resource allocation or monitoring free system resources.)

> [...] and trickle stream it to them [...]

See https://en.wikipedia.org/wiki/Slowloris_(cyber_attack)

Practically all standard libraries have timeouts set for such requests, unless you are explicitly offering streams which they would skip.

This would work, but at times bots pretend not to be bots, so you occasionally do this to a real user

Reply on news.ycombinator.com