Scrapy needs to have sane defaults that do no harm

Is there a pluggable fail2ban-like server for HTTP, to rate-limit and block with a 429 any rude bot?

Also, how come we haven't yet designed a system of pluggable HTTP pipelines? If you have a custom web app, either you implement rate limiting by hand, or you set that up in your frontend nginx/caddy/apache/etc. It would be much nicer if we had standardized composable HTTP server processes that either handle a request, or pass through to the next one. That way rate limiting would just be a matter of plugging a rate-limiter process just before your web app.

I guess I just miss having Elixir's Plug system at the OS level.