Hacker News Reader

Comment by pgraf | Hacker News Reader

Why not adhere to the HTTP standard for once and return a 429 Too many requests or 403 Forbidden? 404 Not Found should only be used for “Not Found” errors. If adhered to this posts advice, you can really screw your search ranking if a major web crawler gets blocked and de-lists all of your pages because of 404

View on HN

I think you can still adhere to the HTTP standards (Rails have built-in keywords for those aswell).

This article just talks about blocking hosts that make too many requests to a 404 page (I guess because it's crawling?) and how to mitigate those using Rack Attack.

Could you please explain how sending a 404 to clients sending too many requests would adhere to the HTTP standards? I’m not against the blocking itself.

For reference: https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.4

When I say you can still adhere to the HTTP standards, I mean that you just have to adjust the suggested settings in the config, so requests reaching over the throttle should give 429 instead of 404:

Like the following:

  # config/routes.rb
  get "429", to: "welcome#429", code: 429


  rescue_from ActiveRecord::RecordNotFound, with: :too_many_requests

  def too_many_requests
    redirect_to "/warning"
  end

  # config/initializers/rack-attack.rb
  class Rack::Attack
  ...
    blocklist("block 429") do |request|
      Allow2Ban.filter("too_many_requests-#{request.ip}", maxretry: 5, findtime: 3.minutes, bantime: 1.day) do
      request.path == '/429'
    end
  ...
  end

  # config/initializers/rack-attack.rb
  Rack::Attack.blocklisted_responder = lambda do |request|
    [ 429, {}, ["You are blocked. If you think are not a bot and you think it was due to a mistake, reach out to us at support@yourdomain.com"]]
  end

The way I interpreted the article wasn't to redirect everything to a 404 page, but how to handle massive amount of requests to a 404 page!

Reply on news.ycombinator.com