Hacker News Reader

Comment by warkanlock | Hacker News Reader

If anyone is receiving these types of logs on AWS, please do yourself a favor and place AWS WAF in front of your VPC.

It's not expensive and can significantly help you, saving you from many headaches in situations like this. While it might not block everything that arrives at your service, it can be a great help!

View on HN

This is a good suggestion, but careful with the default rulesets. We turned on AWS WAF (in our case, the motivation was just SOC 2 compliance).

There were a few overzealous rules that subtly broke parts of our app.

There were request body rules that did things like block requests that contained "localhost" in the request body. There was also a rule that blocked requests without a User-Agent header, which we were not previously requiring on API requests, so we broke our entire API for a few users until we figured that out.

In my experience WAFs are not something that one should ever "just turn on".

Complete due diligence is required to fully understand and realise the impact of the rules and should be tested like any software change by going through a testing phase.

Ideally software teams should be fully trained and be responsible for their lifecycle.

Yes you need to be familiar with the rulesets being applied, and prepared to closely monitor what is being blocked. Ideally I think I’d roll it out one ruleset at a time to limit the number of potential issues being introduced at once.

Had a fun one after turning on the AWS WAF with some default rules–a small number of users reported they couldn’t upload new logo images anymore. Turned out some Adobe product was adding XML metadata to the image files, which the WAF picked up and blocked.

Just to add, but only testing will never work well enough for something like this.

This is one of the cases where you must understand what you are doing. There's no technique for doing it mindlessnessly.

Agree. There are so many ways WAF rules can unintentionally block legitimate traffic. From very long URLs (is that a DoS attempt?), to special characters in a POST with a file upload (is that = part of a SQL injection attempt or is that just part of a base64 encoded file?) and so on.

I use the Azure equivalent of the AWS WAF but I have no direct experience with AWS WAF. Azure WAF leverages the OWASP ruleset[0] and many of those rules throw false-positives, SQL-related rules being one of the top offenders.

As you note, it requires adjustment due to overzealous rules. OWASP has Paranoia Levels[1] which allow you to be more targeted.

[0] https://github.com/coreruleset/coreruleset

[1] https://coreruleset.org/20211028/working-with-paranoia-level...

None of the attacks listed in the post would be an issue for any kind of modern web application. Why should I add a WAF for this?

It is not that easy as using the AWS WAF with default rules for our application led to many valid requests and IPs being blocked. You need to know what is being blocked and verify at first or you will in some cases be losing customers.

Your best plan is to start with all the rules in count mode. Let that sit for awhile and analyze anything that was counted. As you feel good about it, slowly start to move things into block.

Problem is when you get owned during that window and get slammed for not blocking sooner.

Then you block sooner and get slammed for blocking too soon.

Repeat for 1000 web services.

Yeah. Sadly the WAF can't solve organizational problems.

In actuality, WAFs hurt more than help. They give a false sense of security since they are so easily bypassable, plus they have a significant performance cost and a significant chance of blocking legitimate traffic: https://www.macchaffee.com/blog/2023/wafs/

WAF tends to ban widely, sometimes for dubious reasons. For example, researchers at my university study Twitter data, and the mere fact of following links from a small random sample of tweets means that our university's IPs are blocked by most WAF.

Reply on news.ycombinator.com