Hacker News Reader

Comment by jonahss | Hacker News Reader

Ran into a funny bug once from either Cloudflare or AWS's implementation of a similar service.

Marketing made a big push for a new Promo code on the site we were running, but the promo code wasn't working. Reports came in that the promo code worked if you entered it as lowercase, rather than all caps.

Double-checked a dozen times, we take whatever string the user passes in an converted it to upper case, and all the values in the db were upper case. Keep thinking we must have messed this up somewhere.

In the end, the devops guy realizes that AWS (or Cloudflare, I forget) was blocking the uppercase requests, since the promo code was `SELECT` >.<

Assumed it was an injection attack, but only when uppercase.

View on HN

I hate these kind of defenses. If your application is vulnerable to sqli, select is one of many tools an attacker can use and you’re pretty much screwed anyway.

Instead, use sane tooling, like modern ORMs and parameter izers, with some data sanitation if you’re really paranoid.

> Instead

You're misunderstanding the market.

The point of Cloudflare WAF isn't to be a main line of defense for HN readers, it's to stop the low effort automated drive-by attacks for websites that were already hosed. Like WAFs that block /wp-admin/* and instead generate a new segment.

I'd be surprised if there was a single person in the world who is going to go "oh right I should replace Cloudflare WAF and my sqli with some parameterized queries!"

So basically you say that those who care about security and sound engineering practices should quit software development, because it's a lost cause?

I think ‘stop wasting time on dumb stuff and focus on actual security’ is a good take home for the HN crowd. Time and money is finite, so spend it wisely.

It's also "Hey the superstar on the sales team just launched a new Wordpress blog that everyone likes, just wanted to let you know" and you have no time for a detailed security audit. Put it behind Cloudflare and at least you're more protected than you were!

Nah you pull it offline and tell them to follow correct procurement and development practices. If your development teams aren’t talking to your security teams you have bigger problems than Wordpress.

Consider: Your org is more likely to be run by people that are like the sales people than like you. Who do you think they side with, when sales goes up the chain to complain development broke their new initiative and is saying it'll take 4x longer to do the thing they already did themselves, and as a direct result means they won't hit revenue numbers this quarter?

What's even the risk here? Some minor marketing sub-site gets defaced, causing - at worst - an embarassing but instantly-forgotten incident?

No the risk is that somebody has decided to disregard security and general security process and create shadow IT, which if left unchecked will create massive problems within the organisation long term. If the culture is to disregard security, throw a waf infront and call it a day then they’ll pay for it financially (and possibly legally) in the long run and not something I’d want to associate with at all.

We’ve had a couple of similar issues with AWS WAF recently, in one case it blocked any request body containing :// (i.e. any URL) claiming it was a file system path. In the other we had a 3rd party cookie containing `= null` and the WAF thought it was sqli. I guess I can see the argument about layering multiple imperfect defences, but we wasted hours diagnosing these, neither were real issues, so I’m not sold on the value at this point.

Many of WAF's canned rule sets are ridiculously broad. I'm content to use the low-reputation-IP rules, but almost nothing else is advisable, unless you're intent on exposing (and not fixing?) a badly-written service to the world.

The low reputation IP rules are a pain in the arse -- and I say that as someone who frequently falls foul of them. I often have to use tor to get around government blocks. The fact that Google and cloudfare hate tor users is just an extra kick in the teeth.

If your web app is vulnerable being brute forced, fix it, don't blame the users...

Years ago helped handle a support request from a man named "fread" who had a similar issue.

Heaven help Ms. COM1 \\NTDS$\dd if=/dev/scd1

We had a customer named Mr. Echo who was having difficulties paying us. Turns out our credit card processor was blocking the names of common Unix command line programs.

Sounds much like the "PayPal blocking all mentions of tardigrade".

An overreaction is sometimes an overreaction, not a good defence.

It took me far too long puzzling over how "tardigrade" would be a keyword for an injection attack of any sort, and the closest I came to a guess was "tard" (as in "retard") being blocked for offence, so for those like me who didn't see that story: https://news.ycombinator.com/item?id=24450828

Sometimes you need to stop something fast, and the overly broad thing like 'block any mention of tardigrade' so 'we don't go to jail' thing is needed until you can refine the regex to not be so greedy.

you just have to be willing to go back to do the refinement once the bleeding stops.

Wait, does not mean that Cloudflare (or AWS) can read and modify all the user input inluding passwords?

Genuine question, I never used one of these services.

Yes. It depends on your threat model. You have to just someone at some point of you're not running your own datacentres.

Yes, Cloudflare stores the SSL the private keys. CDNs really need to read and modify the requests and responses for most of their functionality, like caching, load balancing, DOS mitigation, ...

You can always use a separate, not CDN protected domain for your API if this is a threat vector you care about.

I guess Marketing learned a valuable lesson about SQL keywords. It's also amusing to me because you know some engineer at Amazon must have brought up an edge case where a valid request would start with SELECT - but they were probably dismissed because how often would that happen - really?

Also amusing is the thought that lowercase "select" is any safer than uppercase "SELECT".

I can't believe anyone who has ever actually used SQL for more than a few hours would not know the keywords are case-insensitive. More likely it was just someone given a list of keywords to filter out, with no mention of case-(in)sensitivity.

is case sensitivity training available through HR we can send the new recruits?

On a case by case basis

More likely an SDE-1 at Amazon 3 months out of college was given a sprint task and because they’ve only ever worked on nosql they had no idea about the edge cases of what they were asked to do.

>SDE1 salaries at Amazon can range from $10,697 - $794,729

Yes, how a person making a minimum of $100K/y could know such things.

Reply on news.ycombinator.com