AWS GuardDuty – the Good, the Bad, and the Ugly | Hacker News Reader

AWS GuardDuty – the Good, the Bad, and the Ugly

badshah.io

132 pointsbnchandrapal56 comments

GuardDuty is another example of brilliance of AWS pricing scheme and how they manage to twist your hand to pay extra which can cost quite a lot in the end of the month.

When comparing EC2 to servers, nobody adds the added premiums of the extras. Things like CloudTrail, Support, GuardDuty, CloudWatch. All of these things have a variable cost that grows with usage and very hard to predict ahead of time.

Just last week I discovered our GuardDuty bills went up from $15/month to $400/month. Inspecting closer, the issue was a small script that did AWS API call at a tight while loop in a couple of EC2 instances.

So if you choose to enable GD, make sure to have your monitoring in place, gradually enable GD across the infra and establish clear baselines and alerting in place for costs.

Until you realize how much the alternatives cost. You think Palo Alto networks is cheap?

Is there a feature and price comparison between the native AWS offering and Palo Alto somewhere? Would be super handy for one of our projects right now. Thanks!

At least you can get a fixed cost if you go that route vs. your bills growing at an exponential rate. From what I have found, PA really wants you to commit to annual pricing with their VM appliances. If you go hourly you pay 73% more.

Do you think Palo Alto's offerings and GuardDuty are comparable?

Na. At some point you scale to the point that $400/m is peanuts for the benefits you get.

We used it at previous job and realised we were under constant attack and and it reduced our cou usage by 15% in reduced requests for the amount of traffic we were getting. No more random spikes.

Improved our overall security and ultimately reduced costs on the AWS bill.

But if you’re gonna switch it on and walk away then you’re not really using it.

Yes of course, $400/month is peanuts so does $8000/month for certain companies.

The point is not the absolute sum but how easy is it to spike your bill by 30 times.

The way GD works, you are pretty much guaranteed to overpay for it sooner or later. If you can afford it.. great.

One thing that we noticed was after switching it on, the EC2 instances were being hit directly, so we moved those into a private security group only accessible to the load balancer. RDS got restricted. S3 buckets fixed. Coupled with AWF to block on inregular activity, resulted in GD bill going down, not up.

This is no different from programming. PHP has some awful code out in the wild, it doesn't mean PHP is shit just because people write bad code.

The issue with AWS is it's far too easy for people to just spin stuff up and it works and they don't look at what they are being billed for, don't analysis their infrastructure, don't optimize. They just throw servers, containers, etc up into the wild then when the bill comes:

"OH AWS BAD I got billed cos I just set it up and forgot about it, then when it worked they charged me for it, AWS is wrong, just go baremetal."

Out of curiosity, did you have any data lakes on S3? Did you find optimization techniques for the same?

Nope, but did realise we had some open buckets we didn't realise were open. Thankfully we didn't store sensitive information in there despite having 2PB of files in there.

I think it is similarly easy to spin it the other way around. "AWS is just selling you the gun and the bullets, you are the one who is shooting yourself in the foot".

I don't think I said AWS is shit or that GD is worthless, after all, I use both by choice. Yet, I do not think that AWS are blameless when it comes to certain decisions of how to bill, how to present data and how to document some of their features.

For example, in order to discover something is wrong with your GD billing, you must have CloudTrail in place, and the appropriate infrastructure to query it. And even tho AWS can easily alert you about weird trend in your API calls (like suspiciously high Describe*), they won't do it. They do it with Trusted Advisor when you have under-utilised EC2 instances (which requires Business+ support plan per account).

Someone mentioned in the thread the need for SCP in order to disable regions. Why should you have go all the route to SCP? Why can't we disable regions by click of a button under root account like it's possible for some of the latest regions?

Is something inherently wrong in it and pure evil? No. But I think the defaults can be better. I think AWS can improve their customer's default posture when it comes to Audit and Security without the need to have to decide between 10 different services with different billing plans and gotchas.

Have you checked out Cost Anomaly Detection[1]? It builds an ML model to alert on anomalous usage and resulting changes in billing.

[1] https://aws.amazon.com/aws-cost-management/aws-cost-anomaly-...

That's exactly my point, that is _yet another_ service you need to go through to get a clear picture of what is going on.

These products have their place, but they don't make sense until you reach a certain size.

The biggest “hidden” cost for me was Elastic Blockstore IO charges! It started adding up quite quickly and I realized I had to think twice about doing IO intensive calculations on EC2! I switched over to lightsail (obviously these are for personal projects).

Just use gp3, shouldn't be needed to request higher IO with it (except for DBs, probably cheaper to use DynamoDB etc then).

Yeah, provisioned IOPS can add up quick. Never had justification to spend on them myself. If I need IO I use instances with attached storage, otherwise I purely use the GP SSDs.

Just make sure everything is multi-az that way if an entire zone goes down you will still be fine /s

Its quite annoying that you can't disable parts of guardduty you don't get much value from. I think the CloudTrail monitoring is quite useful and the VPC flowlog monitoring is basically useless (to me, I have other means of doing host and network based monitoring, I'm sure that there are a lot of people who get meaningful value out of it). I'd like to be able to turn off flowlog monitoring and just use guardduty for cloudtrail monitoring, but that isn't an option. So I can either overpay for a bunch of extra things I don't get any value from or not enable guardduty at all.

I hear you but how is this any different than any alternative approaches? I mean yes you could throw a server under your desk and it would be cheaper. Should you run a server in prod with sensitive workloads without all these extra security bells and whistles (on prem or in the cloud)? No. No you should not. Compared to alternative security appliances and offerings this is still a good deal.

> Should you run a server in prod with sensitive workloads without all these extra security bells and whistles (on prem or in the cloud)? No. No you should not.

Why not? Have people forgotten how to run servers in the past decade?

People haven’t forgotten, companies just don’t hire infra people.

“Fire all your ops people do the cloud” has made it so that organizations have forgotten how to run servers.

Working in infra but not at supermassive scale does really feel like the new COBOL programmer.

Managing and occasionally slashing costs is why AWS consultants can make really good money - if you're into this kinda thing, it's worthwhile learning how to analyze and optimize AWS bills.

Well, before AWS big companies tended to have the same problem, but instead of paying for many AWS features they paid for a thousand different products and trying to make sense of all those bills for each product was hard. I know some people who worked as consultants to clean up all the services the company was paying for but didn't really need any longer, paying for around thousand different software products was the norm and not some unrealistically large number.

how did you debug it? We have this weird thing where the cost of GuardDuty varies on different days of the month and stays the same from month to month. But we can't figure out what causes them. Can you actually see the events that GuardDuty has processed?

> The Ugly > Cost can get sky-high

Is there _any_ service on AWS where you feel like you're getting more value than the dollars you're paying with (other than IAM and Free tier services)? It's no secret that AWS is one of the most successful and profitable modern businesses, but perhaps there's a hidden offering that does something, does it well, and costs very little compared to the value it brings.

RDS is one for me. I’ve set up a regionally replicated, point-in-time backuped DB cluster and have no desire to run one again.

There are a few AWS security services which are free/priced reasonably.

Some free services:

1. AWS Org (Disable services and enforce guardrails)

2. VPC (Create private networks)

3. IAM (User access and IAM policy analyzer to help with least priv)

4. IAM Access Analyzer (Alert on resources with cross account & public access)

5. SSM Inventory & Patch manager (Basic check if all VMs have security updates installed)

Reasonably priced IMO:

1. AWS WAF with free managed rules (when rightly configured you get lesser FP and high ROI)

Those services are gateway, or requirements for using other services though.

A VPC isn't useful without EC2 instances in it. AWS Organizations allows you to create more accounts, with more instances, databases etc in them!

Most if not all services, I feel are a good enough value...when I consider my "time" or hiring consultants time to do the same, etc.

A single example was GuardDuty above. I know how expensive similar to GuardDuty services are per month when you have to have a well planned strategy, implementation, execution, operations for threats... no matter if one does it "in house" or with "consultants" - the cost is very high to implement anything similar to GuardDuty. No matter if it is duct taped open source or enterprise offerings.

And then you have to do that same iterative business process loop across infrastructure (servers/database), data centers, code deployment, DevOps, security, etc etc.

Lambda. I've build multi-region endpoint compute services before. Never again.

Someone else mentioned RDS and I agree with that too.

I use DynamoDB, Lambda, Fargate, and a few other services for a side business. I don't want to spend my time fiddling around with a database or EC2 instances. I'm able to run everything I need for under $100 a month without a lot of overhead, which I think is a good value. This may change as the business grows.

I can't complain about DynamoDB or Lambda pricing. Fargate is a little expensive for what we are running, but the infrastructure management tradeoff still makes it worthwhile.

> Fargate is a little expensive for what we are running

Because of low usage? Lambda supports containers now (as of 2021 I think) so if you have a container to run (or something you could containerise) it's a relatively straightforward usage question which of Lambda/Fargate/EC2 makes sense on price. Lambda doesn't have to complicate comparison by being a completely different architecture/setup any more.

> Lambda supports containers now (...)

The usecases for Fargate vs Lambda are not the same. For instance, Fargate is mainly intended for servers/long-running applications, whereas Lambdas have a hard runtime cap of 15min.

Why is that incompatible with it being a matter of usage, as I said?

Lambda makes sense when usage is a small fraction of the day; 'long-running applications' are not that.

No, the cost of a Fargate vCPU is just higher compared to EC2. An EC2 t3.small instance costs about 2 cents/hour and a similar configuration on Fargate costs about 9 cents/hour. For m6i.large and c6i.large instances, the disparity isn't as bad but it's still 15%-20% more expensive.

There are a few different reasons we're using Fargate. Like the other commenter mentioned, there's the lambda max run time. Our Fargate tasks also have a few sidecar containers running alongside the main services. The ECS Exec integration is also nice for poking around when things aren't working correctly.

Of course it is, if it were cheaper then it would always win over EC2 for container workloads.

This is why I said it's a matter of usage - it's worth paying more per unit time if it's running less and more sporadically making it cheaper over all.

I don't disagree about usage, but a 300% upcharge over a burstable EC2 instance is not reasonable from a pricing perspective.

Fargate should allow users to specify their compute requirements beyond just vCPU count and GB memory.

Lambda/dynamo can work quite well

Dev/staging environments are virtually free

Prod will cost more depending on usage, but you could optimize costs and add caching (like cloudflare - at cdn level)

Route 53

I use Route 53 for some of my domain names. It’s alright, but I think I get far more value on CloudFlare’s free tier. Except that I leave the AWS ecosystem...

Just get the fundamentals:

- Some core databases.

- Compute in the form of EC2 or EKS with EBS.

- IAM roles.

- Secrets management.

- Load balancers.

- S3.

That’s basically it. Build the rest yourself, they tend to be cheaper.

S3 and load balancers (with ACM support). Having a magic load balancing appliance that autoscales and supports TLS termination is nice and reduces a lot of maintenance. There's nice value adds like OpenIDC support.

S3 is incredibly powerful and cost effective for the price. It has insane throughout and a very simple API (compared to, say, setting up SMB or NFS to support public uploads where you need more pieces)

> Disable access to services in all non-active regions using SCPs.

This is key advice anyway. When setting up new AWS infrastructure for a new company, set up an AWS organization, and only enable us-east-1 (required for some global services like CloudFront) and maybe one additional region (if you don't want to put all your eggs in the us-east-1 basket). Don't enable additional regions that you don't need. Because most AWS APIs are regional, it makes finding aberrant infrastructure much, much easier, even if you're just combing through the console manually.

I would not start by default in us-east-1, unless you want chaos monkey as a feature.

Are there any historical metrics out there on uptime/stability by region?

Would be interesting to see a top-line comparison (GCP and Azure regions would also be neat)

It’s not bad advice but I’d do it for latency for western clients more than this — I’ve been running in us-east-1 since the 2000s and there’ve been only a handful of times where we had a production outage on a properly-designed application (a network routing issue in 2011 or 2012, and a couple regional S3 or IAM issues). No, those weren’t perfect but during the same time period our professionally-managed data center resources had multiple weeks of complete downtime versus maybe a day cumulatively.

I recommend AWS Control Tower for getting this all setup. It's also compatible with Terraform in more than one way.

Ahhh. AWS Control tower has not cost but it requires AWS Config to be enabled. Config is yet another AWS service that can get costly over time (if continuous monitoring of changes is enabled)

AWS managed to make things so much easier and cheaper compared to “classic” hosting that you now need twice as many devops employees and spend 10x in bills. Fortunately tech people aren’t financially literate so amazon can keep on squeezing all the while using free software made by the very same people. Congrats.

> AWS managed to make things so much easier and cheaper compared to “classic” hosting that you now need twice as many devops employees and spend 10x in bills.

I'm far from a AWS fan but this take can't even be deemed an apples-to-orange comparison.

"Classical" hosting at best matches EC2. The absolute high-end "classical" hosting offers at best also offer something resembling EC2's VPC. Forget about regions, let alone anything resembling availability zones.

Everything else that AWS offers ends up being nice-to-have conveniences. Stuff like object storage and pub-sub and message queues and managed nosql and classical RDBMS services and managed kubernetes and integrated infrastrucure-as-code systems are way outside what a "classical" hosting company offers

I mean like, this is fun to say, but y’all know this isn’t actually true right? (Well, the free software part is, but the implication in the first sentence isn’t)

The past 10-15 years has seen enormous growth in eCommerce productivity as a portion of the overall economy (just google “gdp attributed to internet commerce” and similar phrases, there’s tons of data on this). The rise in the number of devops engineers and the amounts businesses spend on AWS hosting are often in service of business models that were simply impossible to even try before cloud computing.

Sure, everyone on HN seems to have a story about an organization going all “architecture astronaut”nuts with Kubernetes and then ending up with slower/more expensive infra, and it’s fun to tell those to each other, but there’s clearly a huge amount of economic activity being enabled here that wasn’t happening before.

I actually remember the old days and no bloody thank you. Sometimes you got arcana processes managed by barely technical sysadmins so it took you 5 months to get a VM and 2 months to reimage it if you broke it. Other times you got a free for all where every engineer configures an insecure never updated server that eventually gets hacked. Then you get paged at 2am when the thing explodes and get to spend two days rebuilding it. Especially fun when you didn't build it but simply inherited a magic undocumented machine from someone who left.

Personally I care more about my enjoyment of my day to day job than saving the company a tiny bit of money that they'll never give me.

Depends on the skill of the employees. If you have mostly juniors and people who don't care then AWS is a godsend. If you mostly hire capable seniors to lead and ambitious juniors to follow then you can run things the old way just fine.

My first job had only incompetent seniors that I had to explain SQL query performance to etc. I left very quickly since I realized I wouldn't learn much there. I can see such places benefiting greatly from AWS services.

>If you mostly hire capable seniors to lead and ambitious juniors to follow then you can run things the old way just fine.

Or you can run them even better on AWS assuming you had the same level of competence except in cloud deployments. Granted they'd probably pick a more specialized cloud platform than AWS for the specific problems and scale of the team.

I think this is a natural tendency to get a large bill and want it to be somebody else’s fault. The two cost drivers mentioned are S3 access logs and VPC flow logs. S3 access logs are required by most security standards and are the only way to get that feature: if you need it, you’re going to be setting those up in whatever cloud security tool you pick or build. This is also odd with the request to only monitor some buckets - not enabling logging is exactly how you’re intended to do that.

VPC flow logs are odd, too: you actually don’t need to enable them for Guard Duty - one of its selling points is that you can globally enable it without the possibility of one of your organization’s accounts having it disabled due to accident or malice – but again, if your security policy requires this there’s no shortcut for any tool: you can get figures quickly before you run through the free tier and use those for your budgets.

The DNS logging points are handled by separate products: if you want those features, check out the Route 53 resolver logging and firewall docs:

https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/re...

My personal experience: GuardDuty is a very expensive false positive generator with a bad UI (because AWS).

If money allows, I'd look at wiz.io instead

Reply on news.ycombinator.com