An Open Source Tool to analyze wasted EBS capacity in your AWS environment | Hacker News Reader

An Open Source Tool to analyze wasted EBS capacity in your AWS environment

fittedcloud.com

49 pointsprakashmanden18 comments

Nice tool, thank you for releasing it.

Unfortunately, in my experience it's a pretty normal strategy to over provision EBS gp2 volumes to get the IOPS since they scale linearly up to ~10k. I think that's the break-even point where it's actually cheaper to switch over to provisioned IOPS volumes.

So we'd have a number of little-utilized (in terms of data actually stored) 3TB drives just to avoid paying for provisioned IOPS.

Have you considered getting a single large volume and sharing it over NFS? If this is a bad idea, why?

Nfs can be slow and have consistency and locking issues under heavy write loads. Also security isn't often top notch.

It really depends on the use case.

Instead of downvotes, please explain why I'm wrong. Not only do I want to know, but it's helpful to other.

These have been my experiences with nfs. It's great for certain things, especially read heavy ones. It's very not great at write heavy or ones that require locks.

None of these statements are inherently true about NFS. They may be applicable to some deployments or applications of NFS, but they don't hold generally speaking.

NFS is used heavily throughout HPC for workloads that don't require a parallel filesystem (and even then, pNFS is plugging along, even if it's largely ignored in favor of Lustre et al for accessing parallel filesystems.)

I used to work at a supercomputing center and nfs wasn't the choice for much of anything. Lustre or AFS were the normal options depending on use case. That was a decade ago.

Also, nfs is bad at applications requiring locks. Perhaps some implementations aren't, but I haven't seen one.

We used pnfs at FNAL across 5000+ physical servers doing analysis of CMS detector data. Worked well.

Interesting. I hadn't used that. Thanks!

It is true that there are some very high performance NFS systems out there, throughput optimized like Isilon, and some IOPS optimized like Tintri. A lot of enterprise virtualization systems store their VM disks on NFS and seem to perform decently.

But in a general sense, when I've seen suggestions to use NFS, it hasn't been the correct solution. Like a system that is designed to run on a single server, and someone wants to make it active-active HA by pointing two servers at the same data over NFS. No, that won't work. Or the high IOPS system, and someone wants to move it to a throughput optimized NFS service. Or the team the decided it was a good idea to log from multiple servers to a single file over NFS, and then complained that their log messages were not being written in order. Or having multiple web servers (HA!) serve media off a single NFS server.

In this specific case, if someone is maximizing IOPS per server, it probably isn't a good idea to use a single large volume exported via NFS and share it across them. Max IOPS for gp2 volumes is 10,000, for io1 it is 32,000. Sharing a single 10,000 IOPS volume across a bunch of servers isn't going to get you a bunch of 10,000 IOPS volumes.

Based on my experience, I instinctively question the use of NFS, it is so rarely the right solution in a cloud environment. Sometimes it is the right solution, I'm just saying it is pretty rare.

Another option is to use EFS (AWS version of NFS). It's easy to use/set up. However, EFS costs 3x-12x (gp2-sc1) more than EBS. https://www.fittedcloud.com/blog/aws-elastic-file-system-a-q....

I recommend getting off EBS and switching over to the i3 instance types. AWS pricing for the IOPS is crazy on EBS.

the downside of i3 is you'll lose your data when instance is stopped, then you have to have another place to keep this data permanently.

Fun story: I once heard of a startup that was keeping all their data on ephemeral local storage, mirroring it between instances across AZs for "high availability", and then, er, using spot pricing for all instances to save money.

One day the spot price skyrocketed and all their databases got shot down.

And that, I presume, was the end of the company.

for a second, I've thought it's me, except I don't use spot instance. I use i3 for IOPS with EBS backing it up

It is certainly a good strategy to over provision to achieve performance and reduce cost. We recommend it. Wrote a blog on it a while back which is here - https://www.fittedcloud.com/blog/is-it-possible-to-use-ebs-g...

More than purely looking at disk space, I think it's more useful to look at overall disk write usage which you can get for free by just loading up Trusted Advisor.

Why disk write usage? Assuming the reason for analysis is cost, storage space utilization is more important. GP2 and other non-provisioned IOPS types are priced per GB. I've seen EBS costs amount to 50% of the AWS bill at a couple companies, so reducing disk size can be a good cost saver[1]. You need an agent/script running in the OS to get disk space utilization.

[1] As another poster pointed out, sometimes you want to over provision size to get IOPS.

Agree, disk write usage wouldn't show capacity utilization. One needs to look at the file system utilization and make necessary changes (such as resize) to eliminate wasted space. Our experience is that most customers are way over provisioned from a capacity (and even performance point of view). Various opportunities such as right-sizing (manually doing it is a pain), right-typing (selecting gp2, st1 or sc1) io1 IOPS adjustments are available to significantly the reduce storage cost. Automating those tasks is key so that the solution can scale without user action.

Reply on news.ycombinator.com