Amazon EBS Update – New Elastic Volumes Change Everything
aws.amazon.com
aws.amazon.com
That said... I'd still rather have access to Elastic File System in more regions, and in a perfect world, be able to talk to EFS from Lambda.
It looks like you can shrink a volume, the edit box to change this just a "set a size" box. We have tested this and downsized one volume now and appears to have worked.
While I think I still might do an LVM for adding capacity, the biggest change here would appear the ability to scale provisioned IOPS. While gp2 disks IOPS are primarily tied to storage capacity, which often leads to overprovisvioning for peak loads, the alternatively high cost of smaller and faster storage with PIOPS often times isn't worth it when peak loads only happen occasionally. That changes with this and I am sure this will make a lot of DBAs happy.
AWS in the last year has really impressed with giving you even more knobs to turn to get exactly what you need. While it's still not quite as flexible as GCEs custom instance sizes, the GPU and now storage elasticity sort of make up for it. While it's still pretty expensive, AWS feels more and more like I can use an API to build a custom spec'ed server instead of trying to shove my workloads into only a handful of options.
You can't attach existing (unattached) persistent volumes to an instance at instance creation time.
So for instance, I can't have a machine boot and expect to see ebs vol-xyz on device /dev/xvdg at boot. I have to boot the instance, use the API to attach it and then have my instance mount it.
At first I was a little dubious of the usefulness of this update, but that features is actually really useful. I'm not sure what the granularity is, but it would be nice to crank up he iops on a DB when doing some large batch operations.
> Decreasing the size of an EBS volume is not supported. However, you can create a smaller volume and then migrate your data to it using application-level tools such as robocopy.
[1] - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/consider...
Have you tried EFS? https://aws.amazon.com/efs/
EFS is ridiculous cheap if you use it the way it's intended. It's billed per-byte so you only store things that you actually need shared on there.
If you're looking for an OCFS2 replacement for storing large volumes of data or to run something like Oracle RAC atop then you're probably better off either doing it yourself.
> Moreover, AWS services in general are ridiculously expensive for what has always turned out to be a sub-par experience. Overly complicated to set up and manage, lacklustre performance, and premium pricing. Thanks, but no thanks.
Besides the pricing argument, I'd disagree with all of those points. It's a joy to set up once you know what you're doing and performance is more than adequate for all the services I've used (with plenty of knobs to fiddle). Outside of external bandwidth pricing I don't even consider it that far out of line pricing wise.
It's a joy to set up once you know what you're doing
Which is (in my own long and active AWS experience) another way of saying "You need to make a significant investment in time, effort, and tooling to get to a point where it is fast and simple to bring up a reasonably complex environment. You are re-learning domain knowledge you likely already have, just so that you can use Amazon's product. This in-itself isn't a bad thing, depending on what you get back for it, but the return value is "I now know what I am doing with AWS".
performance is more than adequate
I can get more than adequate pretty much everywhere. I am paying a serious premium, so I really want to be in the amazingly fast bracket, not the adequate.
Outside of external bandwidth pricing I don't even consider it that far out of line pricing wise.
Yeah, they are, both directly as well as in terms of value for money. There are many ways to get increased performance at lower costs with reduced complexity (complexity also cotst real money). I change hosting provider roughly every 12 months to take advantage of investments in new generation hardware and good deals, and everytime I do so, I check out new features, updated pricing, and do the sums to work out value for money (I offer high performance, managed, HA hosting to selected clients)
If you have the data with price/performance comparisons, I'd think this is something the rest of us would be interested in.
That is a straw man argument. AWS isn't the only way to efficiently manage infrastructure, and it can be argued it isn't even efficient, full stop.
Again, if you have numbers and not broad statements we'd appreciate them.
eh, no. You have no idea about my areas of expertise, what features I require, or how I map them, or what I map them to, so you cannot make this kind of statement (well, you can, and you did, but it is a fallacy).
you have numbers and not broad statements we'd appreciate them.
I'm sure you do. Doing what I do today in terms of functional parity, but doing it on AWS, will work out roughly 2.5 times more expensive, and will not gain me any additional features or functionality, and will leave me with about two-thirds of reduction in overall performance against metrics that are of interest to me. Also "we"? You speak for others? Who are the "we" that have appointed you as their speaker?
The we should be anybody bothering to read this comment chain - who wouldn't be interested in seeing AWS' lack of value in your use case / competitors that succeed? Having done migrations to and from AWS for past employers, this would definitely interest me in the least.
AWS Internet traffic costs ~60x as much as renting bare metal servers.
No change in architecture can change that fact.
For a 10 GiB EFS drive you get a base throughput of 0.5 MiB/s and can burst up to 100 MiB/s for 7 minutes per day
Do you want to focus some significant percentage of your time on managing infrastructure or do you want to focus on making what you sell to customers better?
I know it sounds like marketing and I know there's a point at which the cost of cloud doesn't make sense, but I do believe that that inflection point is pretty low if you're in a field that is moving quickly or are say a new startup and need to get to market with a minimum-viable-product to get an idea off the ground -- what a shame it would be to fumble that opportunity re-inventing the wheel with infrastructure.
Yes
Can you turn off unused instances?
Yes
Can you automatically and seamlessly move to the latest hardware running on-prem?
Yes
Do you want to focus some significant percentage of your time on managing infrastructure...
We spend about 10 minutes per day looking after the stack, with about 4 to 6 hours every 6 weeks looking at if and where we can improve things. We then spend about a week or so in depth once every year roughly evaluating different providers, and we typically end up moving to a different provider, which takes a few days.
...or do you want to focus on making what you sell to customers better?
This (infrastructure) is a big part of what we sell to our customers. What we do (and have been doing for about 15 years) is what you kids appear to be calling "devops" now. I was an early adopter (and discarder) of CFEngine...
Secondly, none of my points were specifically about AWS but they were about cloud in general.
The block device interface provides no primitives suitable for concurrency control (no atomic compare-and-set, much less anything more sophisticated). Even if they did, you'd have to pretty much re-write the filesystem to make proper use of it. Even read-only multiple mounts are dodgy: the RO mounts aren't going to do cache coherence, and could trip up hard over intermediate states from the writer, but at least they won't corrupt the filesystem.
Realistically, if you want multiple writers on a network block device, your application is going to have to go to the raw block device, and you get to figure out intra-client co-ordination on your own. This isn't actually terribly useful. As an ex-insider, the nonexistence of this feature is not due to technical difficulty (it should be pretty easy for EBS to let you have multiple writable attachments), it's because 0.1% of customers would find the feature actually useful, and the other 99.9% would use it to shoot themselves in the foot.
If you want multiple machines accessing the same filesystem, you want a network attached filesystem. You can easily put samba / nfs-kernel-server / windows server in front of EBS. There are a few distributed filesystems, FOSS and propriety, if you need more scale. If you don't like any of the above, you're gonna have to build what you like yourself, or wait for somebody else to decide to build it for you.
Yes, it is possible, and even in some cases (mine being one of them) desirable. I mentioned OCFS2, which is designed specifically for doing exactly that. When you do this right, it is an extremely high performance, simple way of sharing storage in a highly available manner.
If you're really clever, you'd set up your AMIs to use LVM and keep adding additional EBS volumes before you reach the 16 TB per volume limit. Though I'm guessing if you have the issue where 16 TB of storage isn't enough for you then you probably can afford to find a real distributed solution to this.
Why? These other m3 instances are listed as current generation here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-...
Even if you go and provision storage on Vultr, Ramnode or Sitehost all of which are a lot cheaper options and just as if not more reliable provide considerably faster storage with a minimum of 20/30,000 IOPs up to 150,000 IOPs per volume and you don't have the vendor lock-in or the hidden costs such as inter-zone data transfer etc...
Amazon really needs to pull finger on their storage performance, but it's not in their best interest. What they want you to do is scale horizontally which means not only do you need to invest in your system / product design to do this but you also have to put more money into Amazons pockets for additional compute / storage nodes and again the associated hidden costs.
I'm not digging on Amazon / outsourced hosting providers but they truly are pulling the wool over people's eyes while still trying to make everyone think that they're not only the best option they're the only option.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-...
Compare this to Google Cloud, where you can attach a local SSD to any instance type and you can increase the storage performance by increasing the volume size.
https://cloudplatform.googleblog.com/2016/03/introducing-Goo... (March 2016)
(Disclaimer: Yes, I work on Google Cloud.)
And the entire article is about AWS customers and it really is game changing for a customer like me.
Your comment is in poor taste.