Open Guide to Amazon Web Services
github.com
github.com
I would as a minimum recommend anybody/everybody considering AWS to read and think about the "When to use AWS" section. Whilst it is an excellent set of tools that have completely changed the economics of deploying software, there are times when you should use Google Cloud, times you should use bare metal, times you should use Heroku. AWS is a complex beast. Heroku is simple, but has limitations.
There are a bunch of apps I'm thinking about building at the moment where I realise a hybrid approach is best: some of GCP's stack, some of AWS', and a small amount of my own bare metal. Knowing when to choose which is not intuitive and comes with time, but there are big, big clues that will help the uninitiated in that section of this open guide.
Also, if you're looking to the future, the AWS Lambda and Google Functions stuff is perhaps the most exciting stuff to start building knowledge up of now if you're a developer, I think.
Kubernetes (and similar technologies) on the other hand, make it possible to get the same economics as cloud functions while still tying your cost directly to the computing resources you use. Also, it gives you the freedom to (with some pain) move your entire platform to a different provider.
unless you have a metric shitton of money to blow, there's never a good reason to start with that.
The most expensive part of any of those cloud providers is networking. If you need to transfer data from bare metal <-> aws, you'll need direct connect which charges basically an arm and a leg. Transferring between aws <-> gce is expensive for the same reason. Sure, if you're apple scale and need better data redundancy maybe it's okay. maybe. But that's not an app you think about building as an individual or small company.
I also don't think GCPs stack has anything whatsoever that AWS's doesn't have, so it's odd to mention it in that phrase.
If you'd be so kind as to provide an example application you're thinking about, and the reason each of those is needed for some part of it, I'd be happy to hear it!
WHY: Google still doesn't handle anchor-links very well. You have 1000 amazing articles on a single page. Each section (e.g.: "High Availability on AWS") would be a great resource for someone searching on that topic in Google. But when you put it all on one page Google infers "1/1000th of this page is about high availability on AWS" and gives better rankings to a page that is 100% about high availability on AWS.
I'm sure it would be pretty simple to write a script that breaks up topics into individual pages. I love the style of having it all on one page but I think it would be a waste of your hard work not to get all this great writing in front of search.
I like the monolithic format, but if the cost is lower SEO, maybe have a paginated version "in addition to" (as opposed to "in place of")
The different versions are automatically generated from a single common source but that would probably require a major change in how you create your guide and so may be more work than you want to take on.
To illustrate why this is useful, I'm a network engineer who primarily works with Cisco gear. Cisco has an absolute wealth of information -- product manuals, configuration guides, etc. -- accumulated over a couple of decades spread out across their web site(s). Unfortunately, their web site team likes to change things -- A LOT! -- and pages "move" frequently and it's often impossible to find them again. Because pretty much everything I'm interested in is available in PDF format, I save these versions locally where I can find them and refer to them later. Quite often, the times that I really need to look up some obscure feature are times when I am somewhere that either 1) I cannot connect my laptop to the network or 2) Internet access is unavailable, heavily filtered, or outright prohibited (of course, that's probably not going to apply for someone working with AWS.)
Regardless, you've put together a wonderful, comprehensive resource here. I'm a "minimal" user of AWS (primarily S3) but I am familiar with the different products and you're done an awesome job of summarizing Amazon's "dense" documentation down to its key points.
I was recommending the one-topic-per-page idea for others who haven't yet found this nugget. I think a lot more people will discover it and benefit from it if they are finding it from specific google searches.
I know HN can be a source of a lot of unfounded flyby critiques, I dont want to contribute to that trend. I see you have a pretty good contributing guide, maybe I'll try and submit a PR with a solution in the spirit of Hacktoberfest!
Many of us have simple goals on AWS. The official AWS docs are thorough, but are too technical. There are blog posts about anything, they can be hard to find or get out of date.
I hope this open guide helps us all get our jobs done faster and easier!
[^0]: https://unop.uk/on-aws-vs-azure-vendor-lock-in-and-pricing-c...
You're right. Their docs are far beyond the scope of what I needed to get started. Interestingly, I would rather have Google searches about AWS show Stack Exchange answers but most of the first results are all Amazon documentation which is far more difficult to read and sort.
* All of the instance types on one page? * All of the per-type facts in one row? * Sorting?
Let me know and I will share it with the team.
* Doesn't take 30+ seconds to load
* Sortable and filterable
* Filters!
* Ability to see pricing cost per hour, daily, weekly, monthly and yearly
* Quick and easy to switch between regions
* The compare selected feature
My wishlist for this site would be a way to easily compare pricing information between regions.
it will take you an hour to do, and you'll be years wiser
this is probably the number one reason people experience extra extra downtime when suffering from rebuild from whatever issue... and EBS volumes in certain regions can and will experience silent deaths
This is why it's important to lock down instance profiles to do only what the application needs to do and no more. For example, you may give the permission to s3:DeleteObject, and in the event that the box is compromised the attacker would be able to delete files in your S3 bucket. However, if you don't give access to s3:DeleteObjectVersion you can evict the attacker and restore the deleted objects with relative ease.
This is why I would not recommend giving access to s3:* to an instance profile (or indeed, any production credentials).
You've also got to go to the trouble of getting the credentials on your box to start with. With instance roles, you can launch an instance and have it immediately capable of doing what your application needs. In the case of most applications my company runs, the instance profile is enough and no further security credentials are required. When database credentials are required, they're retrieved via S3, authenticated by the instance profile.
[0] http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use...
If you'd like to PR or discuss there it'd def help us cover this better.
I guess people don't like Chef? Opsworks has enabled hassle free deployments for us over the past three years or so at no additional cost. :)
It does have some warts.
The necessity of building and maintaining a custom deploy script is the biggest wart for us (though I admit that the API is pretty good, and said script has not had that much maintenance overhead).
It's likely the original authors aren't using Chef or just use Chef server as I do now.
The special sub-category of those are huge RDBMS instances - a pretty common choke point in growing companies with weaker engineering teams. Some of those companies would pay basically any price to keep those DBs running.
IMO theres an awkward space between small data and big data where it isn't really worth spending a long time to treat it like a real "big data" problem, and the x1 instance gives you an easy-out.
Out of date; EBS volumes can be up to 20k IOPS per volume and what is "maximum size"? To get the maximum performance out of a volume depends on workload, the instance size you've attached it to (rather than EBS Optimization) and the number of IOPS provisioned, and whether you've prewarmed it from a snapshot restore or not.
> A standard block size for an EBS volume is 16kb.
A block can be 1kb -> 256kb in size. It depends on the application.
> EBS volumes have a volume type indicating the physical storage type. The types called “standard” (st1 or sc1) are actually old spinning-platter disks, which deliver only hundreds of IOPS — not what you want unless you’re really trying to cut costs. Modern SSD-based gp2 or io1 are typically the options you want.
The ST1/SC1 wording is misleading. You only need '100s' of IOPS when dealing with big blocks for ST1, and SC1 isn't performance oriented at all.
https://github.com/nodesocket/og-aws-pdf/raw/master/the-open...
So overwhelming, in fact, that I decided it was easier to get some VPSs and use common, work anywhere, tools to manage (e.g. saltstack), than have to skill up on AWS specific stuff.
[1] https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_N...
There's still a lot of stuff I don't fully know about (for about Read / Write volumes that is set - I left it at a default of 5) but I guess, I'll learn as I go along.
EFS had completely passed me by. Does anyone have experience with it? I'm wondering what it would be like to use for Whisper / Graphite (just on a single machine). I'm less interested with concurrent access and more interested in not having to resize drives as data grows / overprovision drives all the time.
That's way too much for the use case I was contemplating, so I didn't investigate further.
I've a few questions for AWS experts :
The only container orchestration that is open source seems to be Kubernetes. Is it easy to run on AWS?
What's the equivalent of Azure "Service Fabric" in the AWS world? (and in the Google Cloud?)
Rancher also has k8s as an option and makes deploying it much easier.
We're building an open-source platform at Convox that leverages ECS very successfully. https://github.com/convox/rack
In my experience, ECS is easy to run, as it's a first class part of the platform. Boot up the right "cattle" AMIs with the right ASG configuration and you're good to go.
K8, Docker Swarm, Mesos and Nomad have plenty of documented success but you to stand up and operate the orchestration layer yourself. This is booting up "pet" AMIs and making sure they are monitored, etc. Then you boot up your "cattle" AMIs to run your apps.
The Convox philosophy is that you get application portability by packaging your app correctly with Docker. The orchestration layer should be invisible, something that you shouldn't build or operate yourself.
Kudos to the authors and contributors!
- What are your goals on AWS?
- What topic do you need help with? What articles would you like to be written?
Of course, I'm not 100% sure I could get them to approve contributions to any external repo due to liability concerns, etc.
Luckily, in California the company can't really stop you from working on your own project or contributing to an opensource one.
Simple Storage Service
Rather than making me scroll down to find out what KMS stands for!
Cheers
Also, you might want to wait a day or two before starting and let the dust settle a bit. ;-)
1. That would be very valuable for everyone else.
2. A section, that does not look overwhelmingly empty would attract more and higher quality contributions from others. Kind of a reverse broken windows theory (https://en.wikipedia.org/wiki/Broken_windows_theory).
I've been compiling a lot of tips and tricks personally that I use to help train coworkers. I'm definitely going to cross reference and see if I can open a few useful PR's.