I find there are a lot of high-level abstracted tutorials, but for the new services, there aren't a lot of detailed tutorials.
For instance, an implemented cognito->gateway->lambda->dynamodb is really hard for a newbie to do.
I think there's a big need for a crash course for devs that starts with all the crap they previously ignored or had someone else do for them and I say this as someone who has always written code first and done sysadmin second.
I agree partially. In the end the documentation is what you are going to need to read sooner or later. Or trainings equivalent to that documentation. Good tutorials are good for starting, but doesn't make you a professional.
I guess that in 10 years any one will be able to create websites for billions of internet connected users. But for now, as easy at it is, it is still complicated enough to require an expert. In the same fashion that 20 years ago you needed an expert to make a 3D game and nowadays there is plenty of technologies that allow you to do that with a limited amount of programming knowledge.
I have seen horrible things, usually security related, because non trained people think that they can achieve anything just with standard configurations and quick tutorials. And it looks like they where able to do it, until something really bad happens. Even people with long experience can make mistakes because it is a complex thing.
Yeah that line is blurring, too.
A YC S15 startup, Convox (http://convox.com/), aims to "make AWS as easy as using Heroku." It looks really promising.
Disclaimer: I work for Pivotal, who donate the majority of the engineering effort to CF.
CloudFoundry is a really solid platform, but there is a very important distinction between CloudFoundry and Convox.
Convox is a very thin layer on top of "raw" AWS. It gives you a PaaS abstraction but behind the scenes is well configures VPC, ECS, Kinesis, Lambda, KMS, etc.
For those of us with no need to run on multiple clouds, using pure AWS is simpler, cheaper and more reliable than a middleware like CloudFoundry or Deis.
If you want to run a private platform without bringing in operation dependencies like etcd (Deis) or Lattice (CloudFoundry), give Convox a look.
Every Convox cluster is an autoscale group managed cluster of ECS instances.
Every Convox app gets its own Kinesis stream for logs, ELB for load balancing, and S3 buckets for settings, build artifacts and encrypted environment. And the app processes are run via ECS.
So I'm confident we could handle the 7 apps in a single cluster, scale the cluster instance size and count, scale any individual app process type, and handle any individual app load balancing or log throughput.
You can provision some services like RDS with our tooling which make it really easy to link to apps. You can also bring your own services like elasticsearch or pre-existing RDS and set them in an app's environment to use it.
There are a couple monitoring tools built in.
We automatically monitor AWS events like ECS capacity problems and send Slack notifications.
You can also use our tooling to forward all your logs to Papertrail and configure your searching and alerting there.
More CloudWatch Logs and Metrics work is coming in the near future.
some additional questions though:
1. are security groups opaque within convox or are they exposed to developers?
2. when you say monitoring tools are built in and you have tools for logging does this lock me in to the convox log pipeline and monitoring? what i want to use sensu on my instances? do i have to add sensu to every container? if you run, for instance, five containers per vm do i pay the overhead of five seperate sensu instances? same question for something like logstash?
3. you mention vpc. aws has proven stingy with vpc service limit requests in the past. i have trouble getting them to grant more than low double digits per region/account. can i run multiple convox racks per vpc or is the one vpc per rack a hard requirement?
2. Currently every app gets a Kinesis stream and we tail all Docker logs and put them into Kinesis. Then `convox logs` can stream logs from Kinesis, and `convox services add papertrail` adds a Lambda / Kinesis event source mapping to emit the stream as syslog to Papertrail.
I'm pretty happy with this setup and think it represents a good default infrastructure that is still extensible.
Would Kinesis -> Lambda -> Sensu make sense too? It's a pretty new pattern but this seems a lot saner to me than per-container log agents, or even bothering with custom logging drivers.
That said, one user has been using logstash by bringing a custom AMI with his logstash agent and creds baked in.
3. It's one VPC per rack, but I could see modifying that. We've already started to parameterize some VPC settings like the CIDR block to help integrating with your existing VPC usage.
https://github.com/convox/rack/blob/master/api/dist/kernel.j...
The feedback was that Lattice is not what developers wanted, so it's been wound up in favour of MicroPCF[1], which is a single VM image that runs an entire, actual Cloud Foundry installation.
When developers decide they want to scale up to any size, they simply retarget a regular AWS/vSphere/OpenStack/Azure CF API server and push again.
I'm sure Convox has a single-VM version I can tinker with on my laptop.
This is definitely a goal of Convox: to remove as much AWS complexity as possible.
Our approach matches this guide to a tee. We are using CloudFormation to set up a private app cluster, as well as to create and update (deploy) apps. We are also using ASGs.
The instance utilization point is spot on too. The fist thing convox does to make this easy is a single command to resize your cluster safely (no app downtime).
Coming next is monitoring if ECS and CloudWatch and Slack notifications if we detect over or under utilization.
I strongly believe that these AWS best practices can and should be available for everyone. For anyone starting from scratch or migrating apps off a platform or EC2 Classic onto "modern" AWS.
I'd like to use AWS more, but each time I tried to get into it I felt overwhelmed. I currently use PagodaBox a lot, which is great (most of the time) because it handles a lot of the complexity for me, but it can often be expensive. How does Convox compare to PagodaBox?
Convox has the same goal of a PaaS: to give you and your team an easy way to focus on your code and never worry about your infrastructure.
One big difference with Convox is that we accomplish this with single-tenant AWS things. You and your team's deployment target is an isolated VPC, ECS (EC2 container service), and ELB (load balancers).
If you're asking for a cost comparison, we're building Convox to be extremely cost competitive by unlocking AWS resource costs for everyone.
Its easiest to compare the cost of memory across platforms, though not always apples to apples...
The base Convox recommendation is 3 t2.smalls which is 6 GB of memory which costs about $100 / month. If your app can be sliced up into 512 MB processes, you can easily run 10 processes, which could be 2 to 5 medium traffic PHP apps on the cluster.
I'm finding PagodaBox pricing calculator a bit confusing but 6 512 MB processes, so 3 GB of memory, is $189.
Do you plan to eventually charge a monthly fee for using the command-line tool?
The most straightforward model, and where we are already making some money, is running a Convox as a managed service.
In this setup you and your team get Convox API keys. Convox installs, runs and updates everything for you in our accounts. You get a monthly bill that's your AWS resource costs plus a percentage to Convox for management.
We will be tweaking this model to sell packages so bills are really easy to understand.
Some other experiments we're doing...
We sell support packages and professional services for app setup, migration and custom feature development.
We have a per-seat model for productivity features. Private GitHub repos and Slack integrations are $19 / user / month. There are more closed SaaS tools like this coming.
Infra is trending to commodity prices industry wide.
We'll be selling SLAs, support, productivity tools on top of that infra.
You'll get a cutting edge private platform without hiring and managing your own devops team to build and maintain it.
Open source users will help grow the user base and make the platform better without us running a freemium platform.
The plan is to get the Convox API locked in while mastering advanced AWS like VPC, ECS, ELB, Kinesis and Lambda behind the scenes.
Long term, yes, and in tandem with when the other cloud providers leveling up. For example Google Cloud Logging (for continer logs on GCE) is still in beta.
Making those notions and technologies easy and cheap to access AWS suddenly gave devs the idea that they can roll out complex infrastructure on their own, similar to copy-pasting a piece of code. Well, it is still a bit harder than that, and if you are that kind of dev (which I'd applaud), you'd better dedicate some time to learning those technologies.
Me: "I just said I was interested in DevOps and would like to give it a shot, I don't know how long it'll take, I'm working on it."
We're using gateway -> lambda -> dynamodb, and there are a tons of gotchas and small things that AWS need to iron out, especially with gateway -> lambda.