Just curious, can you go into further detail about this bit?
"we are all developing on top of either AWS or Azure and are actually using the technologies they provide and not just hosting a bunch of VMs"
"we are all developing on top of either AWS or Azure and are actually using the technologies they provide and not just hosting a bunch of VMs"
I didn’t know the first thing about AWS and neither did anyone else - including the infrastructure guys. So we hired some “consultants”. I did a PowerPoint slide of my architecture with the consultants for thier guidance for best practices.
Basically they just set up some VMs, security groups, a VPC etc and they treated AWS like an overpriced colo.
I spent the next few months designing and architecting everything like I would have done on prem:
- 7 servers for Hashicorp’s Consul (1 in Dev,QA, UAT, and a cluster in production). They were used for configuration, service discovery, internal load balancing (with Fabio), and health checks.
- 7 servers for Nomad (same as above) used for orchestration and to schedule jobs. Think Kubernetes but with the flexibility of using raw executables and not just Docker containers.
- 7 servers for Mongo (same as above)
- 12 always running app servers.
- 2 build servers orchestrated with Visual Studio Online and local build agents.
Of course all of these machines had Microsoft agents (?) on them for deployments, Consul agents, and the app servers had Nomad agents.
This would have all been perfectly well architected for an on prem environment. But anyone who knows anything about AWS (and I didn’t then) would know that’s a dumb, overpriced, hard to maintain design and we weren’t taking advantsge of AWS services.
If I were doing that now. I would:
- get rid of all of the Consul servers and use AWS’s Parameter Store for configuration. Use internal AWS application load balancers and route 53 for services along with autoscaling and health checks.
- instead of Nomad, I would have used CloudWatch Events and done a combination of lambda, step functions and Docker. That would have cut out 7 more servers.
- I would have used AWS CodeBuild that basically let’s you use either prebuilt Docker containers for builds or create your own. Cutting down on 2 more servers.
- Today, I would use AWS’s hosted ElasticSearch solution instead of Mongo. But given the needs of the project, I would have used Mlab’s (?) managed offerings.
Of course I would use CloudFormation to manage all of this including the configuration key/values instead of my own bespoked app to source control configuration changes.
I've been curious to learn some of this stuff.
Even if you try to avoid lock-in. It’s usually not worth the risk of regressions and downtime to change your underlying infrastructure once you build on top of it.
The chances of AWS or Azure going out of business in the grand scheme of things is not worth the trade off.
1. Being able to do this would mean lots of work to abstract and polyfill the discrepancies between providers.
2. If AWS or Azure goes down globally, everyone else would be too busy freaking out about their own problems than be worried about the downtime of your SaaS.