Hacker News Reader

AWS is complexity-as-a-service which abstracts away so much that you pay a HUGE price when it's time to scale -- and it's a confusing matrix of tradeoffs. I find it easier to login to a server, "sudo apt-get install" and tail some logs, than to try to manage the monstrosity that is AWS. Products that go horizontal-cloud-first are easy to scale, but burn an order of magnitude more money, with far worse realtime performance. At the end of the day, AWS is just an abstraction on top of CPU, RAM and disk -- if your app makes bad use of underlying CPU, RAM and disk, then no amount of cloud magic is going to fix that, it'll just mask the problem and you'll end up paying an arm and a leg. AWS/GCloud are actually incentivized to make it hard to debug and optimize performance, they would want you to waste resources on their platform.

Exactly, at this decade it's still a ripoff = huge margin for them, but market is there, one can't deny that. It'll even out eventually so they're on par with smaller shops and dedicated/colocation, eventually they'll have no choice but to cut margins and leverage scale to kick out smaller competition, they'll delay it, of course, as much as possible with lock-in dark techniques etc – but at the end there is no reason, in theory, why they should not be more attractive price wise with anything else you can come up with. At this decade-tick timeline they're happy taking huge margins and people just throw money without much of first principle etc thinking.

No:

- They’ll leverage GDPR to make it only possible to comply to all privacy standards (SOC, credit card management…) if you use AWS,

- Worse, they’ll provide special Linux repos with vulnerability fixes that only they provide. Log4j can make all your distributions obsolete in a single day, and only Amazon Linux users will have that patch, this quickly.

- Then insurances will stop covering people who deploy their software manually, because vulnerabilities are too prone to happen.

In the long run and for most higher global nationwide scale AWS/GCP will be cheaper from 1) general reduced price every few quarters/years 2)private contract. It’s important to realize those cloud providers does a whole bunch of work to reduce cost internally so customers can pay less (eventually)

AWS and competing clouds are a non-starter for anything resource intensive such as processing lots of data (needs tons of CPU/GPU) or serving/moving lots of it (needs tons of bandwidth).

The costs are bearable for early-stage startups that are mostly typical web applications with little resource requirements, or bigger startups that have so much VC money they are happy to burn it, but beyond these two scenarios clouds are very bad value for money.

Generally, application level decisions will have a bigger influence on cost than underlying compute storage for large data processing applications.

Tradeoffs that will dominate the costs

1. Are you using a fast language or a slow language? (10-100x cost difference)

2. Are you using an efficient storage format such as parquet? (10-100x cost difference)

3. Are you using a reasonably efficient query planner job processor? (1-10x cost difference).

4. Are you using a good algorithm for the data processing task? meaning a good efficient SQL, An efficient imperitive algorithm etc. (unbounded cost difference)

The above tradeoffs will have a cost difference of up to 10^5 ( or greater depending on point 4). Once you account for risk, utilization, capex, and people costs. The cost difference between different compute platforms is usually negligible to the above points.

What about the cost of optimizing the process? If compute can be obtained very cheap elsewhere, it may be cheaper to just use that with an inefficient process rather than spend time/money optimizing the process and then running it on expensive compute.

Depends,

Cloud providers cut compute/storage costs every few years. You can get long term contracts with them that have heavy discounts for guaranteed spend. Switching compute instance types is pretty cheap, and switching compute platforms is relatively low cost when the time comes. Meaning that investors can generally expect your compute costs to fall YoY all else being equal.

Building out your own compute generally means adding new headcount to deal with colo things such as vender management, technical details such as networking, capes/opex management, and finally hardware selection. The odds that the hardware you select perfectly matches the compute you need in 2 years is low. The odds that a cloud provider has made a cheaper compute option in 2 years are moderate. The odds that your app needs something that’s available trivially in a cloud provider are high.

There is a life beyond AWS. I host applications (including ones responsible for 10s of millions in revenue) on dedicated servers rented from Hetzner and OVH. Do not even do containers as my normal deployment is server per particular business (with standby) and the cost of renting a dedicated server comparatively to revenue is microscopic. CI/CD / setup from scratch / backup / restore is handled by a single bash script and has virtually zero administration overhead. For my model I see zero business benefits doing anything on cloud. I did some deployments on Azure due to a particular client requesting it but management overhead was way higher and do not even start on costs vs performance.

The business benefits of the cloud are ability to scale, an API for everything, cheap automation, cheap managed services for everything, and good customer support. Like you say for your model none of those things may be important, but they're killer features for others

>"ability to scale"

I write C++ business API servers. On the type of hardware I rent from Hetzner /OVH they're capable of processing thousands of requests per second. This would cover probably 90+ percent of real world businesses.

>"an API for everything"

Not sure what you mean here. API for what in particular? My servers talk to partner systems of real businesses. We consume their APIs and they consume ours. Integration is done by little plugins that translate requests should it be needed. I am not sure how Amazon will help in this department. The only generic API that everyone uses is email but you do not need Amazon for that. As for database - I always deploy one local to my business server.

>"good customer support"

Frankly in many years there was like 2-3 times when I needed something resembling support from Hetzner / OVH. In either case the response was prompt and satisfactory.

> > "an API for everything"

> Not sure what you mean here.

I assume referring to APIs for controlling and monitoring the infrastructure, not anything about APIs you may provide in your application or consume from external sources.

What infrastructure? Thanks to my primitive approach I hardly have any.

If you want to dynamically provision storage or grab an extra instance when you’re under load or something. It’s handy to not have to sit on a bunch of storage you’re not using just in case there’s a rush.

Unless the storage (or compute) is so cheap that it's cheaper to just always have enough on hand than bother with autoscaling and the added complexity and potential point of failure.

Old-school dedicated servers are so cheap that you can match a startups' peak autoscaled load and still pay less than their "idle" load.

Exactly my case. My peak load costs are way less than their "idle". Storage expansion if needed is even simpler to handle. Those cloudy people like to create problems where none exists.

Of course there are use cases where not using cloud facilities is cheaper, but unless your business is low margin the difference might still be irrelevant and having the managed services available easily can be handy if you ever need them.

There are many use cases where hosting fee is not the right thing to optimize for.

The last time I compared hosting prices of AWS with self-hosting. AWS based hosting would have cost us every three month the cost of buying over the top speced server hardware. These numbers are not directly comparable, but they are a large difference to account for colocation and maintenance. And we never managed to get a similar performance of postgres on AWS compared to the self-hosted version we use now. But the main advantage is the simplification you get. It is certainly not easier to use you own hardware, but it is far simpler.

I like this. Out of curiosity, could you share your bash script?

I do not want to clean it up from some revealing things, so no. But it is fairly trivial. If you are an experienced programmer you would not have troubles writing one up one yourself in no time.

Understood, it was pure curiosity anyway ;)

What do you use for backup?

I would guess rsync ;)

I would recommend to use borgbackup - it is very convenient for security (it provides very flexible encryption options with safe defaults) and efficiency (deduplication)

scheduled jobs running pg_dump and rsync

Playing devil’s delegate with a normal process how we launch new service in big tech:

1. Have you done a security review and ideally pen-testing from a 3rd party? Are you fully TLS from end to end, are you aware of any exploitable vector from your API/UI and how you mitigate them

2. How do you handle software patching and vulnerabilities?

3. Do you consider your app Operational ready? Can you rollback, do you audit changes, do you need multi-AZ resilience, do you need cell based arch. Did someone other than yourself or outside your team look at your monitoring dashboard? Do you need to be paged at 3am if some of your dependencies or underlying node degrades. We have to answer a 50 questions template and review it multiple times..

4. Did you calculate the cost and forecast it with how the service/app may grow in the next 12-36 months?

While you still need to do all this when using a Cloud providers, you probably _should_ do much more if you manage the bare metal

If you have already done all those, Kudos to you, but I still find it hard to trust everyone who DIY

To beelzebub the devil...how many large scale hacks/data leaks/embarrassment from large companies are because of a misconfigured S3 bucket?

My clients do / pay for whatever the audits are required in their line of business. I do not get involved unless told to do some particular things.

>" but I still find it hard to trust everyone who DIY"

It depends on what is the actual DIY part is. In any way it is your problem. Not mine. Sorry to say but to me your whole post feels like typical FUD scaremongering client to cloud.

>"While you still need to do all this when using a Cloud providers, you probably _should_ do much more if you manage the bare metal"

No I should not probably do much more. I do much less as I have way less moving parts.

I just think all those features are necessary for any serious usage and it’s costly to build all on your own. But perhaps you are right since it fit your business model - you seems not interested in building a platform at all

>"you seems not interested in building a platform at all"

I build products. That is my primary output. And the actual delivery is a binary executable that does the job. And it does it with the stellar performance that makes all that horizontal scaling infra totally unneeded. I serve the needs of my clients, not the needs of Amazon.

> I just find this type of article to be "DYI porn" for a highly specific subset of DYI.

This seems unnecessarily dismissive. (But maybe that means I am part of the 'specific subset'... :) ). IMO there is a big difference between 'tinkering' and 'self-hosting'. Projects you are tinkering with are typically transitory and can be treated as "pets". But in the 'self-hosted' world, stability and ease of maintenance are huge. In that regard I think the overhead of containers makes total sense, especially when running a bunch of stuff on the same server.

I may also be part of that highly specific subset because I enjoy these kinds of articles as well. It’s interesting to explore different solutions and to und west and their pros and cons even if they aren’t motivated by upfront requirements (you’re probably less likely to come up with an interesting configuration in that case anyway).

The problem with AWS is that you can’t really do anything without understanding 1. IAM, and 2. VPCs/networking. And these are probably the two most complicated parts. For DIY you’re probably best off avoiding AWS

You can do a lot with EB for simplistic projects. But yes without that you are in for a world of pain using anything else

This 1000%. It was amazing how much more comfortable I was with my suite of VPS's when I moved them from AWS to Digital Ocean! I always describe on-boarding with AWS as "drinking from a firehouse". There is so much to understand and it is hard to tell what you should actually care about. DO, on the other hand, has a much simpler set of services that seem much more applicable to 'non-enterprise' usage. (Plus the documentation for DO is great!)

I have a recruitment problem. If I tell recruits “We use AWS”, they’ll be happy the be waterboarded when trying to do something, they’ll have the feeling of being in connection with the rest of the world, preparing their resume for the future.

If I tell them “I have Digital Ocean, I maintain the load balancer myself, the database myself, ELK myself, and you’ll be expert in Postgres, Ansible, Nginx and Debian”, I sound like an old fart.

The future for them in AWS is a click-on-the-button deployment and they’ll master nothing.

Amazon tried to emulate Digital Ocean with Lightsail.

It does feel a bit like a Cinderella service however. The versions of the managed databases you can connect are sometimes several years old.

Sounds about right based on my experience with AWS for past 8 years. Does anyone (besides AWS/TF/Pulumi, for whom it’s not their core product) attempt to solve the networking and IAM? These two areas, esp AWS networking, despite being fundamental building blocks, just never get coverage.

I know I’m being a beggar and beggars can’t be choosers, however, I believe that there are folks here who could solve this problem and make a nice living off of it. Hope someone reads this comment and builds something to address this.

There's definitely something to this... but I think in practice most projects end up needing some amount of customization in their networking even if it's 90% "standard". In attempting to provide this, you might end up just gradually recreating TF/Pulumi anyway.

I had this feeling when I first started with AWS years ago. It was hard to find a good overview and all of the Amazon doc on individual services seemed to start in the middle. So, a lot of my initial understanding came through intuition, and trial and error.

For many scenarios, you can completely ignore IAM, but it's definitely not advisable.

On the VPC side, it's actually fairly straightforward, but you may need to come up to speed a bit on some networking concepts if (like me) that's not your background. Nothing too onerous though, especially if you have some technical background.

There are also some gotchas that allow you to too easily do things like create security groups or other resources outside the correct VPC. If you overlook that, you're in for some brick wall head-banging 'til you figure it out.

But how can we trust the DIY stuff meet compliance and hold the right security bar? It’s much easier to do with AWS.

Or maybe as a startup, to-C website you don’t really care

I think the complexity can lead to its own set of security risks, as people just keep opening permissions wider until things connect.

That's actually a really good point. Out of the box, it's hard to screw up because things are pretty locked down. It's really in attempting to open things up that the security risk comes in if people aren't explicitly aware of exactly what they're opening.

EDIT: and this isn't necessarily difficult to grok. A lot of what you'll use from the network side is security groups, and they are straightforward. /EDIT

There are also actually some bad patterns in the AWS Console UI that don't help here. For instance, despite all the warnings they place on S3 buckets about making things public, they still allow you to appear to change subobjects to private. In a traditional hierarchical directory structure, the more granular subobject settings would override, but not so with S3. If you didn't know that, then you've just shot yourself in the foot.

Great and interesting point. I believe the solution is to have “security by default” Infra-as-code construct and some static analyzer

When I am helping out people new to AWS, this is where 90% of the problems are. It suggests to me that they are lacking sensible defaults, when so many people have trouble and I just recommend connecting Beanstalk to RDS. I think some meta recipes that give you a set up where this is already secure and working make sense.

I built a tool for deploying web apps on AWS.. https://github.com/keybittech/awayto If you look at the deployment script app/bin/data/deploy, you can break things down into manageable pieces and understand what's going on probably within a day or two given you have some baseline understanding of frameworks and APIs.

But, I totally agree that the underlying nuance is a lot to take on when you start getting into X is required for Z which connects to A, yadda yadda.

That being said, if you choose a service and invest the time to understand it, you are availing yourself to a very wide world of technology "at your fingertips." You can most certainly say the same for DIY, just different shades of what you want to be responsible for I guess.

I think there's good value to be had from articles like these, depending on where you're starting and how you approach this kind of work.

On the requirements side, one thing I would like to see though is an approach for determining how your applications will scale on cloud infra.

For instance on AWS, which RDS and EC2 instances will suffice out of the gate, and at what IOPS/bandwidth? And when will I need to scale them? The metric is simply how many users can I support with acceptable response times on a given configuration?

Sure, we know that's highly dependent on the applications, stack, etc. But I've often thought there should be some rubric for approaching this that doesn't require going heavy on performance analyzers, load balance testing, etc, which frequently seem like overkill out of the gate when you're making your initial config selections.

2) I think this is a bit like asking, "Why isn't there a comprehensive guide to art"

Infra is massively complicated. Not only do you have competing frameworks/modules, you also have these change in different releases of Linux. Some of these are simpler, some are more comprehensive. Some are really hard to use but powerful/well-maintained, others are simple but might not be so good. Some perform well at a cost, others might perform relatively less well but are easier. Sometimes it is worth the effort setting up ufw on a server, sometimes it isn't.

BY the time you got to something that was simple enough to create a guide from, and which didn't change each week when some vendor renames a configuration option, it would be very high-level and possibly not very useful.

Heroku is "AWS on Rails" taken to an extreme and there have been lots of other in-between services and tools. Cloud 66, Laravel Forge and even AWS's Elastic Beanstalk are all part of that spectrum.

> Is there any sort of comprehensive infra setup guide out there?

This would be the AWS certification at their different levels.

I would tend to disagree. First, because not everyone is on AWS. Second, because even for people on AWS, some of a company's infra isn't on AWS (e.g. gmail). Third because not every infra setup work has an AWS tool.

To take a personal example, we use multiple infra providers.

Some of our infra is gmail (currently working on automating it with terraform).

Some of it is other infrastructure providers that aren't US-based.

etc.

What’s the “automating gmail with terraform” angle? G-suite automation? Or GCP?

g-suite.

Can you elaborate on the terraform automation for gmail?

When we first built our user account system, we had to create a google user and add it to relevant groups in admin console whenever we hired someone, before the automation could kick in.

We're tired of that so we're looking into using a terraform provider [1], so that we can declare new users and instantiate their resources with a simple PR.

[1]: https://registry.terraform.io/providers/hashicorp/googlework...

> It's really nice for a super personal project and I'm sure it helped OP gain a lot of operational experience across several domains, but I just find this type of article to be "DYI porn" for a highly specific subset of DYI.

I share (some parts) of the sentiment.

Fiddling with a VPS and dockers (and Linux generally) landed me my current job but there's a curve and now I feel the need to up my game with a deeper knowledge and understanding of the different pieces and of the overall picture. Otherwise I am just another admin bash monkey.