Hacker News Reader

Comment by danjac | Hacker News Reader

I've made it a habit to absolutely avoid any and all AWS services for any side projects, unless it's on the employer's dime. I'd rather pay a bit more per month for a flat-fee Digital Ocean droplet. Maybe I'll end up paying a few dollars more than I would with the equivalent AWS setup, but I'll rest easy knowing I won't get a surprise bill thanks to the opaque and byzantine billing. I mean, there are consultancies whose entire premise is expertise on AWS billing, so the chance of AWS newbie-me running up many thousands because I forgot to switch off service A or had the wrong setting for service B is non-zero.

And the general advice is "don't worry, call their customer support and they'll refund you". Um, seriously? If I want to spend a morning on hold to deal with a huge unplanned bill I'll call my local tax office, thank you.

Which sucks as I learn best by building things in my spare time, but AWS makes that learning process a bit more stressful than I'd prefer.

View on HN

Pretty much summarises my decision to use Linode, at a small company AWS presents a bigger monetary risk and drain on precious developer time and mental overhead than relatively small savings it might return at smaller scales...

I also actually like Linode as a company and enjoy using their services and management interface; Amazon is challenging to be positive about.

Also use Linode, they're great. Their docs are a treasure

Seems absolutely insane to use AWS for small personal/learning projects (unless the goal is to learn AWS for career purposes, I guess). It'd be like using Unreal Engine to make your 2d indie game

Always use the smallest and simplest solution that'll do the job. Simple solutions are not just as good for simple jobs...they're better

> It'd be like using Unreal Engine to make your 2d indie game

Except that would be free.

Heh, true. So, even worse!

[Disclosure] I'm Co-Founder and CEO of http://vantage.sh/ and was previously the lead PM on DigitalOcean's Droplet product as well as on the product team at AWS for container services.

We try to help out a bit on this with Vantage which essentially gives you a DigitalOcean-esque view of your AWS costs. The first $2,500 in AWS costs are tracked for free which would seemingly cover your side-projects.

It sounds like you've found your home on DigitalOcean, but I'd be curious if something like Vantage would potentially change your decision to build on AWS? In particular what you mention about runaway bills is something that Vantage sends alerts on in advance. We also show you a full inventory of your AWS resources and what they cost you.

What happens if your tracking doesn't end up matching the actual bill?

Vantage integrates at an AWS account level through a mechanism called a Cross account IAM role which allows us to ingest and process the raw data that AWS uses for its own billing systems (Cost and Usage Reports, Service APIs and Cost Explorer)

We haven't seen a single case where we don't end up matching the actual AWS bill. In fact, with a release currently in BETA and rolling out in a week or two, we'll be providing richer data faster than AWS Cost Explorer provides.

You can see a demo of that here (designs still need to be implemented) https://www.loom.com/share/dcb72a921f134e59b19a0dd3d3ab0e2f

Yeah but do you guarantee that you will cover any real billing differences, or you’re not sure enough to put your money where your mouth is?

StratusBen, this is where you could charge an incredible premium for your service.

If you could underwrite price guarantees and show an insurance company or lender that your figures are right in most cases and that you've never lost over a certain amount, you could really hike the cost of your service and provide an incredible utility across the board.

If you have spare bandwidth and can reliably do this, try building this.

I think you are on to something.

The real opportunity here is to start an AWS insurance business.

Where you provide accident insurance for clients by charging 10% of <some estimate of expected monthly running cost>.

And spread the cost of mishaps over the clients you charge.

The trick is to define that estimate of normal cost. And also to certify that clients meet some minimum expectation of clumsiness.

This is important, and will show confidence in your product.

The requirement is insurance against not exceeding a hard limit. We're talking about a rare event, so the offering (vantage.sh) is not good for this specific usecase if it isn't absolutely foolproof.

Having seen firsthand the kind of devious folks who are out there constantly trying to do fraudulent activity on AWS, I don't think any small startup like Vantage would ever want to offer a bill insurance scheme. It would be ripe for exploitation, such as someone trying to spin up 1000 instances in the last couple minutes of the month and then say "Gotcha, your bill prediction didn't match up with the real bill after all!"

At a more general level this may also be one of the most entitled asks I've ever seen in the 12+ years I've been on HN.

This is crazy. You suggest that if AWS changes billing somehow, his startup should shoulder the cost? If you don't trust it don't use it.

This would convert his business to fighting fraud 24/7 practically overnight.

While building CloudAlarm [0] (supports Azure as of now), we found that the usage data on Azure wasn't available for at least a day – in fact, they keep adding the data to it during the next day so technically two days would have passed when the actual usage data is available. I haven't gone deeper in AWS but they also gradually make the data available as per I've read. SO instantly alert we thought cannot be possible with usage and hence we chose a novel route – of 'New Resource' alarm – wherein you can get alerted for all resources created or for anything expensive than the tier you choose. The resources in your Azure subscription are available almost instantly via the API, so this was something a nice workaround we thought.

0: https://cloudalarm.in/

I don't get why you're so stressed out by AWS billing.

From my experience, once you've worked a bit seriously with AWS, billing is not a blackbox anymore and you're able to plan ahead without too much surprise.

If you're still worried, there's also the option of settings alerts on budget spent and forecast of budget, which should settle the debate. (these are also part of the API, so you can deploy and configure these alerts through terraform)

> once you've worked a bit seriously with AWS

Kind of a chicken-and-egg situation, no? Unless you're on the company dime, learning to work with AWS entails that risk. A beginner simply won't know how to configure all of these things.

the risk is minimalized by reading the docs and proper planning.

EDIT: got to love the holy downvoters

That's the point though. A beginner is going to make mistakes and should be able to learn in a safe environment. Think of a student on a tight budget at college or in a bootcamp, who has to learn AWS because it's on the curriculum.

Exactly, it's pretty common for them to shoot their own foot with a couple of hundred dollar bill, just for one tiny instance with additional options that 'you have to do'.

IMO AWS is deliberately make these things happen and reimburse it later with excuses. It's rather a strategy at this point it seems like.

I don't think it's so much a money-grabbing strategy as the problem that AWS is less a suite of unified services and more a litter of puppies fighting in a sack. With that kind of org chart it's difficult to have a unified, simple billing experience with good beginner training-wheels and on-ramps.

>AWS is less a suite of unified services and more a litter of puppies fighting in a sack

Can I quote you on this? I'm going to quote you.

Doesn't the free tier cover a lot of what a student may want to do?

The free tier is just that, a tier (not a hard limit), and will not cover any use above the tier.

Yep. I'm a novice at AWS. Last year I heard about RDS, and tried playing around with it.

I thought I had shut it down because I clicked some button that looked like it was the turn-off button. I let it go on for a few months, only to discover that I had been charged $1,500.

The one thing that really pissed me off was how easy it was to set up vs how hard it was to take it back down. I can't remember the details, but basically you cannot simply turn off an RDS instance only in the UI (even though you can turn it on in the UI). You have to install the SDK and perform some (seemingly complex) commands.

I tried explaining that I was a beginner, and that I made this mistake by accident, and that they could easily see that I had not actually used this instance at all or even put any data into the DB. But they wanted this huge list of things from me in order to refund it, like a super in-depth explanation of how it happened. Really shitty experience overall. So I canceled my AWS account and likely won't go back until I have a job that pays me to learn and use it.

You can ‘Stop’ a DB and ‘Terminate’ a DB through the UI. If you have deletion protection turned on, you can only stop the DB until it’s turned off, which can also be done through the UI.

You most likely stopped the DB, but the problem there is that AWS will automatically turn on the DB after 7 days. You also still get charged for storage for the time your DB is off.

Sorry to hear that though. I know it’s a really sucky situation to be in.

Ah yeah, I remember now. You could turn off the DB, but there was some other kind of scaffolding thing that I was still getting charged for. It seemed like an RDS-specific thing. They had to point me to a tutorial for turning that piece off.

>You could turn off the DB, but there was some other kind of scaffolding thing that I was still getting charged for.

It is a feature! /s

> I let it go on for a few months, only to discover that I had been charged $1,500.

You get billed monthly, so really, letting it go for a few months is on you.

> The one thing that really pissed me off was how easy it was to set up vs how hard it was to take it back down > You have to install the SDK and perform some (seemingly complex) commands.

Hu, no it's not. Really it's litterally one click to shutdown an RDS instance, and always has been.

> But they wanted this huge list of things from me in order to refund it, like a super in-depth explanation of how it happened.

I mean that makes sense to me. They did reserve and partially used these resources for you, so it's only fair that you have to go through the trouble of explaining why they would let it go. If there's no downside everyone would just reserve a bunch of resources all the time.

From your comment, it seems you didn't even bothered to answer their questions to get a refund, I would totally not hesitate to charge you if I was in AWS place.

> You get billed monthly, so really, letting it go for a few months is on you.

right? what a moron, that bill could have been a surprise of only $500!

lol exactly. I've discussed this incident before on HN and had the same kind of "well tough luck but you deserved it" responses. They seem not to understand how off-putting it is to newcomers or students, to basically say "hey well YOU made a mistake, the trillion-dollar corporation SHOULD take your money, idiot!"

>I've discussed this incident before on HN and had the same kind of "well tough luck but you deserved it" responses.

Brush it off. I've worked with AWS pros who get lost in the billing. In my last job, we had a big "hackathon" where the objective was to reduce our AWS spend. Overall, we reduced our annual bill by a couple million dollars.

> From your comment

Pretty stupid to just assume that I "didn't even bother" to answer their questions. I'm not going to write a novel explaining every minutia of my interactions with AWS.

> letting it go is on you

I never said it wasn't. If you pay attention, you'll see that the context of this comment is discussing the "it's on you" culture surrounding AWS and how hostile it is to newcomers.

I did not downvote this, but I have a comment: The risk is not "minimalized by reading the docs and proper planning". To minimize something means to reduce it to the smallest possible amount, and I do not like to take any chances whatsoever when a huge excess bill is a possible outcome of a single misconfigured setting that can only be ruled out by reading hundreds if not thousands of pages of documentation and then following the documentation without mistake to the letter.

There is a clear possible solution for reliably preventing any amount of unintended overpayment, and that would be to configure a hard billing limit that can never be exceeded, no matter what else is being configured. All services that generate additional costs would simply have to stop or be removed if the configured limit is exceeded.

That would truly minimize the risk, because any configuration error I make will then not lead to excess payment if I configure such a limit and the cloud provider respects it.

You're getting downvoted because you're victim blaming; a newbie shouldn't have to master anything to avoid a surprise bill.

I do not have the time for alerts in my personal life, billing PTSD is not fun.

A bad actor could hammer any publicity available services, and you could be hit with an enormous egress bandwidth bill.

Spend alerts can help with this, but spend is only calculated every 24h or so, so it's far from a panacea.

Not if you're doing simple and sensible things like using Cloud Front, and setting limits with API Gateway.

What if you're using something like grpc?

This is also one of the things I fear most when running a service in the cloud: A huge bill due to excessive network usage triggered for example by a search engine, web scraper etc. I consider it very unfortunate that "capped cost" has gone somewhat out of fashion, and nowadays many major cloud providers bill excess usage rather than cutting off or slowing down traffic etc.

Here is a simple Bash script that monitors outgoing eth0 traffic (once per second) and automatically shuts down the instance once it is greater than 1 TB:

    #!/bin/bash

    # shut down instance if outgoing traffic > 1 TB

    # 1 MB
    limit=$((10**6))

    # 10 MB
    limit=$((10**7))

    # 1 GB
    limit=$((10**9))

    # 1 TB
    limit=$((10**12))

    while true
    do
        date
        tx=$(< /sys/class/net/eth0/statistics/tx_bytes)
        echo "$tx (limit: $limit)"
        if (( tx > limit ))
        then
            echo cutting
            systemctl poweroff
        fi
        sleep 1
    done

If you save it as cutnetwork.sh in /home/admin/cutnetwork.sh, you can run it as a systemd service:

    [Unit]
    Description=Cut Network

    [Service]
    UMask=022
    Environment=LANG=en_US.utf8
    Restart=on-abort
    StartLimitInterval=60
    StartLimitBurst=5
    WorkingDirectory=/home/admin
    ExecStart=bash cutnetwork.sh

    [Install]
    WantedBy=multi-user.target

This simplistic approach may require adjustments depending on network settings and operating environment, and will not work for example if the instance is rebooted during the billing period, since that resets the counter. I would much prefer a hard-coded setting that reliable works on the instance itself, or a reliable hard billing limit that reliably turns off the service if the accumulated cost exceeds the set amount.

Thanks for sharing this.

You can mitigate that, for enterprise projects, using AWS Shield Advanced as it comes with DDoS cost protection:

https://aws.amazon.com/shield/features/

I say for enterprise projects because although it's cost is reasonable for corporate projects, not probably something you can justify for most personal/private deployments.

I know experienced people who have woken up to several thousand dollar AWS bill they didn’t expect. And the large cloud providers have clearly indicated by their actions that they’re simply not interested in implementing hard cost circuit breakers.

I use AWS very lightly but I totally understand why someone wouldn’t.

> the large cloud providers have clearly indicated by their actions that they’re simply not interested in implementing hard cost circuit breakers.

I agree, my term for this is “bad faith”.

I recently had a free $200 credit for Azure. I setup their default MariaDB instance for a side project, figuring I’d get my feet wet with Azure. I didn’t spend time evaluating the cost bc I figured, how much could the default be if I haven’t cranked up the instance resources at all? Turns out the answer is more than $10/day which I discovered when authentication failed to my test DB. Back to Digital Ocean.

My term for it is “you’re not their use case.” For better or worse, they’ve prioritized usages that would much rather have an unexpected few thousand dollar bill than have services paused or shutdown unexpectedly.

But computers can behave differently based on user choice. Right? So there could be a user option to cut service beyond a fixed spend. It wouldn't be hard to implement, and tons of people would use it. They don't do it.

It's not a tragic case of priority and limited engineering resources. They like surprise bills, just like hospitals do.

Businesspeople love it when you come to their service and click through their Russian novel of a service agreement that would take a team of lawyers to parse. Once you do that, your money belongs to them! It's their court, their rules! They love it!

> It wouldn't be hard to implement, and tons of people would use it. They don't do it.

Please describe to me, in detail, how this works.

Because every time this comes up everyone claims it's the easiest thing in the world, but if you try and drill into it what they end up actually wanting is generally "pay what you want" cloud services.

There are a _ton_ of resources on AWS that accrue on-going costs with no way to turn them off. A "hard circuit breaker" that brings your newly accruing charges to zero needs to not just shut down your EC2 instances, but delete your EBS volumes, empty your S3 buckets, delete your encryption keys, delete your DNS zones, stop all your DB instances and delete all snapshots and backups, etc, etc.

The only people I see using a feature like this are some individuals doing some basic proof-of-concept work and... a bunch of people that are going to turn it on not understanding the implications and then when they get a burst of traffic that wipes out their AWS account they're going to publish angry blog posts about how AWS killed their startup.

If, like most people, you don't want literally everything to disappear the first time your site gets an unexpected traffic spike, you can already do this by setting up a response tailored to your workload--run a lambda in response to billing alerts that shuts down VMs, or stops your RDS instance but leaves the storage, etc.

> Because every time this comes up everyone claims it's the easiest thing in the world, but if you try and drill into it what they end up actually wanting is generally "pay what you want" cloud services.

Why is it on any (usually a relatively new) user to define how an entire cloud should behave?

Users are asking for a feature that helps them stop accidentally spending more than they intended. This feature request is totally fair. Implementing such a feature would be an act of good faith towards new/onboarding users (also obviously just any user with a very specific budget use-case).

> The only people I see using a feature like this are some individuals doing some basic proof-of-concept work and...

Yes exactly. GCP offers sandboxed accounts for this exact purpose. Why is this such a far reach?

> setting up a response tailored to your workload--run a lambda in response to billing alerts that shuts down VMs, or stops your RDS instance but leaves the storage, etc.

If you're telling every individual user that falls into a specific category to build a specific set of infrastructure, why is it not acceptable to you to just ask AWS to build it?

I think the sandbox idea is a great one. They should just do away with the free tier entirely except for sandbox accounts in which everything just gets shut down the second you go over the free allowance. If you want to build something for real then you pay for whatever resources you use, but if you just want to tinker around and learn a few things then you can get a safe sandbox to do it in.

BUT, I think the parent's point is that such a feature would actually be quite complicated. It's not just a matter of saying "I only want to spend $X in this account per month/total" but defining exactly what you want to do in the case where you hit that limit. Shut everything down? My guess is almost nobody would want to do that. So it ends up being some complicated configuration where you have to deeply understand all of the services and their billing models in order to configure it in the first place. What are the odds that the student who accidentally spins up 100 EC2s for a school project is going to configure this tool correctly?

But I do think the sandbox would be great. Either you are a professional in which case it is your responsibility to manage your system and put in appropriate controls to prevent huge unexpected bills or you are a student (in the general sense of someone learning AWS, not necessarily just someone in school) in which case they provide a safe environment for you to experiment.

> BUT, I think the parent's point is that such a feature would actually be quite complicated.

Sure, but so is making a cloud. Putting the onus of defining a feature like this on users, only after hearing their request ("I want to control my spend"), is IMO unfair.

Not complicated as in "too hard for AWS to build" but complicated as in "really hard to use as someone trying to limit your spend on AWS." So the people most at risk of huge unexpected bills are also not going to be the people knowledgable enough to setup the billing cap correctly. So it would mostly be a feature for enterprises and most enterprises would rather just pay the extra $ rather than potentially turn off a critical system or accidentally delete some user data.

I worked at a company that spent ~$10m per month on AWS. We had a whole "cloud governance" team who built tools to identify both over and underutilized resources. But they STILL never cut any thing off automatically. The risk/reward ratio just wasn't there. You make the right call and shave $10k off a $10m bill every month, but the one time you take down a mission critical service, you give all of that back and then some.

> So the people most at risk of huge unexpected bills are also not going to be the people knowledgable enough to setup the billing cap correctly

Yes, which is why AWS builds it.

> . So it would mostly be a feature for enterprises and most enterprises would rather just pay the extra $ rather than potentially turn off a critical system or accidentally delete some user data.

It would be mostly not Enterprises IMO

I've been there. I shut down a bunch of what looked like idle instances doing nothing to reduce spend. 80% of which were, in fact, doing nothing. I did drop off two vms that were supporting critical infrastructure.

Everyone who had done any work on them was long gone. I had done my due diligence to identify what they could possibly be.

Still, the day of reckoning came, and we got calls of services down a week after I turned them off. I spun them back up, and they were going again without any real impact to the business.

This turned out to be a blessing as the very next week the cert these same services depended on expired and if I hadn't learned about the system by turning them off we never would have known which boxes held up those services.

Also a lesson in what happens when people leave without any documentation on where the work they did lives and how it works.

There are plenty of service limits which you need to request to have lifted. This is a common enough use case that there's an entire console section for requesting increases.

SES has a sandbox mode which you need to explicitly disable.

Metering works perfectly fine for the free tier across a gamut of the most popular products.

Beyond this, the platform captures the necessary information in near enough to realtime to provide an accurate picture of spend.

Yet with all of these capabilities, there is no coherent way to constrain identities or accounts to a certain $ spend on even predictable services.

This has been the case for so long that it must clearly be by design. AWS has the guard rails available but only really wants to use them to stop things that would cause them pain (eg SES, bucket limits).

They totally should and could have the ability to manually _reduce_ service limits.

They totally should and could direct users to a way to restrict regions and services users can use without needing to set things up through IAM.

They could and should have service specific guardrails where they make sense - eg for ec2 provide a sandbox where they limit eg instance types, base hourly spend, use of cpu credits. If they wanted to get crazy they could provide per-service actions to take when thresholds are exceeded (though this might be a bit niche).

They could have a _simple_ billing alarm interface (and enable it by default during provisioning) and they could eat the cost of sending smses or emails when those thresholds are met.

Yet they choose to do none of those things, even when they provide defaults for convenience (eg encryption keys)

While my org has a somewhat nuanced understanding of aws and can set things up to provide this certainty both on our/our clients' environments and to our developers, I have a team of people whose literal job it is to do stuff with AWS, we have had training, certs and other access that beginners do not and cannot have.

IMO it doesnt need to be a tap that people can turn off at $500 a month or whatever, it just needs to be a bit more of a "if you select these default options on these basic services, there's no way that you'll acrue 10k in charges overnight"

It's not bad faith. It's 'providing the resources you signed up for'.

Does it mean you have to go into your planning with more consideration as to cost? Yeah.

But how would you feel if your start-up finally goes viral, you're having your best day ever, and then your app just stops working because someone forgot to remove a hard spend limit?

Most people would rather see their app continue running.

And what does turning off the lights look like? If your database hits your cost limit, do you stop serving requests? Delete the data? To what extent do you want 'cost protection' for resources you signed up for?

It’s not unreasonable to ask for a mechanism to not be billed thousands unexpectedly.

Cloud billing is not easy to understand.

I would know, I work for the part of AWS that calculates people’s bills.

It's not rocket science, either. I would know, I work for premium support.

Every time I see an unexpected bill of thousands of dollars, it's because the customer poorly architected their infrastructure.

People seemingly want the freedom of complete control without the responsibility that comes with having that much control.

> If your database hits your cost limit, do you stop serving requests? Delete the data? To what extent do you want 'cost protection' for resources you signed up for?

Sounds like a reasonable configurable option rather than “you shouldn’t be able to choose at all”.

I am sympathetic to the concern about cost overages--I've hit them in AWS before--but given the way that developers and managers think about SaaS products (generally, not just cloud stuff), I tend to think that even if you required them to click three checkboxes and sign their name in blood, the first time you vaporized somebody's production database because they hit their overages and didn't think it would ever happen would be apocalyptic. And the second, and the third. And you're at fault, not the customer, in the public square.

By comparison, chasing off "cost conscious" (read: relentlessly cheap--and I note that in my personal life I'm one of these, no shade being thrown here) users is probably better for them overall.

Work in AWS Premium Support. This is 100% how it goes.

Take KMS keys for example. You can't outright delete a KMS master key; you have to schedule it for deletion. The shortest period you can schedule for deletion is 7 days (default 30). Once the key is deleted, all encrypted data is orphaned.

Guess who gets blamed for deleted keys?

HINT: It's not the customer.

I am sorry, I might be missing something, but I call bullshit. How much does it cost for Amazon to store several bytes that make a key? 5 cents per decade?

“Yeah, so uhm, you hit zero, so we deleted all your keys in an irrecoverable way, sorry not sorry” — is not a circuit breaker. Make all services inaccessible to public and store the data safely until customer tops up their balance. That’s how VPSes have worked forever.

I don’t argue that “cheapo” clients are worth retaining for AWS, clearly they are not. But this kind of hypocrisy really triggers me.

Edit: a helpful person below suggested I misunderstood the parent, and I now I think I did.

I'm pretty sure they meant that the customer schedules it for deletion and then blames AWS when they can't access their encrypted data.

Oh, in this case I misunderstood what the parent meant, and I replied to a wrong interpretation of their words.

Thank you for the clarification.

AWS doesn't retain anything for you unless you tell them to, and when you tell them to delete something (as in the example relayed by the person you are replying to), they delete it as best as they are able. That's part of the value proposition: when you delete the thing, it goes away. Why would they start now for clients who want their bills to be in the tens of dollars (when if you really care you can do it yourself off of billing alerts[0])?

Going to be real: you aren't "triggered", which is actually a real thing out there that you demean with this usage of the term. You're just not the target market and you're salty that it's more complex than you think it is.

[0]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

I used to run an AWS consultancy, which is how I know. ;) More than once I had a customer go "well support won't help me, how can I get my data back?". And I had to tell them "well, support isn't just not helping you for kicks, you know?".

Stop serving requests until the finances are rectified, delete the data 30 days after it stops. Final migration out/egress requires a small balance for that purpose.

The engineers designing and building these systems are some of the best in the world, this is relatively trivial.

There is absolutely nothing trivial about this.

Relatively trivial. In other words compared to the rest of the infrastructure and billing system this is nothing.

Yes in some cases the default is quite expensive – the same was there with SQL Azure (though they have changed that recently) and it had created a good amount of bill for us (though for their credit, Azure did refund in all such occasions because we didn't use the capacity at all). However, I don't know why the alert system doesn't have an option to say "here's my budget, alert me as soon as when my daily pace is set to exceed the monthly budget" instead, you have % of budget amount consumed based alerts, like you can get email if you say 50% of my budget is consumed, which happens every month so kinds of defeat the purpose of an alert.

We ended up creating a simple solution (cloudalarm.in – in beta) that provides such budgeted pace based alert and more ways to get instant alert which isn't possible with usage based alerts.

Yup. I used AWS at my last job. We had teams of people using AWS, we had fancy 3rd party tools and extraction metrics to track and report on costs. There were still PLENTY of times when I would just scratch my head "well it looks like EC2s cost an extra $1000 this month. I wonder what happened"

> And the large cloud providers have clearly indicated by their actions that they’re simply not interested in implementing hard cost circuit breakers.

Since I enjoy tilting at windmills--how do you propose this works? Like, in detail.

Because every time I try and drill into details of this with someone, it winds up what they really want is "pay what you want" cloud services.

AWS is much, much more than just a place to run a virtual server and many resources accrue on-going costs with no way to "turn them off". When you hit your hard circuit breaker, do they delete all your EBS volumes and data in S3? Your private SSL root? Your user directory? Encryption keys? DNS zones?

The number of people that would want all of that removed when they hit their $X/mo limit is likely minuscule in comparison to the number of people that would turn this on not understanding what it really meant and then publishing angry blog posts about how Amazon killed their startup right when they got popular and traffic spiked.

https://docs.microsoft.com/en-us/azure/cost-management-billi...

e.g. "virtual machines are stopped and de-allocated. The data in your storage accounts are available as read-only."

Most control plane operations will also be blocked. It gets complicated with more complex resource types, but it gets the job done anyway.

Note that this functionality is sort of required to support pre-paid plans without allowing them to exceed specific limits, which do exist on Azure. So there's a business dependency on this functionality today, it's not a hypothetical.

That wasn’t a value judgement on my part. I’ve made the exact same comment as you previously.

Give me a page in billing listing everything that can use money, let me enable/disable them manually (this is mostly to allow restricted dev accounts) and select which, if any, should be allowed over the bill total I set. Opt-in, but yes, do let me kill every EC2 instance if I choose.

>I don't get why you're so stressed out by AWS billing.

Presumably because if you haven't used it and are digging deeper there's so many services with different types of billing that it can be hard to keep track.

And who among us has not left something running way after it should've been shut down accidentally..

I never have left something running like that, but someone else in the org had made a service that would spin a t1.micro for end users to experiment with one of our saas offerings. Iirc the instance would run for 24 hours maybe, and then shut down and delete itself. It didn't delete the ebs volume, though.

So about 6-9 months after I left I get an email, please delete these, I'm too busy to use the UI to delete these 8000 ebs volumes. I don't know how much that cost, even for the final bill month, but it took me an hour and a half to write and test the script, hour and a half to run it, and an hour for my time to get reqs and prepare an invoice.

I've managed AWS for corps that Have spends in the millions of dollars per year (and knowing how split brain org charts are, possibly millions per month) on AWS and other platforms. Actually I know it was in the millions per month range, two transcoder services were costing $250k/m. Obnoxious.

Until you're into EC2-Other and then you have to follow various guides to figure out most of what that means. Even then, it's black box billing that Teams even struggle to explain. I spend a ridiculous amount on egress that's nearly impossible to track.

If I'm using my own money on a personal project, I do not want "alerts". I want a maximum budget spend per X, where X is a small increment of time, like an hour or day.

Supporting hobbyist projects would absolutely lead to higher AWS adoption, at least in smaller companies.

I struggle to understand what's so interesting in X(AWS/Azure/GCP/Alibaba/SAP/Oracle...) Cloud that it appears basically daily on HN

I see it like +-decade old (in mainstream) wrapper/apis over managing VMs/Infra while being proprietary as hard as possible

What's the difference between this and javascript frameworks posts? (except js frameworks being OSS)

How many years have to pass until "Cloud" stops being $hot_topic?

Were "admin" topics (heh you know those guys that were predecessors of "DevOps") 10-15 years ago hot too?

I don't think we are even near the peak "Cloud" before it can slow down. We are just starting to see now entire dev envs in the cloud, and I'm pretty sure we will continue to go in that direction.

> I struggle to understand what's so interesting in X(AWS/Azure/GCP/Alibaba/SAP/Oracle...) Cloud that it appears basically daily on HN

> I see it like +-decade old (in mainstream) wrapper/apis over managing VMs/Infra while being proprietary as hard as possible

Don't know if serious or not, but nevertheless let me try. It's the global and near infinite scale, the enormous amounts of managed services you get behind those APIs. You need a database/message queue/object storage whatever at whatever scale? Have at it, and pay as you go. If you can't see the interest in that, i wonder what it is that you do.

And IMHO there's nothing inherently complex about the APIs of AWS or GCP ( the only ones I've really used). They're as complex as the things they manage.

We are talking about things we re using or want to use. And the usage of cloud is not going to go down anytime soon.

For example, I think AWS spend is likely the biggest spend in my company after the payroll/offices, so it is pretty important topic, business-wise.

And unlike JS frameworks which only matter to a subset of JS frontend developers, everyone can use cloud: JS or Java or Rust or C++ or C; frontent, backend, data science, ML, compilers, embedded.

> I struggle to understand what's so interesting in X(AWS/Azure/GCP/Alibaba/SAP/Oracle...) Cloud...

We're still afraid of the cloud spending all of our money.

The original, main value proposition was that you don't need to physically manage your server anymore. Some products are just wrapped, managed versions of something you're already familiar with. Then there are more "original" offerings like DynamoDB, Bigquery, and Bigtable that bring a lot of value on their own and are significantly easier to operate at large scale than any open source equivalent.

I feel you. I do take the risk – the leverage on automation and manageability that Terraform e.a. give me are just too good to pass, and only with a 100% API approach like AWS provides it can I play the 100% infrastructure-as-code game, and I simply won't play any other game anymore.

Through the very same means, first thing I do with every new AWS-based project is setting up a cleanly organized Org with centralized billing, centralized IAM&Roles, centralized billing alarms, centralized SCP limitations(!!!) (as in “I will never run anything in Southeast Asia, so I disallow anything in Southeast Asia for all Org accounts), and very not-centralized resources per stage/subproject/vertical/whatever.

Plus sensible service limits on everything that has a service limit (request on API gateway etc.).

But as someone here said: your risk will remain > 0, you just have to accept that.

DigitalOcean has a terraform provider you can use for an IaC approach

DO doesn’t offer accounting in services (who accessed what and when), no IAM which is a huge problem and their APIs and services are a bit low on reliability. Spaces has extremely low rate limit and their communication API time outs often. The k8s service works overall but has some annoying hiccups that only support tickets will fix, the CDN returns random 503 errors. All droplets are shared and resource contention is a thing.

That said, AWS support has its own issues, rarely solving the problem even when we pay for a TAM. Services like elasticache are hard to upgrade with zero downtime. Their solutions always involved spending inordinate amounts of money on open source clones with 1/10 of features and their good services (DynamoDB) will cost an arm and a leg.

My 2 cents.

In terms for bang for your buck, DO is much cheaper.

Yeah you can get a cheaper AWS server, but it's a much lower performance one.

From what I can see, it's not a matter of bang for your buck, what matters is that AWS scales lower than a DO, so if your are not fully using your VPS, AWS is cheaper.

Of course, I side with the GP here, it's just not worth the risk. I could save a bit by switching my VPS too, but I won't.

With employers I almost always use AWS, for side projects I almost always use hetzner for cheap servers. I don't even think you need to worry much about learning AWS unless you need it but if you do you can limit your budget, set alerts and hope they go off.

I'd be happy to never touch AWS unless a) I'm not the one paying for it and b) there is a genuine need. Unfortunately it's increasingly a job requirement.

IMO, AWS isn’t competing on cheaper-in-dollars-per-byte but rather in faster and cheaper for your engineering team. If your engineering team is free (as you might decide your time is on a hobby/side project), it’s harder to make the case (I still run my side projects there though), but when they can make the ops team half wearing AWS badges, that offsets a lot of lone-item markups.

> faster and cheaper for your engineering team.

I really wonder how true that is. Sure, for things like S3 or RDS it’s indeed easier, but for most other things I find AWS either very limiting or extremely arcane.

Even “simple” things like Lambda underdeliver, just this week we run into problems using it with VPC, for example. ElasticBeanstalk was another one, fine-ish for simplistic things but problematic with the smallest customization, also lots of undocumented and undebuggable quirks, like breaking if you use UTF-8 characters in your commit messages, for example.

Of course, we now have the problem where some people, both seniors and juniors, only know or only ever worked with AWS, which makes the assertion that it is “faster and cheaper” correct, but is worrisome, as lots of people are not being taught what used to be the basics 10 or 20 years ago.

Curious what your issue was with Lambda and VPCs. We use that combo all the time where I work and it's fine as long as you have the IAM roles correct.

Don't get me wrong, I'm a Lambda fan. But unlike EC2/EBS/etc, talking to some AWS services from a Lambda that's inside a VPC requires additional infra and you have to pay for the egress. In the end it just wasn't worth the price for us. It was a bad surprise money-wise.

Ah, that’s the issue, at least for me - IAM roles are pretty nuanced, and it’s difficult to understand all that’s happening.

I’m working extensively with EventBridge now, and their “security” docs mix “what can access EventBridge” with “what EventBridge can access”. Also, different AWS services all seem to have different requirements - e.g, some are role-based, some are service-based, and some are resource-based. It gets complicated very quickly.

Lambda had some gotchas around the warm-up time the last time I used it to implement something. We had to have some extra workarounds to prevent the functions from going "cold".

I recently switched jobs. At my old company the dev team basically had carte blanche to setup their own AWS infra without any real restrictions (there were some but very few). It was nice, I almost never had to ask anybody outside my team to do anything to unblock us (at least from an infra standpoint).

In my new job we also use AWS for everything BUT I haven't used the AWS cli once. I don't even have credentials. Basically, their is a platform team which is responsible for running a k8s cluster (or clusters really) and some other common infrastructure which is all running on bare EC2 (no RDS, no EKS, we do use S3 for blob storage but that's about it). And I have to say it is pretty amazing. No more dealing with arcane IAM rules, or trying to figure out how to string together a chain of lambdas to do some sort of complex orchestration task).

I've really come to appreciate the model of using k8s as your "cloud platform." Having a dedicated team that manages the k8s clusters and makes sure they are elastically scalable and reliable. Everyone else is just deploying stuff to kubernetes. They could decide to move everything to a colo tomorrow and I would have to change exactly nothing about how I do my job.

> If your engineering team is free (..), it’s harder to make the case (...), but when they can make the ops team half wearing AWS badges, that offsets a lot of lone-item markups.

I have to call bullshit on this claim.

Let's look at the facts. With AWS there are only two scenarios: either you go with the classic "VMs provided by a cloud provider" which is represented by EC2, or you go with hosted services and higher level abstractions like AWS's serverless offerings.

Regarding the EC2, AWS offers absolutely no operational advantage over any other cloud provider, at the expense of being far more expensive. Also, CloudFormation/CDK is arguably far worse and outright developer-hostile than any configuration-as-code alternative. This comparison makes even less sense if we look into AWS' containerization offering, which is either half-baked (ECS) or an afterthought that lags behind alternatives (EKS).

Then we have the higher level abstractions of AWS' managed services and serverless options. Price-gouging runs rampant on this domain, and arguably demands much more training and man-hours to become effective at running production services when compared with just running your own services. This scenario entails higher costs and the only arguments that any ops team can muster revolve around sunk cost and vendor lock-in.

The price gouging services make sense if they avoid you having to hire additional employees. That's the benefit of any managed service provider: they can run it more efficiently than you can (once you add all the people, supervisors of those people, people to cover when those people are on vacation, etc).

It's a way to shift people costs to IT operational expenses. If you don't do that, it's more expensive. If you do, it can easily be less expensive. I'm pretty sure we're at the point where it's less expensive because developers are waiting minutes rather than weeks [or more] for TechOps actions to happen (we were on-prem previously [and I ran TechOps]). That saves time and changes the way you think about TechOps changes. If they're lengthy, you make choices that avoid changes in TechOps. If they're fast, you make choices that make the most sense for the product and customer.

> The price gouging services make sense if they avoid you having to hire additional employees.

But the fact is that it doesn't. It's another service that needs training/experience to develop and operate. Arguing about these hypothetical savings is just a veiled appeal to the sunk cost fallacy and vendor lock-in I've mentioned.

> I'm pretty sure we're at the point where it's less expensive because developers are waiting minutes rather than weeks [or more] for TechOps actions to happen (we were on-prem previously [and I ran TechOps]).

I'm not sure this scenario is remotely realistic for the past decade or so, specially after the inception of containerization. Even in bare metal deployments anyone can get multiple databases configured and going in a matter of minutes.

Containers don’t get you more host machines racked or more disk shelves added to the SAN. On-prem, it solves configuration within a (nearly) fixed scale which, if that’s your only problem, is great. If you’re in your own DC/colo, there’s more advantages to moving out than containers can provide alone.

I literally can’t afford to invest to the level that AWS can to run operations. AWS bandwidth is incredibly expensive right up until the point where you or a neighbor is getting a DDoS attack that Amazon just “handles” for you. My customers don’t care where we’re hosted and won’t pay extra for either on-prem or cloud-hosted. They just want it to be up and transparent. For us, AWS is cheaper/faster all-in. That’s not true for everyone and, if it’s not, please don’t use it.

> Containers don’t get you more host machines racked or more disk shelves added to the SAN.

That's immaterial to the discussion, and reads like a non-sequitur. You want a service. You deploy the service in your infrastructure. If necessary, you scale your infrastructure to meet demand. That's it. If you want to spin up a database instance, just do it. With containerization that takes between minutes and seconds.

And to drive the point home, in case you're not aware, AWS is not the only cloud provider that offers horizontal autoscaling. Some small providers even sell it out of the box, both through their Kubernetes offerings and/or through their own APIs.

Also, the sales brochure for managed services mentions scalability and reliability, and in the case of AWS also global deployments, but the truth of the matter is that it costs a hefty premium and in most cases it's totally irrelevant.

So, pointing out databases in practical terms means close to nothing.

> I literally can’t afford to invest to the level that AWS can to run operations.

And that's perfectly fine because a) AWS really is not full proof (see the latest outage of AWS's US-WEST-2 region which might have single-handedly dropped AWS's reliability to only 99.5 this year), b) operating your own infra already gets you plenty of 9s easily, c) the theoretical difference in the 9s you get and the 9s that are advertised by AWS is more often than not totally irrelevant to the usecases you need to meet.

To sum it up, you may argue all you want about how AWS's Rolls Royce is far superior than any car in the market, but the truth of the matter is that the vast majority has all their needs decisively met and even surpassed by running any other cloud provider's Ford hatchback.

But it doesn't save much engineering time. You now need to manage those services, and the high mark-up means you need to invest effort in scaling things up/down.

Yes, it took me about a week to learn to set-up a Postgresql high availability cluster. But now it saves me $4,000 per MONTH for each of our 10 databases.

And if you are using EC2 instances, AWS saves very little effort compared to bare metal.

So did you send your database logs to a centralized logging system? Did you set up roles and keys to access those systems? Are your roles integrated with the rest of your permissions system? Do you have a perf dashboard where you can see the real-time usage of your DBs? Have you already rehearsed updating your database version?

The thing is that it's totally possible that you learned everything needed to set up the cluster, in my experience most database systems that aren't set up by a professional DBA will sooner or later hit a configuration or maintenance snag. Once that happens, you're pretty much totally on your own and for critical systems that downtime is going to cost you more than the costs of your infra.

So you either need to have a DBA on retainer if you're serious about data integrity, or you pay management costs which means your system was set by literally the most expert people on the planet in the area.

If you're running a cluster where performance is the highest priority and downtime and maintenance isn't a huge issue because you have a nice decent maintenance window and enough dev cycles to spend on staying up to date, for sure, go for it.

But in my experience, if you care more about a system staying up, good managed infra is so much more reliable that it's not even a question.

> So did you send your database logs to a centralized logging system?

Not needed.

> Did you set up roles and keys to access those systems?

Yes.

> Are your roles integrated with the rest of your permissions system?

Not needed. Access to the DB is limited to a very small circle, by design. Postgresql's mechanisms suffice.

> Do you have a perf dashboard where you can see the real-time usage of your DBs?

htop/task manager and Postgresql's internal monitoring have so far been sufficient.

> Have you already rehearsed updating your database version?

Indirectly yes, although AFAIK (happy to learn otherwise) major version update are also not supported by RDS.

This has been working for years now, with actual paying customers (so "in production"). It's always about opportunity-cost and diminishing returns: RDS costs about as 3 developers. The benefit the product gets from 3 more developers far outweighs the benefit of another "9" in the SLA.

Even if all of the savings would have gone to pay for a DBA, we'd still get a better deal: an in-house DBA will focus on addressing our specific needs. RDS also has a serious opportunity cost since it prohibits using extensions that are very useful for us, forcing us to spend time on workarounds.

And frankly, it's not like RDS is even that reliable. We had multiple incidents where the database stopped responding or slowed down to a crawl for several hours, and we had now way to debug this (since we cannot access the server itself). They also don't seem to handle anything about Postgresql tuning.

For the customer, the app slowing down (because RDS has a very limited IO bandwidth) is an outage (and is much more frequent than some system error).

tl;dr: Many people can meet their needs with a Ford, and living on the streets to pay for the extra features of a Ferrari is not a good tradeoff for them.

EDIT: line spacing

That must be some kind of “trick of the trade” because Firebase originally had feature for limiting the bill with a hard cap and they even have videos explaining how to use it however it was removed later on. Now they suggest building a script that monitors your bill and nukes the project if something happens. The catch? Billing is not real time.

Check out https://www.oracle.com/cloud/free/#always-free . I tried to write a comparison https://gitlab.com/ptman/matrix-docs/-/tree/master/free-matr... . Also check out https://free-for.dev/#/

I was recently forced to migrate my hobby FOSS project the other way: from DigitalOcean to AWS. The primary reason being: a generous quota of 60,000 emails per month to send via SES. Most mail providers give only up to 3,000 to 6,000 emails per month.

You can use ses without moving your digital ocean server.

From the AWS SES free tier fine print

> 62,000 Outbound Messages per month to any recipient when you call Amazon SES from an Amazon EC2 instance directly or through AWS Elastic Beanstalk.

wow, they are really tracking from where you're calling the API to give credit, first of its kind I've heard :)

It's not that hard or far-fetched, given that your EC2 instance needs to have permissions to call SES and AWS can tell whether these requests come from inside or outside their network.

Good eye.

You can host your project on DO and connect to SES to send emails. Why do you have to move the complte project over?

Those 60k free emails only apply when SES is invoked from an EC2 instance or Elastic Beanstalk

I think they mean having a seperate small service just for mailing on beanstalk.

> Those 60k free emails only apply when SES is invoked from an EC2 instance or Elastic Beanstalk

What stops you from putting a service running on a free tier EC2 instance to listen to requests to call SES?

Nothing, it's just another layer you have to manage yourself (sidenote: the free tier EC2 instance is free for 12 months since the AWS account creation).

You'd also need to make communication secure, and use some authorization mechanism to avoid abuse and email addresses leaks (or other private information you might be passing to your service to include in the email).

> Nothing, it's just another layer you have to manage yourself

You already have to manage that layer. From your description, the only choice that was really on the table was whether AWS SES was a good enough reason to port a entire system to AWS, or if it was reasonable to just deploy the email firing service to AWS.

> You'd also need to make communication secure, and use some authorization mechanism to avoid abuse and email addresses leaks (or other private information you might be passing to your service to include in the email).

You still need to do that even if the service is running on AWS. Running the SES client in a VPC is also a very poorly researched excuse as you can put up a privatelink connection to get the same effect.

All in all, it's hard to believe that someone thought it was a good idea to port an entire system to a different cloud provider just because they wanted to send emails.

How is 60,000 _a month_ generous?

> I mean, there are consultancies whose entire premise is expertise on AWS billing, so the chance of AWS newbie-me running up many thousands because I forgot to switch off service A or had the wrong setting for service B is non-zero.

That line of reasoning is wrong. I'm sure there are consultancies that specialize in office stationery procurement; doesn't mean anything for your small use case of buying a few pens for your home office.

There's no chance I accidentally buy $10,000 worth of staplers when I walk into Office Depot though, while the opposite is extremely easy in AWS. Plus, when I checkout I'll get an itemized total of what I owe before I pay, I won't be charged an unclear amount at a future date.

You can set a global budget cap to avoid this kind of concern. That said, I've also blown the budget cap by accident - so agree with you on the DO.

What good is a budget cap that can be "blown by accident"? From a budget cap, I expect the key invariant that it reliably caps the budget.

It's the same with Azure. On multiple occasion, I had databases created in tiers several times expensive than the one I use with my subscription. This wasn't a manual mistake; a sleeping app got awakened (may be I'd have hit the run button by mistake) and it ended up creating the database via the ORM framework. Since the ORM framework is only executing create database on the sql server, Azure goes with the default for tier which they had chosen as one with $250 or something monthly price. I've setup the budget alerts on Azure but these are threshold (% of budget consumed) based and they come every month so technically they aren't alert rather information which requires you to do the math whether or not you're in the budget. So you tend to ignore them after a while. Recently, we decided to build a simple solution ourselves [0] which gives alerts based on budgeted pace and not consumption.

0: https://cloudalarm.in/

DO is very amateurish, exactly how to use secure resources with rbac or anything? I mean it took them 10 years to have a load balancer service lol.

I would never use DO or Linode at work, those are for garage project over the week-end.

DO does not have a good support channel. My droplet died. I couldn't reach it, and neither could the world. There was no emergency button for getting help. Just send in an email. After a small time window, I had to just delete and rebuild my droplet. Two days or so later, they got back to me and because I deleted it, they couldn't debug it. Two days to get back to me on a dead, non-reachable droplet. I don't think that is acceptable for anything running in production.

But the whole point of treating cloud servers as cattle rather than pets is that when one dies, you can spin up another one, right? Ironically, that's one reason I prefer AWS over DO and the like, because AWS's EC2 auto-scaling and availability zones are great for this kind of resilience.

Who told you to call their customer support for a refund? AWS (and other cloud vendors) practically never refund. They will give out credits for their platform but that won't help you much as a private individual who just lost hundreds of dollars.

That’s kind of a meme in HN and Reddit: there were a few public occasions where users were refunded and people now just assume AWS will also refund for every instance.

I've never had a refund request rejected myself, and I've made multiple mistakes over multiple accounts. Even things such as "Hey, I forgot to turn off this ec2, i wanted to destroy it, any chance for a refund?"

They always refund mistakes, of private individuals or companies.

The first ones anyway. After that, not really.

> Who told you to call their customer support for a refund? AWS (and other cloud vendors) practically never refund.

I've gotten refunds about a half a dozen times now. Every time I've asked. One was for over a hundred thousand dollars.

I've never paid for support, I have no contacts within the company (like are often necessary at places like Google), I literally just put in a support ticket asking for a refund and got a refund.

They usually require some documentation/explanation of how you're going to avoid making the same mistake again (which is fair), but otherwise have been very cooperative.

I would avoid DO and similar McDonald's-type cloud providers for anything production. Commercial or private.

AWS might make it difficult to figure out the cost (most common complaint) but the services are professional grade and their support is as well. DO on the other hand provided me an instance with an IP that was on a public blacklist and banned my account within 5min of spawning an instance with the explanation that "it was compromised and hacking" failing to accept that they provided me with the OS image and the public IP. Took me two months of arguing to get the account unblocked and balance withdrawn. Lesson learned; you get what you pay for. Back to AWS for me.

Reply on news.ycombinator.com