Slashing data transfer costs in AWS
bitsand.cloud
bitsand.cloud
Lightsail instances can be used to "proxy" data from other AWS resources (eg EC2 instances or S3 buckets). Each Lightsail instance has a certain amount of data transfer included in it's price ($3.5 instance has 1TB, $5 instance has 2TB, $10 instance has 3TB, $20 instance has 4TB, $40 instance has 5TB). The best value (dollar per transferred data) is the $10 instance, which gives you 3TB of traffic.
Using the data provided by the post:
3TB worth of traffic from an EC2 would cost $276.48 (us-east-1). 3TB worth of traffic from a S3 bucket would cost $69.
Note: one downside of using Lightsail instances is that both ingress and egress traffic counts as "traffic".
> 51.3. You may not use Amazon Lightsail in a manner intended to avoid incurring data fees from other Services (e.g., proxying network traffic from Services to the public internet or other destinations or excessive data processing through load balancing or content delivery network (CDN) Services as described in the technical documentation), and if you do, we may throttle or suspend your data services or suspend your account.
Of course if you abuse it, you’re asking for trouble.
In Amazon's case, their bandwidth pricing isn't really defendable. It's just crap. However, sometimes you're trying to offer something reasonable, but need to make sure that a customer doesn't end up abusing something. For example, Chia is a cryptocurrency that will basically wear through SSDs (it's a proof-of-space system). There aren't explicit limits on how frequently you can write to a disk from most hosting providers, but Chia goes beyond what normal usage would do to a disk. Chia farmers would rather burn someone else's SSD that they're renting than their own. But no one at most hosting providers was probably looking at how frequently people were writing before noticing "hey, why are the disks failing faster than we'd expect?"
They probably haven't taken action on it because they probably haven't noticed it being a problem. But if you're a whale of a customer and suddenly your data transfer charges drop off a cliff, someone might end up looking into that and seeing what's going on.
It takes very low hitrates before it pays for itself several times over including management overheads.
Sometimes you can justify a complete replica outside AWS (one of the things I will gladly pay AWS for is durability)
This requires proving the users intent, which is not obvious except in the most blatant of cases (i.e. using Lightsail as a bent-pipe by writing the exact bytes you're reading). If it is a "CSV to Parquet translation layer", how would AWS possibly prove it's anything other than what it claims to be? You'd be paying a few more cents for compute, but that's the price of plausible deniability
Companies are permitted to deny service to anyone at any time for any (non-protected) reason. They typically don't have to justify service terminations to a court of law. Who would they be required to prove user intent to, and why?
Re-posting a bit of the service terms for easy reference:
> 51.3. You may not use Amazon Lightsail in a manner intended to avoid incurring data fees from other Services [...]
As you point out, they may terminate your service without any justification in a court of law. So how do they go about terminating the offenders? Well, one trivial way (from a technology and/or policy perspective): terminate everyone's service! If you blindly terminate everyone's service, that will certainly prevent anyone abusing LightSail.
But that's, uh, not good for business. So they probably want to terminate the service of only those people actually abusing it. But how do you do that?
You'd have to look at each account's usage and do something to determine if that traffic is or isn't a means of avoiding data fees from other services. In other words, you'd have to determine the intent of that traffic. Or, put yet another way: "this requires proving the users intent".
If doing so was as trivial as detecting any traffic between LightSail and the other services, they'd just prevent such connections in the first place. So how can AWS tell if some traffic between services is legitimate or not? The unspoken premise of the person you're replying to is that this probably isn't feasible for AWS to catch any and all people abusing LightSail in this way, with the conclusion being that you can (in practice) probably get away with it unnoticed.
That said, detection is easy. Look for users who spin up a Lightsail instance and use close to 100% of its bandwidth quota before spinning it down. Sort by number of such instances, and tell all users above some cutoff that in your sole discretion you believe they have violated your TOS, and are terminating their service. Doing so is completely legally defensible.
I have not checked my cost and usage reports every time I have some experimental instance for a shorter time, so I am not sure. Just from the general knowledge that AWS is permanently counting every fraction of a peanut. But as the submission shows, exceptions to the rule can exist.
> We disagree on the definition of "prove". I would not object to the claim if it had used "determine" or "detect" instead of "prove".
I do find that a bit odd, though. If I consult the Merriam Webster dictionary, I see precisely one entry under "prove" that says anything related to law and/or courts:
> to establish the existence, truth, or validity of (as by evidence or logic)
> "prove a theorem"
> "the charges were never proved in court"
Even there, the only mentioning of court is in the example sentence, rather than the definition itself -- naturally, we want our court system to be based in reasoning rather than whim.
Additionally, the meaning of "prove" given by this definition is exactly what the study of formal logic sets out to codify, and given that this is hacker news (where many are interested/involved in computer science and/or formal logic itself), it seems counterproductive to ascribe some legal meaning to the word "prove" here, as it would (to my mind, at least) be quite unlikely for others to do so.
There is going to be a program that will have rules to detect patterns in customer traffic and automatically block when those patterns are tripped.
At best you could complain in the forums and maybe if you are lucky a sympathetic community manager may look into your use case.
True. And no one has said that they must prove anything to anyone.
Amazon wants to make money, so they probably don't want to terminate the service of people who are acting in good faith. But that's just another way of saying that they probably want to determine with some certainty that someone is not acting in good faith before terminating their service.
So it's not that Amazon needs to prove anything to anyone. But they do want to prove something to themselves.
The Lightsail style billing model works same way shared vs leased lines works, if everyone fully used their max allocation it won't be possible to offer service at that price point. They can offer 2TB or 4TB for price because the usage modelling of target users supported that.
No company wants a customer to bypass their usage and pricing ToS even if they are not actively enforcing it, it is lost revenue and/or bringing in customers who you don't really want.
Your statement here is absolutely correct (we are in agreement); it is also absolutely orthogonal to what I (and others) have said.
Let me use an analogy.
Marijuana is illegal in most states of the US (and, federally, it is still a controlled substance). And yet a (relatively) recent survey[1] showed that around 7% of respondents grew marijuana at home.
How is this possible? Shouldn't that be 0%? It's almost like the DEA is slacking off or something.
... or maybe it's because they can't practicably round up each and every one these people: the DEA isn't omniscient, and given the 4th amendment they can't ransack every home within the US to catch these people. If you don't do something that gives them sufficient evidence to acquire a search warrant, there's nothing they can do about you growing pot in your domicile.
Back to Amazon. Could you, at a high level, describe a process by which they could, for a given account, determine if that account's use of LightSail is legitimate, or is instead intended to avoid incurring data fees from other services? And you must satisfy some additional, absolutely crucial qualifications: this process must not negatively impact abiding users (because they would abandon AWS, resulting in financial harm to AWS), the cost to AWS of executing this process must not be prohibitive (in terms of compute, human resources, etc), and the process must be applied across all accounts within a reasonable time frame (if it takes 1 year for AWS to comb through 1% of accounts, that means you have a mere 1/100 odds of having your service terminated for abusing LightSail for an entire year).
Something being prohibited doesn't imply that it is practicably, fully enforceable.
If your workload is just Compute or stateless with commodity or standardized API interfaces you could maybe do a move maybe.
Even for those it is fraught with problems and takes a lot of time , time you are not developing features and adding product value.
If you are using S3 or any sort of proprietary stack on AWS to the cost to migrate (retrieval + b/w or rewrite your app ) is just too prohibitive .
All cloud providers know this and plan that in their models , reason why they give out generous free tier or give a ton of money in startup programs or other hooks to get you to start .
——
AWS does not just have an all or nothing suspension policy .
From personal experience I know they do suspend your access to a single service at even single region level - our account still has SES blocked in one region because early on we handled bounces poorly and this was at least 8 years ago they even sent few warnings too, so they have pretty robust framework to handle abuse. Back then I couldn’t get it unblocked with tickets and escalations , I am sure we spending 15/20k a year then so not super small either .
These days we spend more like 250k a year on AWS and I still don’t get a proper account manager. I could perhaps get it unblocked now if I really wanted, not just worth the hassle to jump the hoops, one of the reasons why Azure is our primary cloud partner and we spend most of our money on despite subpar tech compared to AwS (GCP is 10x worse on this) .
I cannot comment on specific controls that is put on lightsail never used the service but they definitely do have a framework to suspend for every service they offer.
Just given the generous free tier, there is huge industry of using stolen credit cards to run scams or send spam on AWS which they constantly have to fight against.
Bandwidth cost is very cheap for AWS and other major cloud vendor, even cheaper than likes of OVH, Hetzner buy at, because they also own good chunk of the deep sea cables in addition to DCs.
B/W specifically is charged high as a competitive moat. Every time I consider migration out of say S3, the migration costs always come out prohibitively high because of b/w pricing, so end up staying put.
That is exactly AWS wants, and so b/w is priced costliest in AWS ( even more so than Azure,GCP) because their market share is far ahead and thus have most risk of customer retention. Hetnzer and OVH have less PaaS service moat they have to worry about unlike AWS,Azure GCP , so they instead compete by setting rock bottom prices.
More market share AWS gains, less incentive they have to drop b/w pricing, only pressure to increase.
You can download 1TB of data for free from AWS each month, as Cloudfront has a free tier [1] with 1TB monthly egress included. Point it to S3 or whatever HTTP server you want and voila.
[1] It used to be 50GB per month for the first 12 months. It was changed to 1TB free forever shortly after Cloudflare posted https://blog.cloudflare.com/aws-egregious-egress
Cloudflare rant was posted in July 2021, new Cloudfront free tier was there Nov 2021. I consider changing pricing like this for AWS in 4 months pretty fast.
Where is your 2 years number comes from?
Nitpick: $5 for 2TB is better than $10 for 3TB.
[1]: https://cloud.google.com/storage/pricing-announce#network
It works, but i have no use case for it. 100TB egress to the internet costs about 7000$ ... i think, i could do it for 20$-50$.
Treat this as an interesting hacking exercise, but do not deploy a solution like this to production (or at least get your account managers blessing first), lest you risk waking up to a terminated AWS account.
However sounds like it was deleted almost right away. In that case the charge might be 0.03 / 60 if the data was around for a minute. Normally I would expect AWS to round this up to $0.01..
The TimedByteStorage value from the cost and usage report would be the ultimate determinant here.
1. Ask for discounts if you are a big AWS customer (e.g., spend $1mln+/year). At some point, they were huge for inter-AZ transfers.
2. Put things in one AZ. Running DB in "b" zone and your only server in "a" is even worse than just standardizing on one zone.
3. When using multiple AZ do load aware AZ balancing.
But in the context of data transfer costs, this would actually increase the costs, because there's a small surcharge for Intelligent Tiering - and the only relevant storage class for sidestepping data transfer costs is standard storage (because it's the only one with free download), so Intelligent Tiering won't provide value.
https://discourse.nixos.org/t/the-nixos-foundations-call-to-...
There must be use cases for this, but I lack imagination. Cost? But not cost?
Many companies run servers without considering AZ. Then you can get the "best" of the worlds:
1. Your service is down if either of AZs gets hiccups.
2. You pay network charges and latency cost.
https://docs.aws.amazon.com/ram/latest/userguide/working-wit...
If too many people do this - AWS will "just close the loophole".
There's not one AWS - there are probably dozens if not hundreds of AWS - each with their own KPIs. One wants to reduce your spend - but not tell you how to really reduce your spend.
If you make something complex enough (AWS) - it will be impossible for customers to optimize in any one factor - as everything is complected together.
Another example is using CloudFront. AWS wants you to use CloudFront, so they make CloudFront cheaper than other types of data egress.
https://www.lastweekinaws.com/blog/aws-cross-az-data-transfe...
Seriously, if you’re at the point that you’re doing sophisticated analysis of cloud costs, consider dropping the cloud.
tilaa.com
vultr.com
hetzner.com
linode.com
If anything you could say that AWS is the closest to actually having unlimited bandwidth (or at least, half-parity there) since they don't charge you for incoming data, where other VPS charge you for data both ways.
Really which has more or less expensive bandwidth comes down to the shape of your data usage.
Which is another way of saying you buy a lump sum of bandwidth included with your VM.
The issue is not paying for bandwidth, the issue is the insane pricing.
Which would mean that you loose part of the reason to use the cloud in the first place... A lot of org move to cloud based hosting because it enable them to go way further in FinOps / cost control (amongst many other thing).
This can make a lot of sense depending on your infra, if you have some fluctuation in you needs (storage, compute, etc...), cloud based solution can be a great fit.
At the end of the day, it is just a tool. I worked in places where I SSH´d into the prod bare-metal server to update our software, manage the firewall, check the storage, ... and all that manually. And I worked in places where we were using a cloud provider for most of our hosting needs. I also handle the transition from one to the other. All I can say is: It's a tool. "The cloud" is no better or worse than a bare-metal server or a VPS. It really depends on your use-case. You should just do your due diligence and evaluate the reason why one would fit you more than the other, and reevaluate it from time to time depending on the changes in your environment.
This whole "cloud bad" is just childish.
Not childish…. it’s a growing line of thought in the IT community that has bought the cloud sell unquestioningly for 20 years.
First they give you a ton of credits, assign you internal resources to help.
Then they encourage you to simply "lift and shift" your workloads onto EC2/EBS/EFS/etc. It's 100% compatible with your current system, you can rollback, etc. This take two years, then you notice your AWS bill is 10x your old infra.
Then they say - of course, that's because you need to rewrite it all to serverless/microservices/etc that are all AWS bespoke branded alphabet soup of services. Now you are fully entrapped, and can not rollback to your own infra, let alone another cloud provider without another rewrite.
A lot of big financial firms are 5+ years into this. Several have rolled back for certain use cases due to cost, especially anything with a lot of data transfer because yeah.. performant storage in the cloud & egress are expensive, duh.
I'd say Cloud lets you do a few things, but the way I think of it ultimately is it lets you spend opex instead of capex. If that means though that your opex will end up higher than your capex, then it would be silly to go with it.
The other thing is in theory your reliability should be higher, but, again, that will depend on your individual situation, and how much reliability matters to you.
Once your org has gotten to that step, it's been so steered by AWS staff it's hard to imagine suddenly finding sense and building with open standard stuff. Very few AWS shops I have encountered avoid the siren call of various AWS-only or AWS-specific services, which they then become heavily ingrained in..
Generally I do think its mostly about transforming CapEx to OpEx, with the rest of the stuff being noise.
We (at least my team) were always pushing for a minimum of modernization when architecting migration projects - even a simple move to containers, managed by whatever orchestrator they want to run. It helps enormously on costs at least by just taking off so many overhead and overprovisioned VMs from the roster of migration candidates.
More often than not, the customer will refuse that and opt for lift + shift. It's either too hard, they don't have the resources or time, etc.
There is virtually nobody saying "cloud bad" without nuance.
AWS is Kafkaesque though
I think a lot of orgs move to cloud simply because it's popular and gartner told them so.
But taking a step away from that, it's really about self-service. When the alternative is logging a ticket for someone to manually misconfigure a VM and then fail to send you the login credentials, then your delivery is slow.
When you're chasing revenue, going slow means you're leaving money on the table. When you're a big bureaucratic org, it means your middle managers can't claim to have delivered a whole bunch of shit. Nobody likes being held up, but that's what infrastructure teams historically do.
Nah, I think it's mostly about the second part of your comment. Everyone hates waiting for months to get a VM or a database or a firewall rule because the infrastructure/DBA teams are stuck ten years in the past and take pride in their artisanal infrastructure building.
So moving to the cloud eliminates a useless layer of time wasting,
Project is a dud? just nuke the cloud project and no more charges for it.
Project is poorly architected and running like a dog? throw more resources at it.
Both of the above are harder to hide when you have to order equipment for on prem.
I think comes down to a couple of things:
- Small orgs don't have the resources to run internal clouds, nor should they be doing so. This limits the pipeline of available candidates. - Large orgs promote the wrong people to management, and they make decisions based on their mental model of the world that was developed 20 years ago. They're filled with people who don't understand the difference between cloud and virtualization. - Large consultancies make more money by throwing raw numbers at the problem rather than smart automation. i.e. it's easier for IBM to bill T&M and a whole project wrapper to patch the server than automate it. - Finance & HR teams want you to bend to their ways of working rather than the opposite.
Of the rest, you get into many of them are simply in ops because they're less skilled software developers, or they're now being asked to assure security, and that scares them so they try to lock everything down.
How is that a negative? Not every project is going to be successful. That's just a basic fact of life. That you don't have to deal with the sunk cost fallacy and just pull the plug is a good thing.
> Project is poorly architected and running like a dog? throw more resources at it.
Another positive...?! You can continue to serve your clients and maintain a revenue stream while you work on a better architecture. Instead of failing completely. And once you need less ressources, you easily scale down.
Even when everything is in IaC and 100% cloud-native, I’ve still seen dev teams bypass the approved methods because ClickOps is easier.
A prior 200+ dev shop went from automated on-prem VM builds happening within hours from when you raise a ticket, to cloud where there was a slack channel to nag&beg for an EC2 which could take a day to a week. This was not a temporary state of affairs either, it was allowed to run like this for 2 years+.
Oh and, worth mentioning, CTO there LOVED him some Gartner.
And more often than not, it had nothing to do with managers, but with individual contributors resisting change because they were set in their ways and were potentially afraid for their jobs. Same applies for firewall changes btw.
I was at a shop that provisions AWS resources via written email requests & clickops, treated fairly similar to a datacenter procurement. Teams don't have access to the AWS console, cannot spin up/down, stop, delete, etc resources.
A year later I found out that all the stuff they provisioned wasn't set up as reserved instances. We weren't even asked. So we paid hourly rates for stuff running 24/7/365.
This was apparently the norm in the org. You have to know reserved instances exist, and ask for them.. you may eventually be granted the discount later. I only realized what they had done when they quoted me rates and I was cross checking ec2instances.info I can guarantee you less than 20% of my org (its not a tech shop) is aware this difference exists, let alone that ec2instances.info exists for cross reference.
No big deal, just paying 2x for no reason on already overpriced resources!
I’m not sure my former coworkers would have done well in an environment with so few constraints. Many of them had grown accustomed to (and been rewarded and praised for) only taking actions that would survive bureaucratic process, or fly underneath it.
An org with actual governance in place really can't deliver infra rapidly, regardless of whether the underlying stuff is cloud or on prem, because whatever form governance takes in practice it tends to be distributed, i.e. everyone wants to be consulted on everything but they also want their own responsibility/accountability to be to be diluted. Bureaucracy 101..
Devs only see ops taking too long to deliver, but ops is generally frozen waiting on infosec, management approving new costs, data stewards approving new copies across ends, architects who haven't yet considered/approved whatever Outlandish new toys the junior devs have requested, etc etc.
Depends on exactly what you're building but with a competent ops team cloud vs on prem shouldn't change that much. Setting aside the org level externalities mentioned above, developer preference for stuff like certain AWS apis or complex services is the next major issue for declouding. From the ops perspective cloud vs on prem is largely gonna be the same toolkit anyway (helm, terraform, ansible, whatever)
I think it helps when people actually take a step back and understand where the money that pays their salary comes from. Often times people are so ensconced in their tech bureaucracy they think they are the tail that wags the dog. Sometimes the people that are the most hops from the money are the least aware of this dynamic. Bureaucracies create an internal logic of their own.
If I am writing some internal software for a firm that makes money selling widgets, and I decide that what we really need is a 3 year rewrite of my app for reasons, am probably not helping in the sale or the production of widgets. If another team is provisioning hardware for me to write the software on, and it now takes 2 weeks to provision virtual hardware that could take seconds, then they are also not helping in the sale or the production of widgets.
These are the kind of orgs that someone may one day walk into, blast 30% of the staff, and find no impact on widget production, and obvious 30% savings on widget costs...
Well in this example, the ops team slowing down pointless dev work by not delivering the platform that work is going to happen on quickly are effectively engaged in costs savings for the org. The org is not paying for the platform, which helps them because the project might be canceled anyway, and plus the slow movement of the org may give them time to organize and declare their real priorities. Also due to the slow down, the dev and the ops team are potentially more available to fix bugs or whatnot in actual widget-production. It's easy to think that "big ships take a while to turn" is some kind of major bug or at least an inefficiency, but there are also reasons orgs evolve in that direction and times when it's adaptive.
> Often times people are so ensconced in their tech bureaucracy they think they are the tail that wags the dog.
Part of my point is that, in general, departments develop internal momentum and resist all interface/integration with other departments until or unless that situation is forced. Structurally, at a lot of orgs of a certain size, that integration point is ops/devops/cloud/platform teams (whatever you call them). Most people probably can't imagine being held responsible for lateness on work that they are also powerless to approve, but for these kind of teams the situation is almost routine. In that sense, simply because they are an integration point, it's almost their job to absorb blame for/from all other departments. If you're lucky management that has a clue can see this happening, introduce better processes and clarify responsibilities.
Summarizing all that complexity and trying to reduce it to some specific technical decision like cloud vs on-prem is usually missing the point. Slow infra delivery could be technical incompetence or technology choices, but in my experience it's much more likely a problem with governance / general org maturity, so the right fix needs to come from leadership with some strong stable vision of how interdepartmental cooperation & collaboration is supposed to happen.
The reality is, a lot of these orgs have likely already discovered devops, pipelines, deployment strategies, observability, and compliance as code.
There's basically little in compliance that can't be automated with patterns and platforms, but in most of these organizations a delivery teams interface with the org is their non-technical delivery manager who folds like a beach chair when they're told no by the random infosec bod who's afraid of automation.
I've cracked this nut a few times though. It requires you be stubborn, talk back, and have the gravitas and understanding to be taken seriously. i.e. yelling that's dumb doesn't work, but asking them for a list of what they'd check, and presenting an automated solution to their group, where they can't just yell no, might.
I haven’t seen this be due to one set of incompetents since the turn of the century. What I have seen is this caused by politics, change management politics, and shortsighted budgetary practices (better to spend thousands of dollars per day on developers going idle or building bizarre things than spend tens on infrastructure!).
In such cases, the only times where firing someone would help would be if they were the C-level people who created and sustained that inefficient system.
You still have to go through your devops (or equivalent) team to make any network configuration/permission changes. Whether that change is implemented by a local firewall rule or some AWS configuration change is not very important.
It's not like you're going to have developers changing AWS access permissions directly. Maybe in a few employee startup, but in any regulated & audited company, you must have separation of duties and audited change control process.
This can be rational and not just following the leader. In particular.. many devs might think that working with an org that does On-prem is bad for their career, and they might be right. So from an org POV you can't hire good engineers if you're perceived as a dinosaur. This actually might be enough to send you towards the cloud even if the price by itself makes no sense
The idea being that devs who lean on cloud excessively do that to masks their lack of fundamentals, which will cause costly fuck ups no matter what technology they use, cloud or on-prem.
Maybe directionally similar idea to hiring ex-Googlers? Some orgs also don't like those. Specific mindset, specific toolbox.
Whatever their fundamental skills are, the most important way they add value is by optimizing things like lambda startup time or EC2 CPU utilization. Does this allow them to mask deep problems with fundamentals? I guess it could, but that sounds a bit gatekeep-y to me.
Devs who came up building software more or less from scratch really do have a different skillset than ones who stick to working in service-rich environments because there's a significant difference between glueing services together vs building out those same services. For example something like using a paginated API is quite a bit easier than designing/implementing one. A developer who is skilled and methodical about reading and understanding service-level documentation may not actually be able to step through debugging in a REPL, and vice versa. (Not to say that either kind of person cannot learn the other persons tricks, but as far as the differences in what they already know, those can be pretty significant.)
Assuming someone only has one of these skillsets, the most valuable one totally depends on the situation. On the one hand it's pretty cool that service-familiarity tends to be language-agnostic, but it's less cool when your S3-API expert barely understands the basics of tooling in the new language.
I ended up years later at AWS, and while I was there I built internet-facing paginated APIs over resources which had a variety of backing stores, each of which was had some behavior I had reason about.
So I don’t doubt the difference between API builder and API user, I’ve been both. I think it’s less about what you are doing and more about how you do it (with curiosity about how things work, vs. as an incurious gluer).
That said, looking at the code inside MySQL is highly instructive for the curious; AWS doesn’t provide that warts-and-all visibility into their implementations, which cuts off the learning journey through the stack.
Granted I dont think thats the norm, but also host your webserver yourself is not as user friendly as AWS.
People always forget that.
Cloud bandwidth pricing is… well the best analogies are very inappropriate for this site.
If you're actually using the features of cloud - i.e. managed services - then this involves building out an IT department with skills that many companies just don't have. And the cost of that will easily negate or exceed any savings, in many cases.
That's a big reason that the choice of cloud became such a no brainer for companies.
This is of course a wildly unrealistic ask, which is why I think the idea of merging jobs to save money is stupid. Let people who like frontend do frontend. Let people who like DBs do DBs. Let people who tolerate YAML do DevOps.
Now as to why the AWS pricing the way it is... we may only guess, but likely to promote one service over the other.
But there is no way that cross-AZ traffic costs AWS anywhere near what they charge for it.
The blog post's solution is relatively simple to put in place ; if you're already locked-in to AWS, dropping it will cost quite a lot and this might be a great middle ground in some cases.
I swear the people who say go on premise have no idea how much the salary costs of someone who will not treat their datacenter like a home lab is. Even Apple iCloud is in AWS and GCP because of how economical it is, you suck at the cloud you think you have to go back on prem, or you just don't give a shit about reliability (start pricing up DDoS protection costs at anything higher than 10G and tell me the cloud is more expensive).
We spend 100k+ on AWS bandwidth and it's still cheaper than our DIA circuits because we don't have to pay network engineers to manage 3 different AZs.
What does this mean? They steal company resources for themselves, or just configure things incompetently?
Over Christmas, everything died, and the brilliant sysadmin was on holiday. Nobody could get things going again for many days and so their entire SaaS business was failing. They lost a lot of business and trust as a result.
The sysadmin is now gone and they are back on AWS.
Your forming a larger dependency on a team lead against a custom system that now is a liability as new people come to the organization don't want to adopt an abandoned poorly understood project.
1. you think it's the sysadmin's fault
2. there are no competent sysadmins out there
what magical things do they have? that every single reasonably sized enterprise doesn't have? it should be extremly easy for a small enterprise to beat any of the main clouds* - they make crap ton of profit from you
*making an assumption that your needs are reasonably static and not MASSIVELY busting up and down your infrastructure
And yes, they pay the costs of those people and take a good profit margin, and yes there are in some ways diminishing returns to go from 3 nines to 6. But most enterprises can't match that depth of concentrated expertise, certainly not most small enterprises.
seriously??!? you think you need thousands of people to to improve your effeciency and performance I would strongly suggest employing a few good infrastructure engineers/ architects who know what they're talking about - there really is no secret sauce!! just lots of kool aid on cloud
the whole cloud thing only looks wonderful and magical if your inexperienced.
re: the whole up time x9 thing is fairly useless in the real world, since the architecture of the application is really the king here!! christ on a stick i got an application running 100% for 3+ years on NT 4 because of good design (the clue is active - active - active)
also to add... availibilty zones are a very very poor mans DR
> entire SaaS business
> [ Unmentioned - Single Point of Failure Service dependent on a single admin ]
If you are fully accounting for vacation, training, sleep etc then you need a minimum of 5 admins for mission critical services. Now, you can engineer around this to reduce your staffing requirement but I wouldn't recommend going under 2 ever because accidents happen.
This business seemed one below that, without the engineering, and I would point to the mgmt, not the brilliant admin as the problem.
This story has nothing to do with AWS or on-prem.
It's a story about incompetent management allowing a single human point of failure. If they don't change that, they'll have the same problem wherever they go.
But if you want something reliable that I can spend 30 seconds writing some terraform for, it will take an entire infra team to set up and maintain it, not to mention an entire procurement process and now having to integrate a new supply chain just for a basic multi-az setup (probably without things like backups and still without basic features the cloud gives you automatically).
Active-active, five nines, fault tolerance. Hard stuff. But managing on-prem is no harder.
This is what we're paid for.
Most people are not paid to manage infra, they are paid to talk to customers, ship features, fix bugs, and other "core business" items; just like most businesses don't build roads, they pay taxes and utilize them because the cost of doing it themselves for their preferred traffic patterns would be much more than they could justify (for now.)
But when your local IT goon says its going to be 8 months to procure the next set of hard drives for your next order of magnitude, it's a real problem and you have real money to invest in solving it, just not owning a data center money.
I worked for a company and would do it again that did the colo route, and it gave immense cost savings compared to public cloud, taking on risks that you can't do elsewhere. Before they started investing in having folks take care of the infra as a raw startup, it was just some servers and some desktop unmanaged switches. But that gave the company breathing room to survive as the business model probably didn't work without it. But also had a reputation for unreliable service.
I've also built the five nines infra at telcos, and yes you can do it with average engineers, but it's going to be time consuming, slow, and expensive in costs and labor. To allow 26 seconds of unplanned outage a month, you're going to be testing every firmware update for every piece of equipment on an ongoing basis, and practicing every operation and change as best as possible. And you need the scale that you get that 26s by having most outages only impact a subset of your customer base, otherwise you're going to blow that outage budget fast.
I am paid to relax on my holidays because I know my team and I don't have to drive to a colo to swap out a failing line card since I realized time is worth money and people quit jobs that take up too much of their time. I can A/B test (something on-prem guys NEVER get the luxury to do) so outages just don't happen at all (fingers crossed).
I have rarely met someone happy with their on-prem DC deployments, but after I moved to the AWS world it's just crazy how backwards it is to be anywhere but the cloud.
It's almost as if you feel the cloud is something magically different, it's actually just servers on racks owned by someone else.
You can own the same thing if you want and do everything exactly the same.
(See e.g. Oxide)
Honestly just sounds like the environment you describe has greater organizational issues not related to on prem vs cloud.
Compare something like rocketry or chemical engineering with running an on-prem DC. I don't see what the complaining is about. It's still a luxury compared to what other professions have to deal with.
For the small/medium businesses infra I manage here in bulgaria , the same thing would cost 5-10 times as much on aws just for the compute, throw in 1tb of bandwidth.. And this makes no financial sense.
On hetzner I pay 40 euros for 2 vms, dedicated ips, daily backups, 100gb of ssd external storage, and firewall.
Pretty sure over these years AWS has experienced a lot more issues overall.
Hetzner is for running homelabs and basic compute, not businesses. That's why EU companies constantly ignore EU rulings on US-EU privacy shield because there's just no alternative to American cloud providers yet.
How do you think AWS or Cloudflare is built?
Some people act like it's some kind of black magic. It's not. We've some customers in our DC and some on AWS for various reasons. AWS isn't less problematic. AWS is about 10x more expensive. Both on prem and cloud require people familiar with them and cloud-engineers are in no way cheaper.
Only meaningful problem is that on-prem requires some up front cost&time. That can be mitigated by leasing and other means, but indeed can be an issue for small businesses.
- small business with at least some reliability expectation, and little to none IT expertise
- huge workload requirement volatility
- having someone else to blame
- solution is already working in cloud, with teams being very comfortable there, and perceiving on-prem as “enemy” (analogy: forcing devs to rewrite stuff from haskell to java)
- that extra cost is small budget line for you
On the other hand, it does not make sense to go cloud, if you are sufficiently big and already have on prem solution and expertise in house. (Extreme case: google, does not use aws for its main load; this upper threshold I wager is couple of orders of magnitude smaller)
It’s in no way the norm for a smaller business.
The story I was hearing internally was that it was too costly to scale infrastructure the way Amazon had been doing it, it was fragile, and the expertise wasn't keeping up with growth. So, set the bar a lot higher, and build infra that is big enough and flexible enough to be everybody's infra, and then Amazon's applications (e.g., retail) could run on AWS' excess capacity. Literally the opposite of what external folks were guessing. I believe they were completely physically separate data centers - even the physical location of AWS data centers were on a need-to-know basis internally (the internal lore was "under a mountain in Virginia" - this was years before Regions and Availability Zones). And any bugs in AWS could be worked out with outside usage before moving Amazon's applications onto it.
Also, Amazon needed the elasticity of AWS because of the nature of their retail business. At the time that the initial AWS services were being developed, a massive chunk of Amazon's traffic came during the holiday season. IIRC, something like half of the year's traffic and revenue, possibly more, came in November/December each year. That meant a lot of capacity was sitting idle most of the year. Selling that excess capacity would mean shutting AWS down every holiday season.
For a time, there was an internal mailing list that wasn't yet locked down that contained reports on S3 bandwidth usage. The growth rate was shockingly high. I would guess that within a year or two of release, S3 was using a few (at least) orders of magnitude more bandwidth than everything else at Amazon combined.
This still isn't the norm for most businesses, even big ones.
As soon as you start factoring in discounts (i.e. bandwidth is nearly free at some point), the math of being on premise completely falls apart to the point you are paying more for licensing and support for the hardware than you are the entire lifecycle of your infrastructure in the cloud. It's just that bad to do it yourself.
Sorry, but who pays for licensing and support of the hardware? I've never done that. You buy a Dell server or whatever, you pay for the 5 or 7 years warranty up front, you put in in a rack and you never literally touch it again until it's EOL'ed. If something breaks Dell touches it, not you. That typically costs around $500/year, albeit paid upfront.
But you usually don't do that. Instead you rent a dedicated metal from someone. The cost if of the order of $1000/year including some storage, no more to pay unless you exceed 100TB/year. A similarly configured AWS EC2 instance is $13,000/year, plus bandwidth, plus whatever other services you get sucked into. And you will get sucked into it because if you ask AWS about any problem (like say, monitoring why your bills are so high), the answer is invariably "use this paid service of ours".
You're kidding yourself if you think using AWS is cheaper than the alternatives. Those discounts you speak of are from an absurdly high starting point. I'm sure there are lots of reasons to use AWS or a similar cloud service, but unless you only need a lot of grunt for at most a few months price isn't one of them.
That's a lot of room to work with for some inconvenience.
Not a good example in my opinion, as Google is also a major provider of cloud services. AWS isn't the only game in town.
I think Disney+ is a more interesting case. A quick google search turns up some articles saying their video streaming is powered by external CDNs. As I understand it, Netflix take the opposite approach and deliver all video data from their own Open Connect CDN, although they use AWS for other workloads (presumably including things like authentication, their recommendation engine, etc).
You probably aren't paying a "cloud engineer" just to fiddle with cloud config full time with <$100k/yr spend. Why would you suddenly need a full time sysop to do the same capability level of on-prem? If you do 10 hours of "cloud engineering" a month to support an application, the same capability level of on-prem work is probably in the same ballpark. 5-30 hours. Yes, it can be lower than cloud. No, it's NOT suddenly 160 hours every month. Yes, it does mean you need someone who can wear the extra hat, cloud engineering skills are literally no different in this regard.
Internet sites ran on basement and closet servers for years, actual server rooms for larger stuff. The jokes were tripping over power cords and backhoes digging up internet lines were the causes of most outages. It's never been easier to run on-prem than today. It cost a fraction of even budget VPS providers like Hetzner.
For certain classes of applications, it's awesome. It's not everyone's cup of tea I'll grant. But if you are inclined to play with hardware or have someone in the org who does, and an extra 2-4 hours of downtime a year isn't that big of a deal (depending on general utility and network available at your chosen site). You can save tons of money.
I think the real story is a bit sordid: office politics. On-prem and cloud are different skillsets. Companies that have been around for a while can end up with both on-prem and cloud experts who end up competing with each other, often on separate teams. Throw in some slick consultants from Amazon who are able to bend the ear of the VP and you've got a real problem. From what I've seen it doesn't end well for the on-prem team!
Next thing you know these multi-million-dollar contracts are signed and the existing teams are just shaking their heads. The smart ones have already put out their resumes and started interviewing elsewhere.
It's an old story but I guarantee you many don't know about it.
“Will my data center run out of floor space & I need to expand?” (years+)
“Will I have enough cooling & power to support the new racks we need?” (6 months+)
“When do I need to get the server order out to ensure we meet our capacity needs?” (6+ weeks)
Every one of those are capital expenditures, so line them up with the annual budget cycle - be sure to keep enough spare capacity to be responsive for last minute asks.
Don’t think my intent is to romanticize the cloud, either. It’s not better, nor worse, just a different way to manage things.
Of course if your company is sufficiently small, do whatever you know and can do quickly - customer acquisition will be more important than debating the cost of either infra in aws or a colo’d server or two in some racks somewhere. But the complexity doesn’t go away if you go to the cloud, OR if you are all on prem. TINSTAAFL.
"They just have to be an AWS expert".
Right. They just have to be experts in: EC2, S3, Aurora, DynamoDB, RDS, Lambda, VPC, LightSail, Athena, EMR, RedShift, MQ, SQS, SNS, ECR, ECS, EKS, ElastiCache, CloudWatch, CloudTrail, IAM, Cognito, and a few more. No big deal.
It may not be true for you and where you work but this is a very real thing in a lot of organizations, where development teams want a quicker turn around on booting up services they need as the product evolves and having some control over how they act with each other. It (sometimes to negative results) opens up more architectural possibilities to solve problems
What I'm saying is some people are trying to portrait rolling your own k8s or on-prem as equivalent to rolling your own crypto - better left to the chosen ones with years of training in secret monastery. This is BS :-)
I also think if a company is unhappy with their current set up and thinks that switching from cloud to on-prem (or vis versa) is the answer, they're probably delusional because "the fault, dear Brutus, is not in our stars, but in ourselves."
Requires some planning ahead of time - yes. Harder - not really.
There is too much nuance to say one is better than the other. In some cases using a IaaS is more economical, in other cases it's not.
For Apple, the same is also true[0] to say "Even Apple is running their own datacenters because of how economical it is"
0 - https://dgtlinfra.com/apple-data-center-locations/#:~:text=a....
I think that's where the disconnect is, a lot of people don't actually realize how cheap cloud compute is because they're only seeing the price for like... 20-30 servers and some basic S3 or load balancer usage. There are entire departments at Amazon that run those numbers on a daily basis to make sure AWS is always competitive with building your own datacenter.
I haven’t yet meet anyone at a company that heavily uses cloud and still doesn’t have the same number of salaried infra people as on-prem.
then load the data into the datacentre and then just pay for the last sync / delta.
So what do I do if I'm at the point where I'm doing sophisticated analysis of on-prem costs? Do I move to the cloud?
That's just not gonna happen for a lot of services.
I remember a bizarre situation where the AWS sales guys wouldn't budge on bandwidth pricing even though the company wouldn't be viable at the standard prices.
Standard discounts start at 10 TB, which is not that much.
If not using HTTP, then it's a no starter.
But yes, the moment you actually produce significant amounts of egress traffic it gets absurdly expensive. And I would expect competitors like R2 to gain ground if they can provide reasonably competitive reliability and performance.
Also, the prices he quotes are label prices, if you are a customer and you pre purchase your bandwidth under an agreement, it gets _significantly_ less expensive.
It’s also possible their billing system can’t detect transient storage usage. Request billing would work differently from how billed storage is tracked. It depends on how billing is implemented but would be my guess. That may change in the future.
Suppose you store the data there for 6 minutes. Then there's an 90% probability that the sampler misses it entirely and you pay $0. But there's a 10% probability that the sampler does catch it. Then you pay for a whole hour even though you used a fraction of that.
Over many events, it averages out close to actual usage[1]. In 9 out of 10 cases, you pay 0X actual usage. In 1 out of 10 cases, you pay 10X actual usage. (But you can't complain because you did agree to 1-hour increments.)
---
[1] Assuming no correlation between your timing and the sampler's timing. If you can evade the sampler by guessing when it runs and carefully timing your access, then you can save a few pennies at the risk of a ban.
> Moreover, uploading to S3 - in any storage class - is also free!
Depending upon how much data you’re transferring in terms of storage class, number of API calls your software makes to do so, and the capacity used, you may incur charges. This is very easy to inadvertently do when uploading large volumes of archival data directly to the S3 Glacier tiers. You absolutely will pay if you end up using millions of API calls to upload tens of millions of objects comprising tens of terabytes or more.
I updated the phrasing.
If all services, then things like whole or most of a single AZ being borked happens fairly often.
I'm back to managing my own systems. So much cheaper and less chance of nonlinear bills.
I’ve on three or four occasions the last three months notices that I got served a completely different SSL certificate than the domain I was visiting, of a domain that often could not be reached on publicly - probably pointing to some organizations internal OTA environment. In all occasions the URL I wanted to visit and the DNS of the site I was then actually visiting were located in AWS. Then less than a minute elapsed the issue is resolved.
I first thought it must be my side, my DNS server malfunctioning or something, but the served sites could not be accessed publicly anyway, and I had the issue on two separate networks with two separate clients and separate DNS servers. I’ve had it with polar.co internal environment, bank of ireland (iirc), multiple times with download.mozilla.org and a few other occassions.
I contacted AWS on Twitter about it, but just got some generic pointless response I should make an incident - but I’m just some random user, I’m not an AWS client. Somehow I could not get it clear to the AWS support on the other side of Twitter.
Maybe raising the cost of transient storage? e.g. If you have to pay for a minimum of a day's storage - but even if that was the case this would still be cost-effective, and at any rate it seems very unnatural for AWS to charge on such granularity.
+ I would guess that S3 is orders of magnitude more profitable for AWS than cross-AZ charges, so I'm not sure they'd consider it a loss-leader.
It would definitely piss a lot of people off as it is adding to their bill, but it could likely be done in a way that makes exploiting this for just data transfer not worth it without adding huge costs to most "real" use cases.
That being said, that'd be sort of "mean" of AWS to do - the data is already replicated across AZs whether you pay for it or not because of how S3 works.
What's more odd now that 1gigabit+ home connections are available, it should be obvious to anyone doing the math that it can't cost that much, otherwise a 200GB CoD install would be costing the ISP $20.
The infrastructure comes at some cost though, right? And there must be some cap on the bandwidth / throughput that a given infrastructure can handle.
So, given these, does it make sense to price bandwidth as a throttle?
Hurricane Electric does 40gig/sec IP transit for $2k/month.
Assuming you used 50% of the capacity of that link that's about 1/200th (I think, numbers are so small) of the cost of AWS for bandwidth.
It serves two purposes for them. One is obviously a nice profit center. The other is that free ingress but expensive egress causes data to flow in but not out, creating a center of gravity and a form of lock in.
I hear you, and that is an egregious margin. Just wondering if part of their bandwidth pricing calculation is driven by a goal of constraining their infrastructure costs (or other considerations beyond profit). I'm actually wondering this exactly because it is so egregious.
There is of course a thing wherein if something is free people mindlessly use it. If all AWS customers did this with bandwidth, I wonder how it would impact total usage and AWS's subsequent infrastructure considerations.
I'm no fan of their pricing and I'm sure there's an unhealthy dose of greed in there. Your phrasing just prompted me to consider what other factors might also be involved. And, if part of the rationale is actually to influence customer behavior with disincentives, then by definition there would have to be some pain involved.
I am almost certain there will be some sort of cartel investigation into this pricing between the big cloud players.
Of course it’s also a zero interest rate phenomenon. We are exiting a >10 year era when the name of the game was simply to grow and anything in your way could be dealt with by just throwing money at it. Nobody cared about cost as long as growth numbers went up.
They're not paying for bandwidth, but their connections are not asymmetric, so they need to balance egress and ingress or they will incur fees or dropped traffic.
The pricing is there to maintain this balance. Since they're obviously egress heavy, it makes sense for them to charge for egress, and make ingress free.
People think AWS is using costs to "tax" you, what they're really doing is using to control the shape and size of their traffic.
Add to that the fact that people often explicitly choose these smaller providers because they have cheap bandwidth, meaning they're going to be a magnet for high bandwidth users like DIY CDNs, streaming, game servers, TURN servers, video conferencing relays, etc.
I find it hard to believe that AWS or GCP are getting core Internet bandwidth on worse terms than much smaller companies like Vultr, Hivelocity, Datapacket, or OVH.
I call BS.
They're not a magnet for these services for the reasons I just described as you reach your per VPS limit very quickly, and to get more "cheap bandwidth" you have to be prepared to run 100s of VMs per provider, and have to consider provisioning VMs you don't need just to get access to another $5 TB of transfer, or you're just going to end up paying the per GB fee anyways.
The terms aren't worse, but the service and their guarantees are different. Again, if you ask AWS for a bandwidth deal, they'll cut you one within a few minutes that will more than halve the price of your transfer if you pay up front. Which is AWSs way of saying, "if you make your usage predictable, we can make it way cheaper."
Why? Because they have fixed _capacity_ on their links. The costs manage that _capacity_.
In my experience GCP and AWS are pretty unwilling to budge on bandwidth pricing unless you are very large and making a long commitment. If you are not spending six figures a month forget about it.
You may be right about SLA but I run large volume services out of bare metal providers and do not experience meaningful packet loss or down time in practice. Bandwidth costs are easily hundreds of times less than AWS or GCP.
I can't speak to GCP, AWS is pretty generous, and they even suggest you contact them for a deal once you're in the low 5 figure range, and that's across all services in a region. If you move enough data the discount is significant and approaches overage pricing at VPS providers.
I'm sure. If I were running things that were more bandwidth heavy as opposed to integration heavy, we would have gone that route as well, and we would have gone through the extra trouble of getting some provider diversity and redundancy built in.
For smaller cases, they can avoid all that overhead and just trade those into bandwidth costs. Which, if your costs do get high, it's much easier to build an external caching network then it is to build a bunch of external dependent infrastructure with bare metal providers.
In any case, I don't think it's that AWS is taxing it's users unfairly, I think the costs are a solid reflection of where their engineering effort and variable costs are concentrated. It seems like maintaining symmetry in bandwidth is one of those.
As a customer I can use petabytes one month and then zero bytes the next month. They have link agreements with multi year terms and possible "balance payments" required if symmetry is not maintained. This type of bandwidth isn't as cheap to provide under these terms.
I'm not sure you how get a 200x multiplier from per GB prices when the list prices are not per GB of transfer but per GB of capacity. Or are you taking a 1Gb/s price average of $1000/mo and then assuming 100% egress activity on this pipe then multiplying a full month of this 100% usage by the AWS price and dividing the two? I get around 230x there, but this is not a practical comparison, and these types of links are quite different than a standard COLO, so you're in for quite a bit of overhead.
Plus, if you actually used this much bandwidth on AWS, 2.5PB, you could get nearly a 10x break in pricing, bringing the multiple down to 23x. If you didn't try to pre purchase the multiple would be something like 80x because of their built in automatic tiering.
In terms of CloudFront I'm getting a global caching layer. In terms of EC2 I get the VPC. I'm getting quite a bit more than just the bandwidth. In terms of luck, we feel we don't need it because we've actually sat down and calculated the costs (even all the above) for running the type of product we're running, and it's /far cheaper/ to do it entirely inside AWS.
This is all assuming you actually wanted to discuss this on merit.
I get there is more to it than just transit. But VPC has loads of additional costs for NAT etc. Same with cloudfront.
Cloudflare can basically give everyone virtually unlimited free bandwidth on their S3 equivalent.
It's so preposterous to think that this reflects real cost that I find it funny, I didn't mean to be obnoxious.
At 7c/GB that would be over 10500$ just for the traffic alone, and probably about the same for the processing power.
Some how AWS has to rip you off so if there is a non rip off gateway to the ripoff then if you can use the non rip off to avoid another rip off, they will close the “loophole”.
Do you setup the same infrastructure in two or more VPS instances and then load balance? (say, [1]). Feels a bit of an ... artisanal solution, compared to using something like AWS ECS.
We've built https://nodeshift.com/ with the idea that cloud is affordable by default without any additional optimization, you focus on your app with no concerns on costs or anything else.