AWS costs every programmer should know
david-codes.hatanian.com
david-codes.hatanian.com
For example I've been following the development of an online manga reading site (mangadex.org) that is now pushing over 1PB/month of images. If they had built it on aws, even at the lowest rate of $0.02/GB with cloudfront/S3, it would be $20,000 per month. But they ended up paying nothing by using cloudflare who gave them unmetered bandwidth.
(well, until they got throttled earlier this year but they got out of that by upgrading to a measly $200/month plan)
OVH Public Cloud[1] offers unmetered bandwidth (250-500 Mbps) in all regions, apart from Asia-Pacific.
Hetzner Cloud[2] offers 20 TB (1 Gbps) for each cloud instance, but has locations only in Europe.
Both of them offer dedicated servers with up to 1-3 Gbps unmetered bandwidth that could be used as exit nodes.
[1] https://www.ovh.com/world/public-cloud/instances/prices/
But there’s always a “secret limit”—the point where your bandwidth usage looks like a DDoS attack, and the tier-1 exchange feeding the cloud provider’s DC decides to blackhole your traffic for the sake of the network.
Why not? It's deliberately misleading, no?
That’s useful to me so I can just not worry about that thing.
Am I missing something? I'm not seeing another side to this. Secret fair-use rules are exactly as dishonest as when the telcos lie about 'unlimited' data plans, no?
That's true, European providers work best for Europe.
But how many global locations does a service really need, if it's sitting behind a CDN? Three locations (East Coast, Western Europe, and Singapore) alone are enough to be within 100-150 ms of the most of the world's population.
OVH probably wouldn't be the best choice for projects focused exclusively on South America or China, but it already covers the rest of the world pretty well, including three locations in North America (East Coast, West Coast, and Canada).
Let's say you spin up 10 instances with OVH at the cheapest level (ie, $200/month). That is supposed to give you dedicated 3Gbps in addition to compute / storage etc or around 1PB data per month + compute and storage.
This compares favorably to google standard egress at $60,000/month. But as soon as you build a business model around this - poof - rate limiting -> some TOS violation claim pops up.
"Oh, we meant unlimited or unmetered, but only if..."
Seriously, with unlimited / unmetered at $20 these would make the great bases of things like CDN networks, image / static asset hosts for big properties etc. But it generally turns out to be total BS.
In contrast - paying for bandwidth with AWS / Google etc -> no one has ever complained to me (though my current usage is minuscule in the distance past had high usage experience)
If bandwidth was at the core of my business model, I would certainly want to pay for it separately to avoid possible interruptions. In case of OVH, I assume, that's what "Bandwidth upgrade"[1] with a "limit of 20 Gbps per customer, and per datacentre" is designed for. It's not unreasonable to consider "spinning up to 10 cheapest cloud instances to avoid bandwidth limits" as a service abuse.
[1] https://www.ovh.com/world/dedicated-servers/bandwidth-upgrad...
The current price per TB is 1 EUR + VAT if applicable, so even if we assume you'll have to pay the German VAT, it's 1.35 USD/TB.
That's true for Hetzner Cloud, but Hetzner's dedicated servers have been unmetered (with a guarantee of 1 Gbps) since October 2018[1].
I've looked at them to use for my email/small web server and like their prices and features, but am not sure of the GDPR implications.
Currently I use a US cloud provider, and I'm in the US, and so am a controller or processor not established in the Union, and all my data processing takes place outside the Union. All my GDPR obligations, if any, are those that arise under the extraterritorial jurisdiction provision of Article 3(2).
If my server was hosted in the EU by an EU company, would that still be the case? Or would GDPR now apply via the in Union jurisdiction provision of Article 3(1)?
It can't actually accomplish that goal, because the EU doesn't have the jurisdiction for that.
For controllers and processors that are in the Union, GDPR applies to their processing of personal data of people regardless of where those people are or what entities they are citizens of.
So, for example, as a US citizen residing in the US who has never set foot within about 7000 km of Europe, but has bought things from vendors in the EU, those vendors need to obey GDPR when dealing with my data.
For controller and processors that are not in the Union, the EU lacks the authority to enforce such a broad requirement on them. Instead, the requirement is that if the person whose data you are processing is "in the Union" and you are offering goods and services to them or monitoring their behavior as far as their behavior takes place within the Union, GDPR applies.
(Whether or not they can actually enforce that is still an open question).
Putting this all together, if I'm in the US, with users in the US, but having my server in the EU makes me count as being in the EU for GDPR purposes, then I have to obey GDPR when dealing with US users. If having my server in the EU doesn't do this, so that for GDPR purposes I'm in the US, then GDPR does not apply to my dealings with people in the US.
This Regulation applies to the processing of personal data of data subjects who are in the Union by a controller or processor not established in the Union, where the processing activities are related to:
a) the offering of goods or services, irrespective of whether a payment of the data subject is required, to such data subjects in the Union; or
b)the monitoring of their behaviour as far as their behaviour takes place within the Union.
So it basically says that the GDPR applies, I don't think that hosting in europe would change anything at all.
Yes, it was not appropriate for situations where you had to pay for upload bandwidth. We were very careful in designing the gui to disclose the fact that it would use your upload bandwidth, explain the possible costs and benefits, let users opt-in if that was ok with them, monitor the download status, and switch the BitTorrent feature on and off.
TomTom real time traffic prediction system depends on users trusting them enough to opt-in to uploading anonymized trip measurements, so it was very important not to do something that broke their trust by abusing their network connection.
https://www.tomtom.com/lib/img/REAL_TIME_TRAFFIC_WHITEPAPER....
TomTom had an "iTunes-like" desktop content management and device control desktop app called TomTom Home, which was implemented in xulrunner (the underlying framework of Firefox and Thunderbird, kind of a predecessor to Electron for writing cross platform desktop apps in JavaScript with C++ XP/COM plugins).
The first thing I tried was to make an XP/COM plugin out of the LibTorrent library. That worked ok, but the idea of increasing the size and complexity of what was already essentially a web browser with a whole bunch more complex code that does lots of memory allocation and networking all the time, didn't seem like a good design. This was long before Firefox supported multiple processes, so the same app handing bittorrent would bloat the app and degrade the user interface responsiveness.
However RedSwoosh, FoxTorrent and BitTorrent DNA all ran in separate processes that just handled all the BitTorrent stuff, and that you could talk to via https (to stream the file with a special url, and retrieve progress telemetry on another url). And it's incredibly easy to integrate xulrunner or any web browser with those servers via http, so no C++ XP/COM plugin required.
Another issue is that you don't want every application to have its own BitTorrent client built in, or you will trash the network and disk, since otherwise they would compete for resources. It needs to be a centralized a system service, shared by any app that needs it.
BitTorrent DNA worked nicely that way. And it could fall back to the CDN to download at full speed if there weren't enough seeders.
Gotcha.
P2P-with-failover-to-CDN is what Spotify famously used to do.
i thought this was obvious to everybody: public clouds want to lock you in, so they make it very cheap and easy to get data in, but charge stupendous amounts for getting your data out.
There’s also engineering around the network involved in your application. A lot of architectures internally reflect traffic around (moreso if you’re also accepting some ridiculous amount of traffic rather than just serving it). In those cases you can save a lot of money by figuring out ways to pass traffic through unmetered connections like internal albs and amazon hosted dbs. Otherwise you can either rack up a lot of cross-az traffic charges or be on the hook for engineering your way to minimize those costs.
A previous site I built (https://hearthstonejson.com/) is pushing on the order of 100TB / month in images and JSON data, with 96% cached requests and 99.9% cloudflare-cached bandwidth. Almost nothing comes out of S3 itself.
Cloudfront is the ridiculously-priced offering IMO. I don't really understand why people use it.
This is probably customized per client and could be more for big companies.
No enterprise contract required. Just for video, obviously.
Even then, just a few terabytes of traffic will make Cloudfront reach the thousands.
For those of us not in the know, what else would you use it for?
"2.8 Limitation on Non-HTML Caching The Service is offered primarily as a platform to cache and serve web pages and websites. Unless explicitly included as a part of a Paid Service purchased by you, you agree to use the Service solely for the purpose of serving web pages as viewed through a web browser or other application and the Hypertext Markup Language (HTML) protocol or other equivalent technology. Use of the Service for the storage or caching of video (unless purchased separately as a Paid Service) or a disproportionate percentage of pictures, audio files, or other non-HTML content, is prohibited."
---
"disproportionate percentage" seems wide open
Companies that get their highly regulated product(health insurance, hospitals, government contractors) "certified" for use in the AWS ecosystem. Once they do that, they are likely to use it for everything, and pass the cost on to the customer.
But as far as I know, kissmanga.com is still abusing blogspot.com/tumblr.com by using them as free image cdns.
Cloudfront’s pricing can absolutely bite you if you are not careful. Which is one of the reasons why I prefer Cloudflare over it.
Of course we started spending all the money on new people and AWS, and soon there was no money.
At one point we were dumping like $15K a month on AWS for a dozen unnecessary over-engineered toys that nobody was using. This is the real cost of AWS.
I'd love to see Amazon's data on money invested vs actual user traffic for small startups, that's got to be some of the most interesting and valuable data on earth. Forget companies, I'll bet Jeff is sitting around predicting when entire industries rise and fall weeks before anyone else based just on this data.
I can now show exactly which departments and dev teams are driving all the costs, and on what (CPU, storage, network). In a way that I never could for on-prem stuff.
For some apps, perhaps a "component" or "service" tag would also be important.
The closest I got to an org that did this well was a big company that ran Cloud Custodian in all their AWS accounts and if you launched an EC2 instance, it would terminate it immediately with extreme prejudice if it didn't have values for three required tags, one to identify the "owner" individually and two for accounting purposes.
The only problem with that is there's no mechanism to make sure that the values of the cost centers values were correct. There was a bit of a scandal when one group (who presumably just copied and pasted a bunch of CloudFormation from another group's repo) was running 5 figures a months of infrastructure under the other group's billing codes.
ALSO, as many have said, bandwidth is a big part of the cost, and at this time it's nearly impossible to do showback/chargeback on bandwidth. There may be a way to do it using Flow Logs by correlating IP addresses to instances and using those tags, but I've never heard of someone doing this successfully.
In this case, a service tag, set in some cases, not in others.
Using AWS as an example, at one of the businesses I worked at, we used Kinesis as an event bus. One shard handles 2MB/sec of output. This worked pretty well for thousands of messages a second, we even got up to the 100's of thousands by compressing the payload of the event message. After that, you can employ any number of strategies that work easily, such as shading and adding additional streams, and use a lambda to pipe the output of one stream to another. It scaled up to millions of messages by essentially pushing buttons in the AWS Console.
Take a look at your architecture. More than likely, outside of a FB/Google/Netflix traffic scenario there is probably an easier and more straightforward architecture you can use that scales to your realistic use cases. Worry about the billions when you get there, which you yourself probably wont at that point because you would have exited, or moved into a higher role most likely by now.
Somebody mentioned 1:3.
Startup I was at burned 400k/month average sometimes when buying RI we spent 800k.
Had a couple engineers route the database connections over the public load balancers for a month, that cost 20k in network alone.
AWS is not cheap at scale, period.
I have many stories about this startup. The idea is great. If the startup succeeds it will be in spite of how its being managed technically
Problems come from many areas... like the fact it was in the data management space and the 1500 or so i3's running Cassandra plus the hybrid cloud approach of front end in AWS talking to APIs in GKE which talked to backends in AWS because cool technologies you know.
No architecting was done. Build this and put it in the cloud. The team from RU didn't even know about autoscaling groups and when I tried to bring them in I ran into resistance.
I have many examples of "opportunities to optimize" from this place, needles to say my stress level is much lower not being there.
Would there be any "conflict of interest" issues is some of the same people requiring AWS spending were also Amazon shareholders?
If you think anyone cares about any conflict of interest among the investing class you're beyond naive, you're just delusional.
- Let's not hire for this idea because it eats into our burn
- Hmm, let's hold off on launching this feature until we have more data
If the company you worked for was profitable, then they could've structured a leveraged debt payoff to the investors to get them off the cap-table. Unless the company took so much money that the investors owned 60%+ and they unanimously do not agree about being profitable, then this is something that can be passed as a board resolution.
It sounds like the founders at your company were just inexperienced.
The real, distinct cost of a vCPU is probably closer to $25/mo. You can look at a m5.large instance, their "general purpose" instance, 2vCPU+8gb @ $70/mo, which would put 1vCPU+4gb at $35/mo. Google Cloud specifically lists the distinct price of a vCPU in custom instances as $24/mo, and AWS feels close to that when you consider the cost of memory in that $35/mo.
The lack of networking cost also seems like a big oversight; if I could force every engineer to know a single "AWS Cost" that affects every vertical of development, from frontend to backend, I want them to know how crazy expensive DC egress is.
I just moved https://vlang.io from Google Cloud (my free credit expired) to an Amazon Lightsail VPS (any VPS will do), and my spending went from ~$70 to $3.5/month.
And the performance actually improved a bit.
There's certainly a huge range of pricing for the exact same services (sometimes worse services cough OVH dedi servers cough)
A properly designed monolith on a dedicated server/VM can easily serve tens of thousands of users just fine and will be far easier to maintain than this mess (plus way cheaper as well).
The main advantage of those tools is when you have a number of distributed teams and you want to scale to millions of users. For nearly anything else, a far smaller setup is what you want.
They know, and try to educate.
The crazy thing is, even used inefficiently, the Cloud is still a very good value prop for a lot of businesses.
AWS creates work, and creates spend, much more than it needs to.
This is all by design.
I can only imagine how the Oracle cloud must go for victims^Wcustomers.
Systems engineering is hard, on-prem or in-cloud.
What is there to say? There are huge scale inefficiencies in maintaining a small datacenter.
At $dayjob we have a small DC (8 racks). We’re facing chiller, UPS, and core switching/cabling replacements all within the next two years.
We did the math, public Cloud is cheaper for us, especially since about 80% of our instances can be shut down outside business hours.
Agreed. I've spoken with a few companies that could save quite a bit of money by buying RIs with a few clicks. With a little bit of work they could save a lot more moving large parts of their workload to spot instances. Almost feels like they want to brag about the AWS bill size...
possibly because that's what most of the job postings are requiring?
Still, they all get a load balance configuration to grab the free TLS certs with auto renewal while keeping configuration simple.
While I would not recommend premature TLS termination for critical business data, it is sufficient for many applications.
Admins get their familiar AWS console to configure the apps and I can make sure these are bundled in a way to be deployed on many different systems.
If I wanted to, I could use AWS identity provider like cognito and have it all in one place.
I think the main argument for cloud provider is the utility and convenience they provide for small projects.
While I do not like the business practices of Amazon and would never like to work for them on AWS, I am also lazy and cannot say I that they didn't develop something very useful.
Generally companies don't really care if a solution is 70$ or 3,50$ a month. They do care about my wage much more. Unjustifiable!
Sure, you could dump it all on a server somewhere without using any external micro services. But in my experience these project are much more likely to start to randomly haunt you from the past. And worse, I so actually have to maintain the servers.
And the price difference for inefficiently used services is mostly between 5$ and 15$. But these extra 10$ a month can easily be set off for reduced development or administration time.
If after ~40 years the service is still active, I would freely admit to that mistake.
Granted, this isn't applicable for application that do actually need to be scaled. But the time to launch for standard solutions that nudge a few bits here and there can be reduced quite a bit thanks to infrastructure like this. And it may be in use anyway since all the Google Home/Alexa IOT stuff.
Why is the cost and size of snapshots so opaque? Is there a way to see how big each one is, and how much it will cost to keep it around for a month?
I understand that snapshots are differential, and deleting one snapshot in a series may move blocks to other undeleted snapshots, of course.
I just want to know how much storage each snapshot requires right now, and to be able to accurately predict how much each one of them will cost at the end of the month, so I can decide how often to make snapshots and how long to keep them.
And is there an easy "serverless" way to automatically archive a snapshot in glacier storage or simply download it, without manually making it into a new volume and using "dd" or "tar" or "dump" or whatever?
I understand they're stored in S3. So why can't I see a list of them in the S3 interface, see how big they all are, and tell how much each of them is going to cost at the end of the month? It's my data. I'm paying for it. I think I deserve to be able to see it and know how much it will cost.
I get the feeling I'm missing something really obvious (probably a big blinking green button on the main page that my brain is filtering out). Surely many other customers must want these features?
Like the one that bit me recently was a $0.05/1000 cost per thing, which is easy to translate to $0.00005/thing and then mentally round to $0. That one added up to real money at 10^8 things, and would have been a big problem at 10^9. The cool thing about horizontal scaling is that doing something 10^6 times or 10^9 times is going to feel pretty similar when you do it -- the only difference is the order of magnitude of the bill.
Most SaaS services have very high markup, but get a few bits of infra wrong, over promise on a few things, and use expensive AWS services, and suddenly you’re selling $500 a month of AWS services for $99 a month. This is an easy mistake to make, I’ve seen it happen with these figures, and it only takes a few wrong assumptions across the engineering and sales teams.
We still use AWS for a few things and still have a small bill with them every month, but we're very careful about putting anything there that's going to push a lot of traffic.
In general I find the big problem with AWS is that cost is handled 'in reverse': developers often get near free reign, and cost only gets handled when someone balks at the size of the AWS bill. Often it turns out to be trivial to cut by changing instance types or renting a server somewhere to act as a cache. At that point people have often spent tens of thousands in unnecessary fees.
There's an underserved niche for people to do AWS cost optimization on a 'no-win no-fee' basis.
I used to help clients with cutting costs on AWS and if people were in doubt I'd often offer that.
And the savings were often staggering (e.g halving hosting costs was common; once we cut costs 90 percent by moving to Hetzner for a bandwidth intensive app even though long-term storage remained on S3).
The biggest challenge in serving that niche is getting people to realise they may be throwing money out the window as surprisingly many people still assume AWS is cheap, and offering to do an initial review for free and not charge if I couldn't achieve the promised savings made it a lot easier. Someone who likes the sales side more could make a killing doing that.
1 vCPU: $17/mo for sustained usage, $10.5/mo for 3yr commit, $5/mo for preemptible (similar to spot)
1 GB RAM: $2.2/mo for sustained usage, $1.4/mo for 3yr commit, $0.7/mo for preemptible
Google Cloud also lets you choose exactly how much CPU and RAM you want, within reasonable limits (you have to allocate 0.9GB RAM per vCPU and you pay a little more for RAM above 6.5GB per vCPU). So these numbers aren't medians/derived values for Google Cloud but numbers that you can find in the docs at https://cloud.google.com/compute/pricing. (I don't work for Google, I'm just a fan of Google Cloud)
In our team, only like two, maybe three programmers are concerned about our AWS architecture at all. I don't really know why everyone in the team should know how much the individual components of AWS cost. They should just know how to not write performance-destroying code that mandates scaling instances up.
[0]: https://people.eecs.berkeley.edu/~rcs/research/interactive_l...
- 10 reasons why AWS costs so much. #7 is mind-boggling.
- AWS costs considered harmful
- "Developer uses one weird trick to reduce cloud bills by 97%. AWS hates him!"
I do think it should be a reasonably common skill to make common tradeoffs without too much investigation.
Besides, the title is an homage to the Latency Numbers article which follows a similar concept. You don't need to memorise numbers, but you probably should be able to reason about them in very vague terms.
[1] https://people.freebsd.org/~lstewart/articles/cpumemory.pdf
These are things developers definitely should know.
I do agree that "EVERY programmer should know.." with regards to some specific software package or service is silly. Given that there are a million ways to provide services similar to AWS, and AWS is really only a means to an end -- I don't get why every programmer should be aware of the specifics.
Latency is one those things that hits almost all of CS. AWS isn't. "Things all programmers should know about AWS.." is , in my opinion, similar to saying something like "Things every programmer should know about the Python GIL." , it includes exactly the reason WHY every programmer needs not know it.
Not a amazon service user? Forget about AWS.
Not a python user? Who cares about the GIL.
I also agree that your linked PDF is overboard -- but it's one of those things that looks overboard now, but that kind of innate cpu/memory knowledge would have been a lot more useful when lower level languages were more widely used. Touching memory is just one of those things that seems to be on its' way out, and I, for one, am thankful for that.
...that's a terrible example, 2007 link nothwithstanding. Nobody cares about AWS unless they use it (and many don't, either because they use something else, or because they're in a different domain), but awareness of true costs of what you're _actually_ doing can be the difference between O(n) and O(n^2).
I must say though, the graphs in this could be a good bit better. There is no reason to use a line here, it makes it look like a time series plot. Each instance type should have its own bar and they should be ordered in some manner, either grouping by types or sorting by price.
Often understated is how difficult it is to set up your local gpu for machine learning. Still haven't figured out how to connect my 8 gpus in a functional way.
Of course, once you have something in production, it might be worth considering buying your own GPUs, but really is all depends.
Your GPU is not licensed in a way that allows AWS to use it.
RDS is super expensive and while you're gaining traction and users you might as well use an EC2 instance.
ElasticBeanstalk seemed like an easy intro to AWS but it steered me into RDS. Of course I've abandoned the whole of ElasticBeanstalk for serverless development lately.
My first intro to AWS was ElasticBeanstalk and serverless seemed too daunting. But now I've been smitten and can't stop thinking serverless.
This is especially true once you factor in all the development overhead, which can ultimately be very expensive to do things correctly with serverless.
Where can I find this? I assume somewhere in the 1060 repos of https://github.com/Azure
I also assume that even though it's open source, it requires licensed MS Windows servers to run.
Still though, this is just another nail in the coffin of the OLD MS image.
https://github.com/Azure/azure-functions-host
Under Satya Nadella the change in Microsoft has been little short of astonishing. A few years ago, I could understand the continued cynicism - but if anything the pace of change has only intensified.
Im not sure this is a good thing, having a generation of developers dependent on a single companies tools seems like the future will be painful.
They are two different problems. One is basically the problem of copy-paste/stackoverflow developers, the other is a walled garden problem that'll make itself worse over time.
The people who learn on AWS and have an actual desire to continue their knowledge will always find ways to continue their knowledge. Learning on AWS first is not going to stop that.
I’m planning on digital ocean because of this: their costs are straightforward and their interface is simple.
Edit: can anyone recommend an easy way to create different-provider backups of a DO server?
Besides, you can never go wrong politically by saying I chose the vendor in the upper right corner of Gartner’s Magic Quadrant.
The technical reason is that if I want my hosting service to do more of the “undifferentiated heavy lifting” and provide managed services. I won’t get as much of an offering from Linode/DO. The last reason is that AWS Business Support is excellent. They’ve helped me out with some head scratchers and when I just didn’t want to figure something complex out myself. Their live chat is awesome.