Hacker News Reader

Comment by jasode | Hacker News Reader

>petabyte scale are doing it on AWS. I would think that by then you could save millions more by migrating to your own colocated hardware.

Usually, those types of judgements are based on thinking of AWS as a "dumb datacenter" such as a bunch of harddrives or just bare cpu.

AWS is more cost-effective if you use high-level AWS services instead of just storing files in the cloud. In this case, it looks like Heap is also using AWS Redshift and probably a bunch of other services in the AWS portfolio. A similar comment I made previously: https://news.ycombinator.com/item?id=28288352

So for self-hosting hardware, Heap would not only build up the petabytes of diskspace, they also have to replicate Redshift functionality and the entire AWS services portfolio they're using. If you use enough AWS _services_, it becomes cheaper than self-hosting because you don't have to reinvent the wheel.

View on HN

My main take aways from this are that cloud vendor lock-ins are real, and they can be hard to break free from.

Perhaps that's more of a cautionary tale for new projects than a justification for the expense though.

What you're calling "vendor lock-ins" I'm calling "providing sufficient value to justify cost."

It's not that migrating out isn't possible, it's that Amazon is providing "Engineering/SiteOps Departments as a Service" at a price that's hard to compete with in house.

What's the newspeak for the high egress fees?

I'm not sure that's what I was arguing against.

From the GP's post - I was trying to say I read this as "the collection of services AWS provides would require several in-house engineering teams to compete with" not "vendor lock-in".

A single service is relatively easy to replicate if it is core to your business, but an entire on-demand datacenter w/ abstractions like Time Series databases, pub/sub services, etc. isn't as trivial to do yourself. Many of these services require teams of engineers to manage at scale, and engineers that _understand_ them well.

The on-demand service catalog of the cloud provides significant value. It's more than "pub/sub" or "blob storage" as a service. It's an entire engineering organization and data center as a service w/ pre-built architectures for you to start using today.

I am not sure the size of your company and the budget for in-house prices, but we realized AWS is not just a lock-in, but also a permanent money drain.

>Perhaps that's more of a cautionary tale for new projects than a justification for the expense though.

You can find case studies for both positions:

- migrate to AWS to save money: Netflix, Guardian newspaper [1]

- migrate away from AWS to save money: E.g. Dropbox [2]

A lot of companies (especially non-tech businesses) don't have the technical skills to run internal datacenters at the same competency as AWS. Thus, they don't want to be "locked in" to their own IT department that's slow and handicaps their business.

Dropbox, Facebook, and Walmart would among the very few that can competently run their own datacenters with advanced services like AWS.

[1] https://web.archive.org/web/20160319022029/https://www.compu...

[2] https://www.google.com/search?q=dropbox+migrates+off+aws+sav...

And then Dropbox shifted kinda back again, at least partially [0], it’s interesting to see the ebb and flow :)

[0]: https://aws.amazon.com/solutions/case-studies/dropbox-s3/

Wait. Why? How? Their in-house system (Magic Pocket / Diskotech) seemed so promising.

https://dropbox.tech/tag-results.magic-pocket

They're for different use cases. Magic Pocket is for storing file block data, and according to the AWS article, they just moved their analytics data to AWS.

>Their in-house system (Magic Pocket / Diskotech) seemed so promising.

The story described Dropbox moving "34 PB of analytics data (Hadoop)" to AWS.

My reading of Dropbox's Magic Pocket / Diskotech appears to be storage for customer raw data -- similar to BackBlaze type of raw storage.

It's 2 different use cases so it's not surprising Dropbox found AWS to be effective for analytics workloads. AWS has an extensive portfolio of software services to analyze data so Dropbox may have concluded paying AWS would cost less than reinventing the analytics pipeline in-house.

> AWS is more cost-effective if you use high-level AWS services instead of just storing files in the cloud.

Mostly this only works when your utilization is low(ish). Once you have high load 24x7, the AWS profit margin will quickly overtake the self-hosted solution.

With spiky utilization, you're buying and powering a lot of hardware to sit idle a good portion of the time.

Exactly, that is why if utilization is particularly spiky or unpredictable, that's a perfect use case for AWS (or similar).

Reply on news.ycombinator.com