Hacker News Reader

Comment by _msw_ | Hacker News Reader

Disclosure: I work for Amazon as a VP and Distinguished Engineer building our cloud infrastructure.

My professional opinion is that is a very poor strategy. Customers want to have freedom, not lock-in. When designing infrastructure services, it's good to carefully consider what abstractions are going to preserve freedom and flexibility. For example, adopting the Arm architecture, and further using the Arm developed Neoverse N1 core, provides the broadest capabilities and compatibility thanks to the investment in a common ecosystem.

The optimizations we are able to put in our silicon design, and the design of the Nitro system, let us deliver better price/performance without resorting to such customer unfriendly tactics of "lock-in".

View on HN

But that's what AWS offers.

All the big products- SQS, DynamoDB, Aurora, S3, Beanstalk. Those are all AWS services you can't use outside AWS (sure, there are API compatible alternatives out there). If you design for those services, you're locked into using AWS.

There is a reason bandwidth in is free while out costs money- it needs to be easy for people to move onto AWS without being east to move out. Customers are clearly choosing vendor lock-in over freedom. And AWS encourages this. If you want to migrate to AWS, Amazon will send an army of developers to help you adopt their technology. Will they do the same if a big customer wants to migrate away down the road?

https://e.lvme.me/kmw2hs1.jpg

As you point out, others implement compatible open source software versions of S3 and DynamoDB. Aurora already adopts (via adapting the open source software) the protocols and semantics of MySQL and PostgreSQL. Elastic Beanstalk helps you orchestrate the deployment of your application via a selection of supported "platforms" that are nearly all open source (except for .NET on Windows Server + IIS). In all of these, you could argue that SQS stands alone as something that doesn't have a "drop in" replacement. Developers that have a concern about that might want to adopt an open source alternative instead. There is nothing that prevents a developer from doing so.

In my experience, developers are adopting open source more than ever, and they also show an adept ability to know (as a crowd) what they should worry about, versus what is modular / abstract enough such that it could be replaced.

You're right. There are open source replacements for most AWS services, but that's like saying Wine under Linux is the same as Windows. For the most part, it works well enough to run most application but there are some that just don't work. PostgreSQL vs CockroachDB. Both speak PostgreSQL, but once you start using any advanced functionality you're going to run into issues.

If you're building on AWS S3 heavily, you're going to encounter features that might not be there in the "open source" version, things that behave differently, or just don't work as well. So while it's a nice thought to have, it's the same logic behind internet competition in the US for some areas. You can pick Comcast cable or DSL/WISP. They're the same, right?

Nothing prevents a developer from not adopting a technology, but we both know there are usually business decisions driving it. If you're already using S3 or DynamoDB heavily, you're already trapped. It's a cult. You're free to leave any time, but why not stay here with all your friends? If you leave you would have to spend millions in bandwidth moving your terabytes of data, hundreds of developer hours to update code, and knowledge building around new products.

You can always replace something with something else. But sometimes it's just not practical.

You're absolutely right with an API/ABI with the size and complexity of Win32 there is a risk to have compatibility challenges. Just like WSL v1 and gVisor, as re-implementations of Linux syscalls and subsystems.

To me, well designed web services are more intrinsically decoupled compared to general purpose operating system runtimes or SQL implementations.

As for when and why a group of people decide to exit various relationships, I've personally observed that sometimes cost and practicality is not evaluated completely by decision makers [1], despite various clear economic benefits of remaining.

[1] https://en.wikipedia.org/wiki/Brexit

> There are open source replacements for most AWS services, but that's like saying Wine under Linux is the same as Windows.

That kinda feels like moving the goalposts. If the alternatives aren't good enough, perhaps a company that cares about avoiding lock-in should fund them and help them get better. AWS isn't obligated to help competing products work better, or even to open-source the stack that is a big part of their competitive advantage.

If you don't want to get locked into SQS or DynamoDB or whatever, then don't use it. You can build a queue out of kafka or redis or whatever, and a DDB-workalike out of cassandra or something similar.

I'm just not really sure what it is you're objecting to. If someone adopts an AWS hosted service and then later complains that they can't move off of it, that's on them for not thinking it through at adoption time.

The only alternative here that sounds like it would make people happy is if AWS didn't build anything interesting at all, and instead only gave us EC2 on commodity hardware running Xen. If they did that, they'd fail as a business to other companies that actually build things customers want.

> "Those are all AWS services you can't use outside AWS"

What does that mean?

You can absolutely run an application hosted on Azure or GCP that uses DynamoDB or SQS as it's backing database, or queueing system?

These are just APIs on the public internet...

> If you design for those services, you're locked into using AWS.

Is that not the case with any software - including open source?

If you design for Active MQ, you're "locked in" to MQ interfaces. If you design for Kafka, you're "locked in" to Kafka interfaces.

Now, your argument about imbalanced data transfer rates MAY have aspects of lock-in - AWS does make it easier to onboard than offboard. But, that's also the case with any service provider, and I don't even mean software. Stock brokerages charge a fee for transfering portfolios out of their management, and the stock brokererage you transfer your stocks TO will usually cover those fees.

Data transfer isn't free, it makes sense that AWS would charge for it, and it would make sense that it would charge more for one direction.

Unless your expectation is that every software or SaaS company's API is open-source and interoperable with every other provider, I just don't see how you can avoid the kind of "lock-in" you describe.

> These are just APIs on the public internet...

I could run my load balance on GCP, send traffic to applications hosted in Azure, which use AWS services to talk to each other. Sure, that's something I _could_ do. but that's drastically more expensive then if I run everything inside of AWS.

They're not just APIs on the public internet. You also need to worry about latency and cost. Talking to/from an AWS service from outside AWS is more expensive from talking inside AWS.

> If you design for Active MQ, you're "locked in" to MQ interfaces.

Being locked into a paid product, vs a free product are two completely different things. I can take the Kafka code base and do whatever I want to it. It's an open system. I can't do the same with DynamoDB or SQS. You can't cut ties with AWS and still have your DyanmoDB dependent systems function, while I could cut all ties with Apache and still have a functional kafka cluster.

> But, that's also the case with any service provider

Yes. That doesn't make it a good thing. Having to pay to move stocks from one broker to another isn't a good thing.

> Unless your expectation is that every software or SaaS company's API is open-source and interoperable with every other provider, I just don't see how you can avoid the kind of "lock-in" you describe.

I'm not presenting a way to avoid it. I'm stating that AWS offers a bunch of vendor lock-in type products. They're like Oracle. Oracle DB is an amazing piece of software, but they know this and charge an arm, a leg, and your soul for it. The only difference between AWS and Oracle is that AWS havn't asked for your soul yet.

If your threat model includes having to address a concern of historically unpredictable or unreliable vendor behavior or performance (which makes sense, in a robust professional engineering practice), it can make sense to take matters in your own hands. Open Source is a great way to do that.

While past performance does not guarantee future results, there is some empirical evidence that you can use when evaluating which vendors you trust, and can rely on to be customer obsessed, including always looking for sustainable ways to save you money. You are, of course, entitled to an opinion about what "sustainable" means to you, and how you measure your choice of vendor. To me, it means specific things in how I professionally practice as an engineer.

[edit: small clarification, saving customers money is only one way to deliver value for their money. But, in my experience ]

> Sure, that's something I _could_ do. but that's drastically more expensive then if I run everything inside of AWS.

oh no! engineering tradeoffs!

Aside from S3, does anybody actually use those?

Software that works with S3 tends to work with competing APIs as well; just change a config option. Even if you’re interfacing with the API directly (you probably shouldn’t do that—use some sort of abstraction layer instead), there are open source alternatives that support the exact same API.

SQS can be useful for getting data out of AWS without worrying about whether your callbacks are down. For example, if you’re using SES, you might want to know when emails bounce; you can have all those events sent to SQS, then pull them out whenever and store them in something more generic. I don’t think it’s particularly common to use SQS beyond that.

DynamoDB—is that even still a thing? Isn’t Cassandra basically DynamoDB 2, but open source?

Aurora is expensive and makes exporting data a pain. I have no intention of using it until it’s easy to set up external replication so I can fail over to other providers (or leave). No external replication, no business, especially at that price point. Aurora was quite a disappointment.

Beanstalk is another one of those things that nobody actually uses. That being said, it doesn’t really lock you into anything; it’s just a way to automate some basic AWS tasks, really. But it’s generally more trouble than it’s worth.

I know where I work we're starting to use SQS, DynamoDB, and Aurora as we poorly expand into the cloud.

> Isn’t Cassandra basically DynamoDB 2, but open source?

Yet you need to deploy it somewhere and maintain it. DynamoDB just works and that’s why people will happily pay extra because they can put effort on something more meaningful that maintaining a database.

Apparently Amazon is offering a hosted version of Cassandra now too.

That isn't relevant to the parent's question. They were complaining about the lack of open source alternatives to DynamoDB. Cassandra qualifies.

"Aurora is expensive and makes exporting data a pain. I have no intention of using it until it’s easy to set up external replication so I can fail over to other providers (or leave). No external replication, no business, especially at that price point. Aurora was quite a disappointment."

Aurora MySQL supports raw access to the binlogs. Aurora Postres supports Activity Streams and Logical Replication. Both are well support by DMS, which can even support replicating and converting between different types of databases - it's designed to support moving into AWS, but nothing's stopping you to use it to replicate in the other direction.

RDS is worth every penny, in my opinion. Of all the technology I've used in my career, it's one of the most impressive.

RDS is fine, but every time I evaluate Aurora, I'm disappointed. As for raw binlog access, that isn't sufficient for external replication, is it?

Edit: Looks like it is now possible to enable external replication.

I hope I'm just being cynical, but I base my attitude on the general trend in the industry at least in any realm related to either cloud or mobile. Everything is about herding people into some kind of closed silo and then locking the door, or on the consumer front about invading peoples' privacy in new and creative ways.

I do agree in principle that it's harder to pull off this kind of lock-in with knowledgeable customers of the sort that deploy things on AWS.

I hope you are too. As technologists and professionals I think we need to be ever-vigilant to build in responsible ways that support the public good. That includes being mindful of safety, security, and privacy concerns. And also creating or extending inequities.

Reply on news.ycombinator.com