Hacker News Reader

Comment by rbranson | Hacker News Reader

My guess is that this is on-demand DynamoDB wrapped with a Cassandra-compatible frontend. Just based on the pricing and stated characteristics. It would be really hard to provide on-demand capacity for compute and storage using anything remotely similar to off-the-shelf Cassandra. The pricing looks like DynamoDB on-demand + a bit extra to cover the cost of operating the frontend.

EDIT: confirmed by an AWS employee here https://twitter.com/_msw_/status/1201924979647905792

View on HN

Disclosure: I work for AWS

Off-the-shelf Cassandra has a fair amount of flexibility in its architecture. Open Source Cassandra is the thing that is in the front end. And the team is excited about the idea of collaborating with the development community around some of the generally useful abstractions needed to build this managed database experience. More at https://aws.amazon.com/blogs/opensource/contributing-cassand...

If you don't mind, can you please elaborate about the backend too.

Has Amazon ever contributed anything meaningful back to open source?

Disclosure: I work at AWS.

Here are some examples from my team’s 2019 work: We contributed numerous changes to containerd. We open sourced firecracker-containerd, and we also created a Go SDK that others are using to work with Firecracker. We contributed to Debian and the Debian kernel team. We contributed to Envoy. We collaborated with a number of communities, including Kata Containers, Red Hat’s Clair, and the Open Container Initiative. All of these examples are sustained investments, not one offs.

Sure, it's contributed where it's convenient. I still don't see a list of committers so that I can look at what those contributions were.

Quite a lot: https://aws.amazon.com/opensource/

Let me give you an example.

"Amazon EMR has been adding Spark runtime improvements since EMR 5.24, and discussed them in Optimizing Spark Performance. EMR 5.28 features several new improvements."

Have these improvements been contributed back to Spark? When I take a look at the improvements themselves, it looks like all Amazon did was upgrade Spark from 2.3 to 2.4.

This seems completely random. EMR isn't an open-source project, it's a proprietary offering.

The page I linked lists plenty of projects if you're looking for actual OSS work.

EMR isn't open source but Spark is. What does the EMR Spark Runtime if not offer Spark as a service? And the changes to optimize spark runtime, why were they not contributed back to upstream Spark?

I'm sure some changes are upstreamed (assuming they're even accepted) but there's no requirement for them to do so.

Again this seems like a random example. What is so important about this particular change over all the other open-source contributions?

This is just an example. I'm sure there are many others. The developers that take the time to contribute to Spark are making Spark a better product. Amazon is not making it better. Amazon should not claim they made improvements to Spark in a newer version. What they did was upgrade Spark to 2.4 and claim that the improvements were done by them whereas in reality they were done by the community.

Again this is all wild assumption. They're not required to contribute to free software, but they do upstream changes.

The improvements they're claiming is for their own EMR product, not to Spark, and they do make updates to software to run better on their own own infrastructure. That's what their customers care about.

You seem dedicated to believing that Amazon has done nothing for open-source even I pointed you to an exhaustive list of their contributions. At this point, there's nothing more to say.

Do you have a list of open source committees accessible somewhere?

What do you mean by committees?

Sorry, I meant committers. I'm asking since the link you posted talks about open source events and has some job postings. I was looking for actual contributions by Amazon employees.

There's far more listed there including entire projects, repos, videos and several blog posts detailing their contributions, including one describing what they're doing for Cassandra: https://aws.amazon.com/blogs/opensource/contributing-cassand...

Amazon, Microsoft, Google, Oracle, IBM and others are all major contributors to open-source software.

That's a remarkable level of openness for Amazon. When Kinesis launched we were explicitly told we couldn't say anything publicly about the implementation (I don't know if the NDA has expired since).

They are way more open now. Before, it was a strong advantage of Google that you could hear from their engineers. AWS seems to have definitely pivoted on that since then.

This explanation does make a lot of sense. But why not just add a Cassandra interface to DynamoDB the way they added a Postgres interface to Aurora RDS?

They didn’t add a Postgres front end to RDS. They took the Postgres open source code and (grossly simplified) changed the storage layer and added more AWS specific features. Aurora/MySQL and Aurora/Postgres or more or less AWS specific forks of the respective projects.

Aurora RDS is a meta product name though, you can't use it without specifying the engine, MySQL or PostgreSQL.

What you are saying would be more like adding a postgres frontend to mysql.

I think what the tweet is saying is the engine is actually Cassandra but the node management is shared with DynamoDB?

Reply on news.ycombinator.com