Hacker News Reader

Comment by ahachete | Hacker News Reader

My bet is that it is built on top of Aurora PostgreSQL. By looking at the "Limits" section (https://docs.aws.amazon.com/documentdb/latest/developerguide...), identifiers are limited to 63 characters and the same characters that PostgreSQL limits identifiers to; and a collection size limit of 32TB, coincidentally maximum PostgreSQL table size.

Edit: I can confirm: does not allow the UTF-8 null character in strings: https://docs.aws.amazon.com/documentdb/latest/developerguide... ... It is written on top of PostgreSQL.

View on HN

It sounds like it is built on top of the Aurora storage subsystem that is used by both Aurora MySQl and Aurora Postgres[1].

I kinda expected them to build it on top of DynamoDb's backend and provide the same kind of "Serverless" on demand experience, but I guess the architecture didn't fit, or maybe this was just faster.

1. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

Definitely because it was faster. Amazon's strategy is to launch new features ASAP and then rely on everyone having to be on-call to fix shit when it inevitably breaks in prod because they rushed to launch. I will admit that while their "operational excellence" is shit, the security engineers do have quite a bit of power to block launches so their security isn't as bad as the reliability.

However, the fact that writes aren't horizontally scalable makes it a laughable nosql database but it probably satisfies the checkmark for enough of their enterprise customers that it will be a mild success and they'll keep it on life support like simpledb forever until they implement a proper solution assuming there is enough demand for it.

I was there for the launch of a major AWS service where they had an entire separate team working on the next iteration since well before launch (because the initial design wasn’t even intended to be sustainable). They are happy to incur technical risk (and in this case, to eat major losses in hardware costs) in order to be first to market.

We used MongoDB in my last job and I just want to say that I would have given up management of that beast in a heartbeat. We didn't stress MongoDB nearly enough to warrant all the effort required to construct it, monitor it, back it up, etc. Even if the performance was crappy, I would have lobbied hard to change to DocumentDB ASAP.

While I don't love MongoDB, I don't find it to be especially difficult to run. I'm running ~200 instances of MongoDB with a small team and it consumes very little of my attention.

ElasticSearch on the other hand...

Why is Elastic so difficult to run for you?

I'm sure they are, so they a pass those lovely negative externalities onto the customer because they know it's in demand and only they provide that service.

If only they had a competitor that could launch the same products a few months later but offered higher reliability off the bat, that could eventually force Amazon to improve their reliability or risk losing customers long term.

Being first to market doesn't ensure eventual market dominance. Sure, it could give you important feedback. But if your product is subpar, the feedback will have a ton of noise and possibly be useless. Plus it's not worth creating negative externalities and earning the reputation.

You think AWS has a reliability problem for their database products? That's news to me. AWS often launches products with limited features, but security, durability and reliability tend to be the standard.

Reliability is the trickiest of the three because it requires the customer to architect their solution with multi-AZ support in mind, but AWS always provides the foundation for that architecture.

Could they, and should they provide more features and a better developer experience around building fault tolerant solutions? Absolutely! But I certainly don't think they have a bad reputation for reliability.

From my perspective, performance and scaling issues are most likely to occur.

> If only they had a competitor that could launch the same products a few months later but offered higher reliability off the bat

Doesn't Azure Cosmos DB do this? From https://docs.microsoft.com/en-us/azure/cosmos-db/introductio...

> You can elastically scale throughput and storage, and take advantage of fast, single-digit-millisecond data access using your favorite API among SQL, MongoDB, Cassandra, Tables, or Gremlin.

Haven't used it though, so would welcome some real world experience.

> If only they had a competitor that could launch the same products a few months later but offered higher reliability off the bat, that could eventually force Amazon to improve their reliability or risk losing customers long term.

They have, it's Azure. I'm even a little bit scared because no one here is mentioning CosmosDB... It seems to me that most of the community only knows aws products.

For how many customers do all of AWS flaws combined represent more than 2% of their production outages? I think it’s a very small number.

Well, they are second to market this time around, Cosmos has had mongo api compatibility for a long time.

It definitely sounds like it sucks from the perspective of an internal AWS developer or SRE, but if the AWS systems are architected such that these internal failures aren't seen by end users then AWS's reliability reputation remains fully intact.

Customers are paying AWS so that their SREs don't get called, they don't care if the AWS SREs do as long as the system keeps running.

Based on the supporting quotes at launch from Capital One, Dow Jones and WaPo it sound like enough customers are ok with vertical write scalability and (pretty awesome) horizontal read scalability for now because it fits their use case and is better than what they had before.

Also consider that since the cluster management overhead has been removed from the customer, they can essentially "shard" by using a separate cluster for each sufficiently large service/org/dept, which might actually work out better for them in some respects.

Perfect is the enemy of good enough, the architecture might be laughable to you, but it is probably miles ahead of what the customer was using before.

I suspect that most MongoDB users never get to the point where they need to horizontally scale (i.e. it gets chosen for fad reasons, not because they actually have something big enough to scale).

And the nice thing about this hypothesis, you can test it by looking how successful DocumentDB will turn out to be. ~

AWS prioritizes launch above EVERYTHING. It is their strategy, to have market tells them what to build.

I think it works, and AWS has yet been brought down by this horizontal complexity. Quite an achievement, but might not be a satisfying experience for the engineers work there.

It makes sense in terms of feeling out the market as well. If this version of the service takes off it validates the decision to proceed with a more complex/scalable version and it gives them more customer feedback. Standard MVP best practices.

The downside is that a lot of their products lack polish which sucks. On the flip side even when they are launched with minimal features, they do tend to be reliable, durable and secure, which is important when it comes to data related services.

This is one of the main reasons why I don't like AWS services, everything just seems so half-finished. There's not a lot in AWS that I would trust enough to use in production.

I wonder how widespread this view is. I suspect it's more widespread than Amazon realise. They may have optimised into a local maximum where they get a lot of value from being first to market, but could potentially get more by being first to "viable to trust a business on".

I certainly agree that they seem half finished in terms of features and developer experience, but from the point of view of security and data durability they have an excellent reputation. They typically have a pretty good reliability story as well, but it relies on the customer architecture their solution to take advantage of multiple AZs/Regions, which is often not trivial.

As far as being "viable to trust a business on" the numbers don't lie, AWS is number one because customers are running their businesses on AWS. The fact that DocumentDB launched with supporting quotes from Capital One, Dow Jones and WaPo shows that customers were clamoring to use it even before GA.

Remember a lot of these customers are coming to AWS because they tried doing themselves and stuggled. When it comes to data, customers trust AWS more than they trust themselves, and rightly so.

What's your definition of production ready? AWS services when launched "half-finished" still do not have outages, data lost or security issues. They also come with metrics and enough monitoring to support them in production. Those are the major checkboxes for production ready.

AWS also has not had a reputation for deprecating services it launches. I find very little risk in taking a dependency on something AWS releases.

You mean if they used a different strategy, they might have more than the entire one third market share of the entire cloud hosting industry?

>> "viable to trust a business on"

They already are viable and trusted by multiple billion-dollar companies and governments.

Apparently SimpleDB is still used quite a lot internally. As for their market tactics, there's no denying it works as their pace is accelerating and leaving everyone else in the dust. Most customers just want to pay some money and have a solution ready to go, they don't need infinite scaling from day 1, if ever.

This focus on actually meeting needs today is what keeps AWS on top while the others take 2 years to launch minor service upgrades.

MongoDB is not horizontally scalable either is it?

Yes it is: https://docs.mongodb.com/manual/sharding/

DynamoDB is written on top of MySQL (more specifically, MySQL's storage engine, not the query engine) so using Aurora which has a newer design would make sense.

Saying DynamoDB is built on top of InnoDB is a pretty big oversimplification of a much more complex distributed system[1] and for all we know they could have switched out the low level the storage engine on the backend to something like RocksDB or WiredTiger.

The Aurora storage subsystem is much more limited in terms of horizontal scalability and performance, they probably chose it because it was a better/quicker fit.

1. https://youtu.be/yvBR71D0nAQ

Yeah, I used to work on DynamoDB, I know it's more complicated (much more complicated than that video makes out - their code quality was atrocious, like 2000-5000 line Java classes in 3 or 4 deep inheritance hierarchies; no unit tests, only "smoke tests" that took 2 hours to run and were so prone to race conditions that common advice was to close everything else on your machine, run them, then leave them alone while you went to meetings)

There was work underway at the time I left to replace InnoDB with WiredTiger. It seemed to be very slow going, and I suspect WiredTiger being acquired by 10gen had a part in it. They also had only 1-2 engineers on the project of ripping out MySQL and replacing it, in a long-lived branch that constantly dealt with merge conflicts from more active feature development happening on mainline.

Aurora, simply by virtue of being newer and learning from DDB's mistakes (in the same way DDB learned from SimpleDB and the original Dynamo) probably has better extension points for supporting (MySQL, Postgres, Mongo) in a sane way.

Interesting, how long ago was that? I would be curious to know if the WiredTiger switch ever happened, and what that support relationship looks like not given the contentious relationship between MongoDB and AWS. The old Wired Tiger Inc website[1] still lists AWS as a customer.

Then again, the relationship between AWS and Oracle is even more contentious and Aurora MySQL is one of AWS's most popular products so I don't think they are terribly worried about building on competitor's technologies.

1. http://www.wiredtiger.com/

3+ years ago, so it's entirely possible that things have changed since I left. I don't have any more recent information on the state of the system.

At least when I was there, the strong focus was always on adding new features (global & local secondary indexes, change streams, cross-region replication, and so on) to keep up with the Joneses (MongoDB et al).

Meanwhile, a bunch of internal Amazon teams were taking a dependency on it instead of being their own DBAs, and those teams didn't care that much about the whiz-bang features, they just wanted a reliable scale-out datastore that someone else would get paged about when some component failed.

Adding features at a breakneck pace while keeping up umpteen-nines reliability and handful-of-milliseconds performance meant tech debt and non-user-facing improvements, including WiredTiger, all got sidelined. Around the time I left, our page load was around 200 per week. That's one page every 50 minutes, 24/7, if you're keeping score at home.

According to this post [1] the WiredTiger project seems to have been cancelled after the acquisition.

https://news.ycombinator.com/item?id=13170746#13173927

Given the scale and popularity of DynamoDB and the distributed nature you would think that they could hire multiple teams just to work on improving it, but I guess it isn't as simple as that.

I would love to get a behind the scenes look at the process of gradually improving the components of DynamoDB with better technologies, while still maintaining reliability and performance.

People downvoting one of the guys who worked on DynamoDB at Amazon, somehow thinking they know better. HN in a nutshell.

You have been downvoted.

It would be nice if Amazon provided an API to access the data via SQL alongside the MongoDB API; I've seen quite a number of organizations migrate from mongo to Postgres once they get out of the rapid development phase. This would make that transition butter smooth.

That would make the internal representation used an "API" and thus won't be able to change it in the future.

Apparently, they are using a 1:1 mapping between a collection and a table. Either by flattening the document or by using jsonb or equivalent. I'm not a big believer this is good for performance reasons, at least compared to a more normalized approach like the one we did for https://www.torodb.com But they may change it in the future --if they don't expose the SQL API to their internal representation.

Anecdote:

I led a C# project where we could seamlessly switch back and forth between Mongo and SQL Server without changing the underlying LINQ expressions.

We sent the expressions to the Mongo driver and they got translated to MongoQuery we sent the expressions to Entity Framework and they got translated to Sql Server.

C# is ahead of the game with LINQ, expression syntax, and the entire Rosyln platform. Passing an IQueryable<> around that can be interpreted and transformed for multiple backends is a incredibly productive. I wish more people knew about this, and .NET in general.

And I’ve seen a few Java and Javascript libraries that purport to “implement LINQ” and they don’t get the power of LINQ is not the syntax, it’s that LINQ translates your code to expressions that can be parsed and translated to any back end query - it’s not just an ORM.

I’ve seen a LINQ to REST API provider.

I doubt that they actually built this on top of Postgres. They probably just integrated the WiredTiger[1] storage engine used by Mongo with their Aurora storage subsystem.

I am however really hoping Amazon provides a MySQL 8.0 compatible version of Aurora with full support for its new hybrid SQL and Document Store interfaces[2] courtesy of the X DevAPI[3] and lightweight "serverless" friendly connections courtesy of the new X Protocol.

That way your don't have to choose just one approach, and you can have your data in one place with high reliability and durability.

My ultimate pipe dream would be that they also provided a redis compatible key/value interface that allows you to fetch simple values directly from the underlying innodb storage engine without going thru the SQL layer, similar to how the memcached plugin currently works[4]

1. https://github.com/wiredtiger/wiredtiger

2. https://mysqlserverteam.com/mysql-8-0-announcing-ga-of-the-m...

3. https://dev.mysql.com/doc/x-devapi-userguide/en/devapi-users...

4. https://dev.mysql.com/doc/refman/8.0/en/innodb-memcached.htm...

What's the motivation for a faster access path to InnoDB: performance?

X DevAPI and X Protocol/X Plugin could team up and map K/V style access to the server internal InnoDB API instead of using a SQL service as it is currently done. They could try to do it "transparently" or let you set hints. Whatever is desired from an application standpoint.

> I doubt that they actually built this on top of Postgres.

Maybe not (but OP makes a lot of good points for why it is), but it is still based on the aurora limits, 64TB of size, 15 low latency read replicas in minutes, and presumably 1 write capacity which makes it a laughable nosql system since it cannot scale past 1 servers write capacity.

Are you aware that they are working on multi-master for Aurora? https://aws.amazon.com/about-aws/whats-new/2017/11/sign-up-f...

And there are organizations who can do rapid development in Postgres.

I think they're built on a common storage system, just like the MySQL compatible version too.

Aurora Postgres isn't really Postgres(only compatible), or is it?

The storage engine is different, but the frontend is actual Postgres

Interesting- Reminds me, I wish Postgres would increase this default identifier limit to 255 - or make it easily user configurable. It can be done by a sophisticated user, but only via special compilation and only when first installed, which is a right pain. I find Long identifier names useful for constraint names and foreign key names auto generated by code.

Corollary: PostgreSQL is also web-scale! ;P

Wasn’t latest Mongo built on Postgres backend too?

I think your thinking of the BI Connector for analytics/SQL compatibility.

From the docs:

Changed in version 2.0: Version 2.0 of the MongoDB Connector for BI introduces a new architecture that replaces the previous PostgreSQL foreign data wrapper with the new mongosqld.

No, MongoDB has its own storage engines, it's not built on top of anything else.

Reply on news.ycombinator.com