Hacker News Reader

Comment by colllectorof | Hacker News Reader

You're not the only one.

Some time ago I worked with a bunch of people who staked their careers on moving everything to AWS and converting it to serverless architecture. Here is just one example of how fishy it was. They used Dynamo. During presentations to the outgroup, they constatly praised Dynamo as being flexible, scalable and fast. However, internally they constantly struggled with various limitations of the database, from limitations of indexing to not being able to get certain types of metadata.

My point is, the architecture they were designing was heavily bent around Dynamo's strength and weaknesses. Considering how extreme and peculiar those strength and weaknesses are, switching to another database would require heavy re-engineering of the rest of their architecture, much more so than, say, switching between Oracle, MS SQL and Postgress.

View on HN

Not sure I understand this. Why did they choose Dynamo? And what does serverless has to do with this?

This seems like a simple case of people choosing the wrong tool for the job, without really understanding its limitations or capabilities.

> say, switching between Oracle, MS SQL and Postgress.

That's because those are all relational DBs that use SQL. If they chose some other NoSQL DB they would likely have encountered similar "peculiar" strengths and weaknesses. And I'm still not sure what this has to do with serverless - they could've have easily implemented a serverless architecture with MySQL or Postgress on RDS instead of Dynamo. But they chose not to.

Aws might have fixed this by now with severless aurora, but as of earlier this year the issue was connection pools didn’t work on lambda functions due to their ephemeral nature not being able to keep the tcp sockets open for more than a few minutes before the functions froze or were terminated.

Connection pools always worked quite well with Lambdas, as long as the connection pool was initiated in the global scope of the running code, and not within the handler function. See [1] and [2]

[1] - https://docs.aws.amazon.com/lambda/latest/dg/running-lambda-...

[2] - https://blog.spotinst.com/2017/11/19/best-practices-serverle...

We use pools in lambdas to Amazon Aurora and it works out fine. Only limit we hit was when the db server was configured way too small.

Are you running your Lambdas inside the VPC (and incurring huge cold-start latency), or are you running your database exposed to the public internet? I've avoided using Lambda for anything that requires relational database connectivity because those appear to be the two options, and neither is viable for me.

The Data API was announced less than a month ago.

https://aws.amazon.com/about-aws/whats-new/2018/11/aurora-se...

> Why did they choose Dynamo? And what does serverless has to do with this?

Possibly because lambda is integrated with dynamodb. You can hook a function to an event like a row being inserted or modified: https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

That lets you skip a lot of explicit queue management code and the change+notify is atomic by default.

You can use lambda within an Aurora (AWS version of MySQL) trigger also.

The advantage of DynamoDB when using lambda is that it is API based instead of based on database connections and pooling and it doesn’t run inside of a VPC - meaning a network interface doesn’t need to be created inside your VPC every time a new lambda process comes online.

But now Aurora Serverless has the Data API that doesn’t use traditional database connections so there is even less of a need for DynamoDB for lambda performance reasons.

Sure, but that's simply a single point in favour of Dynamo in what should have been a comprehensive analysis of strengths and weaknesses with respect to the required use case.

If they chose DynamoDB simply due to this strength (ignoring for the moment that the same thing can be achieved with native functions in Aurora), while ignoring the other things that ultimately turned out to be more important, then this again points to a flawed analysis. It's hard to see how serverless is to blame for this.

> That's because those are all relational DBs that use SQL

I have dealt with all of those, including MySQL. Switching is never easy. Every technology choice, even ones that should be similar, like an RDBMS, have strengths and weaknesses.

If you are actually taking advantage of all of the features of sql server or Oracle, you’re not just going to switch overnight.

I’m no fan of DynamoDB and a multi certified AWS fan, but, I know it’s strengths and weaknesses and so should anyone else.

A product that actually uses all the features of Oracle would be naturally hilarious. Oracle's documentation is over 500 megabytes compressed. It's an alternative computing reality.

Here is a 5559 page book of error codes. Imagine having a physical copy of this on your desk.

https://docs.oracle.com/en/database/oracle/oracle-database/1...

Holy shit

This is exactly right, wow. I am almost speechless.

> alternative computing reality

This such the perfect description. I was working on a Django feature last year that had to work on all DBs and testing the oracle stuff was like entering a new reality lol.

And full of such amazing utility, such as:

" CLSRSC-00009: No value passed as OLR locations

Cause: No value was passed as OLR locations.

Action: None "

SQL Server and Oracle have different features and idiosyncrasies, but use the same fundamental models of data storage that is a product of decades of research and development and isn't controlled or promoted by a single company. You're not going to suddenly run into some fundamental limitation that will suddenly fuck you over on architectural level.

This happens all the time with databases like Dynamo. All of them feel amazing when you use them in the "intended" way, but they also have extreme limitations and those limitations are different for every product. Why? Because they aren't based on some fundamental data storage model, but rather on an aggregation of specific use cases, which are different across different products.

Here is a mild practical example: https://blog.codebarrel.io/why-we-switched-from-dynamodb-bac...

I’m very well aware of that. If they thought about using DynamoDB and they did even cursory research, they would know that it’s basically a glorified key value store with the chance of using multiple key value types - ie global and local indexes.

There are certain (limited) use cases where it is good. Every technology choice takes you down a certain road and limits your implementation choices in certain ways. It’s not the fault of the tool if they choose one that doesn’t meet their needs.

Also, the beauty of something like AWS, is that you can do “polyglot persistence”. You can have different services in your infrastructure use different types of storage depending on the use case.

Dynamo, like many NoSQL dbs, is (partition key, subkey, value). Partition key gets you to a storage node. Subkey gets you range queries on that storage node. That's the "fundamental data storage model" for most of NoSQL. And it's extremely useful for some use cases.

You could have mapped your use case to this storage model and gotten ridiculously fast queries on vast mountains of data, which is largely the point of NoSQL. But this would have required data duplication, home-made management tools for dealing with said data duplication, and other work that was clearly better spent implementing the much simpler SQL solution.

A benefit of the SQL approach is that if your queries start getting bogged down from growth in data size and/or request volume (not entirely unexpected behavior when you have a couple joins), you can move to a different model where you treat the SQL db as a slow-access source of truth and periodically generate key/value into NoSQL or Redis/ES/etc for actually running queries.

> If you are actually taking advantage of all of the features of sql server or Oracle

I doubt any real world application comes anywhere close to doing that. Heck, I’d be mildly surprised if any real world enterprise, considering all their apps together, does that for either SQL Server or Oracle.

This sounds more like an argument to not use a NoSQL database without a specific use case. That's an argument I completely agree with. I'm not sure it's a specific argument against Dynamo or AWS in general though.

This is any system that uses a database. It's just worse because only aws has Dynamo.

I think you’re underestimating the degrees of difference. There are many apps which support multiple SQL databases but far fewer support multiple NoSQL systems because the differences are more significant than an ORM can cover up.

Reply on news.ycombinator.com