Comment by actuator | Hacker News Reader

Comment by actuator | Hacker News Reader

Parent Full thread

He mentions DynamoDB for the data layer instead of regular RDBMS. I have tried to use DynamoDB several times but have been disappointed with it. I am not saying traditional RDBMSs solve the problem that DynamoDB has, but even DynamoDB is a poor fit for systems with rapidly changing load patterns. Some things that have been big pain points while using DynamoDB:

- It has autoscaling built in now but even that leads a lot to be desired, autoscaling is slow and your lambda functions will error out for significant amount of time(I have seen it happen upto 5 minutes) for 5X spikes until it scales up the IOPS. This fills up the queues quite fast and can lead to patterns where your spike increases because of failure retries. You can control this somewhat by limiting lambda invocations though. - Limited number of scale downs allowed. - Scale down doesn't completely reduce the throughput limits. I have seen situations where the write load went to 0 on certain times but scale down reduced it to something like 400 IOPS which can be a huge cost drain.

I am still looking for a DB which has done this well. Essentially what I want is compute layer separated from the storage layer in the database well enough so that both can be scaled independently and the compute layer can react to quick changes in load patterns. Does anyone here have any recommendations?

Similar experience with Dynamo here: it's very easy to get into a situation where it's unusable (rejecting requests) unless you pre-provision enough capacity. The act of scaling up capacity often takes 5-20 minutes, and then there are the limits on how often you can scale. The backup/restore options are very poor. There's no good way to clear a table besides deleting and recreating it, which can be a problem when there are stream listeners. The tooling around Dynamo, both first party and third party, is also pretty shabby.

Maybe I'm just using it wrong. I wish Dynamo came close to its apparent potential.

>Maybe I'm just using it wrong. I wish Dynamo came close to its apparent potential.

Maybe, but you might also be using it for the wrong thing. Any business logic that requires table scans (which are perfectly fine on RDBMS), is going to suck in any document database. The way you win at Dynamo is by making sure that one user operation is as close to one Dynamo call as you can make it. If your workload can't be designed to suit that paradigm, then Dynamo will only bring you pain and suffering.

That said though, migrations are terrible in Dynamo. Altering a table will often involve creating a new one, backfilling it, and then cutting over. Which you have to implement your own logic for. Also, think the GSI limit is 5? In practice it's actually 4, because if you want to recreate a GSI, you need to create a new one, and then drop the old one, so you have to keep one spare to accomodate that process.

Some time ago I read they revamped the provisioning/boosting algorithms, to get around some of the issues.

Not sure what do you mean about scaling storage layer separately from the compute layer. It's how it's usually done currently, with the disk being the storage layer.

As long as there are machines/containers to be started, there will always be some latency, though we can expect it to improve in the next years.

AWS now has the Aurora Serverless, though I don't have any real-life experience with it's latency yet (VPC requirement is a downer for now). https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

> Not sure what do you mean about scaling storage layer separately from the compute layer. It's how it's usually done currently, with the disk being the storage layer.

Yeah, I meant that only. DynamoDB and Aurora Serverless are examples where they are decoupled and can be scaled independently. So, I wanted to know about more products like them but ones that scale with load really well.

> AWS now has the Aurora Serverless, though I don't have any real-life experience with it's latency yet (VPC requirement is a downer for now).

I have tried Aurora Serverless and I like it but it also plagued by the provisioning problems for now. The lack of options to customize database parameters was kind of a downer. Also for some weird reason they only support MySQL in t type instances.

> (VPC requirement is a downer for now)

BTW, you have an use case where you need to use DB from outside VPC and you can't do peering?

> BTW, you have an use case where you need to use DB from outside VPC and you can't do peering?

Lambda in VPC have unacceptable cold-start for interactive applications. This is partly why DynamoDB use is so popular with Lambda, it does not have to be in VPC.

I don't understand what you're looking for here. A network file system? Every database decouples storage from "compute."

Yeah, but not all of them allow you to scale them independently without a downtime.

For example in traditional MySQL setups to scale up the compute capacity your server has to be unavailable for some time.

Google Cloud's BigTable is like this. I don't remember how long scaling up took but instead of telling it how many "IOPS" to provision, you tell BigTable how many compute nodes you want. Once you hit ~80% on a node, the latency would rise, until you added another compute node. Since storage/compute was decoupled you wouldn't have to "wait" for a rebalance either.

I think people way overestimate the business effects of maintenance windows.

Disagree completely. Services that you can mostly depend on to never go down, and scale as far as you will ever want, are a huge relief. Like S3.

We should ask for the same thing with rdbms's.

The overestimation is in whether a company should be using s3 as an operational standard.

If you take them once a quarter/year, they are fine. But if you have load patterns where you want to pay for what you use and not over provision the DB significantly then you can't rely on maintenance windows. I think we will get there eventually with even RDBMS getting solutions like Aurora Serverless but right now most of them don't work well.

Reply on news.ycombinator.com