Running Fast SQL on DynamoDB Tables | Hacker News Reader

Running Fast SQL on DynamoDB Tables

rockset.com

AFAICT, this actually isn't running SQL against data in DynamoDB. Rather, it syncs the the data in DynamoDB with their [Rockset's] own proprietary database, and runs SQL against that data.

It's effectively the same but technically quite different.

Run SQL queries using Athena and a Dynamodb crawler in Glue. It's not that complicated.

Link to a tutorial that is as simple with code examples / screenshots as this one?

I ask because I have looked and haven't found one that I could follow without having to dedicate several hours to it.

To the point where I want to move off of dynamodb to aurora because I can't believe how hard it is to get simple insights like how many rows exist out of dynamo (I inherited the dynamo choice would not have used it myself)

I’ll have a blog post out in AWS Big Data in a few weeks but the parent is right in that it’s not terribly difficult.

Should take about an hour with testing to get a Pyspark script together to read in a DynamoDB table and write it out to S3.

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programm...

You’ll then need to crawl the S3 data to add it to your Glue catalog and then you can query it with Athena.

If you're running this regularly (e.g. once an hour) to dump tables the cost of Glue can really add up due a minimum runtime length.

We switched to a scheduled Fargate task to dump data from DynamoDB into S3 as parquet files. It's really reliable, costs us ~$4/month and completely configurable.

How are you getting data from DynamoDB to S3 so Athena can read it?

ETL job in AWS glue with DynamoDB as source and S3 as sink is one option.

https://aws.amazon.com/about-aws/whats-new/2018/07/aws-glue-...

You could have a lambda hook to trigger on dynamo write, and copy the dynamo change to s3 in the lambda.

Running (expensive) SQL on dynamodb tables.

Need to look at TCO.

Managing a database can be pretty expensive and time consuming.

Reply on news.ycombinator.com