Hacker News Reader

Comment by ta3411 | Hacker News Reader

This seems like an ideal use case for us. I have a naive thinking of my workflow: can someone please comment if I am off track.

I am building an e-commerce product on AWS PostgresSQL. Everyday, I want to be able to do analytics on order volume, new customers, etc. - For us to track internally: we fire client and backend events into Amplitude - For sellers to track: we directly query PostgressQL to export

Now with this, I am thinking of constantly streaming our SQL table to BigQuery. And any analysis can be done on top of this BigQuery instance across both internal tracking and external export.

Is RedShift the AWS equivalent of this?

View on HN

As a heavy BQ user on my side projects, there isn’t really an alternative to BQ in AWS. I find that RedShift does not provide a lot of the functionality and ease of use that BQ provides.

That said the closest thing is Amazon Athena.

The architecture would basically be Kinesis -> S3 <- Athena where S3 is your data lake or you can do it like AWS DMS -> S3 <- Athena.

To accomplish this or the redshift solution you need to implement change data capture from your relational DB, for that you can sue AWS Database Migration Service like this for redshift: https://aws.amazon.com/blogs/apn/change-data-capture-from-on...

Like this for kinesis: https://aws.amazon.com/blogs/big-data/stream-change-data-to-...

The reason you may want to use Kinesis is because you can use Flink in Kinesis Data Analytics just like you can use DataFlow in GCP to aggregate some metrics before dumping them into your data lake/warehouse.

BQ is saas proper vs redshift where you have to pick instance sizes etc. It’s amazing, true superpower stuff in how little you have to think about it to get loads out of it.

Redshift has serverless options.

Exactly this (using bigquery but AWS for everything else) is pretty common. It takes a while to build a service like this, AWS spent too long in the wrong direction (redshift) and haven’t been able to catch up.

basically yes. kinesis -> firehose -> s3

Reply on news.ycombinator.com