Asynchronous API with DynamoDB Streams | Hacker News Reader

Asynchronous API with DynamoDB Streams

medium.com

29 pointskiyanwang9 comments

That's a really use case for DynamoDB Streams! Shameless plug: if you'd like to implement this pattern with other databases such as Postgres or MySQL, take a look at Debezium (debezium.io), which provides open-source change data capture support for a range of databases. Mostly, this is used with Apache Kafka, but you also can run Debezium embedded into your JVM application. Probably should do a blog post too showing how to integrate that with AWS Lambda :)

Disclaimer: I work on Debezium

Can you recommend a guide for embedding Debezium?

The reference documentation is a good starting point: https://debezium.io/documentation/reference/1.5/development/.... It may also be interesting to look into the implementation of Debezium Server (https://github.com/debezium/debezium/tree/master/debezium-se...), which is based on the embedded engine and allows to propagate change events to streaming platforms other than Kafka, such as Apache Pulsar, AWS Kinesis, GCP Cloud Pub/Sub, etc.

What are some general patterns for retreiving the data, client side, for async apis like this?

I.e. client makes a long request in browser, then gets back a response saying the job is logged. The task completes in the background and then what should happen?

I am thinking of JavaScript but more concerned about the patterns.

Something that work for me, have a websocket open to the client and notify him that it's done. Additionally register this notifications in the database, so that if the user had closed the browser he'll see the notification next time that he logs in

Supabase does something like this.

Yes, look at the Supabase Realtime API here:

https://supabase.io/docs/reference/javascript/subscribe

https://supabase.io/docs/reference/javascript/removesubscrip...

https://supabase.io/docs/reference/javascript/getsubscriptio...

one option to avoid client waiting around is to make client-side and server-side queues and maintain an open websocket, then send messages as needed, and have an optimistic switch statement which performs the correct state transition given a certain request and arguments (see useReducer or Redux, zustand, etc).

If the client is offline, then the user requests just build up in the queue, but the local state keeps updating so you can use your app. Then when the net is up, the queue of requests can be sent to the server to update the remote data store.

A big problem is inconsistency, for example if you're making local changes and somebody else makes a remote change to the same data. For that reason the community seems to look at CRDTs, which are just data-structures with explicit order, or where order doesn't matter, because then no matter how changes happen, we can merge them all into an 'eventually-consistent' thing.

IMHO this is a pretty huge amount of complexity to add to your app so if you can avoid it by doing immediate request/response processing it would simplify your stuff dramatically... but "offline first" is pretty cool for certain apps, and some jobs just take way too long. Queueing helps de-couple frontend from backend. That can be useful for zero-downtime database migrations and maintenance, if your client still works when the server is down...

Tools for this, are newer, and evolving rapidly, I haven't seen a whole lot which I consider super mature and impressive...anybody else find a good offline-first state framework?

Interesting! (Typo? Should be `if (error)`?)

Reply on news.ycombinator.com