gdecandia

Committer here. I would love to hear more about this scenario.

Is the proposal to be able to export pg_dump formatted data on some schedule or trigger, entirely hosted in PostgreSQL and with timeouts? There are already extension that can export to blob/file storage and can be combined with pg_durable or pg_cron, so I assume the challenge is pg_dump compatible data export from SQL running in the database?

Please see https://learn.microsoft.com/en-us/azure/horizondb/ai/ai-pipe...

Long-running jobs on Postgres are not new at all. See pg_cron for one example. At the end of the day, these workloads would be running anyway against the database, whether triggered by an external component. HTTP queries from the database have also become more popular to avoid round-trips and failure points from additional components in data or AI pipelines. But yes, whether to bring the compute to the data or vice-versa is a design choice that has a lot of contention.

Contributor here. Good points, we do need to develop some best practices around managing function versioning and lifecycle for pg_durable.

https://github.com/microsoft/duroxide - also OSS, the durable execution framework pg_durable is built on itself supports function versions. We can leverage that to get similar support in pg_durable.

Contributor here. At Microsoft, our Postgres customers seem to split pretty evenly into 2 camps, those that want to do as much as they can in the database, and those that agree with your take - want to keep apps and compute outside the DB.

Contributor here - at Microsoft we've built AI workflows on pg_durable and seen it substantially reduce code and increase reliability. Agree that the DSL ergonomics can be improved. Our pipelines use a higher level language and therefore simplified, but pg_durable is meant to solve a wider array of problems. We're happy to take suggestions for improvements.

Frankly, it's been a while since I worked on Dynamo so I can't say for sure. I don't see a fundamental reason not to support null and zero-length. On the other hand it seems more like a convenience than a necessity, since you can encode those.

You may have different use-cases in mind, but it seems that if you're writing nulls you're using the key-value store to distinguish between keys that do or don't exist - basically storing a large set rather than a large map. And in that case you can write a small value to encode null and you won't change the value (rather you will delete the key-value entry).

I think this takes us to an API design discussion - I don't feel strongly about it. But I'd love to hear if that caused you significant burden or trouble in your application.

Hi Brandur, I'll answer your final question first, then circle back. CitusDB is definitely more akin to a traditional RDMS that to Dynamo because it keeps almost all the goodness of relational databases while providing the ability to scale horizontally via clever partitioning, usually on the tenant and some other primary key (for the multi-tenant, B2B use-cases).

In contrast, Dynamo sacrificed all sorts of guarantees that relational database clients had relied on for decades - partly justifying your comments.

I left Amazon in 2009 so I can't speak to how Dynamo is used internally today (and AFAIK AWS DynamoDB is based on a different design). However, while I was at Amazon the shopping cart data was migrated to Dynamo. That was definitely "core" to the business in the sense that it was a key part of the user's experience and we couldn't lose the contents of a single shopping cart - it's a bad user experience. And for that use-case we could change the per-cart schema on write (with additional scans to clean up carts that were written to old schema versions).

Hi Folks, this is Pino (formally Giuseppe) de Candia. I was the lead developer on Dynamo. I'm happy to answer any questions/comments.