jgraettinger1

Our (Estuary; I'm CTO) streaming transformation product handles this quite well, actually:

https://docs.estuary.dev/concepts/derivations/

Fully managed, UI and config driven. Write SQLite programs using lambdas that are driven by your source events, in order to join data, do streaming transaction processing, and no doubt lots of other things we haven't thought of.

If one-way sync will do, we (Estuary) have been investing in a NetSuite connector that we've _just_ started testing with a few customers. Both SuiteAnalytics and SuiteQL with full support for custom tables & schema.

[1]: https://docs.estuary.dev/reference/Connectors/capture-connec...

They're not literally passing around the hash. Holders of hash(email) <=> browser cookie associations are heavily incentivized for both regulatory and also competitive reasons to not blast that information around the internet -- or even to let direct partners A & B identify overlaps without their being in the middle.

When passing identifiers, there's generally some combination of lookup tables, per-distribution salted hashes, or encryption happening to make reverse mapping as difficult as possible.

(I was in this space up until a few years ago).

Not having to evolve or understand or staff 4,000 micro services.

An ability to easily change the boundaries of your conceptual components, because they WILL be wrong now or in the future.

Hi fellow Dark Tower friend.

Yes, marketing and docs are not our strongest suits and we need to do better. To be fair, though, we're also trying not to scare off less technical users who see a bullet list like above and think "well this is clearly not for me". It's a hard balance

No. But this is neat, and at a glance it looks straight forward to add. Happy to discuss further!

No. We implemented our own [1] for a few reasons:

* Scaling well to multi-TB DBs without pinning the write-ahead log (potentially filling your DB's disk) while the backfill is happening. Instead, our connector constantly reads the WAL and works well in setups like Supabase that have very restrictive WAL sizes (1GB iirc).

* Incremental fault-tolerant backfills that can be stopped and resumed at will.

* Flowing TOAST columns through seamlessly to your materialized destination, without requiring that you resort to REPLICA IDENTITY FULL.

* Being able to offer "precise" captures which are logically consistent in terms of the sequence of create/update/delete events.

The last one becomes really interesting when paired with REPLICA IDENTITY FULL because you can feed the resulting before/after states into an incremental computation (perhaps differential dataflow) for streaming updates of a query.

Our work is based off of the Netflix DBLog paper, which we took and ran with.

[1] https://github.com/estuary/connectors/tree/main/source-postg...

Estuary (https://estuary.dev ; I'm CTO) gives you a real time data lake'd change log of all the changes happening in your database in your cloud storage -- complete with log sequence number, database time, and even before/after states if you use REPLICA IDENTITY FULL -- with no extra setup in your production DB.

By default, if you then go on to materialize your collections somewhere else (like Snowflake), you get synchronized tables that follow your source DB as they update.

But! You can also transform or materialize the complete history of your tables for auditing purposes from that same underlying data-lake, without going back to your source DB for another capture / WAL reader.

> they told me the new CDO was already convinced that said "RT-based" datalake was the way to go forward

Is the desire for "RT-based datalake" itself misplaced? Or just that the implementation isn't up to the job? Nobody _wants_ slow data, and reports that are usually fine with T+1 delay can become time critical (for example, a "what's selling?" report on black friday).

It does involve pinning the WAL while the backfill is happening, though, which is quite different from the incremental backfill achieved by DBLog.

The former can be faster, because you’re able to walk the table in physical storage order, but your DB cannot reclaim WAL segments while this happens (disk can fill).

The latter allows for resumable backfills that can be tuned to the available spare capacity of the source DB, and doesn’t cause WAL segments to back up, but must walk tables in logical order which can sometimes be slower (for example, if you key using a random UUID).

A uuid key would crush performance vs rowid, though, especially for ordered scans, and you’d still have conflicts if parallel r/w accesses are temporally correlated (you wrote X and Y at the ~same time, and tend to read them at the ~same time)

Right, unless they’re marking non-leaf pages as “read” — which would imply marking the root — it would have to be snapshot, right?

Otherwise you can’t properly abort the txn because a “select x where y” that was previously empty, no longer is.

> Postgres has a different issue in that replication slots can grow really fast, esp on AWS! [2]. We ended up writing our own custom snapshotter for Postgres that is Debezium compatible to onboard customers that have a massive dataset and cannot afford to have a read lock on their WAL.

Does Debezium's DDD-3 watermark (DBLog) implementation for Postgres not process the WAL quickly enough? We don't use it ourselves either, but architecturally it appears it would reasonably bound how long the WAL can remain un-read?

Agreed that many production DBs people care about have pretty severe limitations here! Managed Supabase is another good example.

estuary.dev may be a fit for you (am CTO).

(Competitive product and I feel weird about replying in Artie's thread, but tang8330 has said they're not serving this segment)

Hi, I'm Estuary's CTO (https://estuary.dev). Mind speaking a bit more about what didn't work?

We put quite a bit of effort into our CDC connectors, as it's a core competency. We have numerous customers using them at scale successfully, but they can be a bit nuanced to get configured. We're constantly trying to make our onboarding experience more intuitive and seamless... it's a hard problem.

Estuary Flow derivations [1], I suspect (I work here)? You write a program which has messages pushed to it, which maintains and updates an internal state, and which publishes output messages.

You can build cyclic data-flows [2] - not just DAGs - which seems a key ingredient.

You can also write contract-based tests that verify the end-to-end behaviors of your "agents" [3].

Today, you're limited to SQLite or TypeScript. Looking forward, we aim to open it up to any program packaged in a docker container.

[1] https://docs.estuary.dev/concepts/derivations/

[2] https://docs.estuary.dev/concepts/derivations/#approving-tra...

[3] https://docs.estuary.dev/concepts/tests/

futures::join!() and friends do what you’re after admirably. Is the complaint that something is still missing, or that it’s not in the standard library?

We use a macaddr8 that embeds a wall-clock timestamp (so they're ascending order, achieving data locality) with some additional shard and sequence-number bits. It's worked really well for us:

https://github.com/estuary/flow/blob/master/supabase/migrati...

we use macaddr8 instead of bigint, because it has a postgres serialization / JSON encoding which lossless-ly round-trips with browsers and it works well with PostgREST. The same CANNOT be said for bigint, which is a huge footgun.

Building contexts for structured data used in AI / GPT tasks is something I've seen little written about, but is obviously quite important.

Confluent calls it a "customer 360" problem [1] and I don't disagree.

We (Estuary) also wrote up a post showing an approach for Slack => ChatGPT => Google Sheets [2], and have more content coming for Salesforce, HubSpot, and some others.

[1] https://www.confluent.io/blog/chatgpt-and-streaming-data-for...

[2] https://estuary.dev/gpt-real-time-pipeline/

Totally. And 3.5-turbo actually does a great job on this (and lots of other) tasks, in my experience, and is a fraction of the price. The entire endeavor cost us a few bucks of OpenAI tokens.

Also 39, also three young kids. It’s hard.

One thing I’ve noticed, though, is that my difficultly in getting into a flow state is more often than not because of a subconscious hang-up about the approach that I haven’t articulated yet.

I’ve learned to recognize and lean into this, and actively look for what’s bothering me about a design. Usually while doing other things.

I suspect it’s actually saved effort overall.

Do you have an index on `updated_at` ?

I'm having a hard time relating to this comment given our own experience.

We use RLS extensively with PostgREST implementing much of our API. It _absolutely_ uses WHERE clauses and those are evaluated / indexes consulted before RLS is applied. Anything else would be madness.

> Row level security is their "foundational piece", but there is a reason why we moved away from database functions and application logic in database over a decade ago: that stuff in unmaintainable.

Funny. In my experience, application-level authorization checks are very error-prone, easy to accidentally omit, and difficult to audit for correctness."Unmaintainable", I suppose.

Whereas RLS gives you an understandable authorization policy with a baseline assurance that you're not accidentally leaking records you shouldn't be.

Don’t you think billionaires would also rather sip cocktails on a beautiful beach vs their concrete bunker? There’s only so much you can do to spruce up a bunker. They’re not rooting for global warming, as a class.

Tell a person “have as many as you want” and they’ll have none. Tell them they can’t have one and they’ll raise a rebellion.

If you're a SQLite fan who does stream processing, we (Estuary) recently introduced a capability to write transformations as event-driven SQLite [0].

Basically you get a provisioned SQLite DB to which you apply whatever migrations you wish, and write SQL lambdas that are run with each input document, where your lambdas update your tables an/or publish outputs via SELECT.

[0]: https://docs.estuary.dev/concepts/derivations/#sqlite

This is what cost-of-living salary adjustments are intended to address.

A question to answer is, what does orchestration look like if data flows are continuous, updating as quickly as possible? That's where we're headed. Suppose N sources of data, M places where you want that data to be, and T streaming transformations to apply along the way.

In this world, a separate control plane handling time- and job-based polling has some architectural friction with a data-plane that wants to process records the instant they're available.

Disclosure: I have skin in the game / am co-founder of https://www.estuary.dev/

Here's [1] a particularly crazy one which offers synchronous, low-latency (millis) replication of a SQLite database to a distributed recovery log. It does this by instrumenting the file operations SQLite is making and recording their effects into the log.

[1] https://pkg.go.dev/go.gazette.dev/core@v0.89.0/consumer/stor...