vira28

Boundary between OLTP and OLAP is blurring every day.

For folks who want an off shelf version of this you may be interested in https://github.com/viggy28/streambed

(Disclaimer: I am one of the committers)

Super interesting.

I am going to explore integrating this on https://github.com/viggy28/recall

My bad on the phrasing. CJ joined MongoDB. The modern gender pronouns are confusing.

For folks who don’t know, CJ was the president of product and technology at Cloudflare for 6 months and CJ left them and joined MongoDB.

Appreciate it. Will definitely add a CLI option for it.

Thanks you.

Certainly, there is a lot of room to improve the README. Overall the project is very much alpha.

You're right. Currently, it's very binary. The answer is more nuanced and it should classify it based on the profiles like you mentioned.

Also, I noticed parsers for other databases that compiles to WASM. So, all running on client side.

Nice. Integrating it on CI and catching there is still the best way/place. Having said, it doesn't work like that in practice.

Agree. `lock_timeout` will go a long way in terms of damage control.

Author here: Adding some context. I led the Postgres platform team (2019-23) at Cloudflare and we were supporting 170+ growing product teams. One of the constant asks is schema migration review. We published a lot of best practices, added CI checks however, it was still hard to catch. Also, I tried to explain the internals of how the locking (rewrite) works, but I realized most of the devs just want the answer - Is it safe or not safe to run?

Not sure if it rings a bell, the name is a reference to the Silicon Valley Jian Yang's hot dog or not hot dog app.

Also, I understand the decision of safe vs not-safe depends heavily on data/histogram and edge cases, but still quite a lot of low-hanging issues can be easily caught with a deterministic rule engine. So I ported pg_savior[1] and used sql parser from libpg-query-node[2] which compiles as WASM, so it entirely runs on the browser. No telemetry, no login. Source attached [3]

[1] https://github.com/viggy28/pg_savior [2] https://github.com/constructive-io/libpg-query-node [3] https://github.com/viggy28/safe-not-safe

You're right. I could have but this encourages me to seriously consider it.

This will welcome more database/warehouses to be written in Go.

Personally I will implement it in https://github.com/viggy28/streambed

Agree. Until then I will wait.

Still a welcome move that they finally added support from Python.

I use recall https://github.com/viggy28/recall for the following reasons:

1. It's tightly integrated with Pi 2. Creates context automatically from sessions 3. No external dependencies - sqlite and numpy

An analytics store for Postgres on S3 using DuckDB as the query engine

https://github.com/viggy28/streambed

Is Bending Spoons the PE for startups?

Obviously welcome more tools for Postgres and sharding but it’s a tough hill to climb against the hyper clouds. Best wishes to the team.

Instead of managing skills as files, I have been using a simple utility which helps me create, update/attach skills and finally search it across sessions https://github.com/viggy28/recall/

I don’t see mention of pi. Does anyone whether it works? I am going to check out.

Reads like an add to Omnigent or whatever harness (wait it’s meta harness?.

Is that the core/tech behind Kitesurf?

TL;DR

Initial bet didn’t work out and it also became irrelevant.

Company is focusing on computer for agents. Founder CEO replaced with a new CEO.

Building https://github.com/viggy28/streambed - a full Postgres running on S3 using DuckDB as the query engine.

Certainly interested.

Agree. Appreciate the reminder.

Tomas Vondra, a major Postgres contributor recently revived a thread on using Bloom filters - https://www.postgresql.org/message-id/flat/5cd8c20c-14b5-4b0...

So there is more core work happening on support OLAP but I do think it will take some time.

In the meantime, I think we have all the pieces (storage, query engine, table format) to set up a true OLAP. For instance, I created https://github.com/viggy28/streambed to pressure test this idea.

The idea is pretty similar. As per their README, Iceberg support is deprecated.

From what I understand Ducklake needs a dedicated metadata database and it also ties to DuckDB land wherease with Iceberg many engines can query directly.

The challenge with any CDC is making it reliable. Curious, how are you exporting to S3? - Debezium or some service in AWS or home grown tool?

Currently, Strembed expects REPLICA IDENTITY FULL for getting the before and after value of TOAST column. Since we have the data in object storage, we could populate it without the need for REPLICA IDENTITY FULL. Created an issue https://github.com/viggy28/streambed/issues/25 to track this feature.

Welcome. To avoid primary running out of disk space, you can configure max_slot_wal_keep_size https://www.postgresql.org/docs/17/runtime-config-replicatio...

Since Supabase is vanilla Postgres, streambed should work with replica as the source.

reg, Planetscale, I haven't looked at their offerings yet.

Where do you host your DB currently? Happy to try out with that provider as the source.