fixed - sorry about the confusion.
https://github.com/orioledb/orioledb/pull/558
It is now Apache 2.0 which grants patent rights and can be re-licensed to PostgreSQL when the code is upstreamed. I'll amend the blog to make that clearer.
6,216 karma · joined April 6, 2017
copple [at] supabase [dot] io
https://github.com/orioledb/orioledb/pull/558
It is now Apache 2.0 which grants patent rights and can be re-licensed to PostgreSQL when the code is upstreamed. I'll amend the blog to make that clearer.
https://www.orioledb.com/docs#patch-set
The actual storage engine is written as an extension - these patches are mostly to improve the TAM API. If these are accepted by the community then it should be simpler for anyone to write their own Storage extensions.
I think (correctly) it will take a lot longer to upstream the extension - the PG community should take a “default no” attitude with big changes like this. Over time we will prove its worthiness (hopefully beyond just supabase - it would be good to collaborate with other Postgres providers)
We could also change to MIT/Apache but we feel PostgreSQL is more appropriate given our intentions to upstream the code
(if the atlasgo team are reading this feel free to reach out too)
Just to add: if anyone wants to contribute (beyond code) benchmarking and stress-testing is very helpful for us
Our goal now is to ensure that it’s as F/OSS as possible given the pre-existing conditions
I’ll revisit this with legal to try make it clearer.
Our intentions here are clear - if people have examples that we can follow we will do what we can to make this irrevocable (even to the extent of donating the patent if/when the community are ready to bear the cost of the maintainance)
The conversations we have had so far are promising, we just need to make sure we approach it the right way. Projects like Oriole have started and sputtered out several times (zheap, for example). The burden first lies on our side to prove that we can see it through to GA, with meaningful usage. They (correctly) shouldn’t need to entertain the maintenance burden until they know the juice is worth the squeeze.
If it’s not accepted into core, we will work with cloud providers to add it as a supported extension
we respect the level of conservatism in Postgres and expect that this could take a while to upstream. we're committed to any timeline and we'll develop it for self-hosters and on the supabase platform first to iron out all the bugs
that said, even if it remains an extension that's fine: as long as the TAM patches land upstream this enables everyone to create storage engines using an extension approach - a very "postgres" way of doing things
you can follow the progress of these patches here:
one of the possible enhancements Oriole could enable is multi-master (in the presentation here: https://www.orioledb.com/docs#solving-postgresql-wicked-prob...), although that work will come later. The focus for now is getting to GA so that it can be used instead of Heap
We do not push devs not to do migrations - we would strongly prefer if everyone used migrations and declarative schemas.
Especially at the scale that OP is at (see maturity model: https://supabase.com/docs/guides/deployment/maturity-model)
it was great feedback, thank you!
> Supabase's official reply about this question is "I guess you can, but don't worry, RLS."
Here is a more representative view of our official reply:
https://supabase.com/docs/guides/database/hardening-data-api
To avoid both 1 and 2 I would suggest creating a `private` schema. We are now in the planning stages to make this the default
> pushes very hard for RLS to increase adoption by non-technical users
We are tailoring what we're doing for this audience. The challenge is that they appeared out of nowhere about 6 months ago and the LLMs that are used by this audience is trained on 5 years of content tailored for developers
this is not an excuse, I'm just adding color. We've made a lot of changes with tools, alerts, email warnings etc. We are in planning-mode for changing defaults and working with the AI Builder platforms. We will likely change the schema configuration and advocate for Edge Functions (serverside Typescript) where appropriate.
https://airtable.com/app7jp5t0dEHyDpa8/shr00etqywoDW2N6N
thanks for running the benchmarks, it helps to have external parties verifying the progress
https://pganalyze.com/blog/postgres-18-async-io
> Postgres has historically operated under a synchronous I/O model, meaning every read request is a blocking system call. The database must pause and wait for the operating system to return the data before continuing.
> Postgres 18 introduces a new configuration parameter: io_method. This setting determines how read operations are dispatched under the hood, and whether they’re handled synchronously, offloaded to I/O workers, or submitted directly to the kernel via io_uring.
This is a big deal - the blog post has benchmarks showing 2-3x improvements
We just released a new version which solves this problem: https://github.com/supabase/pg_crdt
The new approach uses UNLOGGED tables for documents and logged tables for changes. This enables us to merge changes efficiently in-memory, then persists just the changes directly to the WAL (previously it was sending the entire CRDT).
The new version also takes advantage an advanced in-memory object persistence feature in Postgres called "expanded datum"[1] and some optimizations to this feature that are coming in Postgres 18[2]. This allows for more complex operations without serializing and deserializing the documents between every operation, which gets progressively slower as documents grow.
What is a CRDT? A CRDT (Conflict-free Replicated Data Type) is a data structure that syncs across multiple devices or users. They are used in multi-player applications like Figma and Notion. For example: A shared text editor where multiple people type at once, and everyone sees the same final result without manual conflict resolution.
The end goal here is that you will be able to store CRDTs as a data type in your database, and treat Postgres simply as a "peer". A good example would be storing text for a blog post:
create table posts (
id serial primary key,
title text not null
content autodoc not null default '{}'
);
Docs: https://supabase.github.io/pg_crdt/automerge/
----
[0] initial release/discussion: https://news.ycombinator.com/item?id=33931971
[1] expanded datum: https://www.postgresql.org/docs/current/storage-toast.html#S...
[2] PG18 optimizations: https://www.postgresql.org/message-id/3363452.1737483125%40s...
1/ Add a new column to the declarative file
2/ Generate a new migration: `supabase db diff -f my_new_migration`
3/ Review/edit the generated migration, check it into git
4/ Apply migration to database
> Of course, you're generally supposed to do that with these tools
this seems to be the crux of the comments - we'll try to make it much clearer than declarative schemas don't skip any migrations/review process, they just provide another way of generating the migrations (note: you can also generate them directly from the database using the CLI)
I'm curious what makes it better than the approach in the blog?
If it's this:
> also maintains a schema.sql file that can be loaded as a one-off
That's exactly what the declarative files provide
In the past we only offered tools to create a sql migration to make a change to your database. Now you can write the state of your database into files, then the migration is generated for you.
This is very similar to something like Rails/Phoenix/whatever, where you write your models and then it generates a change. The difference is that here you write your models in raw SQL, rather than an ORM
yeah - this is definitely the intended flow here. We won't be recommending anyone blindly applying generated migrations.
As you mention, it is expected that you generate & review on your local development machine, check into source control, push & merge. We've also been using this internally for ~2 years now and it works great
1/ Add a new column to the declarative file
2/ Generate a new migration: `supabase db diff -f my_new_migration`
3/ Review/edit the generated migration
Docs are here:
https://supabase.com/docs/guides/local-development/declarati...
Despite the title "UI library", this is more like a "component registry" for developers to bootstrap their applications and it will work for everything from Web & Mobile, to database scaffolding. Perhaps some poor naming/positioning on our part.
If you aren't familiar with shadcn, it works by dumping a bunch of files into your application which you can modify at your leisure. This is a different from something the "Bootstrap" approach where you could only do minimal tweaks to the theming.
> The idea to use tree-sitter in addition to libpg_query came from feedback[0] on our previous HN post, so thank you for that
supabase team here - can you share more about the challenges? We'd love your feedback so that we can fix any difficulties for the future