Show HN: Epsio – Incremental views for your existing database
docs.epsio.io
docs.epsio.io
Also a big no-go you want to fix is after clicking around on the page, I still have no clue what or who Epsio is. There's no business address, no info on company etc. Judging by the "website terms of use" I finally found an address, though, but that was really really hard.
Also, still no clue if these "epsio website" terms of use are supposed to govern my usage of the product or only me browsing the website or whether the entity named in the "website" terms of use will be the one providing the software to me at all. Seriously. As great as your product is, it basically has access to the most sensitive thing any company has - its databases - so you have to be a tad bit more forthcoming with building trust.
- What underlying data flow technology is this based on? Is it Timely dataflow a la Materialize, something else a la RisingWave, etc.? - What can the materialized views not have? Ie. Does it support window functions? Unions? User-defined functions? - Finally I’m curious about how you think of the landscape versus RisingWave, Readyset, Materialize, etc.!
- Actually, Epsio has its own dataflow engine. It's based on much of the same theory as these other technologies, we just found that their implementation doesn't quite fit our use case neatly.
Just as an example, Timely/Differential dataflow requires saving state in memory. While this results in extremely low latency (beneficial for its original purpose), the trade-off is the high memory footprint required to save state for complex SQL queries (JOINS, DISTINCT ON, etc.). We found this to be quite costly for the use case of replacing heavy SQL queries in PG/MySQL.
We actually recently published a blog post on how we see the streaming landscape -https://www.epsio.io/blog/epsios-diff-in-the-streaming-lands... , welcome to take a look for our full thoughts :)
- Regarding SQL support, we support today DISTINCT [ON], UNIONs, JOINs, CTEs, ORDER BYs, GROUP BYs, and some additional operators/functions (full list is in this link: https://docs.epsio.io/sql-support/queries/)
Window functions are supposed to be released soon :)
One thing that became clear to us while working on this project is the significant difference in how incremental views (or stream processing) store and access data, compared to how PostgreSQL looks at and processes data (batch processing). So, it made a lot of sense for us to separate the technologies (i.e., have a transformation layer, and a "serving" layer), but still try to integrate the two as seamlessly as possible.