buremba

I'm developing https://github.com/buremba/universql for this use-case, though DuckDB is used under the hood.

There is also this one from Supabase: https://www.orioledb.com/

I thought Crunchy Data Warehouse was their main product, looking at most of their marketing posts. What's the advantage of using their managed PostgreSQL offering on the cloud, compared to native offerings such as AWS RDS and GCP Cloud SQL?

It's interesting that Snowflake went shopping for Crunchy Data over Neon. While Neon focused on bringing compute and storage separation to OLTP, Crunchy Data focused more on bringing OLTP/PostgreSQL closer to OLAP with DuckDB and Iceberg.

In a way, Crunch Data was a competitor to Snowflake as they literally name themselves as "Postgresql Data Warehouse" but correct me if I'm wrong. Neon sounds more complementary to Snowflake as they were struggling with an OLTP backend, namely their Unistore product, which was announced 3 years ago but never went into general availability due to its scalability issues.

Maybe Neon was 4x more expensive, but this acquisition sounds more like an answer to Databricks than a strategic acquisition if I'm being honest. Apparently, Crunchy had $30M ARR, so it's 8x ARR, which is a cheaper answer to Databricks.

If the human here is the creator of Redis, probably not.

You are right, you can only scale vertically for single query execution but given that cloud providers now have beefy machines available with TBs of memory and hundreds of CPUs, it’s not a blocker unless you are querying petabyte level raw data. Trino/Spark is still unbeatable in that way but in most cases, you partition the data and use predicate pushdown anyways.

You can scale horizontally if you have many concurrent queries pretty well, that’s what I was referring to.

My understanding was that MotherDuck was focusing on providing the "multiplayer mode" for DuckDB. It's interesting to see DuckDB Labs supporting data lakes natively. I guess MotherDuck is potentially moving to the UI layer by providing the notebook interface for DuckDB.

You don't need to store the metadata in DuckDB; it can live in your own PostgreSQL/MySQL, similar to Iceberg REST Catalog. They solve query parallelism by allowing you to perform computations on the edge, enabling horizontal scaling the compute layer.

They don't focus on solving the scalability problem in the metadata layer; you might need to scale your PostgreSQL independently as you have many DuckDB compute nodes running on the edge.

I had the same impression but I wouldn't call it competitor after watching their video: https://youtu.be/zeonmOO9jm4?t=4032

They support syncing to Iceberg by writing the manifest and metadata files on demand, and they already have read support for Iceberg. They just fixed Iceberg's core issues but it's not a direct competitor as you can use DuckLake along with Iceberg in a very nice and bidirectional way.

Not just for building a new one, it can also complement existing data-warehouse/lakehouses: https://github.com/buremba/universql

The flight extension is excellent as it removes the need to write C++ extensions and lets you use your favorite language to develop native DuckDB catalogs. It's straightforward to build data lake connectors and plug them in as a flight catalog, thanks to Airport!

> People that believe that this is going to be a done deal in six months are dreaming. It's more like ten years.

You might be underestimating how fast the current ETL / integration companies can pivot to provide reliable MCP servers as the lift is pretty small.

This indeed sounds like closing the loop, congrats to the team! Boris’s announcement is pretty interesting: https://www.getcensus.com/blog/census-joins-forces-with-five...

The refactoring feature in Cursor barely works, unlike any Intellij products

They have much superior product compared to VSCode in terms of pretty much everything, except AI.

Not sure why it’s so hard for them to catch up with Cursor. They have everything they need but somehow they focus on just something that they don’t have much expertise, building models instead of better integration. It’s a shame seeing such good product going downhill considering AI is becoming fundamental for dev productivity.

I remember your demos of visualizing the CTEs of a huge query in the editor. I'm looking forward to trying it!

You might like MCP then.

GoDaddy was always scam, I bought a domain like 15 years ago and they had same mindset back then.

Just look at the names; “Go Daddy” and “Wild West”.. It’s alarming already.

Is there a doc for Python apps?

The assumption with low-code tooling was that AI is so good at writing actual code in a way that it will make low-code tools redundant. Spending time with Windsurf, Cursor, and a bunch of VSCode extensions, while it was so impressive to see new projects being created autonomously, asking for new requirements or fixing bugs after >10 iterations was more complex.

I had to audit the code and give specific directions on how to restructure the code to avoid getting stuck when the project gets more complex. That makes me think autonomous agents will do much better on low-code tools, as their restrictions ensure the agent is on track. The problem with low-code tools is that they also get more complicated to scale after maybe like >200 iterations. (for a medium-sized project, on average 6 months)

It’s much intuitive to provide drag and drop support rather than the user asking AI to change the layout, which is hars bolt.new or v0 works at the moment.

Same here, but with Claude. I often just ask it to rewrite the CSS to fix issues & make it look the way I want, and it does it for me much faster than going through the docs.

I like the middle ground where I generate the models from OpenAPI but stick to my preferred HTTP library of choice in the language I’m using.

You need the full name & passport to be able to book a flight though. I'm not sure how the agencies can re-sell the flights later on?

I'm assuming agencies do that to lock the prices early on for the flights that they know will be fully booked. If airlines didn't do the nasty tricks with the prices changing every minute, this approach wouldn't work, though.

Cool use of WebSockets! Have you tried deploying it to AWS Lambda? I would like to have something similar on AWS, but SSL support and scale-to-zero are hard and tricky with EC2 Load Balancer and ECS.

I agree but I also don't think you can have $3B ARR with only good sales.

Exactly. Let's say you have a data pipeline in dbt or your favourite transformation tool. All the source data will be in your cold storage, AWS S3, all the intermediate data (TEMP tables) is written to NVME, and then the final tables are created in S3 & registered in your catalog. AWS recently released S3 Tables, which aims to make the maintenance easier with Apache Iceberg as well: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tab...

You can use spot instances if you can afford more latency or might prefer on-demand instances and keep them warm if you need low latency. Databricks (compute) and Snowflake (warehouse) do that automatically for you for the premium price.

I don't recommend overpriced SaaS solutions; I highlight that there is a lesson to be learned on why they're so popular, and open-source tech could be as simple as them.

That's why I started Universql in the first place; use their interface/protocol as an open-source tech and reduce the compute cost (often ~90% of the total cost of DWHs) by using DuckDB. You still get all the catalog features without paying the premium price.

The comparison is with Snowflake and Databricks, which takes a button or a query (https://docs.snowflake.com/en/sql-reference/sql/create-wareh...) to deploy new clusters. No disk, network, or permission setup is needed for each cluster.

I have rarely seen data people familiar with K8s as they mostly use managed services, but feel free to prove me wrong!

Clickhouse has excellent performance and it scales out indeed. Unfortunately, it's hard to deploy and maintain a Clickhouse cluster and its catalog features are pretty basic compared to Databricks and Snowflake.

Great post but it seems like you still rely on Fabric to run Spark NEE. If you're on AWS or GCP, you should probably not ditch Spark but combine both. DuckDB's gotcha is that it can't scale horizontally (multi-node), unlike Databricks. A single node can get you as far as you or can rent 2TB memory + 20TB NVME in AWS, and if you use PySpark, you can run DuckDB until it doesn't scale with its Spark integration (https://duckdb.org/docs/api/python/spark_api.html) and switch to Databricks if you need to scale out. That way, you get the best of the two worlds.

DuckDB on AWS EC2's price performance rate is 10x that of Databricks and Snowflake with its native file format, so it's a better deal if you're not processing petabyte-level data. That's unsurprising, given that DuckDB operates in a single node (no need for distributed shuffles) and works primarily with NVME (no use of object stores such as S3 for intermediate data). Thus, it can optimize the workloads much better than the other data warehouses.

If you use SQL, another gotcha is that DuckDB doesn't have advanced catalog features in cloud data warehouses. Still, it's possible to combine DuckDB compute and Snowflake Horizon / Databricks Unity Catalog thanks to Apache Iceberg, which enables multi-engine support in the same catalog. I'm experimenting this multi-stack idea with DuckDB <> Snowflake, and it works well so far: https://github.com/buremba/universql