My Laptop Is Faster Than Your Cloud: Announcing MotherDuck
tomtunguz.com
tomtunguz.com
"The cloud" runs on servers with dozens (or even hundreds) of cores, optimized for reliability. The cost of that optimization is each individual core is less fast than a core on a laptop or gaming computer.
The cloud can bring a lot of computing power to gratuitously parallelizable problems, and it allows more effectively sharing and reusing resources, so you can act as a computing utility company instead of every user needing to buy their own on-prem gear.
It doesn't mean the hardware works by magic, which is what is really being expected if you thought optimizations never involve trade-offs.
On the other hand, Postgres can be shockingly fast. With a little care I've been able to make it beat the stuffing out of most other DBs and various big data tools like Spark(running locally, not massively scaled out of course). The article starts out saying it's 80x faster than Postgres. That level of difference from Postgres doesn't impress me, it makes me stop reading and become deeply suspicious that the author either doesn't know anything about how to make PG run well or is cherry picking one of the very few tasks PG doesn't do well or has had a regression in.
MotherDuck is an analytics database optimized for reading with columnar compression. Postgres is more of a transactional/general purpose database. Tuning it well for analytics would surely improve performance.
Most people won't tune a db with custom indices though because it can be hard, so purpose built solutions like this offer value in those cases.
The article isn't purely about performance but also ease of use. MD is an in process database so starting with it is very easy.
Glad we agree on the power within a laptop! It's underappreciated.
The situation where it seems useful is if I'm using a jupyter notebook and I want to use it to pull data in but it's too much to fit into memory, and sqlite is too slow... which seems like a pretty specific situation to be in? And it's not data that is part of some business process that needs updating frequently?
Or I'm just missing the point entirely? I see another post about huge parquet datasets - but how do I get them? Is everyone at my company comfortable with me moving them onto my machine even if I can? Is the dataset so big it won't fit into memory? Can't I just connect my jupyter notebook to snowflake anyway? Then I'm only pulling down the data I want instead of all the data.
I guess maybe the real question is - exactly who is the buyer of this product, how much will they spend on it, and who is the user?
It's usually because most tasks I do end up being limited by the network connection to other resources (Postgres DBs, etc). The dev server being in the same datacenter as the resource is a huge advantage in terms of both latency and bandwith.
Again, this is mostly observations from my own workloads, and other people's use cases may be different.
In this case, most of the analysis is happening locally without a ton of network use.
go ahead and host your production apps on your laptop then. enjoy yourself!
I once had a customer where I've actually used a ~3y old middle of the line laptop (not even a dev one) to prove that our software is not at fault for performance problems because it was running all nodes of the production setup/load better than their cloud setup.
It already replaced pandas and dask for most of my common data eng tasks.
You’re querying cloud data locally? Wouldn’t egress costs get expensive quickly?
This is interesting - why did you replace pandas/dask with this? What are you getting out of it?
I mostly use it to extract data from a production DB, do basic preprocessing for ML training (which requires joins and filtering that gets ugly in pandas) and typical OLAP queries to compute statistics.
Is it this thing? So it’s a non existent service that has 47m in funding? Or am I missing something.
the existence of DuckDB?
https://github.com/duckdb/duckdb
you've got an already popular open source tool being extended and built by the guy who helped make BigQuery and people from Snowflake. VCs are going to be willing to throw money at that
sort of off-topic, I read the "why duckdb" page and found this really nice in this day and age:
"DuckDB has NO external dependencies, neither for compilation nor during run-time.
> For development, DuckDB requires CMake, Python3 and a C++11 compliant compiler.
I guess they mean libraries? But specifying for compilation implies to me they wrote their own compiler or something strange.
How do I get the data on my "laptop" in a consistent (ie, daily updates or whatever) way? How do I share queries and results with other people?