Perf Is Not Enough
motherduck.com
motherduck.com
This is frustrating to read: "completely blind" to years of customer complaints, and they don't eat their own dogfood.
This is a part of an argument people will often use to talk about why end to end testing is superior to unit testing. It isn't that you don't want the units tested. Nor is it that end to end is easier. Rather, it is harder to focus on an end to end test for the sake of the end to end test. (No panacea, mind.)
This is not about focusing on the metrics itself, it's about focusing on what matters to your customers: the metric is a tool, not a end goal.
It's obvious in the airplane example OP has, what matters is end to end time (and why HSR are more successful than you would think they would be at the travel game btw).
What's weird to me about the claim "performance is not enough" (and your longer-winded quote at the end) is the idea that the idea of trying to look at what customers are actually doing and finding what their pain points are is somehow best stated by decrying performance as a metric rather than decrying _any_ single factor as "enough". If I go to a restaurant, order spaghetti, and then get sauce spilled on my lap by the server, the takeaway isn't "having high quality sauce is not enough", it's "spilling things on customers makes for a bad restaurant experience".
Google built a database, works great internally, all well.
They subcontract out to build an adapter layer for the wider world which doesn't work properly, so the wider world gets to use a crappy database. A carefully engineered core Google uses, wrapped in broken nonsense, creating an emergent whole that is unnecessarily rubbish. Noone internally notices they've done this and the external people are poorly placed to work it out.
That seems absolutely on point for Google's open source strategy.
The problem is that people can do a bad enough job at the non-core stuff that it doesn't matter that you're good at your core competency. Outsourcing is not a free ride.
I was trying to say that any monopoly organization will have similar dysfunctions that a state has.
My thought is that states aren't dysfunctional primarily because they're states, but because they're monopolies.
It seems like Google is a bit allergic to this.
Not enough founders are moving at one hundred and seventy nine miles per hour anymore but I guess that’s what happens when the fed jacks up interest rates
- I don't think performance is secondary as put here, but perhaps closer to binary. As in: does it perform well enough? Check. Now let's move onto judging everything else. In this sense, it's not secondary because before you can work on anything else you need to work on performance. Else you're not even at the table. Once you're at the table it's a different story though. The author points this out themselves: "I should mention that DuckDB is fast". If it wasn't, then you probably would be competing on performance, at least until you got that "performant" tick out of the way.
- "The one [database engine] who is moving most quickly will be the one that wins in the end": Perhaps a valid-ish point, but certainly not practical. Acceleration isn't constant. Your progress will be much faster as a new player in the space, but if you manage to get to where e.g. Snowflake is, your progress will certainly slow down. So as someone choosing a system today, I can't just judge by today's acceleration and extrapolate that it will continue into the future.
So overall some good points in there, but the conclusions feel a bit off for me. Although this: "how quickly you can go from idea to answer, not query to result" is a point I'd love to see explored. Feels interesting enough to warrant its own investigation.
Unless we are talking about user interfaces designed to give the user a feeling of greater speed. With fast moving status indicators that communicate great efforts and the user feels like the response was fast for all the work. But that is an interface thing, not a database thing.
Relative would be if there's no way to put a number on performance other than a comparison between systems, which is simply not true.
Looking for good literature on this topic
- unstable execution (random crashes)
- out-of-memory errors where I would've hoped for DuckDB to gracefully take the slow route to completion if no more memory is available (tried all the different conf settings)
Why not just make shims to migrate dbs for future compatibility? So you could read db 1.0 in v2.0 but only insofar as to migrate it to v2. The implication that you don't want to promise backwards read compatibility feels antithetical to a db driver.
For example, if I have an ancient mssql db that was started in 2001, I'm confident that I can grab the latest mssql driver and still use it. I don't have to track down mssql 2007 to migrate incrementally. Not sure about postgres or mysql but I assume it's the same there. Sqlite is definitely backwards read compatible.
> Major versions usually change the internal format of system tables and data files. These changes are often complex, so we do not maintain backward compatibility of all stored data.
You mentioned OOMs, this has been a focus for a while and ha gotten steadily better over the past few releases. 0.9 added spill to disk to prevent most OOMs. And 0.10, released a couple of weeks ago, fixes a bunch more memory usage problems. The storage format, which another commenter brought up, is now fully backwards compatible.
I'd suggest giving it another try, especially once 1.0 comes out.
Example of a query that should never, ever, out-of-memory, but absolutely will in the latest DuckDB:
COPY
(
SELECT
rs.my_int,
rs.my_bigint
FROM
READ_PARQUET('s3://some/folder/my-large-files-*.parquet')
AS rs
)
TO
'/my/home/folder/my-large-file.parquet'
(
FORMAT PARQUET,
ROW_GROUP_SIZE 100000,
COMPRESSION 'ZSTD'
)
;
And yet it will try to load data in memory before crashing.
There were some recent fixes: https://github.com/duckdb/duckdb/issues/10737
Three separate occasions with different uses all leading to crashes in the first hour of using DuckDB is enough that I frankly see no point in trying it again; I don't expect it to ever magically become reliable.
For example, there might be a library which performs 2x faster than its closest alternative... Yet it might not necessarily be the best option for your project. There are other factors to consider like compatibility. E.g. If the library breaks every time you upgrade your Node.js engine version, it may add some risk to your project and require additional maintenance. Also, maybe the library itself is 2x faster, but once you've added all your application logic on top of it, your product might only end up being 5% faster than if you had used the alternative because, as is often the case, most of the workload is in the business layer.
Also, often, performance is at odds with scalability. You often need to make some performance sacrifices to achieve scalability. For example load balancing with consistent hashing adds overhead on a per-machine basis but you can't scale to multiple machines without them.
Sometimes choosing worse performance is not a tradeoff in favor of ease of use, sometimes the technology just has strict performance ceiling or critical deficiency that cannot be addressed.
All I got was a piece that more or less states that generic benchmarks are not as useful as one might expect and that other stuff is important too. Which TBH is not really that surprising...
I didn't just measure the time between the API call and the results returning. I measured from the time the user pressed the 'go button' and saw the results displayed on the screen.
I wasn't satisfied until mine was faster in both cases across a wide range of queries.
I think this is where we have still a lot of unrealised opportunity
I'm working on a database system right now, and my favorite paragraph was this one:
> Performance must be measured from the user’s perspective, not the database’s. It is a UX problem and, like any UX problem, can’t really be described in a single number. This is surprising to many people, since they think performance, like car racing, is an objective thing. Just because you can say that a Lamborghini is faster than a Prius, they believe you should also be able to say that My database is faster than Your database. But just like a Lamborghini might not get me to work any faster than a Prius (or a bicycle, if there is traffic), the actual workload for a database is going to determine which one is faster.
So, comparing BigQuery to some small data warehouse product, while BigQuery could not even get off the ground yet, that I call bs and put in the same drawer as the above. If the intention was to say "product x" vs "product y", this is what should have been done. But the author deliberately put the proper product names in a lame and dishonest perspective, which makes the whole thing not credible.
(Ok, I guess I should be glad they don't use "p9e" instead)