Ryan Boyd, co-founder at MotherDuck here. love to hear it- thanks for sharing :-)
240 karma · joined August 17, 2012
Would love to hear your feedback!
FROM table SELECT foo, bar WHERE zoo=“goo”
Awesome video of feature: https://youtu.be/aFDUlyeMBc8
Disclaimer: I’m a co-founder at MotherDuck.
this is a first release. we know there are going to be tons of feature requests (including @antman’s request for simple charts). feel free to chime in on this thread and we’ll keep an eye on it!
meanwhile, hope you enjoy this release! we had lots of fun building it.
enter duckdb with columnar vectorized execution and full SQL support. :-)
disclaimer: i work with the author at motherduck and we make a data warehouse powered by duckdb
we actually have a panel discussion on this at coalesce featuring matt housley and others. https://coalesce.getdbt.com/agenda/panel-discussion-fixing-t...
dbt folks actually brought this idea to us because it just makes so much sense -- no translating of sql between dev and prod.
highlights from panel brainstorming from winnie @ dbt: Local development: should be fast, representative but not comprehensive of prod data, and independent of cloud services if I want that.
The alternative has been that people typically use a schema in their production platform for development, CI workflows, etc. Costly and cloud-dependent.
DuckDB and its role in facilitating the MDS-in-a-box pattern has been a game changer for local data development, which ramped up the ability to share data, modeling work, and analysis the way we share open source tooling, but syntax remained an issue for deployment.
calling ourselves motherduck means we can either embrace the duck, or shy away from it. we've embraced it.
tino's board slides do refer to it as the Produck team :-)
over my career i’ve seen countless cases where the complexity of a distributed system introduces as many (or more) reliability concerns as it solves. when everything works well, it can be beautiful. but when things go wrong, they can quickly become a cascading mess of hard-to-debug issues.
Lots of users: (a) mentally align better with SQL than pandas APIs, regardless of whether they know both or not (b) want decent performance on their analyses, which they aren't getting from pandas, and won't get from many OLTP-databases they're using over-the-wire (c) want ease of accessing parquet and csv locally and remotely with minimal development overhead. it's super simple with duckdb.
Nonetheless, some of the other things you pointed out are some tradeoffs. We're building a serverless cloud capability at motherduck on top of duckdb in order to address some of these and optimize compute and storage based on data locality, bandwidth and the need for collaboration amongst multiple people.
I have no doubt that there are situations where the cloud will be faster, especially when provisioned for max usage [which many companies do not]. However, there are a lot of these situations even where the local machine can supplement the cloud resources [think re decisions a query planner can make].
Feel free to reach out at ryan at motherduck if you want to chat more.
I've worked on many product-led-growth initiatives in the software industry. The software industry is probably the biggest 'believer in data' there is -- many scientific-forward minds who understand the value. However, even in the software industry, it's really hard to convince folks that if you make 5 improvements that net 1% conversion gain each, you can dramatically improve revenue.
The expertise we need in the industry is people who understand applications in-and-out and make great decisions on what data is worth keeping for present and future applications. And what data is needed to be kept, but only in aggregates (or anonymized, which reduces costs of maintenance)
DuckDB Foundation is a membership-based organization to push the awareness, adoption and development of the DuckDB open source project.
DuckDB Labs is a commercial entity spun out of CWI that provides consulting services around DuckDB. The founders of this organization are the original creators of DuckDB [Mark and Hannes].
MotherDuck is a VC-funded startup that has a formal partnership with DuckDB Labs to develop and market a serverless analytics platform using DuckDB.
Hope this helps!
And Sacha of Lyft was previously using similar techniques on the red team at Microsoft.
The awesome Bloodhound project productizes some of the red team use cases: https://github.com/BloodHoundAD/Bloodhound/wiki
https://gist.github.com/nicolewhite/f1980f01dc4342b369a6b3b6...
This paper is a bit unrelated to Neo4j. Neo4j is an ACID-compliant native graph database. It is not a "specialized graph analytic engine."
Neo4j stores the graph data on disk (and caches in memory) as nodes and relationships. After index lookups to find the start points in the graph, all traversals of relationships are done in constant time -- allowing it to scale with linear performance characteristics, regardless of the size of the graph.
The referenced paper cites two main reasons that RDBMS would be better for the analytics use cases: (1) ability to express graph queries in SQL and (2) performance of executing those queries.
(1) Cypher is, like SQL, a declarative language. However, it represents graph constructs in a much more natural way -- "ASCII art for graphs." There's significant praise from developers on the web of the benefits of Cypher for traversing graphs, which is why we decided to open up the language: http://www.opencypher.org/
(2) As Neo4j isn't really intended as an analytics engine, its performance characteristics are not included in this paper. However, (expensive) indexes do not need to be created and maintained for every relationship in Neo4j. Similarly, these indexes do not need to be accessed for traversal (also expensive).
@iskander - do you have other approaches for uploading which you'd prefer?