Show HN: Splitgraph - Build and share data with Postgres, inspired by Docker/Git
splitgraph.com
splitgraph.com
I’m Miles, co-founder of Splitgraph along with Artjoms (mildbyte). We met back in 2018, when I reached out to him after reading his blog on HN and realizing we lived right next to each other. Neither of us had a “real job,“ and we both wanted to build something truly innovative and cool. We tossed around a few ideas, but ultimately we couldn’t resist the idea of building “GitHub for data,” which seemed like an obvious gap in the market. After nearly two years of development, we are finally ready — and extremely excited — to share it with the world.
We are not the first to notice this gap or try to build this product. So we wanted to make sure we did it right. We made sure to start from “first principles” and really analyze the problem space. We ended up realizing that it’s not strictly Git or GitHub that people want “for data.” Rather, people just want to be able to work with data as easily as they can work with code. They want to experiment, build and maintain data without needless overhead.
Tools like Git and Docker are ubiquitous in any software engineer’s workflow, and we took a lot of inspiration from them when designing Splitgraph. We thought about why people like and use these tools, and tried to translate their benefits to the domain of data science. Our core philosophy is to stay out of the way, and work with existing abstractions instead of introducing new ones. You can version your code with Git without switching filesystems. You can build Docker images without changing your code to work in Docker. Our goal with Splitgraph is to provide an easy path to incremental adoption, so you can introduce it into your existing workflows where and when it makes sense.
Splitgraph is powered by Postgres, and provides an easy way to build and share versioned datasets, along with a whole bunch of other benefits. We encourage you to read the landing page which (hopefully) explains it well. The documentation goes into much more detail, and if you have ten minutes and Docker installed, you can try Splitgraph for yourself. [0] If you work with data, we really hope you’ll give Splitgraph a try.
We’re here to answer any questions, and we’ve also created a Discord server [1] to hopefully build a bit of a community around Splitgraph.
[0] https://www.splitgraph.com/docs/getting-started/five-minute-...
To be honest, since you introduce a new workflow and a few new concepts it's not that easy to get the right perspective in 5 minutes (I know the same problems exists with DVC and we've been iterating on docs a lot). Mind a few questions?
Do I understand it right, that is mostly focused on tabular data? Kinda git checkout for an SQL table?
> Kinda git checkout for an SQL table
This is basically right, yes. We implement some Git- and Docker- like operations on top of the SQL standard. For example, we borrow the idea of "delta compression" (storing only changes) from Git. Like Git, we store commits as a set of objects representing the changes since the last commit. But whereas Git versions files with lines as the unit of change, Splitgraph versions tables with rows as the unit of change. Our "objects" are actually cstore files that represent fragments of a table, with content addressable hashes generated with LTHash [1]. If you want to read about this in more detail, have a look at the documentation for "objects." [2]
[0] https://www.splitgraph.com/docs/getting-started/frequently-a...
[1] LTHash has the useful property that the sum of all hashes of individual fragments composing a table is equal to the content hash of a whole table. This is how we are able to have content addressable objects, which unlocks a lot of tricks in terms of efficiency and allows techniques like "layered querying".
People who use flat files and they are large enough and/or need some provenance? Or people who already have Postgres and they want to version it? Or is it about production use case (like Docker) - make snapshot to deliver it consistently?
Btw, regarding the FQA - thanks! From my take on Splitgraph though one the main DVC's difference is that it deals with flat files. From software engineering perspective - it is very close to Git lfs on steroids + some higher level features similar to makefiles, ML metrics, etc.
We hope that anyone who works with data on a daily basis can benefit from Splitgraph. When we first started, we gave a presentation at a Docker meetup called "Docker for Data" [0] with the idea that we wanted to do for data scientists what Docker did for DevOps. We hope that people will be able to throw out some of their fragile ETL scripts in favor of using Splitfiles, just like DevOps engineers could throw out their Salt and Chef scripts when they switched to Docker.
> Or people who already have Postgres and they want to version it?
It's important to note that in many cases, Postgres is really just an intermediary. Splitgraph allows you to "mount" data from any source, not just Postgres databases, by leveraging Postgres foreign data wrappers (FDWs) [1]. This idea of "mounting" is one of the core abstractions of Splitgraph that makes it really powerful, because it lets you use a common format (Splitfiles) to transform and query data from anywhere. And once you've built an image from a bunch of disparate data sources, you can use a core set of your favorite tools to query it, since as far as they're concerned, it's just a Postgres schema. So in this sense, Splitgraph can serve as a sort of universal translation layer for data from all over the place.
For a really powerful example using this idea, see the example where we mount two tables from two separate data portals (Chicago and Cambridge), and join between them. [2]
[0] We gave two presentations in 2018, both similar. A lot of details have changed since then, but the core abstractions are the same, and they might give some insight into our direction:
[0.a] https://www.slideshare.net/splitgraph/splitgraph-docker-for-...
[0.b] https://www.slideshare.net/splitgraph/splitgraph-ahl-talk
[1] https://www.splitgraph.com/docs/ingesting-data/foreign-data-...
[2] https://www.splitgraph.com/docs/ingesting-data/socrata#using...
I somewhat don't like this analogy btw, see how you mentioned Docker changed life for DevOps in the first place (vs engineers), the same here - data scientists don't care about packaging data - there should be strong incentive to do so.
A few specific questions:
1. where does query execution happen - always client or remote as well? 2. in a global "Github for data" case is there some discovery mechanism for existing data? 3. do you provide a public storage to cover the case for Github for data? or is it now more like torrent - peers host and pay for data storage?
Btw, what is major direction for you - Github (public collaboration and sharing) or internal versioned warehouses (or some other internal case?).
> Have you seen Quilt btw?
Yes, we have. It seems in this space that everything has been done or pitched before, but in our opinion nobody has hit the exact right execution yet. A big problem with a lot of existing tools is that they disrupt your workflow, or are otherwise hard to adopt without major adjacent changes. Our core philosophy with Splitgraph is to stay out of the way. As long as we can keep this up, and as long as we can continue building on a core set of simple abstractions, we think we stand a pretty good chance.
> data scientists don't care about packaging data
Indeed. It's worth noting that packaging data with Splitfiles is entirely optional. You can also run ad-hoc queries against a database with change-tracking enabled (meaning, Splitgraph audit triggers are installed), and periodically commit or checkout different versions as you see fit. This workflow would be more similar to a git workflow. But we encourage the use of Splitfiles because of the advantages they add; namely reproducibility due to provenance. It's sort of like how you can build a docker image by running arbitrary commands in a container and then `docker commit`. The problem with that workflow is that you lose all the benefits of Dockerfiles. The same logic applies to `sgr commit` and Splitfiles. Our bet is that data scientists will find Splitfiles to be the path of least resistance to accomplishing their goals.
> where does query execution happen - always client or remote as well?
At the moment, most of it happens on the client. But in Splitgraph Cloud, we do have the capability to execute queries on the remote. In a public setting, it's obviously more desirable to push down query execution to the client (or, if it's done remotely, to charge them for it). But in a corporate setting, you could imagine a shared remote cluster that executes queries on behalf of thin clients. So, it's possible to support both, but at the moment we're focused on the client.
> in a global "Github for data" case is there some discovery mechanism for existing data
Splitgraph Cloud includes discovery mechanisms including search and topics. We'll be adding a lot more features around this. We intend for the "data catalog" to be a core part of our offering.
> do you provide a public storage to cover the case for Github for data? or is it now more like torrent - peers host and pay for data storage?
At the moment, for simplicity and while we're in beta, Splitgraph Cloud is providing storage at our discretion. However, Splitgraph is designed so that data storage is decoupled from metadata storage. You can configure `sgr` to upload objects to any S3 compatible store. Currently it's configured to upload to object storage at Splitgraph Cloud, but there is no reason we could not introduce some kind of federation protocol where users can upload to independent silos of S3-compatible storage. But this raises a lot of questions with reliability and responsibility, so we have not fully explored it yet. In the near term, the easier solution will probably be charging clients for storage at Splitgraph Cloud. But, federation is something that is technically possible and at least academically interesting.
Also, note that Splitgraph Cloud does not host all the data it includes in its index. For example, the 40,000+ datasets currently in the Splitgraph index are not hosted by Splitgraph [0], but we index them, and provide value added services like a REST API that does some remote execution of queries on your behalf. Currently these use the Socrata mount point, but you could imagine a situation in a corporate environment where the catalog might index lots of databases that are not Splitgraph images, but can be mounted with an FDW in the same way.
> what is major direction for you - Github (public collaboration and sharing) or internal versioned warehouses (or some other internal case?).
Most likely, both. We will probably follow the GitHub model of offering a public and on-premise version of the same product. In an ideal world, companies or universities might pay to license an on-premise version of Splitgraph Cloud that includes all the same features as the public version. We've done a lot of work on our backend to make deployments like this possible, so it's an appealing direction for us.
[0] https://www.splitgraph.com/docs/splitgraph-cloud/external-re...
I have been looking around for databases that have any sort of cryptographic digest of data to ensure integrity. And this is the first time I have seen something do that.
Could the snapshots and content addressability be used for regular backups of application databases?
Theoretically, yes, you can use Splitgraph as a PostgreSQL replication client and then occasionally run a commit to produce a new delta. But we're currently focused on the OLAP use case and so Splitgraph images don't yet support storing things like indexes, views, triggers, stored procedures etc.
Writing to PostgreSQL tables that are change-tracked by Splitgraph is almost 2x slower than writing to untracked tables (Splitgraph uses audit triggers to record changes rather than diffing the table at commit time).
For things like schema migrations, PostgreSQL itself has transactional DDL: column deletions/additions can be wrapped in a transaction, so you can ROLLBACK if your migration fails. This might be more appropriate for your use case?
(Note that you can still add DDL commands to a Splitgraph table after you check it out, since at that point it's just a Postgres table. In theory it would be possible to track DDL changes with some other mechanism, and apply them after loading a version of data)
The thing Dolt does that splitgraph does not do is support branches, diffs, merges and conflicts on data and schema. Dolt has its own storage engine to do this efficiently whereas Splitgraph relies on Postgres.
Having native Postgres with a versioning layer on top has other advantages so we're excited to see how Splitgraph's approach works in practice. Excited to play with it. We love to see more tools in this underserved space.
With data versioning, we cherry-picked (no pun intended!) only a few concepts/commands from Git that we think are applicable. In particular, like Tim mentioned, we don't support cell-level merges. However, we offer a higher level DSL (Splitfiles[-1]) to collaborate on data.
After a Splitgraph image is checked-out, it's just a set of ordinary tables, so you get PostgreSQL feature parity and read-write performance out of the box. That means we're compatible with any existing PostgreSQL clients (DataGrip, pgcli, pgAdmin, DBeaver...), tools (Metabase, dbt...) and extensions, including those that define custom types (e.g. PostGIS for geospatial data)[0].
We also offer a way of querying Splitgraph images without checking them out[1]. This lets it download required table regions on the fly (IIRC with Dolt, you have to fully clone the whole dataset and all of its history before querying it) and in a lot of cases can be faster than PostgreSQL itself[2]. This is also completely transparent to and compatible with existing clients.
Splitgraph is decentralized and can treat any instance as a remote and push data there, with authorization provided by PostgreSQL, so you can use methods like LDAP/RADIUS/Kerberos to control access. It looks like you can spin up a Dolt remote with [3] but it's not very well documented so I can't comment on it.
Finally, we offer a lot of features beyond data versioning (reproducible dataset builds with provenance tracking, mounting other databases, automatically generated REST API for all datasets on Splitgraph Cloud etc).
When I tried it out a couple of months ago, Dolt's MySQL server didn't work with mysql_fdw. But their MySQL compatibility is progressing at an impressive pace and if it works now, you'll be able to query Dolt datasets directly from Splitgraph (by running sgr mount mysql[4]) and use them in Splitfiles. In the meantime, you can import data from Dolt into Splitgraph by using their dolt-to-pg adapter.
[-1] https://www.splitgraph.com/docs/concepts/splitfiles
[0] https://www.splitgraph.com/product/splitgraph/integrations
[1] https://www.splitgraph.com/docs/large-datasets/layered-query...
[2] https://github.com/splitgraph/splitgraph/blob/master/example...
[3] https://github.com/liquidata-inc/dolt/tree/master/go/utils/r...
[4] https://www.splitgraph.com/docs/ingesting-data/foreign-data-...