Pachyderm (YC W15) Challenges Hadoop with Containerized Data Lakes

We make money by offering support and services for Pachyderm. This includes a couple of things. Implementation services in which we help people setup a working Pachyderm cluster on their own infrastructure, help train their engineers on how to use the system and establish a plan for getting the company's data and analysis running on the new system. Support, where we help people with any issues that arise using the system. And consulting, where we get a lot more involved and help people actually build things on top of Pachyderm that their company needs.

If you're interested in these services feel free to drop me a line at jdoliner@pachyderm.io

We totally agree that containers aren't a suitable interface for everyone. We see containers as the ideal interface for developers. Over time we're expecting developers to build simpler interfaces on top of it. We'll also be building a lot of those ourselves. For example one feature we have on our near term roadmap is a SQL layer. We're going to implement that in terms of our container API which means that it should be very possible for a developer to implement a totally new access pattern just by throwing some code in a container.

Good question. It's funny how much collaboration is overlooked. And you're right - it's not obvious how a data store can enable collaboration.

In the software engineering world, collaboration by means of git is so prevalent its like breathing air. There's no such thing today for data scientists! That's crazy! Because doing data science involves more variables than writing software alone.

Pachyderm stores your data in a git-like manner. We store the deltas and version the data so that its consistently reproducible. We also give you some nice tools to run any code alongside your data.

This enables some very basic workflows:

1 - You're trying to develop your analysis - so work on your code & lock your data

2 - You're trying to vet new data - develop and version your feature extraction and data together

3 - You're trying to work on some analysis w colleagues - fork the data + analysis to do your work ... then merge to make sure your work is compatible before deploying

There are many more ... but hopefully that makes it a bit more concrete

The biggest difference between data science tutorials and data science at a company is that you have to work with people, lots of people all of whom have different backgrounds, skill sets etc. In a setting like this it gets very easy for people to step on each other's toes. For example Alice is doing some analysis on a data set and she realizes that there's a format for this data that would greatly simplify her work. So she reformats the data, but unbeknownst to her this same data was being fed into our fraud model every night... which is now broken. So reformatting data that you don't own is a bad idea, Alice learns her lesson and the next time she's in this situation instead she makes a copy of the data first so that she won't mess things up for others. This is a pretty standard thing to with a Hadoop based data lake but it leads to much more subtle problems down the road.

By making a copy Alice is setting up her own little data silo, a month later when she goes to report her findings she'll discover that in suddenly contradicts with other data scientists work. Why? Because each data scientist has his/her own completely separate copy of the data. We had cases of side by side reports which greatly disagreed about how many users we had. People stop trusting data really quickly in an organization when there are glaring flaws like that.

In Pachyderm this workflow makes a lot more sense, if Alice wants to modify data then Pachyderm will give her a clean room to work in that won't affect any of her coworkers analysis. As data outside the clean room changes it can be automatically pulled in and processed using the same code she wrote before and when she's ready to present it to others she can share it publicly.

I agree very strongly with what you've written here too. I think it's unavoidable that programs wind up taking on some of the cultural qualities of the language that they're written. It's a very interesting phenomenon that I could probably talk about at length... but I digress. This was a very conscious part of our decision to use Go. Its simplicity, consistency and reasonable resource requirements have been hugely influential, but the biggest one has been its pragmatism. Go is laser focussed on actually getting things done, and the most prominent things written in it clearly have taken on this ethos, etcd, Docker, Kubernetes to name a few. All are relatively new tools with alternatives written in other languages that have existed for a while. But they've all found use because of how well they've embrace this ethos and how appealing tools like that are to people. We use those tools internally and have very much tried to make Pachyderm fit in with the ethos as well.

We get a lot of questions about why we choose to implement our own filesystem and computation layer and the answer we gave is the one you have here, it allows us to make sure they work well together because they're built from the ground up with containerized workloads in mind. We could have tried to containerize Hadoop, but I doubt it would have resulted in the fully integrated system that we have today. Too much in the Hadoop ecosystem was written for a precontainer world for it to make sense to containerize it after the fact.

For local development - we recommend connecting to a dev k8s cluster, which is easy enough to setup.

CI is actually something I get really excited about. Similary for CI you can have a separate k8s cluster for any tests you need to run. But because we version everything, you can run 'unit tests' on sample data sets that you know should be rock solid ... and also 'integration tests' on the _same_versioned_data_ you would see in production.