Show HN: Pfs – A Git-like distributed file system for the Docker ecosystem | Hacker News Reader

Show HN: Pfs – A Git-like distributed file system for the Docker ecosystem

pachyderm-io.github.io

74 pointsjdoliner28 comments

> Pachyderm will eventually be a complete replacement for Hadoop, built on top of a modern toolchain instead of the JVM.

Is the JVM legitimately not considered a modern toolchain? I understand the benefits of leveraging Docker, but this comparison seems short-sighted.

This is a fair criticism and I think we used unclear phrasing here. It's not that we don't consider the JVM to be a modern toolchain in general. Rather we think it's not a modern toolchain for implementing MapReduce because it forces users to work in a JVM compatible way. Docker is a better solution because it gives users the freedom to use what they're already using.

Doesn't this approach make users send 1GB+ Docker images instead of small 5-10MB .jar files?

Useful Docker images can actually be made quite a bit smaller than 1GB+ by using small distros like busybox and following best practices such as cleaning up temporary data between commits. That being said this is a legitimate concern because it affects how tight a developers loop is so it's something we'll probably have to address longterm.

That sounds relatively easy to fix. I'd imagine a diff between two Docker images is relatively small compared to the size of the image. I'd imagine a deduplication scheme based on widely available tools (e.g. rsync) would solve this problems fairly easily. (Though I have no idea how hard it would be to implement; it doesn't sound too hard off the top of my head)

This is a very good point. In addition Docker itself tries to address this by using btrfs's diffing. Like a lot of things in Docker it's not perfect but it's getting there.

How is schlepping around non-portable Linux-specific binaries packaged into what is essentially a VM a "more modern toolchain"?

(just my personal opinion below, I'm not affiliated with the grand-parent) The tooling around the JVM is much more mature the Docker ecosystem in a lot of ways. However, one way to justify the "more modern" label is that Docker can be used to package and manage every component of a Linux-based application stack, whereas JVM tools can only run the JVM-backed pieces. The modern application architecture is decisively service-oriented and polyglot, which makes a JVM-only toolbox less relevant.

The bummer about the JVM is that the first piece of advice you get when things are going wrong is "you're using Oracle Java, not OpenJDK, right?"

There are situations now where the JVM is unavoidable. It's an increasingly poor choice for greenfield development.

Yeah it can be a big pain. Especially for companies who aren't already using the JVM and are forced in to it so they can leverage the Hadoop ecosystem.

I've had some very painful experiences trying to figure out what's going wrong on my JVM that keeps dying somewhere over the course of MapReduce job and not leaving behind any log files.

Since Java 7, Oracle Java and OpenJDK are nearly identical [0]. There's little reason to use the former instead of the latter. Previous to 7 OpenJDK had a much worse performance than the Sun/Oracle JDK, but that hasn't been true for a while now.

It remains that the JVM is incredibly mature and performant, and almost any library you could want has been implemented on it. It also remains a fruitful ground for new language development, although LLVM is an increasingly popular alternative.

[0] http://stackoverflow.com/questions/22358071/differences-betw...

One of the highest performing VMs with some of the most advanced JIT and GC technology is a poor choice for greenfield development?

In what universe?

The issue with OpenJDK arose when open source folks "backported" the unstable Java 7 OpenJDK source drops to create OpenJDK6, which was ostensibly compatible with (but not identical to) Java 6.

As of OpenJDK7/OpenJDK8, none of this hackery is required, and we're seeing convergence of the implementations.

Very creative way to combine the pros of recent fault-tolerant, distributed systems. Looks like you are the same jdoliner behind RethinkDB?

The link "git-like semantics" points to an article about ZFS -- not btrfs. I was already thinking of ZFS because Pfs reminds me of Joyent's Manta. Manta also provides virtualized containers for data where compute/MapReduce jobs can be run directly. I love ZFS and have been relying on it for years. Which makes me wonder:

* Why Btrfs and not ZFS? Licensing? Features? Lack of availability of ZFS hosts? [1]

* Joyent just open-sourced Manta, is the timing a coincidence?

* Manta provides "SnapLinks" for version control but otherwise does not really leverage ZFS to provide a history. What other differences and advantages does Pfs aim to provide?

Looking forward to how this project evolves.

[1] I own zfshosting.com since I see a need for affordable, semi-managed, online ZFS storage. If you share my vision please contact me.

> Looks like you are the same jdoliner behind RethinkDB There are many great people behind RethinkDB, 2 of them are involved in pfs actually. I am however the only one named jdoliner :)

> Why Btrfs and not ZFS? Licensing? Features? Lack of availability of ZFS hosts? [1] This came down to minimizing dependencies, Docker uses btrfs so we know it's going to be installed anywhere we deploy. I actually think that ZFS is better and am interested in using it in the future.

> Joyent just open-sourced Manta, is the timing a coincidence? Yeah total coincidence, I haven't gotten a chance to read a ton about it but it seems interesting. One big difference is that we intend to fill out a full set of git-like with things such as branching and merging. Another is that our model of MapReduce is very deeply intertwined with this set of git-like semantics.

Thanks for the kind words and zfshosting looks awesome. I'd love to chat about ZFS with you sometime.

Very neat. Btrfs is a practical choice. Looks like the Go http server listens for both changes and jobs?

Could you tell me about the advantages git semantics add to MapReduce?

Last, how does Pfs choose to distribute data across containers?

You've inspired me to put up an Unbounce page. Let's talk ZFS soon!

> Very neat. Btrfs is a practical choice. Looks like the Go http server listens for both changes and jobs?

Right now pfs is just a file system so servers only listen for changes. However longterm this is going to be the case because submitting a job is actually going to be implemented as making a commit in the file system that tells pfs which data to run the job over and the container to use to do so.

> Could you tell me about the advantages git semantics add to MapReduce?

One is that it gives you an audit log which is very useful for debugging. I often find myself with data in HDFS that doesn't look like what I expect it to and wish that I know what process changed it.

Another killer feature is that it removes the distinction between stream processing and static MapReduce. Since our file system is commit based we think of MapReduce as an operation over a stream of commits rather than a static set of files. When more commits come in you can recompute your job for only the changes.

> Last, how does Pfs choose to distribute data across containers?

Right now pfs uses a simple static hashing algorithm based on filename. This probably isn't going to work for production deployments. We're planning to improve it soon.

> You've inspired me to put up an Unbounce page. Let's talk ZFS soon!

Yes please.

Is there a distributed, ZFS-like filesystem usable in clustered production environments today?

I'm intrigued by the idea of Git-like filesystems, but am not aware of any great clustered filesystems at all.

I don't know of one and I did a decent amount of looking while developing pfs. Hopefully we can eventually meet your needs!

I am not sure I see the point in implementing MapReduce when most of the current work seems to be in generalising MapReduce - i.e. Apache Spark and YARN. Is there any reasoning behind this?

This is a very good question. There's a growing sentiment in the Hadoop ecosystem that MapReduce is in someway passe and I think it's somewhat unfair. A lot of the confusion comes from the fact that people don't distinguish between Hadoop's implementation of MapReduce and MapReduce the paradigm. As a paradigm MapReduce is actually very general. A good example of this is stream processing. Hadoop has completely separate implementations for stream processing in Storm. However there's no inherent reason MapReduce can't operate on streams. In fact, in pfs where the file system can be thought of as a stream of commits that's the only thing that it does operate on.

So tl;dr we think that a better implementation of MapReduce can be a much more general tool than Hadoop's MapReduce is.

Not the author of the project but I can think of two reasons.

Firstly, you can think of map/reduce as the infrastructure for higher level operations (sort of like the assembly language of large scale data processing that higher-level data processing systems compile to). A breakthrough in the quality of the operational engine significantly impacts the experience of doing higher-level work, so if someone finds a better way to run map/reduce jobs, it's a win for everyone. Shipping jars instead of docker containers, and not having snapshots are serious drawbacks in the existing map/reduce infrastructure that significantly impact users in negative ways.

Secondly, an easier way to specify map/reduce jobs (via a simple web server that exposes API endpoints to do data grouping, mapping, and reduction) is a dramatically simpler, more composable way to expose map/reduce jobs. Building higher level infrastructure on top of this abstraction is an order of magnitude easier than doing it on top of Hadoop, so it could be a better underlying platform for the generalization work being done in the community.

This says what I was trying to say a lot better than I did. +1 coffeemug.

What's the license? I don't see anything in any of the source files, nor the README, nor any files such as LICENSE/COPYING/etc at the top of the repo.

We actually haven't spent time thinking about this yet. Sorry about that we'll have one up soon.

I tend not to read tutorial/documentation until license is clear. Hope to see LICENSE file soon.

Q: It looks like could do `curl` on pfs. Is that mean pfs act as basic http server like AWS S3?

Hi,

Just added a LICENSE file (Apache 2.0).

> Q: It looks like could do `curl` on pfs. Is that mean pfs act as basic http server like AWS S3?

Yes!! Pfs has a microservice architecture which means it's made entirely of http servers which communicate with each other and users over RESTful APIs. To your point, this means pfs can be used as a simple webserver for static content via the `/pfs` route.

One of the features we'll be building in the near term is a web interface that users can point their browsers at to check on the status of the cluster and browse data stored in it. This is most likely going to just be implemented by injecting the .html, .css, .js etc files directly in to the file system.

Maybe I'm misunderstanding how these pieces will fit together, but from what I can see, the file system part is a fragile implementation of master-slave replication, which isn't what I would describe as a "distributed file system" like HDFS.

So I could potentially run pf and pfs on the same server?

Reply on news.ycombinator.com